# Chapel deployment **URL:** \ **Category:** Users\ **Created:** [October 26, 2021, 2:12pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860 "2021-10-26T14:12:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![twesterhout](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/twesterhout/32/242_2.png) [@twesterhout](https://chapel.discourse.group/u/twesterhout)\ **Post date:** [October 26, 2021, 2:12pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/1 "2021-10-26T14:12:42Z")
Are there any standard ways how Chapel applications are deployed? We have discussed it very briefly with Brad (@bradcray), and it seems that distributing sources is the most common approach. It so happens that scientists who use my code are not to experienced with various toolchains/compilers/etc. If I tell them to compile a pure-Chapel program and provide detailed instructions, they just might agree to do it. However, the moment external libraries get involved and users need to compile or `module load` them, they'll give up... Potential solutions include - **Static executables** : one just has to download a file, give it exec permissions, and run. Then, as long as the application bundles a bunch of microkernels and dispatches between them automatically, one can nicely benefit from SIMD when it's supported. This is the approach that I've used in the past for single-node applications. It doesn't seem to work for multi-locale case though since now we have at least two executables (launcher & worker). AppImage and similar tools can be used to bundle those files together, but what about communication? We have to recompile our application if we change the value of `CHPL_COMM` and bundling different communication protocols together inside one executable doesn't seem to be supported. This brings us to - **Containers** (Singularity): it would be great if we could bundle all external dependencies and have pre-compiled binaries for various `CHPL_COMM` values. The container would then automatically pick the right one depending on the underlying hardware. And compared to Docker, Singularity doesn't put a lot of stuff between network libraries and the actual application, so very little (if any) performance is lost. That's all wishful thinking though as I have no idea how to achieve this in practice. I know Machine Learning folk do it with e.g. bundling CUDA toolkit, but using the system CUDA driver. That way, one only needs to compile and test the code for a specific CUDA toolkit version rather than trying to locate it on every platform. And no performance is lost since the right driver is used in the end. Could we perhaps do something similar with Chapel? I.e. compiling GASNet once and then somehow letting the dynamic linker choose the proper libraries? These are just some thoughts and I'd be very curious to hear what other people with more experience in Chapel think and have come up with in the past. ---
**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\ **Post date:** [October 28, 2021, 2:06am UTC](https://chapel.discourse.group/t/chapel-deployment/7860/2 "2021-10-28T02:06:33Z")
Hi Tom — Welcome to Discourse! With respect to Singularity, @ronawho mentioned that one of our users has done a bit with Chapel and Singularity, and shared his notes, but it's not immediately obvious to me whether it was a Chapel executable within Singularity or Chapel itself (at a glance, it seems to at least be the second). I was hoping to tag him on this thread, but am not finding his Discourse ID quickly, which is surprising to me. I'll mail and see if he can tag on or if it's OK to share his notes. One reminder I'm hoping the development team can help me with is how well or poorly Chapel binaries relying on GASNet can be moved between systems, where my memory is that GASNet ties itself pretty closely to a system's capabilities, paths, and network capabilities. For a binary, the "paths" part probably doesn't matter, but the others may? This may be a question to take to the GASNet team themselves, but maybe Elliot or others have a quick opinion. -Brad ---
**Author:** ![aconsroe-hpe](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/aconsroe-hpe/32/217_2.png) [@aconsroe-hpe](https://chapel.discourse.group/u/aconsroe-hpe)\ **Post date:** [November 1, 2021, 6:10pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/3 "2021-11-01T18:10:51Z")
I've been thinking about this a bit and I'm curious how many or which configurations would you want to support in a single Chapel executable bundle? Is it just different `CHPL_COMM_SUBSTRATE`s for various Gasnet? Or also supporting `CHPL_COMM=ofi`? And what about different launchers? The configuration space gets large very quickly! I'm less familiar with portability of libfabric that requires providers. Gasnet is linked to the chapel executable statically so except for host vs. target compatibility I wouldn't expect major issues. Other bundled third party libraries are linked dynamically (gmp comes to mind) and perhaps whatever else your specific program is bringing in from the outside. As it is today, I don't think we would be able to use appimage bundles with our bundled shared third party libraries because eg their RUNPATH is not relative (a requirement for appimage). That points to using a container that would bundle these third party libraries and the various compiled chapel executables (and their `_real` counterparts) along with an entrypoint script that would dispatch appropriately depending on the end user's system. I'm not sure we could do this dispatching 100% accurately but perhaps the entrypoint script could at least list all the options. And then the only kicker is it also has to support running the `*_real` when it gets scheduled. I think we could set something up to build these various configurations in a container to produce a container (ie. the build process happens in a container and the output is the distributable container). If the build and run environment have the same base container, I'd feel more confident something wouldn't get crossed when running the final product. ---
**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\ **Post date:** [November 1, 2021, 6:16pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/4 "2021-11-01T18:16:00Z")
> [@aconsroe-hpe](#): > > Other bundled third party libraries are linked dynamically (gmp comes to mind) Andrew's comment here makes me realize: It could make sense to disable third-party library capabilities that you're not relying on, like GMP (assuming you're not using `bigint`) or RE2 (assuming you're not using regular expressions). Tougher might be `qthreads` or `jemalloc` (if they are dynamically linked... I'm not sure offhand) because we like those option over the fallbacks of `CHPL_TASKS=fifo`/pthreads or `CHPL_MEM=cstdlib` due to the performance benefits. But if you found that for your application these options were a wash, that would reduce the dependencies (and size of the binary you were shipping). -Brad ---
**Author:** ![mppf](https://avatars.discourse-cdn.com/v4/letter/m/94ad74/32.png) [@mppf](https://chapel.discourse.group/u/mppf)\ **Post date:** [November 1, 2021, 6:26pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/5 "2021-11-01T18:26:13Z")
FWIW I'd expect Singlularity containers to neatly handle the static vs dynamic linking issues. But, I myself don't have any experience trying to run one of these across different Infiniband systems. ---
**Author:** ![twesterhout](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/twesterhout/32/242_2.png) [@twesterhout](https://chapel.discourse.group/u/twesterhout)\ **Post date:** [November 1, 2021, 7:24pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/6 "2021-11-01T19:24:35Z")
@aconsroe-hpe I think different `CHPL_COMM_SUBSTRATE`s should suffice (`ibv`,`udp`, and `smp`). Such that testing on a laptop can be done with `smp`, running on local clusters can be done with `udp`, and people who need larger-scale jobs and have access to proper machines can use `ibv`. We don't often encounter Cray XC systems, hence no `aries` in the above list. I think it's perfectly fine if the dispatching cannot be done automatically and the user has to pass in a command-line option to choose a specific substrate. Could you explain why running `*_real` could be problematic? Using the same base container is definitely not a problem. @bradcray dynamic libraries such as GMP require no special care when used inside Singularity containers. In my experience, they just work (or maybe I just got lucky 😃 ) ---
**Author:** ![aconsroe-hpe](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/aconsroe-hpe/32/217_2.png) [@aconsroe-hpe](https://chapel.discourse.group/u/aconsroe-hpe)\ **Post date:** [November 2, 2021, 5:35pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/7 "2021-11-02T17:35:32Z")
I am just thinking that if you have a single container executable, the `runscript` ([docs](https://sylabs.io/guides/3.0/user-guide/definition_files.html#runscript)) would be a script that does the dispatch to the right binary inside the container. That exe is just the launcher, so it is then doing something like `srun ... /absolute/path/to/foo_real`, but `foo_real` is inside the container. So the launcher would actually need to launch something like `srun ... /absolute/path/to/foo_container --real --substrate=ibv` (or maybe those go in env vars, but same idea). The `runscript` would then do the same dispatch logic as before, but call the `foo_real` binary. Does that clarify? Were you imagining a different setup? I would need some help from @ronawho perhaps to shed some light on what it looks like to customize a launcher script today and/or how hard it would be to add support for that. ---
**Author:** ![twesterhout](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/twesterhout/32/242_2.png) [@twesterhout](https://chapel.discourse.group/u/twesterhout)\ **Post date:** [November 2, 2021, 10:41pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/8 "2021-11-02T22:41:46Z")
Thanks, Andrew! Yes, that makes total sense. I'm still getting used to the launcher-worker architecture you guys have (launcher calling `srun` seems extremely confusing to me...). Indeed it'd be great to learn a bit more about how launchers work under the hood. Also, launchers seem to have minimal footprint in terms off dynamic libraries which they need to run. Could we come up with a hack which would first build `foo` launcher inside the container, copy it to the outside world, and then create a script named `foo_real` which would simply call the container (without `srun` and such)? Would something like this fool `foo` into thinking `foo_real` is a normal Chapel executable? ---
**Author:** ![aconsroe-hpe](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/aconsroe-hpe/32/217_2.png) [@aconsroe-hpe](https://chapel.discourse.group/u/aconsroe-hpe)\ **Post date:** [November 2, 2021, 11:25pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/9 "2021-11-02T23:25:37Z")
Ooh that sounds like a great hack! I initially thought an even bigger hack would be to symlink `foo_real` to the container itself and let the runscript take care of everything, but I think for maximal portability we don't want to rely on the underlying launcher to preserve env variables which would be necessary to know whether `runscript` should act as `foo` or `foo_real` and which substrate to use. I think generating a script like ```bash #!/usr/bin/env bash CHPL_COMM_SUBSTRATE=ibv CHPL_REAL=1 exec ./container $@ ``` might do the trick. (note that `CHPL_REAL` is not a real chapel env var and would be checked by the `runscript`) In case you're not aware, running `foo -v` will at least print out the launcher command like `srun ... foo_real` that gives a bit of insight into what it's doing. ---
**Author:** ![twesterhout](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/twesterhout/32/242_2.png) [@twesterhout](https://chapel.discourse.group/u/twesterhout)\ **Post date:** [November 8, 2021, 10:51pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/10 "2021-11-08T22:51:14Z")
@aconsroe-hpe that `-v` option is really nice, thanks! Now I wonder, do we actually need a launcher if we're using a container? I didn't know that `CHPL_LAUNCHER=none` was an option, but now I'm kind of liking it. It would allow us to generate just one executable (thus avoiding the "container calling container"-type problems), and we can ask the user to write a simple Slurm script, e.g. ```Chapel #!/bin/bash #SBATCH -N 2 --exclusive -n 2 -c 128 -t 1-00:00:00 ... /path/to/container $@ ``` The documentation says that `CHPL_LAUNCHER=none` doesn't work with Infiniband. What is the reason for it and is there a way to overcome it? ---
**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\ **Post date:** [November 8, 2021, 11:40pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/11 "2021-11-08T23:40:26Z")
Hi Tom — Not directly related to your latest response, but rather to this earlier mention of mine: > [@bradcray](#): > > I'll mail and see if he can tag on or if it's OK to share his notes. The following notes are from @npadmana about his experience running singularity with Chapel: [My notes on running singularity with Chapel · GitHub](https://gist.github.com/npadmana/577db4b48a58d86562345c8604c3c6b8) in hopes that they may be of some benefit. W.r.t. your latest post: > The documentation says that `CHPL_LAUNCHER=none` doesn't work with Infiniband. What is the reason for it and is there a way to overcome it? My guess is that the mechanisms that we rely on to launch GASNet/ibv programs rely on some hidden arguments to be threaded from the GASNet launcher script to the executable (which woudl suggest that the launcher could be avoided, just at additional complexity for the user?). Let's check with @ronawho who has the most experience with Chapel on IB-based systems to see what he can tell us. -Brad ---
**Author:** ![ronawho](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/ronawho/32/221_2.png) [@ronawho](https://chapel.discourse.group/u/ronawho)\ **Post date:** [November 9, 2021, 4:50pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/12 "2021-11-09T16:50:36Z")
> > The documentation says that `CHPL_LAUNCHER=none` doesn't work with Infiniband. What is the reason for it and is there a way to overcome it? > My guess is that the mechanisms that we rely on to launch GASNet/ibv programs rely on some hidden arguments to be threaded from the GASNet launcher script to the executable (which would suggest that the launcher could be avoided, just at additional complexity for the user?). Let's check with @ronawho who has the most experience with Chapel on IB-based systems to see what he can tell us. Can you send a pointer to this documentation? Disclaimer that I don't have any experience with containers or running gasnet-ibv without a launcher, but at a high level GASNet needs something to tell it what nodes are participating and how it should exchange early set up information. The 3 main supported ways of doing this today are with ssh, mpi, or pmi. With ssh gasnetrun\_ibv will launch onto the compute nodes with ssh and then exchange set up information over sockets. With mpi, mpirun is used to launch and MPI is used to exchange set up information. With pmi something like slurm is used to launch and PMI (process management interface) is used to exchange set up information. Today our recommended approach is to use ssh because mpi can add overhead and pmi isn't always available. Looking at the ssh code for an example, it builds up the list of nodes that will participate and stuffs those into an environment variable. It also builds up `GASNET_SPAWN_ARGS`, which determines which is the root node and other settings that can be different for different nodes. Maybe we could figure out what all environment variables need to be set and set those ourselves, but we'd have to reach out to the gasnet team to figure out how feasible this is. Do you know how you're expecting to launch the executable onto the compute nodes? If you're imagining a manual srun or something, maybe PMI support could be added to the container and gasnet could use that to exchange set up information. ---
**Author:** ![twesterhout](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/twesterhout/32/242_2.png) [@twesterhout](https://chapel.discourse.group/u/twesterhout)\ **Post date:** [November 9, 2021, 10:44pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/13 "2021-11-09T22:44:29Z")
@ronawho I got this impression from the last bullet point here [Chapel Launchers — Chapel Documentation 1.25](https://chapel-lang.org/docs/usingchapel/launcher.html#bypassing-the-launcher) Regarding PMI support in a container (to be honest, I've never done or even tried to do it before) and the link which Brad shared, let me play with it for a few days and I'll be back with an update/further questions. ---
**Author:** ![twesterhout](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/twesterhout/32/242_2.png) [@twesterhout](https://chapel.discourse.group/u/twesterhout)\ **Post date:** [November 17, 2021, 3:41pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/14 "2021-11-17T15:41:35Z")
So I've played around with Singularity and different launchers for a while, and here's what I came up with so far: [GitHub - twesterhout/chapel-deployment-playground: Experiments with using Singularity to deploy multi-locale Chapel executables](https://github.com/twesterhout/chapel-deployment-playground). README in the repository contains more information, but the gist is that I managed to create a Singularity container which compiles `hello6-taskpar-dist` example and which can be run on a laptop, on a cluster with Ethernet between nodes, on a cluster with Infiniband between nodes. The container is heavily inspired by the work of @npadmana, but there are a few differences. OpenMPI is used instead of Intel MPI because a) I don't have a license to use Intel MPI and b) the clusters I'm which targeting, have AMD Epyc. I also really like the fact that `apt` package manager does the right thing, and I didn't have to compile MPI & PMIX from source. Finally, this configuration doesn't require setting any environment variables or creating binds. However, when running on an Infiniband system, I get some weird error messages related to PMIX which I have no idea how to fix. They aren't fatal though, and the jobs run fine. I'd really appreciate any feedback or suggestions on this. The final step which is currently missing is creating a "release" container with just the necessary libraries to actually run the code. I expect this to greatly reduce the size of the container. ---
**Author:** ![twesterhout](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/twesterhout/32/242_2.png) [@twesterhout](https://chapel.discourse.group/u/twesterhout)\ **Post date:** [November 17, 2021, 7:03pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/15 "2021-11-17T19:03:24Z")
The repo now also contains a `release.def` file which can build a minimal container with just enough libraries to run the executables. This reduces the container size from ~635MB to just 16MB. ---
**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\ **Post date:** [November 17, 2021, 10:38pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/16 "2021-11-17T22:38:51Z")
This is very cool, thanks for the update and notes Tom. The PMIX message you mention isn't at all familiar to me, nor is the source file it's referring to. On our team, @ronawho has the most experience with IB, but I don't know whether he would've seen it either. Googling, it looks like it is probably OpenMPI related? [PMIX ERROR: ERROR in file gds\_ds12\_lock\_pthread.c at line 206 · Issue #7516 · open-mpi/ompi · GitHub](https://github.com/open-mpi/ompi/issues/7516) -Brad ---
**Author:** ![ronawho](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/ronawho/32/221_2.png) [@ronawho](https://chapel.discourse.group/u/ronawho)\ **Post date:** [November 17, 2021, 10:57pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/17 "2021-11-17T22:57:43Z")
I'm not familiar with this offhand, but I found the same link Brad did and it seems like the warning is harmless and can be quieted with `export PMIX_MCA_gds=hash` Elliot ---
**Author:** ![justSomeCoderGuy](https://avatars.discourse-cdn.com/v4/letter/j/7feea3/32.png) [@justSomeCoderGuy](https://chapel.discourse.group/u/justSomeCoderGuy)\ **Post date:** [April 29, 2022, 2:37am UTC](https://chapel.discourse.group/t/chapel-deployment/7860/18 "2022-04-29T02:37:55Z")
This may be too late for the original question, but might help going forward. Recently I've been able to compile multi-locale Chapel on LXD containers. Linux containers are lighter than docker, and fully open-source. I have used stock lxd fedora 25 amd64, opensuse leap 15.3, and opensuse tumbleweed containers to build OpenMPI and chapel with great success. The process was relatively simple. lxc init images:/fedora/35/amd64 my-bare-container lxc start my-bare-container lxc exec my-bare-container /bin/bash Now you're root inside the container Install binutils, autotools, and the gcc compiler if not already present If you want all openmpi packages for example: dnf install openmpi-\* dnf install wget git If you want to build chapel, install wget to fetch it or use git to clone the latest. You must set up ssh for sign in with no password prompt. This was tedious. I was able to automate the ssh-keygen, ssh-copy-id and ssh-scan steps to some degree with a bash script. Still trying to get it fully automated. Some steps can be performed with lxc exec commands, but working on this too. I use the bundled llvm and gasnet with udp for my home lab. export CHPL\_HOME= (unpacked tarball or git clone) export CHPL\_LLVM=bundled export CHPL\_COMM=gasnet export CHPL\_COMM\_SUBSTRATE=udp ./configure --help Select one of two installation options: standard /usr/local autotools or chapel all-in-one-directory-tree installation. I found the all-in-one install was easier to deal with. After installation set your CHPL\_HOME environment variable to the installed location instead of the source tree. The other CHPL\_and GASNET variables remain the same. Set the path to your chpl executable as well. You will only need these later when actually compiling and running multi-locale chapel programs: export GASNET\_SPAWNFN=L (for local task parallel testing) export GASNET\_SPAWNFN=S (for multi-container communication). Now you can publish an lxc image with all your work saved to this point. lxc publish my-bare-container my-custom-image Now make new containers from your custom image lxc init my-custom-image custom-container-101 lxc init my-custom-image custom-container-102 ... Put that in a loop if you need 5 or more Attach each custom container to the network bridge lxc network attach lxdbr0 custom-container-101 eth0 Set the static ip for each custom container - your network prefix will be different. lxc config device set custom-container-101 eth0 ipv4.address 10.0.3.11 lxc start custom-container-101 lxc exec custom-container-101 /bin/bash Your chapel environment vars (put in .bashrc) and compiler will be ready for use. You can use bash scripting to automate all of this if you need many containers. You can dig into LXD and script the creation of whole containers, or script the customization of various parts. There is a lot of information on how to do that on the LXD website, stack overflow and youtube. I have used a simpler method for now. ---
**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\ **Post date:** [May 10, 2022, 8:24pm UTC](https://chapel.discourse.group/t/chapel-deployment/7860/19 "2022-05-10T20:24:17Z")
Belatedly, thanks for capturing your experience and notes here, @justSomeCoderGuy! -Brad ---
**Author:** ![justSomeCoderGuy](https://avatars.discourse-cdn.com/v4/letter/j/7feea3/32.png) [@justSomeCoderGuy](https://chapel.discourse.group/u/justSomeCoderGuy)\ **Post date:** [May 11, 2022, 1:37am UTC](https://chapel.discourse.group/t/chapel-deployment/7860/20 "2022-05-11T01:37:32Z")
You're welcome! Will update the post as I automate more of the steps that copy configurations and binaries to all containers. Eventually I'm hoping to script the building of a custom lxd container from scratch using distrobuilder. There are also some github projects that claim to be turnkey lxd image building frameworks. [LXD Distrobuilder](https://github.com/lxc/distrobuilder) [LICB](https://github.com/Jens-H/licb) Will post a github repository address here once the pieces are all in place Fraser