Are there any standard ways how Chapel applications are deployed?
We have discussed it very briefly with Brad (@bradcray), and it seems that distributing sources is the most common approach.
It so happens that scientists who use my code are not to experienced with various toolchains/compilers/etc. If I tell them to compile a pure-Chapel program and provide detailed instructions, they just might agree to do it. However, the moment external libraries get involved and users need to compile or module load
them, they'll give up...
Potential solutions include
-
Static executables: one just has to download a file, give it exec permissions, and run. Then, as long as the application bundles a bunch of microkernels and dispatches between them automatically, one can nicely benefit from SIMD when it's supported. This is the approach that I've used in the past for single-node applications. It doesn't seem to work for multi-locale case though since now we have at least two executables (launcher & worker). AppImage and similar tools can be used to bundle those files together, but what about communication?
We have to recompile our application if we change the value ofCHPL_COMM
and bundling different communication protocols together inside one executable doesn't seem to be supported. This brings us to -
Containers (Singularity): it would be great if we could bundle all external dependencies and have pre-compiled binaries for various
CHPL_COMM
values. The container would then automatically pick the right one depending on the underlying hardware. And compared to Docker, Singularity doesn't put a lot of stuff between network libraries and the actual application, so very little (if any) performance is lost. That's all wishful thinking though as I have no idea how to achieve this in practice.
I know Machine Learning folk do it with e.g. bundling CUDA toolkit, but using the system CUDA driver. That way, one only needs to compile and test the code for a specific CUDA toolkit version rather than trying to locate it on every platform. And no performance is lost since the right driver is used in the end.
Could we perhaps do something similar with Chapel? I.e. compiling GASNet once and then somehow letting the dynamic linker choose the proper libraries?
These are just some thoughts and I'd be very curious to hear what other people with more experience in Chapel think and have come up with in the past.