# Chapel is using MPI! **URL:** \ **Category:** Users\ **Created:** [July 28, 2023, 12:18pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907 "2023-07-28T12:18:06Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![MarjanAsgari](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@MarjanAsgari](https://chapel.discourse.group/u/MarjanAsgari)\ **Post date:** [July 28, 2023, 12:18pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/1 "2023-07-28T12:18:07Z")
Hello Chapel team, I am trying to run Chapel hello world (the example), but it seems that it is using MPI while I haven't set MPI env. Would you please help? My batch file #! /bin/bash -l #SBATCH --partition=standard #SBATCH --account=accc #SBATCH --nodes=8 #SBATCH --cpus-per-task=8 #SBATCH --time=7:00:00 #SBATCH --comment="image=registry.maze.science.gc.ca/ssc-hpcs/generic-job:ubuntu22.04" #SBATCH -o stdout.txt #SBATCH --verbose export CHPL\_LAUNCHER\_WALLTIME=03:00:00 export CHPL\_LAUNCHER\_PARTITION=standard export CHPL\_LAUNCHER\_ACCOUNT=accc /home/maa004/hello2 -nl 8 What I get is: ## warning: refusing to reload ordenv; ORDENV\_SETUP already set WARNING: Open MPI accepted a TCP connection from what appears to be a another Open MPI process but cannot find a corresponding process entry for that peer. This attempted connection will be ignored; your MPI job may or may not continue properly. ## Local host: ib13be-087 PID: 111 [ib13be-065.science.gc.ca:00120] 3 more processes have sent help message help-mpi-btl-tcp.txt / server accept cannot find guid [ib13be-065.science.gc.ca:00120] Set MCA parameter "orte\_base\_help\_aggregate" to 0 to see all help / error messages ---
**Author:** ![jhh67](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/jhh67/32/169_2.png) [@jhh67](https://chapel.discourse.group/u/jhh67)\ **Post date:** [July 28, 2023, 1:33pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/2 "2023-07-28T13:33:31Z")
Please post the output of "$CHPL\_HOME/util/printchplenv --anonymize --all". John ---
**Author:** ![MarjanAsgari](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@MarjanAsgari](https://chapel.discourse.group/u/MarjanAsgari)\ **Post date:** [August 1, 2023, 11:23am UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/3 "2023-08-01T11:23:20Z")
@jhh67 Hi John, here is the output CHPL\_HOST\_PLATFORM: linux64 CHPL\_HOST\_COMPILER: gnu CHPL\_HOST\_CC: gcc CHPL\_HOST\_CXX: g++ CHPL\_HOST\_ARCH: x86\_64 CHPL\_TARGET\_PLATFORM: linux64 CHPL\_TARGET\_COMPILER: gnu CHPL\_TARGET\_CC: gcc CHPL\_TARGET\_CXX: g++ CHPL\_TARGET\_LD: mpicxx CHPL\_TARGET\_ARCH: x86\_64 CHPL\_TARGET\_CPU: unknown CHPL\_LOCALE\_MODEL: flat CHPL\_COMM: gasnet + CHPL\_COMM\_SUBSTRATE: ibv + CHPL\_GASNET\_SEGMENT: large CHPL\_TASKS: qthreads CHPL\_LAUNCHER: slurm-srun + CHPL\_TIMERS: generic CHPL\_UNWIND: none CHPL\_HOST\_MEM: jemalloc CHPL\_MEM: jemalloc CHPL\_ATOMICS: cstdlib CHPL\_NETWORK\_ATOMICS: none CHPL\_GMP: bundled CHPL\_HWLOC: bundled CHPL\_RE2: bundled CHPL\_LLVM: none + CHPL\_LLVM\_SUPPORT: bundled CHPL\_LLVM\_CONFIG: /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/third-party/llvm/install/support-only-linux64-x86\_64/bin/llvm-config CHPL\_LLVM\_VERSION: 15 CHPL\_AUX\_FILESYS: none CHPL\_LIB\_PIC: none CHPL\_SANITIZE: none CHPL\_SANITIZE\_EXE: none ---
**Author:** ![jhh67](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/jhh67/32/169_2.png) [@jhh67](https://chapel.discourse.group/u/jhh67)\ **Post date:** [August 1, 2023, 2:07pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/4 "2023-08-01T14:07:49Z")
I believe the problem is that `CHPL_LAUNCHER` should be `slurm-gasnet_ibv`, not `slurm-srun` since you are using `gasnet` as your communication layer. It looks like that variable was set by a configuration file. Try setting `CHPL_LAUNCHER=slurm-gasnet_ibv` in your environment. If that works, update the value in the configuration file. See [Chapel Launchers — Chapel Documentation 1.31](https://chapel-lang.org/docs/usingchapel/launcher.html?highlight=launchers#:~:text=View%20page%20source-,Chapel%20Launchers,-%C2%B6) for imore nformation on Chapel launchers and [Setting up Your Environment for Chapel — Chapel Documentation 1.31](https://chapel-lang.org/docs/usingchapel/chplenv.html#chapel-configuration-files:~:text=selects%20that%20flag.-,Chapel%20Configuration%20Files,-%C2%B6) for more information on configuration files. ---
**Author:** ![MarjanAsgari](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@MarjanAsgari](https://chapel.discourse.group/u/MarjanAsgari)\ **Post date:** [August 1, 2023, 4:31pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/5 "2023-08-01T16:31:02Z")
Thank you so much @jhh67 , I'll set that env correctly every time but when I run $CHPL\_HOME/util/printchplenv --anonymize --all, it still shows slurm-srun, In fact I follow these steps for Chapel installation, I appreciate any insights: # There is folder where you have Chapel files extracted # Run these env commands export CHPL\_LLVM=none export CHPL\_COMM=gasnet export CHPL\_COMM\_SUBSTRATE=ibv export CHPL\_LAUNCHER=slurm-gasnet\_ibv # cd to the directory where you have extracted Chapel # run the following commands ./configure make make install # Share the address where you have installed Chapel (extracted) --\> it is basically CHPL\_HOME # chpl command will be automatically added to /usr/local/bin/ # set the CHPL\_HOME export CHPL\_HOME= # try to compile chpl -o hello $CHPL\_HOME/examples/hello6-taskpar-dist.chpl After that when I run it, it keeps telling me about MPI ---
**Author:** ![MarjanAsgari](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@MarjanAsgari](https://chapel.discourse.group/u/MarjanAsgari)\ **Post date:** [August 2, 2023, 12:44pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/6 "2023-08-02T12:44:27Z")
Does it mean that something is missing on the cluster? chpl -o hello /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/examples/hello6-taskpar-dist.chpl /usr/bin/ld: cannot find /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/lib/linux64/gnu/x86\_64/loc-flat/comm-gasnet/ibv/large/tasks-qthreads/launch-slurm-gasnet\_ibv/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/lib\_pic-none/san-none/main\_launcher.o: No such file or directory /usr/bin/ld: cannot find -lchpllaunch: No such file or directory collect2: error: ld returned 1 exit status gmake[1]: \*\*\* [/space/partner/nrcan/geobase/work/opt/chapel-1.31.0/runtime/etc/Makefile.launcher:52: all] Error 1 gmake: \*\*\* [/space/partner/nrcan/geobase/work/opt/chapel-1.31.0/runtime/etc/Makefile.exe:44: /tmp/chpl-maa004.deleteme-25NKW6/hello.tmp] Error 2 error: compiling generated source ---
**Author:** ![jhh67](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/jhh67/32/169_2.png) [@jhh67](https://chapel.discourse.group/u/jhh67)\ **Post date:** [August 2, 2023, 7:48pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/7 "2023-08-02T19:48:22Z")
If you set the `CHPL_LAUNCHER` environment variable it should override the default setting and any configuration file. The output of `printchplenv` will show a '\*' after the value to indicate that it was taken from the environment. You can use `printenv | grep CHPL_LAUNCHER` to make sure it is set correctly in your environment. Your instructions seem reasonable, although instead of setting `CHPL_HOME` directly I suggest sourcing `util/setchplenv.` which will set `CHPL_HOME` and set your `PATH` and other environment variables correctly. ---
**Author:** ![jhh67](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/jhh67/32/169_2.png) [@jhh67](https://chapel.discourse.group/u/jhh67)\ **Post date:** [August 2, 2023, 7:52pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/8 "2023-08-02T19:52:42Z")
I'm not sure what went wrong with the build and why `main_launcher.o`. is not found. If you change a configuration variable such as `CHPL_LAUNCHER` you have to rebuild Chapel before compiling a program, although it should give you a reasonable error message in that case. I suggest running `make clobber` followed by `make`. ---
**Author:** ![MarjanAsgari](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@MarjanAsgari](https://chapel.discourse.group/u/MarjanAsgari)\ **Post date:** [August 3, 2023, 11:25am UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/9 "2023-08-03T11:25:40Z")
Thank you so much John. Do you think it may because either gasnet is not insatlled or it cannot be found? I do change the CHPL\_LAUNCHER every time and it always gets beack to slurm-srun, without any error.. ---
**Author:** ![jhh67](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/jhh67/32/169_2.png) [@jhh67](https://chapel.discourse.group/u/jhh67)\ **Post date:** [August 3, 2023, 3:45pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/10 "2023-08-03T15:45:18Z")
No, `printchplenv` just prints the values of the configuration variables. In general, it does not catch mistakes such as gasnet not being installed. That would be caught later, during the build. We bundle `gasnet` with Chapel, so it should always be found. Please send the output from: % printenv | grep CHPL % $CHPL\_HOME/util/printchplenv --anonymize --all ---
**Author:** ![MarjanAsgari](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@MarjanAsgari](https://chapel.discourse.group/u/MarjanAsgari)\ **Post date:** [August 3, 2023, 3:50pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/11 "2023-08-03T15:50:01Z")
Thank you so much! I appreciate it! $printenv | grep CHPL CHPL\_HOME=/space/partner/nrcan/geobase/work/opt/chapel-1.31.0 $CHPL\_HOME/util/printchplenv --anonymize --all CHPL\_HOST\_PLATFORM: linux64 CHPL\_HOST\_COMPILER: gnu CHPL\_HOST\_CC: gcc CHPL\_HOST\_CXX: g++ CHPL\_HOST\_ARCH: x86\_64 CHPL\_TARGET\_PLATFORM: linux64 CHPL\_TARGET\_COMPILER: gnu CHPL\_TARGET\_CC: gcc CHPL\_TARGET\_CXX: g++ CHPL\_TARGET\_LD: mpicxx CHPL\_TARGET\_ARCH: x86\_64 CHPL\_TARGET\_CPU: unknown CHPL\_LOCALE\_MODEL: flat CHPL\_COMM: gasnet + CHPL\_COMM\_SUBSTRATE: ibv + CHPL\_GASNET\_SEGMENT: large CHPL\_TASKS: qthreads CHPL\_LAUNCHER: slurm-gasnet\_ibv + CHPL\_TIMERS: generic CHPL\_UNWIND: none CHPL\_HOST\_MEM: jemalloc CHPL\_MEM: jemalloc CHPL\_ATOMICS: cstdlib CHPL\_NETWORK\_ATOMICS: none CHPL\_GMP: bundled CHPL\_HWLOC: bundled CHPL\_RE2: bundled CHPL\_LLVM: none + CHPL\_LLVM\_SUPPORT: bundled CHPL\_LLVM\_CONFIG: /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/third-party/llvm/install/support-only-linux64-x86\_64/bin/llvm-config CHPL\_LLVM\_VERSION: 15 CHPL\_AUX\_FILESYS: none CHPL\_LIB\_PIC: none CHPL\_SANITIZE: none CHPL\_SANITIZE\_EXE: none then I run --\> chpl -o hello /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/examples/hello6-taskpar-dist.chpl I get :: /usr/bin/ld: cannot find /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/lib/linux64/gnu/x86\_64/loc-flat/comm-gasnet/ibv/large/tasks-qthreads/launch-slurm-gasnet\_ibv/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/lib\_pic-none/san-none/main\_launcher.o: No such file or directory /usr/bin/ld: cannot find -lchpllaunch: No such file or directory collect2: error: ld returned 1 exit status gmake[1]: \*\*\* [/space/partner/nrcan/geobase/work/opt/chapel-1.31.0/runtime/etc/Makefile.launcher:52: all] Error 1 gmake: \*\*\* [/space/partner/nrcan/geobase/work/opt/chapel-1.31.0/runtime/etc/Makefile.exe:44: /tmp/chpl-maa004.deleteme-ZHB7nQ/hello.tmp] Error 2 error: compiling generated source ---
**Author:** ![jhh67](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/jhh67/32/169_2.png) [@jhh67](https://chapel.discourse.group/u/jhh67)\ **Post date:** [August 3, 2023, 4:53pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/12 "2023-08-03T16:53:36Z")
Let's switch over to email as this thread is getting a bit long, [john.hartman@hpe.com](mailto:john.hartman@hpe.com). Please email me the output from running `chpl --print-comands -o hello /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/examples/hello6-taskpar-dist.chpl` Also `ls /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/lib/linux64/gnu/x86_64/loc-flat/comm-gasnet/ibv/large/tasks-qthreads/launch-slurm-gasnet_ibv/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/lib_pic-none/san-none` John ---
**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\ **Post date:** [August 11, 2023, 6:00pm UTC](https://chapel.discourse.group/t/chapel-is-using-mpi/24907/13 "2023-08-11T18:00:51Z")
Hi Marjan and John — Catching up after vacation, I wanted to see whether you were able to resolve this offline. Thanks, -Brad