warning: refusing to reload ordenv; ORDENV_SETUP already set
WARNING: Open MPI accepted a TCP connection from what appears to be a
another Open MPI process but cannot find a corresponding process
entry for that peer.
This attempted connection will be ignored; your MPI job may or may not
continue properly.
Local host: ib13be-087
PID: 111
[ib13be-065.science.gc.ca:00120] 3 more processes have sent help message help-mpi-btl-tcp.txt / server accept cannot find guid
[ib13be-065.science.gc.ca:00120] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
I believe the problem is that CHPL_LAUNCHER should be slurm-gasnet_ibv, not slurm-srun since you are using gasnet as your communication layer. It looks like that variable was set by a configuration file. Try setting CHPL_LAUNCHER=slurm-gasnet_ibv in your environment. If that works, update the value in the configuration file. See Chapel Launchers — Chapel Documentation 1.31 for imore nformation on Chapel launchers and Setting up Your Environment for Chapel — Chapel Documentation 1.31 for more information on configuration files.
Thank you so much @jhh67 , I'll set that env correctly every time but when I run $CHPL_HOME/util/printchplenv --anonymize --all, it still shows slurm-srun, In fact I follow these steps for Chapel installation, I appreciate any insights:
There is folder where you have Chapel files extracted
If you set the CHPL_LAUNCHER environment variable it should override the default setting and any configuration file. The output of printchplenv will show a '*' after the value to indicate that it was taken from the environment. You can use printenv | grep CHPL_LAUNCHER to make sure it is set correctly in your environment.
Your instructions seem reasonable, although instead of setting CHPL_HOME directly I suggest sourcing util/setchplenv.<shell> which will set CHPL_HOME and set your PATH and other environment variables correctly.
I'm not sure what went wrong with the build and why main_launcher.o. is not found. If you change a configuration variable such as CHPL_LAUNCHER you have to rebuild Chapel before compiling a program, although it should give you a reasonable error message in that case. I suggest running make clobber followed by make.
Thank you so much John. Do you think it may because either gasnet is not insatlled or it cannot be found? I do change the CHPL_LAUNCHER every time and it always gets beack to slurm-srun, without any error..
No, printchplenv just prints the values of the configuration variables. In general, it does not catch mistakes such as gasnet not being installed. That would be caught later, during the build. We bundle gasnet with Chapel, so it should always be found. Please send the output from:
Let's switch over to email as this thread is getting a bit long, john.hartman@hpe.com. Please email me the output from running chpl --print-comands -o hello /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/examples/hello6-taskpar-dist.chpl
Also ls /space/partner/nrcan/geobase/work/opt/chapel-1.31.0/lib/linux64/gnu/x86_64/loc-flat/comm-gasnet/ibv/large/tasks-qthreads/launch-slurm-gasnet_ibv/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/lib_pic-none/san-none