Thanks for giving it a try! If you still have it handy, what was the failure mode or error message on the omnipath system?
Thanks again,
-Brad
I did this more carefully by setting up the following environment variables.
#!/bin/sh
module purge && module load gcc/11 cmake/3.28 doxygen/1.10.0
export PATH=/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:$PATH
unset GASNET_BACKTRACE
unset CHPL_COMM_DEBUG
export CHPL_HOME=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=ofi
export FI_PROVIDER=psm2
export CHPL_LAUNCHER=gasnetrun_ofi
export GASNET_OFI_SPAWNER=pmi
export HFI_NO_CPUAFFINITY=1
export CHPL_LLVM=bundled
export CHPL_TARGET_CPU=skylake
export CHPL_GASNET_CFG_OPTIONS="--enable-debug"
export CHPL_COMM_DEBUG=1
export GASNET_BACKTRACE=1
Looks like Chapel compiles successfully but during runtime it also requires openmpi/4 to be loaded to run else it gives the following error.
adutt@freya01:pmi-test$ cat ./output.chapel
Currently Loaded Modulefiles:
1) gcc/11(default) 2) cmake/3.28 3) doxygen/1.10.0
Key:
(symbolic-version)
Using /freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin/chpl
CHPL_HOME=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0
CHPL_LAUNCHER=gasnetrun_ofi
GASNET_OFI_SPAWNER=pmi
Running Chapel program with 2 locales...
srun: spank: option "enable-coredump" provided by both coredumpsize.so and coredumpsize.so
WARNING: Don't know how to control process->node layout with your mpirun
WARNING: PROCESS LAYOUT MIGHT NOT MATCH YOUR REQUEST
error running /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none-debug/substrate-ofi/seg-everything/bin/gasnetrun_ofi-mpi.pl:
gasnetrun: exec(mpirun -np 2 /usr/bin/env CONDA_SHLVL=0 LS_COLORS=no=00:fi=00:di=01%03b34:ln=00%03b36:pi=40%03b33:so=01%03b35:do=01%03b35:bd=40%03b33%03b01:cd=40%03b33%03b01:or=41%03b33%03b01:ex=00%03b32:%02a.cmd=00%03b32:%02a.exe=01%03b32:%02a.com=01%03b32:%02a.bat=01%03b32:%02a.btm=01%03b32:%02a.dll=01%03b32:%02a.tar=00%03b31:%02a.tbz=00%03b31:%02a.tgz=00%03b31:%02a.rpm=00%03b31:%02a.deb=00%03b31:%02a.arj=00%03b31:%02a.taz=00%03b31:%02a.lzh=00%03b31:%02a.lzma=00%03b31:%02a.zip=00%03b31:%02a.zoo=00%03b31:%02a.z=00%03b31:%02a.Z=00%03b31:%02a.gz=00%03b31:%02a.bz2=00%03b31:%02a.tb2=00%03b31:%02a.tz2=00%03b31:%02a.tbz2=00%03b31:%02a.xz=00%03b31:%02a.avi=01%03b35:%02a.bmp=01%03b35:%02a.dl=01%03b35:%02a.fli=01%03b35:%02a.gif=01%03b35:%02a.gl=01%03b35:%02a.jpg=01%03b35:%02a.jpeg=01%03b35:%02a.mkv=01%03b35:%02a.mng=01%03b35:%02a.mov=01%03b35:%02a.mp4=01%03b35:%02a.mpg=01%03b35:%02a.pcx=01%03b35:%02a.pbm=01%03b35:%02a.pgm=01%03b35:%02a.png=01%03b35:%02a.ppm=01%03b35:%02a.svg=01%03b35:%02a.tga=01%03b35:%02a.tif=01%03b35:%02a.webm=01%03b35:%02a.webp=01%03b35:%02a.wmv=01%03b35:%02a.xbm=01%03b35:%02a.xcf=01%03b35:%02a.xpm=01%03b35:%02a.aiff=00%03b32:%02a.ape=00%03b32:%02a.au=00%03b32:%02a.flac=00%03b32:%02a.m4a=00%03b32:%02a.mid=00%03b32:%02a.mp3=00%03b32:%02a.mpc=00%03b32:%02a.ogg=00%03b32:%02a.voc=00%03b32:%02a.wav=00%03b32:%02a.wma=00%03b32:%02a.wv=00%03b32: LD_LIBRARY_PATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux/lib64: CONDA_EXE=/u/adutt/anaconda3/bin/conda HOSTTYPE=x86_64 SLURM_NODEID=0 SLURM_TASK_PID=30610 SSH_CONNECTION=130.183.12.24%02055814%020130.183.183.24%02022 SPACK_PYTHON=/usr/bin/python3 LESSCLOSE=lessclose.sh%020%s%020%s SLURM_PRIO_PROCESS=0 XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB GASNET_BACKTRACE=1 CHPL_COMM_DEBUG=1 LANG=en_US SLURM_SUBMIT_DIR=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test WINDOWMANAGER=/usr/bin/icewm-session LESS=-M%020-I%020-R HOSTNAME=freyag01 OLDPWD=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0 CHPL_TARGET_CPU=skylake __MODULES_SHARE_MODULEPATH=/mpcdf/soft/SLE_15/modules/compilers:9:/mpcdf/soft/SLE_15/modules/tools:9:/mpcdf/soft/SLE_15/modules/ml:9:/mpcdf/soft/SLE_15/modules/java:9:/mpcdf/soft/SLE_15/modules/visualization:9:/mpcdf/soft/SLE_15/modules/gpu:9:/mpcdf/soft/SLE_15/modules/python:9:/mpcdf/soft/SLE_15/modules/libs:9:/mpcdf/soft/SLE_15/modules/third-party-compilers:9 CSHEDIT=emacs ENVIRONMENT=BATCH PROG=./test-locales GPG_TTY=/dev/pts/8 LESS_ADVANCED_PREPROCESSOR=no GASNET_OFI_SPAWNER=pmi MPI_PATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux COLORTERM=1 CHOLLA_DIR=/u/adutt/cholla/src SLURM_CELL=FREYA ROCR_VISIBLE_DEVICES=0,1 SLURM_PROCID=0 SLURM_JOB_GID=10100 CHPL_LAUNCHER=gasnetrun_ofi MACHTYPE=x86_64-suse-linux SLURMD_NODENAME=freyag01 JOB_TMPDIR=/tmp MINICOM=-c%020on SLURM_TASKS_PER_NODE=1%028x2%029 _CE_M= QT_SYSTEM_DIR=/usr/share/desktop-data OSTYPE=linux XDG_SESSION_ID=c12683 MODULES_CMD=/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/libexec/modulecmd.tcl HFI_NO_CPUAFFINITY=1 SLURM_NNODES=2 USER=adutt PAGER=less DOMAIN=bc.rzg.mpg.de PLUTO_DIR=/u/adutt/jan23/pluto-clcrush/PLUTO MORE=-sl CHPL_COMM_SUBSTRATE=ofi PWD=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test SLURM_JOB_NODELIST=freyag%05b01-02%05d HOME=/u/adutt SLURM_CLUSTER_NAME=freya CONDA_PYTHON_EXE=/u/adutt/anaconda3/bin/python SLURM_NODELIST=freyag%05b01-02%05d SLURM_GPUS_ON_NODE=2 HOST=freya01 SSH_CLIENT=130.183.12.24%02055814%02022 CHPL_COMM=gasnet XNLSPATH=/usr/X11R6/lib/X11/nls CPATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux/include64 XDG_SESSION_TYPE=tty KRB5CCNAME=FILE:/tmp/krb5cc_28744_ZAlP7ZPti6 SLURM_JOB_CPUS_PER_NODE=40%028x2%029 INTERACTIVE=TRUE XDG_DATA_DIRS=/usr/share MPCDF_SUBMODULE_COMBINATIONS=/mpcdf/soft/SLE_15/sub/gcc_11/combinations SLURM_TOPOLOGY_ADDR=opasw%05b5-6%05d.opasw4.freyag01 _CE_CONDA= LIBGL_DEBUG=quiet SLURM_WORKING_CLUSTER=freya:freyaio2:6817:9472:101 __MODULES_LMALTNAME=gcc/11%026gcc/default%026gcc%026as%07cgcc/latest:cmake/3.28%026as%07ccmake/latest:doxygen/1.10.0%026as%07cdoxygen/default%026as%07cdoxygen/latest GCC_HOME=/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0 SLURM_JOB_NAME=chapel-job PROFILEREAD=true TMPDIR=/tmp LIBRARY_PATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux/lib64: SLURM_JOB_GPUS=0,1 SLURM_JOBID=760538 SLURM_CONF=/etc/slurm/slurm.conf LOADEDMODULES=gcc/11:cmake/3.28:doxygen/1.10.0 FI_PROVIDER=psm2 SLURM_NODE_ALIASES=%028null%029 SLURM_JOB_QOS=normal SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node SSH_TTY=/dev/pts/8 FROM_HEADER= MAIL=/var/spool/mail/adutt SLURM_CPUS_ON_NODE=40 SLURM_JOB_NUM_NODES=2 SLURM_MEM_PER_NODE=9500 LESSKEY=/etc/lesskey.bin SPACK_ROOT=/u/adutt/spack SHELL=/bin/bash TERM=xterm-256color XDG_SESSION_CLASS=user CMAKE_HOME=/mpcdf/soft/SLE_15/packages/x86_64/cmake/3.28 SLURM_JOB_UID=28744 ARGS=-nl%0202 __MODULES_LMCONFLICT=gcc/11%026gcc XCURSOR_THEME=DMZ LS_OPTIONS=-N%020--color=tty%020-T%0200 SLURM_JOB_PARTITION=p.test SLURM_JOB_USER=adutt CUDA_VISIBLE_DEVICES=0,1 CHPL_LLVM=bundled SHLVL=2 SLURM_SUBMIT_HOST=freya01 G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252 SLURM_JOB_ACCOUNT=mpa MANPATH=/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0/share/man:/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/share/man:/usr/local/man:/usr/share/man AFS=localhost CELL=ipp-garching.mpg.de MODULEPATH=/mpcdf/soft/SLE_15/sub/gcc_11/modules/ml:/mpcdf/soft/SLE_15/sub/gcc_11/modules/applications:/mpcdf/soft/SLE_15/sub/gcc_11/modules/mpi:/mpcdf/soft/SLE_15/sub/gcc_11/modules/libs:/mpcdf/soft/SLE_15/sub/gcc_11/modules/tools:/mpcdf/soft/SLE_15/modules/third-party-compilers:/mpcdf/soft/SLE_15/modules/java:/mpcdf/soft/SLE_15/modules/visualization:/mpcdf/soft/SLE_15/modules/gpu:/mpcdf/soft/SLE_15/modules/ml:/mpcdf/soft/SLE_15/modules/compilers:/mpcdf/soft/SLE_15/modules/python:/mpcdf/soft/SLE_15/modules/libs:/mpcdf/soft/SLE_15/modules/tools CHPL_HOME=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0 SLURM_GTIDS=0 LOGNAME=adutt DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/28744/bus CLUSTER=FREYA XDG_RUNTIME_DIR=/run/user/28744 SYS=amd64_sles15 CHPL_GASNET_CFG_OPTIONS=--enable-debug XDG_CONFIG_DIRS=/etc/xdg PATH=/mpcdf/soft/SLE_15/packages/x86_64/doxygen/1.10.0/bin:/mpcdf/soft/SLE_15/packages/x86_64/cmake/3.28/bin:/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0/bin:/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:/u/adutt/anaconda3/condabin:/u/adutt/spack/bin:/u/adutt/.local/bin:/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/bin:/u/adutt/.local/bin:/u/adutt/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/afs/ipp/amd64_sles15/bin:/mpcdf/soft/SLE_15/packages/x86_64/find-module/1.0/bin SLURM_JOB_ID=760538 _LMFILES_=/mpcdf/soft/SLE_15/modules/compilers/gcc/11:/mpcdf/soft/SLE_15/modules/tools/cmake/3.28:/mpcdf/soft/SLE_15/modules/tools/doxygen/1.10.0 MODULESHOME=/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1 INFOPATH=/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0/share/info JOB_SHMTMPDIR=/dev/shm G_BROKEN_FILENAMES=1 HISTSIZE=1000 CPU=x86_64 DOXYGEN_HOME=/mpcdf/soft/SLE_15/packages/x86_64/doxygen/1.10.0 SLURM_LOCALID=0 CVS_RSH=ssh GPU_DEVICE_ORDINAL=0,1 LESSOPEN=lessopen.sh%020%s BASH_FUNC_module%%=%028%029%020%07b%020%020_module_raw%020%022%024@%022%0202%03e%0261%00a%07d BASH_FUNC_spack%%=%028%029%020%07b%020%020:%020this%020is%020a%020shell%020function%020from:%020/u/adutt/spack/share/spack/setup-env.sh%03b%00a%020:%020the%020real%020spack%020script%020is%020here:%020/u/adutt/spack/bin/spack%03b%00a%020_spack_shell_wrapper%020%022%024@%022%03b%00a%020return%020%024%03f%00a%07d BASH_FUNC__module_raw%%=%028%029%020%07b%020%020eval%020%060/usr/bin/tclsh%020/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/libexec/modulecmd.tcl%020bash%020%022%024@%022%060%03b%00a%020_mlstatus=%024%03f%03b%00a%020return%020%024_mlstatus%00a%07d BASH_FUNC__spack_shell_wrapper%%=%028%029%020%07b%020%020for%020var%020in%020DYLD_LIBRARY_PATH%020DYLD_FALLBACK_LIBRARY_PATH%03b%00a%020do%00a%020eval%020%022if%020%05b%020-n%020%05c%022%05c%024%07b%024%07bvar%07d-%07d%05c%022%020%05d%03b%020then%020export%020SPACK_%024var=%05c%024%07b%024%07bvar%07d%07d%03b%020fi%022%03b%00a%020done%03b%00a%020if%020%05b%020-n%020%022%024%07bZSH_VERSION:-%07d%022%020%05d%03b%020then%00a%020emulate%020-L%020sh%03b%00a%020fi%03b%00a%020_sp_flags=%022%022%03b%00a%020while%020%05b%020%021%020-z%020%024%07b1+x%07d%020%05d%020%026%026%020%05b%020%022%024%07b1%023-%07d%022%020%021=%020%022%024%07b1%07d%022%020%05d%03b%020do%00a%020_sp_flags=%022%024_sp_flags%020%0241%022%03b%00a%020shift%03b%00a%020done%03b%00a%020if%020%05b%020-n%020%022%024_sp_flags%022%020%05d%020%026%026%020%05b%020%022%024%07b_sp_flags%023%02ah%07d%022%020%021=%020%022%024%07b_sp_flags%07d%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_sp_flags%023%02aV%07d%022%020%021=%020%022%024%07b_sp_flags%07d%022%020%05d%03b%020then%00a%020command%020spack%020%024_sp_flags%020%022%024@%022%03b%00a%020return%03b%00a%020fi%03b%00a%020_sp_subcommand=%022%022%03b%00a%020if%020%05b%020%021%020-z%020%024%07b1+x%07d%020%05d%03b%020then%00a%020_sp_subcommand=%022%0241%022%03b%00a%020shift%03b%00a%020fi%03b%00a%020case%020%024_sp_subcommand%020in%020%00a%020%022cd%022%029%00a%020_sp_arg=%022%022%03b%00a%020if%020%05b%020-n%020%022%0241%022%020%05d%03b%020then%00a%020_sp_arg=%022%0241%022%03b%00a%020shift%03b%00a%020fi%03b%00a%020if%020%05b%020%022%024_sp_arg%022%020=%020%022-h%022%020%05d%020%07c%07c%020%05b%020%022%024_sp_arg%022%020=%020%022--help%022%020%05d%03b%020then%00a%020command%020spack%020cd%020-h%03b%00a%020else%00a%020LOC=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020spack%020location%020%024_sp_arg%020%022%024@%022%029%022%03b%00a%020if%020%05b%020-d%020%022%024LOC%022%020%05d%03b%020then%00a%020cd%020%022%024LOC%022%03b%00a%020else%00a%020return%0201%03b%00a%020fi%03b%00a%020fi%03b%00a%020return%00a%020%03b%03b%00a%020%022env%022%029%00a%020_sp_arg=%022%022%03b%00a%020if%020%05b%020-n%020%022%0241%022%020%05d%03b%020then%00a%020_sp_arg=%022%0241%022%03b%00a%020shift%03b%00a%020fi%03b%00a%020if%020%05b%020%022%024_sp_arg%022%020=%020%022-h%022%020%05d%020%07c%07c%020%05b%020%022%024_sp_arg%022%020=%020%022--help%022%020%05d%03b%020then%00a%020command%020spack%020env%020-h%03b%00a%020else%00a%020case%020%024_sp_arg%020in%020%00a%020activate%029%00a%020_a=%022%020%024@%022%03b%00a%020if%020%05b%020%022%024%07b_a%023%02a%020--sh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--csh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020-h%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--help%07d%022%020%021=%020%022%024_a%022%020%05d%03b%020then%00a%020command%020spack%020env%020activate%020%022%024@%022%03b%00a%020else%00a%020stdout=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020command%020spack%020%024_sp_flags%020env%020activate%020--sh%020%022%024@%022%029%022%020%07c%07c%020return%03b%00a%020eval%020%022%024stdout%022%03b%00a%020fi%00a%020%03b%03b%00a%020deactivate%029%00a%020_a=%022%020%024@%022%03b%00a%020if%020%05b%020%022%024%07b_a%023%02a%020--sh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--csh%07d%022%020%021=%020%022%024_a%022%020%05d%03b%020then%00a%020command%020spack%020env%020deactivate%020%022%024@%022%03b%00a%020else%00a%020if%020%05b%020-n%020%022%024%02a%022%020%05d%03b%020then%00a%020command%020spack%020env%020deactivate%020-h%03b%00a%020else%00a%020stdout=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020command%020spack%020%024_sp_flags%020env%020deactivate%020--sh%029%022%020%07c%07c%020return%03b%00a%020eval%020%022%024stdout%022%03b%00a%020fi%03b%00a%020fi%00a%020%03b%03b%00a%020%02a%029%00a%020command%020spack%020env%020%024_sp_arg%020%022%024@%022%00a%020%03b%03b%00a%020esac%03b%00a%020fi%03b%00a%020return%00a%020%03b%03b%00a%020%022load%022%020%07c%020%022unload%022%029%00a%020_a=%022%020%024@%022%03b%00a%020if%020%05b%020%022%024%07b_a%023%02a%020--sh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--csh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020-h%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--list%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--help%07d%022%020%021=%020%022%024_a%022%020%05d%03b%020then%00a%020command%020spack%020%024_sp_flags%020%024_sp_subcommand%020%022%024@%022%03b%00a%020else%00a%020stdout=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020command%020spack%020%024_sp_flags%020%024_sp_subcommand%020--sh%020%022%024@%022%029%022%020%07c%07c%020return%03b%00a%020eval%020%022%024stdout%022%03b%00a%020fi%00a%020%03b%03b%00a%020%02a%029%00a%020command%020spack%020%024_sp_flags%020%024_sp_subcommand%020%022%024@%022%00a%020%03b%03b%00a%020esac%00a%07d BASH_FUNC_mc%%=%028%029%020%07b%020%020.%020/usr/share/mc/mc-wrapper.sh%00a%07d BASH_FUNC_ml%%=%028%029%020%07b%020%020module%020ml%020%022%024@%022%00a%07d SRUN_DEBUG=3 SLURM_UMASK=0022 SLURM_JOB_CPUS_PER_NODE_PACK_GROUP_0=40%028x2%029 SLURM_NTASKS=2 SLURM_NPROCS=2 SLURM_DISTRIBUTION=cyclic SLURM_STEP_ID=0 SLURM_STEPID=0 SLURM_SRUN_COMM_PORT=40307 SLURM_STEP_NODELIST=freyag%05b01-02%05d SLURM_STEP_NUM_NODES=2 SLURM_STEP_NUM_TASKS=2 SLURM_STEP_TASKS_PER_NODE=1%028x2%029 SLURM_STEP_LAUNCHER_PORT=40307 SLURM_SRUN_COMM_HOST=130.183.183.160 SLURM_CPU_BIND=quiet,mask_cpu:0x0000000001 SLURM_CPU_BIND_LIST=0x0000000001 SLURM_CPU_BIND_TYPE=mask_cpu: SLURM_CPU_BIND_VERBOSE=quiet SLURM_LAUNCH_NODE_IPADDR=130.183.183.160 GASNET_BACKTRACE=1 GASNET_OFI_SPAWNER=pmi GASNET_PSHM_ENABLED=no GASNET_SPAWN_HAVE_PMI=1 GASNET_SPAWN_HAVE_MPI=0 GASNET_PLATFORM=generic GASNET_ENVCMD=/usr/bin/env GASNET_SPAWN_CONDUIT=OFI /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/test-locales_real -nl 2) failed: No such file or directory
After loading mpi but not setting OMPI_MCA_btl=tcp,self I get this error:
*** FATAL ERROR (proc 0): in do_kvs_init_inner() at net/gasnet-src/other/pmi-spawner/gasnet_bootstrap_pmi.c:484: PMI2_Job_GetId() failed
*** Details for bug reporting (proc 0): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,debug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=CLANG/19.1.3 sys=x86_64-pc-linux-gnu
[0] Invoking GDB for backtrace...
[0] /usr/bin/gdb -nx -batch -x /tmp/gasnet_SgHRJr '/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/test-locales_real' 31333
[0] [Thread debugging using libthread_db enabled]
[0] Using host libthread_db library "/lib64/libthread_db.so.1".
[0] 0x00001512a5c2f736 in wait4 () from /lib64/libc.so.6
[0] Id Target Id Frame
[0] * 1 Thread 0x1512a77bcd00 (LWP 31333) "test-locales_re" 0x00001512a5c2f736 in wait4 () from /lib64/libc.so.6
[0]
[0] Thread 1 (Thread 0x1512a77bcd00 (LWP 31333) "test-locales_re"):
[0] #0 0x00001512a5c2f736 in wait4 () from /lib64/libc.so.6
[0] #1 0x00001512a5ba6bc7 in do_system () from /lib64/libc.so.6
[0] #2 0x000055ea090fa994 in gasneti_system_redirected (cmd=0x55ea094dfa80 <gasneti_bt_gdb.cmd> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_SgHRJr '/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/test-locales_real' 31333", stdout_fd=3) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:1679
[0] #3 0x000055ea090fa4a7 in gasneti_bt_gdb (fd=3) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:1944
[0] #4 0x000055ea090f23e5 in gasneti_print_backtrace (fd=2) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:2228
[0] #5 0x000055ea090f2d45 in _gasneti_print_backtrace_ifenabled (fd=2) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:2359
[0] #6 0x000055ea090f0682 in gasneti_error_abort () at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:807
[0] #7 0x000055ea090ef77a in _gasneti_fatalerror (msg=0x55ea092809ec "PMI2_Job_GetId() failed") at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:843
[0] #8 0x000055ea09230ae5 in do_kvs_init_inner () at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/other/pmi-spawner/gasnet_bootstrap_pmi.c:484
[0] #9 0x000055ea0922fbed in bootstrapExchange (src=0x55ea094e8698 <gasneti_my_hostid>, len=4, dest=0x55ea0a03e1f0) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/other/pmi-spawner/gasnet_bootstrap_pmi.c:608
[0] #10 0x000055ea091ef9ff in gasneti_nodemapInit (exchangefn=0x55ea0922fba0 <bootstrapExchange>, ids=0x55ea094e8698 <gasneti_my_hostid>, sz=4, stride=0) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_internal.c:2297
[0] #11 0x000055ea090b25cc in gasnetc_init (client_p=0x55ea094dad58 <myclient>, ep_p=0x55ea094dad60 <myep>, tm_p=0x55ea094dad48 <myteam>, clientName=0x55ea0923d44b "chapel", argc=0x7ffed35bed88, argv=0x7ffed35bed90, flags=1) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/ofi-conduit/gasnet_core.c:123
[0] #12 0x000055ea090b2256 in gex_Client_Init_GASNET_202450PARnopshmEVERYTHINGdebugtracestatsdebugmallocsrclines (client_p=0x55ea094dad58 <myclient>, ep_p=0x55ea094dad60 <myep>, tm_p=0x55ea094dad48 <myteam>, clientName=0x55ea0923d44b "chapel", argc=0x7ffed35bed88, argv=0x7ffed35bed90, flags=1) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/ofi-conduit/gasnet_core.c:276
[0] #13 0x000055ea09013cc2 in chpl_comm_init ()
[0] #14 0x000055ea0900b5c5 in chpl_rt_init ()
[0] #15 0x000055ea090015a6 in main ()
[0] [Inferior 1 (process 31333) detached]
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node freya01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Changing OMPI_MCA_btl=openib,vader,self to OMPI_MCA_btl=tcp,self I get the following error.
*** FATAL ERROR (proc 0): in do_kvs_init_inner() at net/gasnet-src/other/pmi-spawner/gasnet_bootstrap_pmi.c:484: PMI2_Job_GetId() failed
*** Details for bug reporting (proc 0): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,debug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=CLANG/19.1.3 sys=x86_64-pc-linux-gnu
[0] Invoking GDB for backtrace...
[0] /usr/bin/gdb -nx -batch -x /tmp/gasnet_FNO2TL '/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/test-locales_real' 32951
[0] [Thread debugging using libthread_db enabled]
[0] Using host libthread_db library "/lib64/libthread_db.so.1".
[0] 0x000014d7e2fd8736 in wait4 () from /lib64/libc.so.6
[0] Id Target Id Frame
[0] * 1 Thread 0x14d7e4b65d00 (LWP 32951) "test-locales_re" 0x000014d7e2fd8736 in wait4 () from /lib64/libc.so.6
[0]
[0] Thread 1 (Thread 0x14d7e4b65d00 (LWP 32951) "test-locales_re"):
[0] #0 0x000014d7e2fd8736 in wait4 () from /lib64/libc.so.6
[0] #1 0x000014d7e2f4fbc7 in do_system () from /lib64/libc.so.6
[0] #2 0x000055f836373994 in gasneti_system_redirected (cmd=0x55f836758a80 <gasneti_bt_gdb.cmd> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_FNO2TL '/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/test-locales_real' 32951", stdout_fd=3) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:1679
[0] #3 0x000055f8363734a7 in gasneti_bt_gdb (fd=3) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:1944
[0] #4 0x000055f83636b3e5 in gasneti_print_backtrace (fd=2) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:2228
[0] #5 0x000055f83636bd45 in _gasneti_print_backtrace_ifenabled (fd=2) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:2359
[0] #6 0x000055f836369682 in gasneti_error_abort () at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:807
[0] #7 0x000055f83636877a in _gasneti_fatalerror (msg=0x55f8364f99ec "PMI2_Job_GetId() failed") at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_tools.c:843
[0] #8 0x000055f8364a9ae5 in do_kvs_init_inner () at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/other/pmi-spawner/gasnet_bootstrap_pmi.c:484
[0] #9 0x000055f8364a8bed in bootstrapExchange (src=0x55f836761698 <gasneti_my_hostid>, len=4, dest=0x55f83704d1f0) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/other/pmi-spawner/gasnet_bootstrap_pmi.c:608
[0] #10 0x000055f8364689ff in gasneti_nodemapInit (exchangefn=0x55f8364a8ba0 <bootstrapExchange>, ids=0x55f836761698 <gasneti_my_hostid>, sz=4, stride=0) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/gasnet_internal.c:2297
[0] #11 0x000055f83632b5cc in gasnetc_init (client_p=0x55f836753d58 <myclient>, ep_p=0x55f836753d60 <myep>, tm_p=0x55f836753d48 <myteam>, clientName=0x55f8364b644b "chapel", argc=0x7ffe66c43b08, argv=0x7ffe66c43b10, flags=1) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/ofi-conduit/gasnet_core.c:123
[0] #12 0x000055f83632b256 in gex_Client_Init_GASNET_202450PARnopshmEVERYTHINGdebugtracestatsdebugmallocsrclines (client_p=0x55f836753d58 <myclient>, ep_p=0x55f836753d60 <myep>, tm_p=0x55f836753d48 <myteam>, clientName=0x55f8364b644b "chapel", argc=0x7ffe66c43b08, argv=0x7ffe66c43b10, flags=1) at /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/gasnet-src/ofi-conduit/gasnet_core.c:276
[0] #13 0x000055f83628ccc2 in chpl_comm_init ()
[0] #14 0x000055f8362845c5 in chpl_rt_init ()
[0] #15 0x000055f83627a5a6 in main ()
[0] [Inferior 1 (process 32951) detached]
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node freya01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
For all of these I used the following slurm job script.
#!/bin/bash
#SBATCH -t 0:10:0
#SBATCH --nodes=2
#SBATCH --exclusive
#SBATCH --partition=p.test
#SBATCH --output=output.chapel
source /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/setenv_chapel-debug-pmi.sh
module purge && module load gcc/11 cmake/3.28 doxygen/1.10.0 openmpi/4
module list
echo "Using $(which chpl)"
chpl test-locales.chpl --fast -o test-locales
# $CHPL_HOME/util/chplenv/printchplbuilds.py
echo "CHPL_HOME=$CHPL_HOME"
echo "CHPL_LAUNCHER=$CHPL_LAUNCHER"
echo "GASNET_OFI_SPAWNER=$GASNET_OFI_SPAWNER"
# Set the Chapel program and dynamic number of locales
export PROG="./test-locales"
export ARGS="-nl $SLURM_NNODES --verbose" # Dynamically set the number of locales
# Run the Chapel program
echo "Running Chapel program with $SLURM_NNODES locales..."
$PROG $ARGS
rm test-locales test-locales_real
Thanks for these additional notes, @dutta-alankar , I appreciate you giving it a try. I was not anticipating the pmi option to have a dependence on mpirun, and am not sure what to make of that.
I think if the mpi-based approach is working well enough for you, it probably makes sense to keep using that rather than to push on this harder, but let me know if you disagree.
Again, thanks for entertaining my question and sharing your results,
-Brad
I agree and mpi looks like the best option given the limitations, except that I have nothing to compare against and don't know how much will the performance be improved when switching to pmi or ssh.
1 Like
@dutta-alankar If you can afford to spend a little more time on this, I think there is still value to the Chapel and GASNet teams if we can determine what is causing your current issues with PMI-based job launch.
Can you please try adding export PMI_RUNCMD='srun %V --mpi=pmi2 -n %N -- %C' to the SLURM job? This will instruct gasnetrun_ofi to use srun, versus what I suspect is a default use of OpenMPI's mpirun.
1 Like
I also want to take a moment to respond to "I have nothing to compare against and don't know how much will the performance be improved".
We are currently trying to switch among three different mechanisims for getting the Chapel application running across the nodes managed by SLURM. Except for small differences in how fast the applicaiton starts, there should be no performance impact to be concerned over.
1 Like
I tried this, but it still seems to be choosing MPI.
Environment:
#!/bin/sh
module purge && module load gcc/11 cmake/3.28 doxygen/1.10.0
export PATH=/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:$PATH
unset GASNET_BACKTRACE
unset CHPL_COMM_DEBUG
export CHPL_HOME=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=ofi
export FI_PROVIDER=psm2
export CHPL_LAUNCHER=gasnetrun_ofi
export GASNET_OFI_SPAWNER=pmi
export HFI_NO_CPUAFFINITY=1
export CHPL_LLVM=bundled
export CHPL_TARGET_CPU=skylake
# export OMPI_MCA_btl=tcp,self
export CHPL_GASNET_CFG_OPTIONS="--enable-debug"
export CHPL_COMM_DEBUG=1
export GASNET_BACKTRACE=1
job-script:
#!/bin/bash
#SBATCH -t 0:10:0
#SBATCH --nodes=2
#SBATCH --exclusive
#SBATCH --partition=p.test
#SBATCH --output=output.chapel
export PMI_RUNCMD="srun %V --mpi=pmi2 -n %N -- %C"
source /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/setenv_chapel-debug-pmi.sh
module purge && module load gcc/11 cmake/3.28 doxygen/1.10.0
module list
echo "Using $(which chpl)"
chpl test-locales.chpl --fast -o test-locales
# $CHPL_HOME/util/chplenv/printchplbuilds.py
echo "CHPL_HOME=$CHPL_HOME"
echo "CHPL_LAUNCHER=$CHPL_LAUNCHER"
echo "GASNET_OFI_SPAWNER=$GASNET_OFI_SPAWNER"
# Set the Chapel program and dynamic number of locales
export PROG="./test-locales"
export ARGS="-nl $SLURM_NNODES --verbose" # Dynamically set the number of locales
# Run the Chapel program
echo "Running Chapel program with $SLURM_NNODES locales..."
$PROG $ARGS
rm test-locales test-locales_real
output result:
Currently Loaded Modulefiles:
1) gcc/11(default) 2) cmake/3.28 3) doxygen/1.10.0
Key:
(symbolic-version)
Using /freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin/chpl
CHPL_HOME=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0
CHPL_LAUNCHER=gasnetrun_ofi
GASNET_OFI_SPAWNER=pmi
Running Chapel program with 2 locales...
/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none-debug/substrate-ofi/seg-everything/bin/gasnetrun_ofi -n 2 -N 2 -c 0 -E CONDA_SHLVL,LS_COLORS,LD_LIBRARY_PATH,CONDA_EXE,HOSTTYPE,SLURM_NODEID,SLURM_TASK_PID,SSH_CONNECTION,SPACK_PYTHON,LESSCLOSE,SLURM_PRIO_PROCESS,XKEYSYMDB,GASNET_BACKTRACE,CHPL_COMM_DEBUG,LANG,SLURM_SUBMIT_DIR,WINDOWMANAGER,LESS,HOSTNAME,OLDPWD,CHPL_TARGET_CPU,__MODULES_SHARE_MODULEPATH,CSHEDIT,ENVIRONMENT,PROG,GPG_TTY,LESS_ADVANCED_PREPROCESSOR,GASNET_OFI_SPAWNER,MPI_PATH,COLORTERM,CHOLLA_DIR,SLURM_CELL,ROCR_VISIBLE_DEVICES,SLURM_PROCID,SLURM_JOB_GID,CHPL_LAUNCHER,MACHTYPE,SLURMD_NODENAME,JOB_TMPDIR,MINICOM,SLURM_TASKS_PER_NODE,_CE_M,QT_SYSTEM_DIR,OSTYPE,XDG_SESSION_ID,MODULES_CMD,HFI_NO_CPUAFFINITY,SLURM_NNODES,USER,PAGER,DOMAIN,PLUTO_DIR,MORE,CHPL_COMM_SUBSTRATE,PWD,SLURM_JOB_NODELIST,HOME,SLURM_CLUSTER_NAME,CONDA_PYTHON_EXE,SLURM_NODELIST,SLURM_GPUS_ON_NODE,HOST,SSH_CLIENT,CHPL_COMM,XNLSPATH,CPATH,XDG_SESSION_TYPE,KRB5CCNAME,SLURM_JOB_CPUS_PER_NODE,INTERACTIVE,XDG_DATA_DIRS,MPCDF_SUBMODULE_COMBINATIONS,SLURM_TOPOLOGY_ADDR,_CE_CONDA,LIBGL_DEBUG,SLURM_WORKING_CLUSTER,__MODULES_LMALTNAME,GCC_HOME,SLURM_JOB_NAME,PROFILEREAD,TMPDIR,LIBRARY_PATH,SLURM_JOB_GPUS,SLURM_JOBID,SLURM_CONF,LOADEDMODULES,FI_PROVIDER,SLURM_NODE_ALIASES,SLURM_JOB_QOS,SLURM_TOPOLOGY_ADDR_PATTERN,SSH_TTY,FROM_HEADER,MAIL,SLURM_CPUS_ON_NODE,SLURM_JOB_NUM_NODES,SLURM_MEM_PER_NODE,LESSKEY,SPACK_ROOT,SHELL,TERM,XDG_SESSION_CLASS,CMAKE_HOME,SLURM_JOB_UID,ARGS,__MODULES_LMCONFLICT,XCURSOR_THEME,LS_OPTIONS,PMI_RUNCMD,SLURM_JOB_PARTITION,SLURM_JOB_USER,CUDA_VISIBLE_DEVICES,CHPL_LLVM,SHLVL,SLURM_SUBMIT_HOST,G_FILENAME_ENCODING,SLURM_JOB_ACCOUNT,MANPATH,AFS,CELL,MODULEPATH,CHPL_HOME,SLURM_GTIDS,LOGNAME,DBUS_SESSION_BUS_ADDRESS,CLUSTER,XDG_RUNTIME_DIR,SYS,CHPL_GASNET_CFG_OPTIONS,XDG_CONFIG_DIRS,PATH,SLURM_JOB_ID,_LMFILES_,MODULESHOME,INFOPATH,JOB_SHMTMPDIR,G_BROKEN_FILENAMES,HISTSIZE,CPU,DOXYGEN_HOME,SLURM_LOCALID,CVS_RSH,GPU_DEVICE_ORDINAL,LESSOPEN,OMPI_MCA_btl,BASH_FUNC_module%%,BASH_FUNC_spack%%,BASH_FUNC__module_raw%%,BASH_FUNC__spack_shell_wrapper%%,BASH_FUNC_mc%%,BASH_FUNC_ml%%,_, /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/test-locales_real -nl 2 --verbose
WARNING: Don't know how to control process->node layout with your mpirun
WARNING: PROCESS LAYOUT MIGHT NOT MATCH YOUR REQUEST
error running /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none-debug/substrate-ofi/seg-everything/bin/gasnetrun_ofi-mpi.pl:
gasnetrun: exec(mpirun -np 2 /usr/bin/env CONDA_SHLVL=0 LS_COLORS=no=00:fi=00:di=01%03b34:ln=00%03b36:pi=40%03b33:so=01%03b35:do=01%03b35:bd=40%03b33%03b01:cd=40%03b33%03b01:or=41%03b33%03b01:ex=00%03b32:%02a.cmd=00%03b32:%02a.exe=01%03b32:%02a.com=01%03b32:%02a.bat=01%03b32:%02a.btm=01%03b32:%02a.dll=01%03b32:%02a.tar=00%03b31:%02a.tbz=00%03b31:%02a.tgz=00%03b31:%02a.rpm=00%03b31:%02a.deb=00%03b31:%02a.arj=00%03b31:%02a.taz=00%03b31:%02a.lzh=00%03b31:%02a.lzma=00%03b31:%02a.zip=00%03b31:%02a.zoo=00%03b31:%02a.z=00%03b31:%02a.Z=00%03b31:%02a.gz=00%03b31:%02a.bz2=00%03b31:%02a.tb2=00%03b31:%02a.tz2=00%03b31:%02a.tbz2=00%03b31:%02a.xz=00%03b31:%02a.avi=01%03b35:%02a.bmp=01%03b35:%02a.dl=01%03b35:%02a.fli=01%03b35:%02a.gif=01%03b35:%02a.gl=01%03b35:%02a.jpg=01%03b35:%02a.jpeg=01%03b35:%02a.mkv=01%03b35:%02a.mng=01%03b35:%02a.mov=01%03b35:%02a.mp4=01%03b35:%02a.mpg=01%03b35:%02a.pcx=01%03b35:%02a.pbm=01%03b35:%02a.pgm=01%03b35:%02a.png=01%03b35:%02a.ppm=01%03b35:%02a.svg=01%03b35:%02a.tga=01%03b35:%02a.tif=01%03b35:%02a.webm=01%03b35:%02a.webp=01%03b35:%02a.wmv=01%03b35:%02a.xbm=01%03b35:%02a.xcf=01%03b35:%02a.xpm=01%03b35:%02a.aiff=00%03b32:%02a.ape=00%03b32:%02a.au=00%03b32:%02a.flac=00%03b32:%02a.m4a=00%03b32:%02a.mid=00%03b32:%02a.mp3=00%03b32:%02a.mpc=00%03b32:%02a.ogg=00%03b32:%02a.voc=00%03b32:%02a.wav=00%03b32:%02a.wma=00%03b32:%02a.wv=00%03b32: LD_LIBRARY_PATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux/lib64: CONDA_EXE=/u/adutt/anaconda3/bin/conda HOSTTYPE=x86_64 SLURM_NODEID=0 SLURM_TASK_PID=24318 SSH_CONNECTION=130.183.12.24%02039374%020130.183.183.24%02022 SPACK_PYTHON=/usr/bin/python3 LESSCLOSE=lessclose.sh%020%s%020%s SLURM_PRIO_PROCESS=0 XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB GASNET_BACKTRACE=1 CHPL_COMM_DEBUG=1 LANG=en_US SLURM_SUBMIT_DIR=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test WINDOWMANAGER=/usr/bin/icewm-session LESS=-M%020-I%020-R HOSTNAME=freyag01 OLDPWD=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0 CHPL_TARGET_CPU=skylake __MODULES_SHARE_MODULEPATH=/mpcdf/soft/SLE_15/modules/compilers:15:/mpcdf/soft/SLE_15/modules/tools:15:/mpcdf/soft/SLE_15/modules/ml:15:/mpcdf/soft/SLE_15/modules/java:15:/mpcdf/soft/SLE_15/modules/visualization:15:/mpcdf/soft/SLE_15/modules/gpu:15:/mpcdf/soft/SLE_15/modules/python:15:/mpcdf/soft/SLE_15/modules/libs:15:/mpcdf/soft/SLE_15/modules/third-party-compilers:15 CSHEDIT=emacs ENVIRONMENT=BATCH PROG=./test-locales GPG_TTY=/dev/pts/0 LESS_ADVANCED_PREPROCESSOR=no GASNET_OFI_SPAWNER=pmi MPI_PATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux COLORTERM=1 CHOLLA_DIR=/u/adutt/cholla/src SLURM_CELL=FREYA ROCR_VISIBLE_DEVICES=0,1 SLURM_PROCID=0 SLURM_JOB_GID=10100 CHPL_LAUNCHER=gasnetrun_ofi MACHTYPE=x86_64-suse-linux SLURMD_NODENAME=freyag01 JOB_TMPDIR=/tmp MINICOM=-c%020on SLURM_TASKS_PER_NODE=40%028x2%029 _CE_M= QT_SYSTEM_DIR=/usr/share/desktop-data OSTYPE=linux XDG_SESSION_ID=c13363 MODULES_CMD=/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/libexec/modulecmd.tcl HFI_NO_CPUAFFINITY=1 SLURM_NNODES=2 USER=adutt PAGER=less DOMAIN=bc.rzg.mpg.de PLUTO_DIR=/u/adutt/jan23/pluto-clcrush/PLUTO MORE=-sl CHPL_COMM_SUBSTRATE=ofi PWD=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test SLURM_JOB_NODELIST=freyag%05b01-02%05d HOME=/u/adutt SLURM_CLUSTER_NAME=freya CONDA_PYTHON_EXE=/u/adutt/anaconda3/bin/python SLURM_NODELIST=freyag%05b01-02%05d SLURM_GPUS_ON_NODE=2 HOST=freya01 SSH_CLIENT=130.183.12.24%02039374%02022 CHPL_COMM=gasnet XNLSPATH=/usr/X11R6/lib/X11/nls CPATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux/include64 XDG_SESSION_TYPE=tty KRB5CCNAME=FILE:/tmp/krb5cc_28744_vv8BTgmpDp SLURM_JOB_CPUS_PER_NODE=40%028x2%029 INTERACTIVE=TRUE XDG_DATA_DIRS=/usr/share MPCDF_SUBMODULE_COMBINATIONS=/mpcdf/soft/SLE_15/sub/gcc_11/combinations SLURM_TOPOLOGY_ADDR=opasw%05b5-6%05d.opasw4.freyag01 _CE_CONDA= LIBGL_DEBUG=quiet SLURM_WORKING_CLUSTER=freya:freyaio2:6817:9472:101 __MODULES_LMALTNAME=gcc/11%026gcc/default%026gcc%026as%07cgcc/latest:cmake/3.28%026as%07ccmake/latest:doxygen/1.10.0%026as%07cdoxygen/default%026as%07cdoxygen/latest GCC_HOME=/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0 SLURM_JOB_NAME=chapel-job PROFILEREAD=true TMPDIR=/tmp LIBRARY_PATH=/afs/@cell/common/soft/intel/ics2018.2/18.0/linux/lib64: SLURM_JOB_GPUS=0,1 SLURM_JOBID=762502 SLURM_CONF=/etc/slurm/slurm.conf LOADEDMODULES=gcc/11:cmake/3.28:doxygen/1.10.0 FI_PROVIDER=psm2 SLURM_NODE_ALIASES=%028null%029 SLURM_JOB_QOS=normal SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node SSH_TTY=/dev/pts/0 FROM_HEADER= MAIL=/var/spool/mail/adutt SLURM_CPUS_ON_NODE=40 SLURM_JOB_NUM_NODES=2 SLURM_MEM_PER_NODE=9500 LESSKEY=/etc/lesskey.bin SPACK_ROOT=/u/adutt/spack SHELL=/bin/bash TERM=xterm-256color XDG_SESSION_CLASS=user CMAKE_HOME=/mpcdf/soft/SLE_15/packages/x86_64/cmake/3.28 SLURM_JOB_UID=28744 ARGS=-nl%0202%020--verbose __MODULES_LMCONFLICT=gcc/11%026gcc XCURSOR_THEME=DMZ LS_OPTIONS=-N%020--color=tty%020-T%0200 PMI_RUNCMD=srun%020%V%020--mpi=pmi2%020-n%020%N%020--%020%C SLURM_JOB_PARTITION=p.test SLURM_JOB_USER=adutt CUDA_VISIBLE_DEVICES=0,1 CHPL_LLVM=bundled SHLVL=2 SLURM_SUBMIT_HOST=freya01 G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252 SLURM_JOB_ACCOUNT=mpa MANPATH=/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0/share/man:/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/share/man:/usr/local/man:/usr/share/man AFS=localhost CELL=ipp-garching.mpg.de MODULEPATH=/mpcdf/soft/SLE_15/sub/gcc_11/modules/ml:/mpcdf/soft/SLE_15/sub/gcc_11/modules/applications:/mpcdf/soft/SLE_15/sub/gcc_11/modules/mpi:/mpcdf/soft/SLE_15/sub/gcc_11/modules/libs:/mpcdf/soft/SLE_15/sub/gcc_11/modules/tools:/mpcdf/soft/SLE_15/modules/third-party-compilers:/mpcdf/soft/SLE_15/modules/java:/mpcdf/soft/SLE_15/modules/visualization:/mpcdf/soft/SLE_15/modules/gpu:/mpcdf/soft/SLE_15/modules/ml:/mpcdf/soft/SLE_15/modules/compilers:/mpcdf/soft/SLE_15/modules/python:/mpcdf/soft/SLE_15/modules/libs:/mpcdf/soft/SLE_15/modules/tools CHPL_HOME=/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0 SLURM_GTIDS=0 LOGNAME=adutt DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/28744/bus CLUSTER=FREYA XDG_RUNTIME_DIR=/run/user/28744 SYS=amd64_sles15 CHPL_GASNET_CFG_OPTIONS=--enable-debug XDG_CONFIG_DIRS=/etc/xdg PATH=/mpcdf/soft/SLE_15/packages/x86_64/doxygen/1.10.0/bin:/mpcdf/soft/SLE_15/packages/x86_64/cmake/3.28/bin:/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0/bin:/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:/freya/ptmp/mpa/adutt/chapel-multi_locale/pmi-debug/bin:/u/adutt/anaconda3/condabin:/u/adutt/spack/bin:/u/adutt/.local/bin:/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/bin:/u/adutt/.local/bin:/u/adutt/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/afs/ipp/amd64_sles15/bin:/mpcdf/soft/SLE_15/packages/x86_64/find-module/1.0/bin SLURM_JOB_ID=762502 _LMFILES_=/mpcdf/soft/SLE_15/modules/compilers/gcc/11:/mpcdf/soft/SLE_15/modules/tools/cmake/3.28:/mpcdf/soft/SLE_15/modules/tools/doxygen/1.10.0 MODULESHOME=/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1 INFOPATH=/mpcdf/soft/SLE_15/packages/x86_64/gcc/11.2.0/share/info JOB_SHMTMPDIR=/dev/shm G_BROKEN_FILENAMES=1 HISTSIZE=1000 CPU=x86_64 DOXYGEN_HOME=/mpcdf/soft/SLE_15/packages/x86_64/doxygen/1.10.0 SLURM_LOCALID=0 CVS_RSH=ssh GPU_DEVICE_ORDINAL=0,1 LESSOPEN=lessopen.sh%020%s OMPI_MCA_btl=tcp,self BASH_FUNC_module%%=%028%029%020%07b%020%020_module_raw%020%022%024@%022%0202%03e%0261%00a%07d BASH_FUNC_spack%%=%028%029%020%07b%020%020:%020this%020is%020a%020shell%020function%020from:%020/u/adutt/spack/share/spack/setup-env.sh%03b%00a%020:%020the%020real%020spack%020script%020is%020here:%020/u/adutt/spack/bin/spack%03b%00a%020_spack_shell_wrapper%020%022%024@%022%03b%00a%020return%020%024%03f%00a%07d BASH_FUNC__module_raw%%=%028%029%020%07b%020%020eval%020%060/usr/bin/tclsh%020/mpcdf/soft/SLE_15/packages/x86_64/Modules/5.0.1/libexec/modulecmd.tcl%020bash%020%022%024@%022%060%03b%00a%020_mlstatus=%024%03f%03b%00a%020return%020%024_mlstatus%00a%07d BASH_FUNC__spack_shell_wrapper%%=%028%029%020%07b%020%020for%020var%020in%020DYLD_LIBRARY_PATH%020DYLD_FALLBACK_LIBRARY_PATH%03b%00a%020do%00a%020eval%020%022if%020%05b%020-n%020%05c%022%05c%024%07b%024%07bvar%07d-%07d%05c%022%020%05d%03b%020then%020export%020SPACK_%024var=%05c%024%07b%024%07bvar%07d%07d%03b%020fi%022%03b%00a%020done%03b%00a%020if%020%05b%020-n%020%022%024%07bZSH_VERSION:-%07d%022%020%05d%03b%020then%00a%020emulate%020-L%020sh%03b%00a%020fi%03b%00a%020_sp_flags=%022%022%03b%00a%020while%020%05b%020%021%020-z%020%024%07b1+x%07d%020%05d%020%026%026%020%05b%020%022%024%07b1%023-%07d%022%020%021=%020%022%024%07b1%07d%022%020%05d%03b%020do%00a%020_sp_flags=%022%024_sp_flags%020%0241%022%03b%00a%020shift%03b%00a%020done%03b%00a%020if%020%05b%020-n%020%022%024_sp_flags%022%020%05d%020%026%026%020%05b%020%022%024%07b_sp_flags%023%02ah%07d%022%020%021=%020%022%024%07b_sp_flags%07d%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_sp_flags%023%02aV%07d%022%020%021=%020%022%024%07b_sp_flags%07d%022%020%05d%03b%020then%00a%020command%020spack%020%024_sp_flags%020%022%024@%022%03b%00a%020return%03b%00a%020fi%03b%00a%020_sp_subcommand=%022%022%03b%00a%020if%020%05b%020%021%020-z%020%024%07b1+x%07d%020%05d%03b%020then%00a%020_sp_subcommand=%022%0241%022%03b%00a%020shift%03b%00a%020fi%03b%00a%020case%020%024_sp_subcommand%020in%020%00a%020%022cd%022%029%00a%020_sp_arg=%022%022%03b%00a%020if%020%05b%020-n%020%022%0241%022%020%05d%03b%020then%00a%020_sp_arg=%022%0241%022%03b%00a%020shift%03b%00a%020fi%03b%00a%020if%020%05b%020%022%024_sp_arg%022%020=%020%022-h%022%020%05d%020%07c%07c%020%05b%020%022%024_sp_arg%022%020=%020%022--help%022%020%05d%03b%020then%00a%020command%020spack%020cd%020-h%03b%00a%020else%00a%020LOC=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020spack%020location%020%024_sp_arg%020%022%024@%022%029%022%03b%00a%020if%020%05b%020-d%020%022%024LOC%022%020%05d%03b%020then%00a%020cd%020%022%024LOC%022%03b%00a%020else%00a%020return%0201%03b%00a%020fi%03b%00a%020fi%03b%00a%020return%00a%020%03b%03b%00a%020%022env%022%029%00a%020_sp_arg=%022%022%03b%00a%020if%020%05b%020-n%020%022%0241%022%020%05d%03b%020then%00a%020_sp_arg=%022%0241%022%03b%00a%020shift%03b%00a%020fi%03b%00a%020if%020%05b%020%022%024_sp_arg%022%020=%020%022-h%022%020%05d%020%07c%07c%020%05b%020%022%024_sp_arg%022%020=%020%022--help%022%020%05d%03b%020then%00a%020command%020spack%020env%020-h%03b%00a%020else%00a%020case%020%024_sp_arg%020in%020%00a%020activate%029%00a%020_a=%022%020%024@%022%03b%00a%020if%020%05b%020%022%024%07b_a%023%02a%020--sh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--csh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020-h%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--help%07d%022%020%021=%020%022%024_a%022%020%05d%03b%020then%00a%020command%020spack%020env%020activate%020%022%024@%022%03b%00a%020else%00a%020stdout=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020command%020spack%020%024_sp_flags%020env%020activate%020--sh%020%022%024@%022%029%022%020%07c%07c%020return%03b%00a%020eval%020%022%024stdout%022%03b%00a%020fi%00a%020%03b%03b%00a%020deactivate%029%00a%020_a=%022%020%024@%022%03b%00a%020if%020%05b%020%022%024%07b_a%023%02a%020--sh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--csh%07d%022%020%021=%020%022%024_a%022%020%05d%03b%020then%00a%020command%020spack%020env%020deactivate%020%022%024@%022%03b%00a%020else%00a%020if%020%05b%020-n%020%022%024%02a%022%020%05d%03b%020then%00a%020command%020spack%020env%020deactivate%020-h%03b%00a%020else%00a%020stdout=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020command%020spack%020%024_sp_flags%020env%020deactivate%020--sh%029%022%020%07c%07c%020return%03b%00a%020eval%020%022%024stdout%022%03b%00a%020fi%03b%00a%020fi%00a%020%03b%03b%00a%020%02a%029%00a%020command%020spack%020env%020%024_sp_arg%020%022%024@%022%00a%020%03b%03b%00a%020esac%03b%00a%020fi%03b%00a%020return%00a%020%03b%03b%00a%020%022load%022%020%07c%020%022unload%022%029%00a%020_a=%022%020%024@%022%03b%00a%020if%020%05b%020%022%024%07b_a%023%02a%020--sh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--csh%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020-h%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--list%07d%022%020%021=%020%022%024_a%022%020%05d%020%07c%07c%020%05b%020%022%024%07b_a%023%02a%020--help%07d%022%020%021=%020%022%024_a%022%020%05d%03b%020then%00a%020command%020spack%020%024_sp_flags%020%024_sp_subcommand%020%022%024@%022%03b%00a%020else%00a%020stdout=%022%024%028SPACK_COLOR=%022%024%07bSPACK_COLOR:-always%07d%022%020command%020spack%020%024_sp_flags%020%024_sp_subcommand%020--sh%020%022%024@%022%029%022%020%07c%07c%020return%03b%00a%020eval%020%022%024stdout%022%03b%00a%020fi%00a%020%03b%03b%00a%020%02a%029%00a%020command%020spack%020%024_sp_flags%020%024_sp_subcommand%020%022%024@%022%00a%020%03b%03b%00a%020esac%00a%07d BASH_FUNC_mc%%=%028%029%020%07b%020%020.%020/usr/share/mc/mc-wrapper.sh%00a%07d BASH_FUNC_ml%%=%028%029%020%07b%020%020module%020ml%020%022%024@%022%00a%07d GASNET_OFI_SPAWNER=pmi GASNET_BACKTRACE=1 GASNET_ENVCMD=/usr/bin/env GASNET_SPAWN_HAVE_MPI=0 GASNET_SPAWN_CONDUIT=OFI GASNET_SPAWN_HAVE_PMI=1 GASNET_PLATFORM=generic GASNET_PSHM_ENABLED=no /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-test/pmi-test/test-locales_real -nl 2 --verbose) failed: No such file or directory
Odd. That is not what I was expecting to see for output.
I appreciate your help trying that out, but feel like I am now wasting your time with no apparent benefits.
Let me know if I can do something that helps. Happy to contribute to it.