The only config that partially works is
export CHPL_LAUNCHER=gasnetrun_ofi
export GASNET_OFI_SPAWNER=ssh
and gives the output
Running Chapel program with 1 locales...
/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none/substrate-ofi/seg-everything/bin/gasnetrun_ofi -n 1 -N 1 -c 0 -E SLURM_MPI_TYPE,CONDA_SHLVL,LC_ALL,LS_COLORS,LD_LIBRARY_PATH,CONDA_EXE,HOSTTYPE,SLURM_NODEID,SLURM_TASK_PID,SSH_CONNECTION,SPACK_PYTHON,LESSCLOSE,OMPI_MCA_btl_openib_allow_ib,SLURM_PRIO_PROCESS,XKEYSYMDB,GASNET_BACKTRACE,LANG,SLURM_SUBMIT_DIR,WINDOWMANAGER,LESS,OMPI_MCA_io,HDF5_HOME,HOSTNAME,OLDPWD,CHPL_TARGET_CPU,__MODULES_SHARE_MODULEPATH,CSHEDIT,HDF5_ROOT,SLURM_DISTRIBUTION,ENVIRONMENT,PROG,GPG_TTY,OPENMPI_HOME,LESS_ADVANCED_PREPROCESSOR,GASNET_OFI_SPAWNER,MPI_PATH,COLORTERM,CHOLLA_DIR,SLURM_CELL,ROCR_VISIBLE_DEVICES,SLURM_PROCID,SLURM_JOB_GID,CHPL_LAUNCHER,MACHTYPE,SLURMD_NODENAME,JOB_TMPDIR,MINICOM,SLURM_TASKS_PER_NODE,_CE_M,QT_SYSTEM_DIR,OSTYPE,XDG_SESSION_ID,MODULES_CMD,HFI_NO_CPUAFFINITY,SLURM_NNODES,USER,PAGER,DOMAIN,PLUTO_DIR,MORE,CHPL_COMM_SUBSTRATE,PWD,SLURM_JOB_NODELIST,HOME,SLURM_CLUSTER_NAME,CONDA_PYTHON_EXE,LC_CTYPE,SLURM_NODELIST,SLURM_GPUS_ON_NODE,HOST,SSH_CLIENT,CHPL_COMM,XNLSPATH,CPATH,XDG_SESSION_TYPE,KRB5CCNAME,SLURM_JOB_CPUS_PER_NODE,INTERACTIVE,XDG_DATA_DIRS,MPCDF_SUBMODULE_COMBINATIONS,SLURM_TOPOLOGY_ADDR,_CE_CONDA,LIBGL_DEBUG,SLURM_WORKING_CLUSTER,__MODULES_LMALTNAME,GCC_HOME,SLURM_JOB_NAME,PROFILEREAD,TMPDIR,LIBRARY_PATH,SLURM_JOB_GPUS,SLURM_JOBID,SLURM_CONF,LOADEDMODULES,FI_PROVIDER,SLURM_NODE_ALIASES,SLURM_JOB_QOS,SLURM_TOPOLOGY_ADDR_PATTERN,SSH_TTY,OMPI_MCA_pml,FROM_HEADER,MAIL,SLURM_CPUS_ON_NODE,SLURM_JOB_NUM_NODES,SLURM_MEM_PER_NODE,LESSKEY,SPACK_ROOT,SHELL,TERM,XDG_SESSION_CLASS,CMAKE_HOME,SLURM_JOB_UID,ARGS,__MODULES_LMCONFLICT,XCURSOR_THEME,LS_OPTIONS,SLURM_JOB_PARTITION,SLURM_HINT,SLURM_JOB_USER,CUDA_VISIBLE_DEVICES,CHPL_LLVM,SHLVL,SLURM_SUBMIT_HOST,G_FILENAME_ENCODING,SLURM_JOB_ACCOUNT,MANPATH,AFS,CELL,MODULEPATH,CHPL_HOME,OMPI_MCA_btl_openib_if_include,SLURM_GTIDS,LOGNAME,DBUS_SESSION_BUS_ADDRESS,CLUSTER,XDG_RUNTIME_DIR,SYS,XDG_CONFIG_DIRS,PATH,SLURM_JOB_ID,_LMFILES_,MODULESHOME,PKG_CONFIG_PATH,INFOPATH,JOB_SHMTMPDIR,G_BROKEN_FILENAMES,OMPI_MCA_mtl,HISTSIZE,CPU,DOXYGEN_HOME,SLURM_LOCALID,CVS_RSH,GPU_DEVICE_ORDINAL,LESSOPEN,OMPI_MCA_btl,BASH_FUNC_module%%,BASH_FUNC_spack%%,BASH_FUNC__module_raw%%,BASH_FUNC__spack_shell_wrapper%%,BASH_FUNC_mc%%,BASH_FUNC_ml%%,_, /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real -nl 1 --verbose
0: using core(s) 0-39
oversubscribed = False
QTHREADS: Using 40 Shepherds
QTHREADS: Using 1 Workers per Shepherd
QTHREADS: Using 8384512 byte stack size.
comm task bound to accessible PUs
PSHM is disabled.
executing locale 0 of 1 on node 'freyag01'
(freyag01, 40)
But on requesting more than one node it fails
Running Chapel program with 2 locales...
/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none/substrate-ofi/seg-everything/bin/gasnetrun_ofi -n 2 -N 2 -c 0 -E SLURM_MPI_TYPE,CONDA_SHLVL,LC_ALL,LS_COLORS,LD_LIBRARY_PATH,CONDA_EXE,HOSTTYPE,SLURM_NODEID,SLURM_TASK_PID,SSH_CONNECTION,SPACK_PYTHON,LESSCLOSE,OMPI_MCA_btl_openib_allow_ib,SLURM_PRIO_PROCESS,XKEYSYMDB,GASNET_BACKTRACE,LANG,SLURM_SUBMIT_DIR,WINDOWMANAGER,LESS,OMPI_MCA_io,HDF5_HOME,HOSTNAME,OLDPWD,CHPL_TARGET_CPU,__MODULES_SHARE_MODULEPATH,CSHEDIT,HDF5_ROOT,SLURM_DISTRIBUTION,ENVIRONMENT,PROG,GPG_TTY,OPENMPI_HOME,LESS_ADVANCED_PREPROCESSOR,GASNET_OFI_SPAWNER,MPI_PATH,COLORTERM,CHOLLA_DIR,SLURM_CELL,ROCR_VISIBLE_DEVICES,SLURM_PROCID,SLURM_JOB_GID,CHPL_LAUNCHER,MACHTYPE,SLURMD_NODENAME,JOB_TMPDIR,MINICOM,SLURM_TASKS_PER_NODE,_CE_M,QT_SYSTEM_DIR,OSTYPE,XDG_SESSION_ID,MODULES_CMD,HFI_NO_CPUAFFINITY,SLURM_NNODES,USER,PAGER,DOMAIN,PLUTO_DIR,MORE,CHPL_COMM_SUBSTRATE,PWD,SLURM_JOB_NODELIST,HOME,SLURM_CLUSTER_NAME,CONDA_PYTHON_EXE,LC_CTYPE,SLURM_NODELIST,SLURM_GPUS_ON_NODE,HOST,SSH_CLIENT,CHPL_COMM,XNLSPATH,CPATH,XDG_SESSION_TYPE,KRB5CCNAME,SLURM_JOB_CPUS_PER_NODE,INTERACTIVE,XDG_DATA_DIRS,MPCDF_SUBMODULE_COMBINATIONS,SLURM_TOPOLOGY_ADDR,_CE_CONDA,LIBGL_DEBUG,SLURM_WORKING_CLUSTER,__MODULES_LMALTNAME,GCC_HOME,SLURM_JOB_NAME,PROFILEREAD,TMPDIR,LIBRARY_PATH,SLURM_JOB_GPUS,SLURM_JOBID,SLURM_CONF,LOADEDMODULES,FI_PROVIDER,SLURM_NODE_ALIASES,SLURM_JOB_QOS,SLURM_TOPOLOGY_ADDR_PATTERN,SSH_TTY,OMPI_MCA_pml,FROM_HEADER,MAIL,SLURM_CPUS_ON_NODE,SLURM_JOB_NUM_NODES,SLURM_MEM_PER_NODE,LESSKEY,SPACK_ROOT,SHELL,TERM,XDG_SESSION_CLASS,CMAKE_HOME,SLURM_JOB_UID,ARGS,__MODULES_LMCONFLICT,XCURSOR_THEME,LS_OPTIONS,SLURM_JOB_PARTITION,SLURM_HINT,SLURM_JOB_USER,CUDA_VISIBLE_DEVICES,CHPL_LLVM,SHLVL,SLURM_SUBMIT_HOST,G_FILENAME_ENCODING,SLURM_JOB_ACCOUNT,MANPATH,AFS,CELL,MODULEPATH,CHPL_HOME,OMPI_MCA_btl_openib_if_include,SLURM_GTIDS,LOGNAME,DBUS_SESSION_BUS_ADDRESS,CLUSTER,XDG_RUNTIME_DIR,SYS,XDG_CONFIG_DIRS,PATH,SLURM_JOB_ID,_LMFILES_,MODULESHOME,PKG_CONFIG_PATH,INFOPATH,JOB_SHMTMPDIR,G_BROKEN_FILENAMES,OMPI_MCA_mtl,HISTSIZE,CPU,DOXYGEN_HOME,SLURM_LOCALID,CVS_RSH,GPU_DEVICE_ORDINAL,LESSOPEN,OMPI_MCA_btl,BASH_FUNC_module%%,BASH_FUNC_spack%%,BASH_FUNC__module_raw%%,BASH_FUNC__spack_shell_wrapper%%,BASH_FUNC_mc%%,BASH_FUNC_ml%%,_, /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real -nl 2 --verbose
*** SSH-SPAWNER (freyag01:2567): Failed to start processes on freyag02, possibly due to an inability to establish an ssh connection from freyag01 without interactive authentication.
*** FATAL ERROR (freyag01:2567): in reap_one() at net/gasnet-src/other/ssh-spawner/gasnet_bootstrap_ssh.c:525: One or more processes died before setup was completed
*** WARNING (freyag01:2567): Ignoring call to gasneti_print_backtrace_ifenabled before gasneti_backtrace_init
test-locales_real:2567 terminated with signal 6 at PC=154a412fbcdb SP=7ffcce7bae60. Backtrace:
/lib64/libc.so.6(gsignal+0x10d)[0x154a412fbcdb]
/lib64/libc.so.6(abort+0x177)[0x154a412fd375]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x407291]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x406d5c]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x5a6079]
/lib64/libc.so.6(+0x4ad70)[0x154a412fbd70]
/lib64/libpthread.so.0(accept+0x13)[0x154a4230d811]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x5a0c0e]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x59f9bf]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x59f3ec]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x5967c4]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x505180]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x46bd5e]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x4656b4]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x45b726]
/lib64/libc.so.6(__libc_start_main+0xef)[0x154a412e62bd]
/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x407e7a]
Using
export GASNET_OFI_SPAWNER=mpi
fails with even one node
Running Chapel program with 1 locales...
/freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none/substrate-ofi/seg-everything/bin/gasnetrun_ofi -n 1 -N 1 -c 0 -E SLURM_MPI_TYPE,CONDA_SHLVL,LC_ALL,LS_COLORS,LD_LIBRARY_PATH,CONDA_EXE,HOSTTYPE,SLURM_NODEID,SLURM_TASK_PID,SSH_CONNECTION,SPACK_PYTHON,LESSCLOSE,OMPI_MCA_btl_openib_allow_ib,SLURM_PRIO_PROCESS,XKEYSYMDB,GASNET_BACKTRACE,LANG,SLURM_SUBMIT_DIR,WINDOWMANAGER,LESS,OMPI_MCA_io,HDF5_HOME,HOSTNAME,OLDPWD,CHPL_TARGET_CPU,__MODULES_SHARE_MODULEPATH,CSHEDIT,HDF5_ROOT,SLURM_DISTRIBUTION,ENVIRONMENT,PROG,GPG_TTY,OPENMPI_HOME,LESS_ADVANCED_PREPROCESSOR,GASNET_OFI_SPAWNER,MPI_PATH,COLORTERM,CHOLLA_DIR,SLURM_CELL,ROCR_VISIBLE_DEVICES,SLURM_PROCID,SLURM_JOB_GID,CHPL_LAUNCHER,MACHTYPE,SLURMD_NODENAME,JOB_TMPDIR,MINICOM,SLURM_TASKS_PER_NODE,_CE_M,QT_SYSTEM_DIR,OSTYPE,XDG_SESSION_ID,MODULES_CMD,HFI_NO_CPUAFFINITY,SLURM_NNODES,USER,PAGER,DOMAIN,PLUTO_DIR,MORE,CHPL_COMM_SUBSTRATE,PWD,SLURM_JOB_NODELIST,HOME,SLURM_CLUSTER_NAME,CONDA_PYTHON_EXE,LC_CTYPE,SLURM_NODELIST,SLURM_GPUS_ON_NODE,HOST,SSH_CLIENT,CHPL_COMM,XNLSPATH,CPATH,XDG_SESSION_TYPE,KRB5CCNAME,SLURM_JOB_CPUS_PER_NODE,INTERACTIVE,XDG_DATA_DIRS,MPCDF_SUBMODULE_COMBINATIONS,SLURM_TOPOLOGY_ADDR,_CE_CONDA,LIBGL_DEBUG,SLURM_WORKING_CLUSTER,__MODULES_LMALTNAME,GCC_HOME,SLURM_JOB_NAME,PROFILEREAD,TMPDIR,LIBRARY_PATH,SLURM_JOB_GPUS,SLURM_JOBID,SLURM_CONF,LOADEDMODULES,FI_PROVIDER,SLURM_NODE_ALIASES,SLURM_JOB_QOS,SLURM_TOPOLOGY_ADDR_PATTERN,SSH_TTY,OMPI_MCA_pml,FROM_HEADER,MAIL,SLURM_CPUS_ON_NODE,SLURM_JOB_NUM_NODES,SLURM_MEM_PER_NODE,LESSKEY,SPACK_ROOT,SHELL,TERM,XDG_SESSION_CLASS,CMAKE_HOME,SLURM_JOB_UID,ARGS,__MODULES_LMCONFLICT,XCURSOR_THEME,LS_OPTIONS,SLURM_JOB_PARTITION,SLURM_HINT,SLURM_JOB_USER,CUDA_VISIBLE_DEVICES,CHPL_LLVM,SHLVL,SLURM_SUBMIT_HOST,G_FILENAME_ENCODING,SLURM_JOB_ACCOUNT,MANPATH,AFS,CELL,MODULEPATH,CHPL_HOME,OMPI_MCA_btl_openib_if_include,SLURM_GTIDS,LOGNAME,DBUS_SESSION_BUS_ADDRESS,CLUSTER,XDG_RUNTIME_DIR,SYS,XDG_CONFIG_DIRS,PATH,SLURM_JOB_ID,_LMFILES_,MODULESHOME,PKG_CONFIG_PATH,INFOPATH,JOB_SHMTMPDIR,G_BROKEN_FILENAMES,OMPI_MCA_mtl,HISTSIZE,CPU,DOXYGEN_HOME,SLURM_LOCALID,CVS_RSH,GPU_DEVICE_ORDINAL,LESSOPEN,OMPI_MCA_btl,BASH_FUNC_module%%,BASH_FUNC_spack%%,BASH_FUNC__module_raw%%,BASH_FUNC__spack_shell_wrapper%%,BASH_FUNC_mc%%,BASH_FUNC_ml%%,_, /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real -nl 1 --verbose
*** FATAL ERROR (proc 0): in gasnetc_ofi_init() at /third-party/gasnet/gasnet-src/ofi-conduit/gasnet_ofi.c:1336: fi_endpoint for rdma failed: -22(Invalid argument)
*** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Details for bug reporting (proc 0): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=CLANG/19.1.3 sys=x86_64-pc-linux-gnu
[0] Invoking GDB for backtrace...
[0] /usr/bin/gdb -nx -batch -x /tmp/gasnet_GNlyhn '/freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real' 2951
[0] [New LWP 2952]
[0] [New LWP 2953]
[0] [New LWP 2954]
[0] [Thread debugging using libthread_db enabled]
[0] Using host libthread_db library "/lib64/libthread_db.so.1".
[0] 0x000014b70ddcd76f in wait4 () from /lib64/libc.so.6
[0] To enable execution of this file add
[0] add-auto-load-safe-path /freya/u/system/soft/SLE_15/packages/x86_64/gcc/10.3.0/lib64/libstdc++.so.6.0.28-gdb.py
[0] line to your configuration file "/u/adutt/.config/gdb/gdbinit".
[0] To completely disable this security protection add
[0] set auto-load safe-path /
[0] line to your configuration file "/u/adutt/.config/gdb/gdbinit".
[0] For more information about this security protection see the
[0] "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
[0] info "(gdb)Auto-loading safe path"
[0] Id Target Id Frame
[0] * 1 Thread 0x14b70fc36200 (LWP 2951) "test-locales_re" 0x000014b70ddcd76f in wait4 () from /lib64/libc.so.6
[0] 2 Thread 0x14b70a5dd700 (LWP 2952) "test-locales_re" 0x000014b70ddf71e9 in poll () from /lib64/libc.so.6
[0] 3 Thread 0x14b7036eb700 (LWP 2953) "test-locales_re" 0x000014b70de03e1f in epoll_wait () from /lib64/libc.so.6
[0] 4 Thread 0x14b6f75ac700 (LWP 2954) "test-locales_re" 0x000014b70ddf71e9 in poll () from /lib64/libc.so.6
[0]
[0] Thread 4 (Thread 0x14b6f75ac700 (LWP 2954) "test-locales_re"):
[0] #0 0x000014b70ddf71e9 in poll () from /lib64/libc.so.6
[0] #1 0x000014b70d18e7e5 in ips_ptl_pollintr (rcvthreadc=0x14b6f75abd80) at /home/scm/gitrepo/ifs-all/components/psm/temp.build/BUILD/libpsm2-11.2.228/ptl_ips/ptl_rcvthread.c:379
[0] #2 0x000014b70ed3d6ea in start_thread () from /lib64/libpthread.so.0
[0] #3 0x000014b70de03a8f in clone () from /lib64/libc.so.6
[0]
[0] Thread 3 (Thread 0x14b7036eb700 (LWP 2953) "test-locales_re"):
[0] #0 0x000014b70de03e1f in epoll_wait () from /lib64/libc.so.6
[0] #1 0x000014b70bdb85e7 in epoll_dispatch () from /mpcdf/soft/SLE_15/packages/skylake/openmpi/gcc_10-10.3.0/4.0.7/lib/libopen-pal.so.40
[0] #2 0x000014b70bdbb519 in opal_libevent2022_event_base_loop () from /mpcdf/soft/SLE_15/packages/skylake/openmpi/gcc_10-10.3.0/4.0.7/lib/libopen-pal.so.40
[0] #3 0x000014b703d8c0fe in progress_engine () from /mpcdf/soft/SLE_15/packages/skylake/openmpi/gcc_10-10.3.0/4.0.7/lib/openmpi/mca_pmix_pmix3x.so
[0] #4 0x000014b70ed3d6ea in start_thread () from /lib64/libpthread.so.0
[0] #5 0x000014b70de03a8f in clone () from /lib64/libc.so.6
[0]
[0] Thread 2 (Thread 0x14b70a5dd700 (LWP 2952) "test-locales_re"):
[0] #0 0x000014b70ddf71e9 in poll () from /lib64/libc.so.6
[0] #1 0x000014b70bdc34ad in poll_dispatch () from /mpcdf/soft/SLE_15/packages/skylake/openmpi/gcc_10-10.3.0/4.0.7/lib/libopen-pal.so.40
[0] #2 0x000014b70bdbb519 in opal_libevent2022_event_base_loop () from /mpcdf/soft/SLE_15/packages/skylake/openmpi/gcc_10-10.3.0/4.0.7/lib/libopen-pal.so.40
[0] #3 0x000014b70bd7f24e in progress_engine () from /mpcdf/soft/SLE_15/packages/skylake/openmpi/gcc_10-10.3.0/4.0.7/lib/libopen-pal.so.40
[0] #4 0x000014b70ed3d6ea in start_thread () from /lib64/libpthread.so.0
[0] #5 0x000014b70de03a8f in clone () from /lib64/libc.so.6
[0]
[0] Thread 1 (Thread 0x14b70fc36200 (LWP 2951) "test-locales_re"):
[0] #0 0x000014b70ddcd76f in wait4 () from /lib64/libc.so.6
[0] #1 0x000014b70dd44bc7 in do_system () from /lib64/libc.so.6
[0] #2 0x0000000000524b7a in gasneti_system_redirected ()
[0] #3 0x0000000000524520 in gasneti_bt_gdb ()
[0] #4 0x000000000051e36f in gasneti_print_backtrace ()
[0] #5 0x000000000040725b in gasneti_error_abort ()
[0] #6 0x0000000000406d5c in _gasneti_fatalerror ()
[0] #7 0x0000000000512a9c in gasnetc_ofi_init ()
[0] #8 0x00000000005051d6 in gex_Client_Init_GASNET_202450PARnopshmEVERYTHINGnodebugnotracenostatsnodebugmallocnosrclines ()
[0] #9 0x000000000046bd5e in chpl_comm_init ()
[0] #10 0x00000000004656b4 in chpl_rt_init ()
[0] #11 0x000000000045b726 in main ()
[0] [Inferior 1 (process 2951) detached]
[freyag01:02951] *** Process received signal ***
[freyag01:02951] Signal: Aborted (6)
[freyag01:02951] Signal code: (262373036)
[freyag01:02951] [ 0] /lib64/libc.so.6(+0x4ad70)[0x14b70dd36d70]
[freyag01:02951] [ 1] /lib64/libc.so.6(gsignal+0x10d)[0x14b70dd36cdb]
[freyag01:02951] [ 2] /lib64/libc.so.6(abort+0x177)[0x14b70dd38375]
[freyag01:02951] [ 3] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x407291]
[freyag01:02951] [ 4] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x406d5c]
[freyag01:02951] [ 5] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x512a9c]
[freyag01:02951] [ 6] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x5051d6]
[freyag01:02951] [ 7] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x46bd5e]
[freyag01:02951] [ 8] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x4656b4]
[freyag01:02951] [ 9] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x45b726]
[freyag01:02951] [10] /lib64/libc.so.6(__libc_start_main+0xef)[0x14b70dd212bd]
[freyag01:02951] [11] /freya/ptmp/mpa/adutt/chapel-multi_locale/test-locales_real[0x407e7a]
[freyag01:02951] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 2951 on node freyag01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Setting
export CHPL_LAUNCHER=gasnetrun_mpi
I get an error that some binary is non-existent
internal error: execvp() failed for command /freya/ptmp/mpa/adutt/chapel-multi_locale/ssh/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none/substrate-ofi/seg-everything/bin/gasnetrun_mpi: No such file or directory
ls
gives
adutt@freya01:chapel-2.3.0$ ls /freya/ptmp/mpa/adutt/chapel-multi_locale/chapel-2.3.0/third-party/gasnet/install/linux64-x86_64-skylake-llvm-none/substrate-ofi/seg-everything/bin/
gasnetrun_ofi gasnetrun_ofi-mpi.pl gasnetrun_ofi.pl gasnet_trace gasnet_trace.pl ident