18186, "mppf", "intermittent gasnet error in local UDP configuration", "2021-08-09T18:46:43Z"
Summary of Problem
I'm seeing errors along the lines of
*** FATAL ERROR(Node 0): An active message was returned to sender,
and trapped by the default returned message handler (handler 0):
Error Code: EBADENDPOINT: Destination endpoint does not exist
Message type: AM_REQUEST_M
Destination: (127.0.0.1:59597) (1)
Handler: 64
Tag: 0x7f000101000167f3
Arguments(5): 0x00000000 0x00000001 0x00000000 0x00000000 0x00000009
Aborting...
*** Caught a fatal signal (proc 0): SIGABRT(6)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace.
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
I have already tried setting CHPL_MASTERIP and CHPL_WORKERIP but that does not seem to help.
Steps to Reproduce
On my system, which is runing Ubuntu 21.04, with Chapel main (I am on 60571a055b but I have also reproduced it with 1.24), I am able to reproduce with the following commands:
source util/quickstart/setchplenv.bash
export CHPL_COMM=gasnet
make
chpl examples/hello6-taskpar-dist.chpl
for i in `seq 1 10` ; do ./hello6-taskpar-dist -nl 2; done
I have also seen this failure when using qthreads.
I am seeing the failure in something like half of the trials.
Configuration Information
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: gnu
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: native *
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet *
CHPL_COMM_SUBSTRATE: udp
CHPL_GASNET_SEGMENT: everything
CHPL_TASKS: fifo *
CHPL_LAUNCHER: amudprun
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: cstdlib *
CHPL_ATOMICS: cstdlib
CHPL_NETWORK_ATOMICS: none
CHPL_GMP: none *
CHPL_HWLOC: none
CHPL_RE2: none *
CHPL_LLVM: none *
CHPL_AUX_FILESYS: none
chpl version 1.25.0 pre-release (60571a055b)
Copyright 2020-2021 Hewlett Packard Enterprise Development LP
Copyright 2004-2019 Cray Inc.
(See LICENSE file for more details)
gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Stack Trace
Here is a run that worked followed by one that did not, with additional gasnet debug output enabled (GASNET_VERBOSEENV=1 and GASNET_BACKTRACE=1).
zeus:11 [(HEAD detached at 60571a055b)] $ ./hello6-taskpar-dist -nl 2
ENV parameter: GASNET_NETWORKDEPTH = 128 (default)
ENV parameter: GASNET_SPAWNFN = L
GASNET: master host name: zeus
ENV parameter: GASNET_MASTERIP = 127.0.0.1
ENV parameter: GASNET_WORKERIP = 127.0.0.0
ENV parameter: GASNET_ROUTE_OUTPUT = 0
ENV parameter: GASNET_WORKER_RANK = -1 (default)
ENV parameter: GASNET_WORKER_RANK = -1 (default)
GASNET: worker connecting to 127.0.0.1:58419
GASNET: worker connecting to 127.0.0.1:58419
ENV parameter: GASNET_LINEBUFFERSZ = 1024 (default)
GASNET: worker using IP 127.0.0.1
GASNET: worker using IP 127.0.0.1
ENV parameter: GASNET_FAULT_RATE = 0.0 (default)
GASNET: Endpoint table (nproc=2):
ENV parameter: GASNET_RECVDEPTH = 256 (default)
GASNET: P#0: (127.0.0.1:39878) tag: 0x7f0000010000bd34
ENV parameter: GASNET_SENDDEPTH = 128 (default)
GASNET: P#1: (127.0.0.1:60566) tag: 0x7f0000010001bd34
ENV parameter: GASNET_REQUESTTIMEOUT_MAX = 30000000 (default)
ENV parameter: GASNET_REQUESTTIMEOUT_INITIAL = 100000 (default)
ENV parameter: GASNET_REQUESTTIMEOUT_BACKOFF = 2 (default)
GASNET(Node 1): UDP SO_RCVBUF buffer successfully set to 413039 bytes
GASNET(Node 0): UDP SO_RCVBUF buffer successfully set to 413039 bytes
GASNET(Node 1): UDP SO_SNDBUF buffer successfully set to 413039 bytes
GASNET(Node 0): UDP SO_SNDBUF buffer successfully set to 413039 bytes
GASNET(Node 1): Worker 1/2 starting (tag=0x7f0000010001bd34)...
ENV parameter: GASNET_FS_SYNC = NO (default)
GASNET(Node 0): Worker 0/2 starting (tag=0x7f0000010000bd34)...
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INITVAL = NAN (default)
ENV parameter: GASNET_MALLOC_CLOBBER = NO (default)
ENV parameter: GASNET_MALLOC_CLOBBERVAL = NAN (default)
ENV parameter: GASNET_MALLOC_LEAKALL = NO (default)
ENV parameter: GASNET_MALLOC_SCANFREED = NO (default)
ENV parameter: GASNET_MALLOC_EXTRACHECK = NO (default)
ENV parameter: GASNET_DISABLE_ARGDECODE = NO (default)
ENV parameter: GASNET_DISABLE_ENVDECODE = NO (default)
ENV parameter: GASNET_BACKTRACE = YES
ENV parameter: GASNET_BACKTRACE_NODES = *not set* (default)
ENV parameter: GASNET_TMPDIR = *not set* (default)
ENV parameter: TMPDIR = *not set* (default)
ENV parameter: GASNET_BACKTRACE_TYPE = GDB,EXECINFO (default)
ENV parameter: GASNET_FREEZE_ON_ERROR = NO (default)
ENV parameter: GASNET_FREEZE_SIGNAL = *not set* (default)
ENV parameter: GASNET_BACKTRACE_SIGNAL = *not set* (default)
ENV parameter: GASNET_TRACEFILE = *empty* (default)
ENV parameter: GASNET_STATSFILE = *empty* (default)
ENV parameter: GASNET_TRACEFLUSH = NO (default)
ENV parameter: GASNET_TRACELOCAL = YES (default)
ENV parameter: GASNET_MALLOCFILE = *empty* (default)
ENV parameter: GASNET_USE_GETHOSTID = NO (default)
ENV parameter: GASNET_NODEMAP_EXACT = YES (default)
ENV parameter: GASNET_BARRIER = DISSEM (default)
ENV parameter: GASNET_COLL_SCRATCH_SIZE = 2 MB (default)
ENV parameter: GASNET_MAX_SEGSIZE = 13 GB (default)
ENV parameter: GASNET_CATCH_EXIT = YES (default)
ENV parameter: GASNET_DISABLE_MUNMAP = NO (default)
ENV parameter: GASNET_FS_SYNC = NO (default)
ENV parameter: GASNET_MAX_THREADS = 256 (default)
ENV parameter: GASNET_COLL_P2P_EAGER_MIN = 16 (default)
ENV parameter: GASNET_COLL_P2P_EAGER_SCALE = 16 (default)
ENV parameter: GASNET_COLL_ROOTED_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_BROADCAST_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_SCATTER_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_GATHER_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_GATHER_ALL_DISSEM_LIMIT_PER_THREAD = 1 KB (default)
ENV parameter: GASNET_COLL_GATHER_ALL_DISSEM_LIMIT = 1 KB (default)
ENV parameter: GASNET_COLL_EXCHANGE_DISSEM_LIMIT_PER_THREAD = 1 KB (default)
ENV parameter: GASNET_COLL_EXCHANGE_DISSEM_LIMIT = 1 KB (default)
ENV parameter: GASNET_COLL_EXCHANGE_DISSEM_RADIX = 2 (default)
ENV parameter: GASNET_COLL_PIPE_SEG_SIZE = 31 KB (default)
ENV parameter: GASNET_COLL_AUTOTUNE_WARM_ITERS = 5 (default)
ENV parameter: GASNET_COLL_AUTOTUNE_PERF_ITERS = 10 (default)
ENV parameter: GASNET_COLL_AUTOTUNE_ALLOW_FLAT_TREE = 1 (default)
ENV parameter: GASNET_COLL_TUNING_FILE = *not set* (default)
ENV parameter: GASNET_COLL_PRINT_AUTOTUNE_TIMER = NO (default)
ENV parameter: GASNET_COLL_PRINT_COLL_ALG = NO (default)
ENV parameter: GASNET_COLL_ENABLE_SEARCH = NO (default)
ENV parameter: GASNET_COLL_ENABLE_PROFILE = NO (default)
ENV parameter: GASNET_COLL_REDUCE_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_NO_SCRATCH_WARN_THRESHOLD = 8 (default)
ENV parameter: GASNET_VIS_AMPIPE = YES (default)
ENV parameter: GASNET_VIS_MAXCHUNK = 63 KB (default)
ENV parameter: GASNET_VIS_PUT_MAXCHUNK = 63 KB (default)
ENV parameter: GASNET_VIS_GET_MAXCHUNK = 63 KB (default)
ENV parameter: GASNET_VIS_REMOTECONTIG = YES (default)
Hello, world! (from locale 0 of 2 named zeus-0)
Hello, world! (from locale 1 of 2 named zeus-1)
GASNET: Waiting for child processes...
GASNET: result=48438 status=0 errno=Success
GASNET: result=48437 status=0 errno=Success
GASNET: Exiting after AMUDP_SPMDExit(0)...
zeus:11 [(HEAD detached at 60571a055b)] $ ./hello6-taskpar-dist -nl 2
ENV parameter: GASNET_NETWORKDEPTH = 128 (default)
ENV parameter: GASNET_SPAWNFN = L
GASNET: master host name: zeus
ENV parameter: GASNET_MASTERIP = 127.0.0.1
ENV parameter: GASNET_WORKERIP = 127.0.0.0
ENV parameter: GASNET_ROUTE_OUTPUT = 0
ENV parameter: GASNET_WORKER_RANK = -1 (default)
GASNET: worker connecting to 127.0.0.1:34749
ENV parameter: GASNET_WORKER_RANK = -1 (default)
GASNET: worker connecting to 127.0.0.1:34749
ENV parameter: GASNET_LINEBUFFERSZ = 1024 (default)
GASNET: worker using IP 127.0.0.1
GASNET: worker using IP 127.0.0.1
ENV parameter: GASNET_FAULT_RATE = 0.0 (default)
ENV parameter: GASNET_RECVDEPTH = 256 (default)
ENV parameter: GASNET_SENDDEPTH = 128 (default)
ENV parameter: GASNET_REQUESTTIMEOUT_MAX = 30000000 (default)
ENV parameter: GASNET_REQUESTTIMEOUT_INITIAL = 100000 (default)
GASNET: Endpoint table (nproc=2):
ENV parameter: GASNET_REQUESTTIMEOUT_BACKOFF = 2 (default)
GASNET: P#0: (127.0.0.1:56584) tag: 0x7f0000010000bd45
GASNET: P#1: (127.0.0.1:58486) tag: 0x7f0000010001bd45
GASNET(Node 0): UDP SO_RCVBUF buffer successfully set to 413039 bytes
GASNET(Node 0): UDP SO_SNDBUF buffer successfully set to 413039 bytes
GASNET(Node 1): UDP SO_RCVBUF buffer successfully set to 413039 bytes
ENV parameter: GASNET_FS_SYNC = NO (default)
GASNET(Node 0): Worker 0/2 starting (tag=0x7f0000010000bd45)...
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
ENV parameter: GASNET_MALLOC_INIT = NO (default)
GASNET(Node 1): UDP SO_SNDBUF buffer successfully set to 413039 bytes
ENV parameter: GASNET_MALLOC_INITVAL = NAN (default)
ENV parameter: GASNET_MALLOC_CLOBBER = NO (default)
ENV parameter: GASNET_MALLOC_CLOBBERVAL = NAN (default)
ENV parameter: GASNET_MALLOC_LEAKALL = NO (default)
GASNET(Node 1): Worker 1/2 starting (tag=0x7f0000010001bd45)...
ENV parameter: GASNET_MALLOC_SCANFREED = NO (default)
ENV parameter: GASNET_MALLOC_EXTRACHECK = NO (default)
ENV parameter: GASNET_DISABLE_ARGDECODE = NO (default)
ENV parameter: GASNET_DISABLE_ENVDECODE = NO (default)
ENV parameter: GASNET_BACKTRACE = YES
ENV parameter: GASNET_BACKTRACE_NODES = *not set* (default)
ENV parameter: GASNET_TMPDIR = *not set* (default)
ENV parameter: TMPDIR = *not set* (default)
ENV parameter: GASNET_BACKTRACE_TYPE = GDB,EXECINFO (default)
ENV parameter: GASNET_FREEZE_ON_ERROR = NO (default)
ENV parameter: GASNET_FREEZE_SIGNAL = *not set* (default)
ENV parameter: GASNET_BACKTRACE_SIGNAL = *not set* (default)
ENV parameter: GASNET_TRACEFILE = *empty* (default)
ENV parameter: GASNET_STATSFILE = *empty* (default)
ENV parameter: GASNET_TRACEFLUSH = NO (default)
ENV parameter: GASNET_TRACELOCAL = YES (default)
ENV parameter: GASNET_MALLOCFILE = *empty* (default)
ENV parameter: GASNET_USE_GETHOSTID = NO (default)
ENV parameter: GASNET_NODEMAP_EXACT = YES (default)
ENV parameter: GASNET_BARRIER = DISSEM (default)
ENV parameter: GASNET_COLL_SCRATCH_SIZE = 2 MB (default)
ENV parameter: GASNET_MAX_SEGSIZE = 13 GB (default)
ENV parameter: GASNET_CATCH_EXIT = YES (default)
ENV parameter: GASNET_DISABLE_MUNMAP = NO (default)
ENV parameter: GASNET_FS_SYNC = NO (default)
ENV parameter: GASNET_MAX_THREADS = 256 (default)
ENV parameter: GASNET_COLL_P2P_EAGER_MIN = 16 (default)
ENV parameter: GASNET_COLL_P2P_EAGER_SCALE = 16 (default)
ENV parameter: GASNET_COLL_ROOTED_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_BROADCAST_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_SCATTER_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_GATHER_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_COLL_GATHER_ALL_DISSEM_LIMIT_PER_THREAD = 1 KB (default)
ENV parameter: GASNET_COLL_GATHER_ALL_DISSEM_LIMIT = 1 KB (default)
ENV parameter: GASNET_COLL_EXCHANGE_DISSEM_LIMIT_PER_THREAD = 1 KB (default)
ENV parameter: GASNET_COLL_EXCHANGE_DISSEM_LIMIT = 1 KB (default)
ENV parameter: GASNET_COLL_EXCHANGE_DISSEM_RADIX = 2 (default)
ENV parameter: GASNET_COLL_PIPE_SEG_SIZE = 31 KB (default)
ENV parameter: GASNET_COLL_AUTOTUNE_WARM_ITERS = 5 (default)
ENV parameter: GASNET_COLL_AUTOTUNE_PERF_ITERS = 10 (default)
ENV parameter: GASNET_COLL_AUTOTUNE_ALLOW_FLAT_TREE = 1 (default)
ENV parameter: GASNET_COLL_TUNING_FILE = *not set* (default)
ENV parameter: GASNET_COLL_PRINT_AUTOTUNE_TIMER = NO (default)
ENV parameter: GASNET_COLL_PRINT_COLL_ALG = NO (default)
ENV parameter: GASNET_COLL_ENABLE_SEARCH = NO (default)
ENV parameter: GASNET_COLL_ENABLE_PROFILE = NO (default)
ENV parameter: GASNET_COLL_REDUCE_GEOM = KNOMIAL_TREE,2 (default)
ENV parameter: GASNET_NO_SCRATCH_WARN_THRESHOLD = 8 (default)
ENV parameter: GASNET_VIS_AMPIPE = YES (default)
ENV parameter: GASNET_VIS_MAXCHUNK = 63 KB (default)
ENV parameter: GASNET_VIS_PUT_MAXCHUNK = 63 KB (default)
ENV parameter: GASNET_VIS_GET_MAXCHUNK = 63 KB (default)
ENV parameter: GASNET_VIS_REMOTECONTIG = YES (default)
*** GASNET ERROR(Node 0): I just refused a message and returned to sender. Reason: EBADENDPOINT
*** FATAL ERROR(Node 1): An active message was returned to sender,
and trapped by the default returned message handler (handler 0):
Error Code: EBADENDPOINT: Destination endpoint does not exist
Message type: AM_REQUEST_M
Destination: (127.0.0.1:56584) (0)
Handler: 64
Tag: 0x7f0000010000bd45
Arguments(5): 0x00000000 0x00000001 0x00000000 0x00000000 0x00000009
Aborting...
*** Caught a fatal signal (proc 1): SIGABRT(6)
[1] Invoking GDB for backtrace...
[1] /usr/bin/gdb -nx -batch -x /tmp/gasnet_UsRI6O '/home/mppf/w/11/hello6-taskpar-dist_real' 48454
[1] 0x00007fd997c8283a in __GI___wait4 (pid=48456, stat_loc=stat_loc@entry=0x7ffea5106938, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[1] Id Target Id Frame
[1] * 1 process 48454 "hello6-taskpar-" 0x00007fd997c8283a in __GI___wait4 (pid=48456, stat_loc=stat_loc@entry=0x7ffea5106938, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[1]
[1] Thread 1 (process 48454 "hello6-taskpar-"):
[1] #0 0x00007fd997c8283a in __GI___wait4 (pid=48456, stat_loc=stat_loc@entry=0x7ffea5106938, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[1] #1 0x00007fd997c827fb in __GI___waitpid (pid=<optimized out>, stat_loc=stat_loc@entry=0x7ffea5106938, options=options@entry=0) at waitpid.c:38
[1] #2 0x00007fd997bf46ab in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:172
[1] #3 0x000055d7c719d859 in gasneti_system_redirected (cmd=0x55d7c754f580 <cmd> "/usr/bin/gdb -nx -batch -x /tmp/gasnet_UsRI6O '/home/mppf/w/11/hello6-taskpar-dist_real' 48454", stdout_fd=8) at /home/mppf/w/11/third-party/gasnet/gasnet-src/gasnet_tools.c:1279
[1] #4 0x000055d7c719e168 in gasneti_bt_gdb (fd=8) at /home/mppf/w/11/third-party/gasnet/gasnet-src/gasnet_tools.c:1535
[1] #5 0x000055d7c719eb2e in gasneti_print_backtrace (fd=2) at /home/mppf/w/11/third-party/gasnet/gasnet-src/gasnet_tools.c:1813
[1] #6 0x000055d7c719f1dc in _gasneti_print_backtrace_ifenabled (fd=2) at /home/mppf/w/11/third-party/gasnet/gasnet-src/gasnet_tools.c:1946
[1] #7 0x000055d7c7398ff7 in gasneti_defaultSignalHandler (sig=6) at /home/mppf/w/11/third-party/gasnet/gasnet-src/gasnet_internal.c:938
[1] #8 <signal handler called>
[1] #9 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
[1] #10 0x00007fd997bcb864 in __GI_abort () at abort.c:79
[1] #11 0x000055d7c7436669 in AMX_FatalErr (msg=0x55d7c74d2b68 "An active message was returned to sender,\n and trapped by the default returned message handler (handler 0):\nError Code: %s\nMessage type: %s\nDestination: %s (%i)\nHandler: %i\nTag: %s\nArguments(%i): %"...) at /home/mppf/w/11/third-party/gasnet/gasnet-src/other/amudp/../amx/amx_internal.c:96
[1] #12 0x000055d7c74482a2 in AMUDP_DefaultReturnedMsg_Handler (status=7, opcode=1, token=0x55d7c8fa62f8) at /home/mppf/w/11/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:1458
[1] #13 0x000055d7c7440cd3 in AMUDP_processPacket (buf=0x55d7c8fa62f8, isloopback=0) at /home/mppf/w/11/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:616
[1] #14 0x000055d7c7444750 in AMUDP_ServiceIncomingMessages (ep=0x55d7c8f45a30) at /home/mppf/w/11/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:857
[1] #15 0x000055d7c7444941 in AM_Poll (eb=0x55d7c8f448e0) at /home/mppf/w/11/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:886
[1] #16 0x000055d7c7444bfb in AMUDP_RequestGeneric(amudp_category_t, ep_t, amudp_node_t, handler_t, void *, size_t, uintptr_t, int, typedef __va_list_tag __va_list_tag *, uint8_t, uint8_t) (category=amudp_Short, ep=0x55d7c8f45a30, reply_endpoint=0, handler=64 '@', source_addr=0x0, nbytes=0, dest_offset=0, numargs=5, argptr=0x7ffea5109f08, systemType=0 '\000', systemArg=0 '\000') at /home/mppf/w/11/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:933
[1] #17 0x000055d7c744648d in AMUDP_RequestVA(ep_t, amudp_node_t, handler_t, int, typedef __va_list_tag __va_list_tag *) (request_endpoint=0x55d7c8f45a30, reply_endpoint=0, handler=64 '@', numargs=5, argptr=0x7ffea5109f08) at /home/mppf/w/11/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:1186
[1] #18 0x000055d7c7169230 in gasnetc_AMRequestShort (tm=0xffffaa28370b96ff, rank=0, handler=64 '@', flags=0, numargs=5, argptr=0x7ffea5109f08) at /home/mppf/w/11/third-party/gasnet/gasnet-src/udp-conduit/gasnet_core.c:772
[1] #19 0x000055d7c7169ad8 in gasnetc_AMRequestShortM (tm=0xffffaa28370b96ff, rank=0, handler=64 '@', flags=0, numargs=5) at /home/mppf/w/11/third-party/gasnet/gasnet-src/udp-conduit/gasnet_core.c:792
[1] #20 0x000055d7c7193db4 in gasnete_amdbarrier_send (team=0x55d7c8f46c70, phase=0, step=0, value=103, flags=0) at /home/mppf/w/11/third-party/gasnet/gasnet-src/extended-ref/gasnet_extended_refbarrier.c:743
[1] #21 0x000055d7c7194106 in gasnete_amdbarrier_notify (team=0x55d7c8f46c70, id=103, flags=0) at /home/mppf/w/11/third-party/gasnet/gasnet-src/extended-ref/gasnet_extended_refbarrier.c:910
[1] #22 0x000055d7c7196a10 in gasnete_barrier_notify_common (team=0x55d7c8f46c70, id=103, flags=0) at /home/mppf/w/11/third-party/gasnet/gasnet-src/extended-ref/gasnet_extended_refbarrier.c:1935
[1] #23 0x000055d7c7196f5f in gasnet_barrier_notify (id=103, flags=0) at /home/mppf/w/11/third-party/gasnet/gasnet-src/extended-ref/gasnet_extended_refbarrier.c:2007
[1] #24 0x000055d7c71267a1 in chpl_comm_barrier (msg=0x55d7c7460b70 "getting ready to broadcast addresses") at comm-gasnet.c:1056
[1] #25 0x000055d7c7125cca in chpl_comm_init (argc_p=0x7ffea510a33c, argv_p=0x7ffea510a330) at comm-gasnet.c:898
[1] #26 0x000055d7c7102d02 in chpl_rt_init (argc=3, argv=0x7ffea510a478) at chpl-init.c:178
[1] #27 0x000055d7c70f52c9 in main (argc=3, argv=0x7ffea510a478) at main.c:30
[1] [Inferior 1 (process 48454) detached]
GASNET(Node 0): Exiting after exit signal from master (-1)...