Branch: refs/heads/main
Revision: 344b24c
Author: ronawho
Log Message:
Merge pull request #18243 from ronawho/upgrade-gasnet-ex-2021-8.0
Upgrade to GASNet-EX 2021.8.0 snapshot
The next official GASNet release will be 2021.9.0, but that comes out after our 1.25.0 release and there have been some some performance improvements on InfiniBand that we want to include in our release. Upgrade to a snapshot of the "stable" branch that the GASNet team has added 2021.8.0 version markers to.
This improves performance on InfiniBand by serializing polling on completion queues (CQs). Previously, we we getting high contention and false-sharing on the underlying locks that verbs uses to protect CQs. CQ polling is now protected by a cacheline aligned try-lock in gasnet, which reduces contention and resolves the false-sharing issue. For more info see https://bitbucket.org/berkeleylab/gasnet/pull-requests/468.
This serialization improves performance of concurrent communication. On the cascade lake nodes of 16-node-cs-hdr we see large improvements for RA-on, minor improvements for RA-rmo, as well as modest improvements for aggregated indexgather and NPB-FT. On the rome nodes of 16-node-cs-hdr performance for RA and NPB didn't change much, but indexgather improved more.
16-node-cs-hdr with 48-core CascadeLake CPUs:
config RA-on RA-rmo Indexgather NPB-FT before 0.0018 GUPS 0.0010 GUPS 1075 MB/s/node 470 GFLOPS after 0.0080 GUPS 0.0011 GUPS 1150 MB/s/node 490 GFLOPS
16-node-cs-hdr with 128-core Rome CPUs
config Indexgather before 1600 MB/s/node after 1875 MB/s/node
We originally saw this issue on an HPE Apollo system with HDR-100 InfiniBand. We no longer have access to that system but we do have some performance results from an early version of the serialization patch and we expect performance results for the final version included here to be similar. There we two partitions that I ran on -- one with 13 112-core CascadeLake nodes and the other with 240 128-core Rome nodes.
Here are Arkouda gather/scatter results on the 112-core Intel nodes:
And results for Arkouda argsort on the 128-core Rome nodes for large and massive problem sizes:
This also extends Cray PMI support to work on HPE Cray EX systems. See the ChangeLog for additional details and other improvements.
Resolves https://github.com/Cray/chapel-private/issues/2040
Helps Improve gasnet-ibv performance · Issue #14438 · chapel-lang/chapel · GitHub
Modified Files:
A third-party/gasnet/gasnet-src/tests/Makefile
M third-party/gasnet/README
M third-party/gasnet/gasnet-src/Bootstrap
M third-party/gasnet/gasnet-src/ChangeLog
M third-party/gasnet/gasnet-src/Makefile.am
M third-party/gasnet/gasnet-src/Makefile.in
M third-party/gasnet/gasnet-src/README
M third-party/gasnet/gasnet-src/README-tools
M third-party/gasnet/gasnet-src/acconfig.h
M third-party/gasnet/gasnet-src/acinclude.m4
M third-party/gasnet/gasnet-src/aries-conduit/README
M third-party/gasnet/gasnet-src/aries-conduit/gasnet_aries.c
M third-party/gasnet/gasnet-src/aries-conduit/gasnet_aries.h
M third-party/gasnet/gasnet-src/aries-conduit/gasnet_core.c
M third-party/gasnet/gasnet-src/aries-conduit/gasnet_core.h
M third-party/gasnet/gasnet-src/aries-conduit/gasnet_core_fwd.h
M third-party/gasnet/gasnet-src/aries-conduit/gasnet_extended.c
M third-party/gasnet/gasnet-src/aries-conduit/gasnet_extended_fwd.h
M third-party/gasnet/gasnet-src/configure
M third-party/gasnet/gasnet-src/configure.in
M third-party/gasnet/gasnet-src/docs/GASNet-EX.txt
M third-party/gasnet/gasnet-src/docs/memory_kinds.md
M third-party/gasnet/gasnet-src/extended-ref/coll/gasnet_coll_internal.h
M third-party/gasnet/gasnet-src/extended-ref/coll/gasnet_reduce.c
M third-party/gasnet/gasnet-src/extended-ref/coll/gasnet_refcoll.c
M third-party/gasnet/gasnet-src/extended-ref/coll/gasnet_scratch.c
M third-party/gasnet/gasnet-src/extended-ref/coll/gasnet_scratch.h
M third-party/gasnet/gasnet-src/extended-ref/coll/gasnet_team.c
M third-party/gasnet/gasnet-src/extended-ref/coll/gasnet_trees.c
M third-party/gasnet/gasnet-src/extended-ref/gasnet_extended.h
M third-party/gasnet/gasnet-src/extended-ref/gasnet_extended_help.h
M third-party/gasnet/gasnet-src/extended-ref/gasnet_extended_refbarrier.c
M third-party/gasnet/gasnet-src/extended-ref/ratomic/gasnet_ratomic.h
M third-party/gasnet/gasnet-src/extended-ref/ratomic/gasnet_refratomic.c
M third-party/gasnet/gasnet-src/extended-ref/vis/gasnet_strided.c
M third-party/gasnet/gasnet-src/gasnet.h
M third-party/gasnet/gasnet-src/gasnet_am.c
M third-party/gasnet/gasnet-src/gasnet_am.h
M third-party/gasnet/gasnet-src/gasnet_asm.h
M third-party/gasnet/gasnet-src/gasnet_atomic_bits.h
M third-party/gasnet/gasnet-src/gasnet_atomic_fwd.h
M third-party/gasnet/gasnet-src/gasnet_atomicops.h
M third-party/gasnet/gasnet-src/gasnet_basic.h
M third-party/gasnet/gasnet-src/gasnet_config.h.in
M third-party/gasnet/gasnet-src/gasnet_event_internal.h
M third-party/gasnet/gasnet-src/gasnet_fwd.h
M third-party/gasnet/gasnet-src/gasnet_help.h
M third-party/gasnet/gasnet-src/gasnet_internal.c
M third-party/gasnet/gasnet-src/gasnet_internal.h
M third-party/gasnet/gasnet-src/gasnet_legacy.c
M third-party/gasnet/gasnet-src/gasnet_membar.h
M third-party/gasnet/gasnet-src/gasnet_mmap.c
M third-party/gasnet/gasnet-src/gasnet_pshm.c
M third-party/gasnet/gasnet-src/gasnet_tm.c
M third-party/gasnet/gasnet-src/gasnet_toolhelp.h
M third-party/gasnet/gasnet-src/gasnet_tools.c
M third-party/gasnet/gasnet-src/gasnet_tools.h
M third-party/gasnet/gasnet-src/gasnet_trace.h
M third-party/gasnet/gasnet-src/gasnetex.h
M third-party/gasnet/gasnet-src/ibv-conduit/README
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_core.c
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_core.h
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_core_connect.c
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_core_fwd.h
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_core_internal.h
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_core_sndrcv.c
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_core_thread.c
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_extended.c
M third-party/gasnet/gasnet-src/ibv-conduit/gasnet_extended_fwd.h
M third-party/gasnet/gasnet-src/license.txt
M third-party/gasnet/gasnet-src/mpi-conduit/gasnet_core.c
M third-party/gasnet/gasnet-src/mpi-conduit/gasnet_core.h
M third-party/gasnet/gasnet-src/mpi-conduit/gasnet_core_fwd.h
M third-party/gasnet/gasnet-src/ofi-conduit/gasnet_core.c
M third-party/gasnet/gasnet-src/ofi-conduit/gasnet_core.h
M third-party/gasnet/gasnet-src/ofi-conduit/gasnet_core_fwd.h
M third-party/gasnet/gasnet-src/ofi-conduit/gasnet_ofi.c
M third-party/gasnet/gasnet-src/other/Makefile-libgasnet.mak.in
M third-party/gasnet/gasnet-src/other/gasnet_portable_platform.h
M third-party/gasnet/gasnet-src/other/kinds/gasnet_cuda_uva.c
M third-party/gasnet/gasnet-src/other/kinds/gasnet_refkinds.c
M third-party/gasnet/gasnet-src/other/mpi-spawner/README
M third-party/gasnet/gasnet-src/other/spawner/gasnetrun.pl
M third-party/gasnet/gasnet-src/other/ssh-spawner/gasnet_bootstrap_ssh.c
M third-party/gasnet/gasnet-src/other/valgrind/gasnet.supp
M third-party/gasnet/gasnet-src/smp-conduit/gasnet_core.c
M third-party/gasnet/gasnet-src/smp-conduit/gasnet_core.h
M third-party/gasnet/gasnet-src/smp-conduit/gasnet_core_fwd.h
M third-party/gasnet/gasnet-src/smp-conduit/gasnet_extended_fwd.h
M third-party/gasnet/gasnet-src/tests/test.h
M third-party/gasnet/gasnet-src/tests/testam.c
M third-party/gasnet/gasnet-src/tests/testcoll.c
M third-party/gasnet/gasnet-src/tests/testcontend.c
M third-party/gasnet/gasnet-src/tests/testcudauva.c
M third-party/gasnet/gasnet-src/tests/testgasnet.c
M third-party/gasnet/gasnet-src/tests/testlarge.c
M third-party/gasnet/gasnet-src/tests/testlegacy.c
M third-party/gasnet/gasnet-src/tests/testreduce.c
M third-party/gasnet/gasnet-src/tests/testslice.c
M third-party/gasnet/gasnet-src/tests/testsplit.c
M third-party/gasnet/gasnet-src/tests/testtools.c
M third-party/gasnet/gasnet-src/tests/upcr-harness/external-chapel/Makefile
M third-party/gasnet/gasnet-src/tests/upcr-harness/external-chapel/harness.conf
M third-party/gasnet/gasnet-src/tests/upcr-harness/external-upcxx/Makefile
M third-party/gasnet/gasnet-src/tests/upcr-harness/external-upcxx/harness.conf
M third-party/gasnet/gasnet-src/tests/upcr-harness/gasnet-tests/harness.conf
M third-party/gasnet/gasnet-src/ucx-conduit/README
M third-party/gasnet/gasnet-src/ucx-conduit/gasnet_core.c
M third-party/gasnet/gasnet-src/ucx-conduit/gasnet_core.h
M third-party/gasnet/gasnet-src/ucx-conduit/gasnet_core_fwd.h
M third-party/gasnet/gasnet-src/ucx-conduit/gasnet_core_internal.h
M third-party/gasnet/gasnet-src/ucx-conduit/gasnet_core_sndrcv.c
M third-party/gasnet/gasnet-src/ucx-conduit/gasnet_extended.c
M third-party/gasnet/gasnet-src/ucx-conduit/gasnet_extended_fwd.h
M third-party/gasnet/gasnet-src/udp-conduit/gasnet_core.c
M third-party/gasnet/gasnet-src/udp-conduit/gasnet_core.h
M third-party/gasnet/gasnet-src/udp-conduit/gasnet_core_fwd.h
M third-party/gasnet/gasnet-src/version.git
Compare: https://github.com/chapel-lang/chapel/compare/e5b2a7f098ef...344b24c14f9a