18287, "ronawho", "Support gasnet-ibv segment everything", "2021-08-24T19:28:34Z"
Improve gasnet-ibv performance · Issue #14438 · chapel-lang/chapel · GitHub discusses our current use of gasnet-ibv segment large and fast. In both of those configurations gasnet provides us with a fixed segment that we can communicate out of. Segment fast uses static registration where the entire segment is registered at start up and segment large uses dynamic registration where memory is registered on the fly with the firehose registration library. Segment everything is a mode without a true segment where everything is communicable. This also uses dynamic registration but there is a potential for a performance hit because there's not just a single communicable range so tacking registration can be more involved.
Having a segment with large/fast modes puts us into a mode where we use a "fixed heap" and that can lead to fragmentation as detailed in Using a fixed heap can lead to fragmentation · Issue #18286 · chapel-lang/chapel · GitHub so we want to experiment with using segment everything as a way to avoid that fragmentation.
Today segment everything seems to fail when doing GETs from the readonly data segment (Stop communicating from read-only memory · Issue #18284 · chapel-lang/chapel · GitHub). We'd like to address that and add nightly correctness and performance testing.