[Chapel Merge] Fix an MCM issue between AMs and PUTs in ofi msg-o

Branch: refs/heads/main
Revision: 2542039
Author: ronawho
Link: Fix an MCM issue between AMs and PUTs in ofi msg-order-fence by ronawho · Pull Request #19575 · chapel-lang/chapel · GitHub
Log Message:

Merge pull request #19575 from ronawho/fix-ofi-fenced-am

Fix an MCM issue between AMs and PUTs in ofi msg-order-fence

[reviewed by @jhh67]

Fix an MCM issue between AMs and PUTs in ofi msg-order-fence

Fix a race where an AM wasn't waiting for outstanding PUTs to complete
for ofi message-order-fence. For a case like:

var a = 0;
on Locales[numLocales-1] {
  a = 1;
  on Locales[0] {
    assert(a == 1);
  }
}

the AM (on Locales[0]) wasn't waiting for the pending PUT (a = 1) to
complete. The comments in the code indicate we thought we were handling
this correctly by issuing our AM with FI_FENCE to wait for any pending
operations, but that does not appear to be happening.

Here I'm applying a hotfix to extend how we force message visibility to
stop skipping the target node. We want to understand what's going on
here better, but for now I just want to get a base fix in to address
failures we're seeing.

Also add a CHANGES.md entry since we're tying to land this hotfix in the
1.26 release.

Modified Files:
M CHANGES.md

M runtime/src/comm/ofi/comm-ofi.c

Compare: https://github.com/chapel-lang/chapel/compare/9f57eacdf5c3...254203979ca4