[Chapel Merge] Fix sporadic issue with COMM=ofi where puts are not fully visible

Branch: fix-put-cache
Revision: c5014d192384e6348bf1e9b94c9f00bdf6b8381b
Author: jabraham17
Link: Fix sporadic issue with COMM=ofi where puts are not fully visible by jabraham17 · Pull Request #29558 · chapel-lang/chapel · GitHub
Log Message:
Fix sporadic issue with COMM=ofi where puts are not fully visible

Fixes a sporadic issue which can occur where the runtime assumes a non-blocking put is complete and relies on its result, but it may not actually be.

With the OFI comm layer (not using delivery complete), a non-blocking put is considered complete when the comm layer is done with it, but that does not mean the put's affects are actually visible to a given thread. I saw issues with this in the remote cache (compiling with --no-cache-remote made the issue go away) where sporadically a value would be out of date.

This is not a problem with other comm layers or even with other providers in OFI which use delivery complete

The fix is to force visibility of the operation, but only in the select cases we need it (namely this cache operation) to avoid unintended side effects

[Reviewed by @jhh67]

Compare: Fix sporadic issue with COMM=ofi where puts are not fully visible by jabraham17 · Pull Request #29558 · chapel-lang/chapel · GitHub

Diff:
M runtime/include/chpl-comm.h
M runtime/include/comm/ofi/chpl-comm-impl.h
M runtime/src/chpl-cache.c
M runtime/src/comm/ofi/comm-ofi.c
https://github.com/chapel-lang/chapel/pull/29558.diff