[Chapel Merge] Update IB/Omni-Path docs to talk about GASNet and MPI playing nice together / point to GASNet READMEs

Branch: refs/heads/main
Revision: 28a3cdceb4837f5217462a15282531f285fc8b06
Author: bradcray
Link: Update IB/Omni-Path docs to talk about GASNet and MPI playing nice together / point to GASNet READMEs by bradcray · Pull Request #26901 · chapel-lang/chapel · GitHub
Log Message:
Update IB/Omni-Path docs to talk about GASNet and MPI playing nice together / point to GASNet READMEs (#26901)

[reviewed by @jabraham17 and @PHHargrove]

This PR is motivated by a desire to talk more about how GASNet and MPI
can conflict with one another in their desire for resources, leading to
either performance hits or simply an error trying to acquire the desired
resources. It includes an error message that we've seen a few times
(ourselves, and by users) in recent Omni-Path runs. It primarily points
off to GASNet documentation for remedies.

While here, I also did some wordsmithing (including an improvement
suggested by Paul), added additional links to GASNet documentation about
mpi/pmi/ssh spawners, and made sure the OmniPath and IB cases are still
in sync (a few things had drifted apart).

Related to this PR: