[Chapel Merge] Don't try to use CQ wait/poll on macOS in comm=ofi

gbtitus1 · June 15, 2021, 10:27pm

Branch: refs/heads/master
Revision: 82f6f88
Author: gbtitus
Log Message:

Merge pull request #17897 from gbtitus/no-wait-poll-on-macOS

Don't try to use CQ wait/poll on macOS in comm=ofi.

(Reviewed by @jhh67.)

The libfabric wait/poll support requires epoll, which is a Linux thing
rather than a POSIX one and thus isn't present on macOS. But because
macOS is a rather secondary target for libfabric, they haven't coded any
sort of workaround for that either. The result is that depending on the
circumstances you can see a variety of unpleasant behaviors (internal
error, hang, segfault) if you turn on waiting and/or polling for CQs. So
here, just avoid wait/poll with CQs entirely under macOS.

There is some background information in the comments on this libfabric
issue, though the issue itself isn't directly about this problem:

github.com/ofiwg/libfabric

endpoint enumeration/ ipv6 endpoints unusable

opened 12:05AM - 22 Nov 19 UTC

u-u-h

I have been trying to set up a server/multi-client use case as follows: The ser…ver process does `fi_getinfo()` with the appropriate hints to identify all suitable endpoint candidates (basically `caps=(FI_MSG|FI_RMA|FI_TAGGED)`, `mode=FI_CONTEXT`, `ep_type=FI_EP_RDM`). Then it creates an endpoint for all of the entries and enables each of them. (I'll typically see some socket/IPv4, socket/IPv6 and some GNI endpoints on my test system.) As it turns out, the IPv4 and GNI endpoints have an address that is reasonable: IPv4 has the right local address and some (high) port number, GNI is a nice Aries address. There are however some IPv6 endpoints (based on link-local addresses) which have a port number of `0` even after enabling the endpoint. Why does the socket provider not assign a port number at this stage for IPv6 endpoints? In a second step I'm serializing the endpoint addresses and pass them OOB to client processes. The goal is that a client can pick the most suitable provider/interface to connect to the server process. As it turns out, a client process may successfully insert the server's IPv6 address (even with `sin6_port==0`) into its address vector of an ipv6 endpoint, but `fi_send()` will subsequently fail (with errno -2). Is this an error on my side? How do I check that an endpoint obtained from an entry in the `fi_getinfo()` result where I did bind completion and event queue, address vector etc. and enabled it is actually reachable, in particular for the socket/IPv6 provider? What I really want to obtain is a server process with enabled RDM endpoints on all suitable network interfaces, and a list of these that a client can pick the most suitable from to talk to (where you have to imagine a multi-homed server and clients with heterogeneous network capabilities, e.g., some on simple IP, some on HPC interconnects).

Modified Files:
M runtime/src/comm/ofi/comm-ofi.c

Compare: https://github.com/chapel-lang/chapel/compare/21a5fe46de72...82f6f885d4d9

[Chapel Merge] Don't try to use CQ wait/poll on macOS in comm=ofi

Related Topics