Building a distributed chapel array with multiple pointers


I recently reached out to the chapel team over email about the function makeArrayFromPtr() with regards to creating a chapel array at memory location returned from mmap for c interoperability. Thank you for the replies.

Now I'm curious if it is possible to create a single, distributed chapel array with multiple pointers. For example, I have N locales on N separate nodes with an mmap region. Is it possible to take the region pointer from each locale and have them accessible by a 1D chapel array like a block distribution or achieve similar functionality?


Hi Trevor -

It's definitely possible to do that but I don't see a way to do it unless you create a domain map.

Domain maps are really implementations of different array types. We talk about two kinds of domain maps - layouts and distributions. A layout is describing how an array is implemented within a locale (aka compute node). A distribution describes how an array is implemented across many locales. So Block is a distribution but if you are talking about having an array that wraps some mmap'd region then I'd say you're talking about a layout.

Now, the Block distribution is implemented as a bunch of local arrays. Each of these arrays has a layout - it's normally the default layout but Block has some support for specifying the layout for sparse arrays specifically (see the sparseLayoutType initializer argument). So, you could create a layout and then seek to make adjustments to Block to allow it to use your layout.

Alternatively, you could create a domain map that handles the distribution as well as your mmap issue. That would more or less amount to making something similar to Block. (I wouldn't favor this approach for that reason).

Anyway, maybe others will have a better idea, but my advice would be to try to create a domain map that is a layout for your mmap'd arrays and then to try to get Block to use it. Here I think one of the main challenges is that the domain map standard interface ("dsi") is neither stable or well documented. We do have this dsi documentation and you can learn a lot by looking at the source code for other layouts and distributions.


Hi Trevor —

Following up on Michael's response with a few additional comments:

  • I haven't tried this yet, but have recently been postulating that it ought to be possible for us to create an "expert interface" to create a block-distributed array in which the user would supply a C pointer per locale to the data buffer to use on that locale. That might help with a case where Chapel was "adopting" memory from external code, though in order to work correctly, the per-locale sizes of the chunks would need to conform to how Block distributes data in the event that the number of locales doesn't evenly divide the problem size. I have no experience with mmap, so am not sure whether it would bring additional challenges or pain points beyond doing this for a normal C pointer (but Michael does, so I'm feeling cautious due to his mention of needing a new layout).

  • As mentioned in response to your previous question, this theme resonates strongly with a project that one of our interns is working on this summer, to see what it would take to create a Chapel interface to Apache Arrow / Parquet. We're still in the early stages there, but my general sense is that the theme of wanting to "adopt" memory from another source is similar between the efforts. So I expect we'd be wrestling with and making progress with your style of question this summer even if you hadn't asked it.

  • Michael's correct that our current documentation on writing domain maps isn't spectacular. I have a longstanding intent to write a friendly, step-by-step "So you want to write a domain map?" document and should really try to make that happen this summer. If you were to decide to write your own domain map, we can use that as pressure on me to buckle down and get it done.