20749, "bradcray", "Optimize rank change slices to support a lazy creation mode, similar to rank-preserving slices", "2022-09-29T21:17:08Z"
Awhile back in #12985 and related PRs, we added the ability to
lazily communicate (rank-preserving) array slices like
A[2..n-1, 2..n-1] or A[3..4, 3..4] rather than proactively
privatizing them and communicating their descriptions to all
involved locales owning a piece of A. This gave us big wins,
but has still not been enabled by default due to it making a
few cases worse that we weren't sure we wanted to tolerate.
@e-kayrakli worked in this area recently as well, further
improving it in #17212 and related PRs. Since this is not
enabled by default, today it must be opted into using
-schpl_seriallizeSlices.
Recent user use cases have been trying to use rank-change
slices of distributed arrays and ended up moving away from
them due to the significant communication requirements for
setting up the slices. Where in some cases we've been
able to replace an A[i, 1..n] with, say A[i..i, 1..n] to keep
it a rank-preserving slice, in at least some of these user cases,
this has not been possible, due to the expression being more
like A2[i, 1..n] = A1[1..n] where A2 is a 2D array and A1 is
a 1D array. Specifically A2[i..i, 1..n] = A1[1..n] is not legal
because it's a 1D-to-2D assignment. As a result of these
overheads, these users have switched to writing these paterns
using loops and indexing rather than slicing, which works,
but feels like a shame.
In retrospect, the potential benefits for lazily privatizing
rank-change slices are at least as great as for rank-preserving
slices, because given a 2D array block-distributed in both
dimensions (say) to a j x k grid of locales, a 1D slice of that
array will necessarily only need to involve j or k of the locales
rather than all j*k of them.
To that end, it seems it would be worthwhile to apply similar
serialization support to our rank-change array views as to the
rank-preserving cases in hopes of reaping similar benefits.
Note that the scalar code cost of indexing into a rank-change
slice may still be a bit more expensive than a rank-preserving
slice (?), but given that the communication costs are what's
killing things now for distributed arrays, we should start there.