External Issue: Unexpected behavior of task intents with distributed forall loops

19864, "twesterhout", "Unexpected behavior of task intents with distributed forall loops", "2022-05-23T20:31:09Z"

Consider the following example:

use CyclicDist;

record R {
  var payload : int;

  proc init=(const ref other : R) {
    if other.locale == here then
      writeln("copy constructor called with a local object");
    else
      writeln("copy constructor called with a remote object");
    this.payload = other.payload;
  }
}

proc main() {
  const D = {0 ..# 4} dmapped Cyclic(startIdx=0);
  var xs : [D] int = [1, 2, 3, 4];
  var r = new R(123);

  writeln("Test #1");
  forall x in xs with (in r) {
    assert(r.payload == 123);
  }

  writeln("Test #2");
  forall x in xs {
    var localR = r;
    assert(localR.payload == 123);
  }
}

The idea here is to emulate the case when the user wants to implement their custom marshaling of objects. I expected tests one and two to behave very similarly (modulo the actual number of calls to the copy constructor), but they do not.
In the first test, the second branch in the constructor is never taken. The copy constructor is always invoked with a local object.
In the second test, however, the behavior is trivial: depending on the number of locales on which you run the test, up to 3 calls to the copy constructor will be made with remote data.

@bradcray provided some explanation on Gitter: