# Array initialization by expression

**URL:** <https://chapel.discourse.group/t/array-initialization-by-expression/18059>\
**Category:** Users\
**Created:** [December 11, 2022, 1:30pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059 "2022-12-11T13:30:26Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![damianmoz](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/damianmoz/32/319_2.png) [@damianmoz](https://chapel.discourse.group/u/damianmoz)\
**Post date:** [December 11, 2022, 1:30pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/1 "2022-12-11T13:30:26Z")

</div>

The declaration and initialization sequence

```chapel
var e : [eD] real;

forall (i,j,k) in eD do e[i, j, k] = expression;

```

is way faster than the far more compact

```chapel
const e = forall (i,j,k) in eD do expression;

```

Any ideas why?

---

<div class="post-metadata">

**Author:** ![vass](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/vass/32/692_2.png) [@vass](https://chapel.discourse.group/u/vass)\
**Post date:** [December 13, 2022, 2:40am UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/2 "2022-12-13T02:40:06Z")

</div>

Damian, how big is your `eD` ? Asymptotically the two approaches should have identical performance.

While the first approach performs parallel iteration over a domain, the second approach does parallel zippered iteration over the resulting array and the iterable object for the forall expression. [Reference: it is the loop 'forall (r, src) in zip(result, ir)' in chpl\_\_initCopy\_shapeHelp() in modules/internal/ChapelArray.chpl]

So it is likely that the additional overhead in the second approach is observable for a smaller eD.

---

<div class="post-metadata">

**Author:** ![damianmoz](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/damianmoz/32/319_2.png) [@damianmoz](https://chapel.discourse.group/u/damianmoz)\
**Post date:** [December 13, 2022, 2:04pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/3 "2022-12-13T14:04:31Z")

</div>

eD is {1..130, 1..130, 1..130},

It is heavily parallelized. Running on a 2\*12 core system, the first takes 30 seconds. The second takes 76 seconds.

EDIT - The elapsed time consumed by the offending routing which does a

```Chapel
max reduce abs(e);

```

and nothing else, goes up by approximately a factor of 4.

EDIT - actually a factor of nearly 5 (which I investigate a bit more).

If you tell me how to attach a file to a Discourse post, I can upload the code.

---

<div class="post-metadata">

**Author:** ![cassella](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/cassella/32/53_2.png) [@cassella](https://chapel.discourse.group/u/cassella)\
**Post date:** [December 21, 2022, 11:53pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/4 "2022-12-21T23:53:57Z")

</div>

> [@damianmoz](#):
>
> If you tell me how to attach a file to a Discourse post, I can upload the code.

It seems discourse has a very short list of allowed attachment file types, and .chpl isn't on it (nor is .txt). You could format the code as a .pdf or .png. 🙂

I think I read somewhere that maybe someone with chapel.discourse admin rights could add .chpl or .txt to the allowed list?

Another option if the code is too long to post inline might be to use a gist: [https://gist.github.com/](https://gist.github.com/) or even create a github repo for it if it's even more complex.

---

<div class="post-metadata">

**Author:** ![damianmoz](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/damianmoz/32/319_2.png) [@damianmoz](https://chapel.discourse.group/u/damianmoz)\
**Post date:** [December 22, 2022, 12:30am UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/5 "2022-12-22T00:30:30Z")

</div>

See `calc_erroX` below. These are tweaks of what exists in Anna Jesus's finite difference code as seen in the Discourse posting about "Code Optimization". For performance reasons, they return the worst scalar error, **NOT** the array of errors as Anna does. The one which initialises the array in the declaration has woeful performance. I got rid of the divisions from the original code.

```chapel
    proc calc_erroY(f, tf, dx2: real, dy2: real, dz2: real)
    {
        param one = 1.0;
        const rdx2 = one / dx2;
        const rdy2 = one / dy2;
        const rdz2 = one / dz2;
        const (_i, _j, _k) = f.domain.dims();
        const innerIJK = {_i.expand(-1), _j.expand(-1), _k.expand(-1)};
        var e : [innerIJK] real;

        forall (i,j,k) in innerIJK do
        {
            const fijk = f[i,j,k], t = fijk + fijk;

            e[i,j,k] = ((f[i,j,k-1] + f[i,j,k+1]) - t)*rdz2 - tf[i,j,k]
                + ((f[i,j-1,k] + f[i,j+1,k]) - t)*rdy2
                + ((f[i-1,j,k] + f[i+1,j,k]) - t)*rdx2;
        }
        return max reduce(abs(e));
    }
    proc calc_erroZ(f, tf, dx2: real, dy2: real, dz2: real)
    {
        param one = 1.0;
        const rdx2 = one / dx2;
        const rdy2 = one / dy2;
        const rdz2 = one / dz2;
        const rxyz = 2.0 * (rdx2 + rdy2 + rdz2);
        const (_i, _j, _k) = f.domain.dims();
        const innerIJK = {_i.expand(-1), _j.expand(-1), _k.expand(-1)};
        var e : [innerIJK] real;

        forall (i,j,k) in innerIJK do
        {
            const _f = f[i,j,k], t = _f + _f;

            e[i,j,k] = tf[i,j,k] -
                (f[i,j,k-1] - t + f[i,j,k+1])*rdz2 -
                (f[i,j-1,k] - t + f[i,j+1,k])*rdy2 -
                (f[i-1,j,k] - t + f[i+1,j,k])*rdx2;
        }
        return max reduce(abs(e));
    }
    proc calc_erroX(f, tf, dx2: real, dy2: real, dz2: real)
    {
        param one = 1.0;
        const rdx2 = one / dx2;
        const rdy2 = one / dy2;
        const rdz2 = one / dz2;
        const rxyz = 2.0 * (rdx2 + rdy2 + rdz2);
        const (_i, _j, _k) = f.domain.dims();
        const innerIJK = {_i.expand(-1), _j.expand(-1), _k.expand(-1)};
        const e = forall (i,j,k) in innerIJK do
            ((f[i,j,k-1] + f[i,j,k+1]))*rdz2 - tf[i,j,k] +
            ((f[i,j-1,k] + f[i,j+1,k]))*rdy2 +
            ((f[i-1,j,k] + f[i+1,j,k]))*rdx2 - rxyz * f[i,j,k];

        return max reduce(abs(e));
    }
    inline proc calc_erro(f, tf, dx2: real, dy2: real, dz2: real)
    {
        // enable or disable to taste
        // return calc_erroY(f, tf, dx2, dy2, dz2); // 30 seconds
        return calc_erroZ(f, tf, dx2, dy2, dz2); // 30 seconds
        // return calc_erroX(f, tf, dx2, dy2, dz2); // 76 seconds
    }

```

---

<div class="post-metadata">

**Author:** ![vass](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/vass/32/692_2.png) [@vass](https://chapel.discourse.group/u/vass)\
**Post date:** [December 23, 2022, 2:53am UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/6 "2022-12-23T02:53:55Z")

</div>

Damian,

Sorry for the uploading trouble. We will look into allowing uploads of Chapel code.

Nothing in your calc\_erroXYZ jumps at me as a plausible cause of performance difference, and I do not have the time to dig deeper.

Given how nicely you organized the code, I am curious about the performance of a version that does not create the array `e` altogether, instead reducing directly the values produced by the forall loop:

```Chapel
    return max reduce abs(
      forall (i,j,k) in innerIJK do
        ((f[i,j,k-1] + f[i,j,k+1]))*rdz2 - tf[i,j,k] +
        ((f[i,j-1,k] + f[i,j+1,k]))*rdy2 +
        ((f[i-1,j,k] + f[i+1,j,k]))*rdx2 - rxyz * f[i,j,k]
    );

```

---

<div class="post-metadata">

**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\
**Post date:** [January 3, 2023, 9:27pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/7 "2023-01-03T21:27:13Z")

</div>

> [@cassella](#):
>
> someone with chapel.discourse admin rights could add .chpl or .txt to the allowed list?

This should be enabled now.

-Brad

---

<div class="post-metadata">

**Author:** ![cassella](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/cassella/32/53_2.png) [@cassella](https://chapel.discourse.group/u/cassella)\
**Post date:** [January 4, 2023, 12:27am UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/8 "2023-01-04T00:27:29Z")

</div>

[checking.chpl](https://chapel.discourse.group/uploads/short-url/8rojfPJwBjvFLEp7t6jKrrXnwD1.chpl) (42 Bytes)

---

<div class="post-metadata">

**Author:** ![vass](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/vass/32/692_2.png) [@vass](https://chapel.discourse.group/u/vass)\
**Post date:** [January 4, 2023, 1:03am UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/9 "2023-01-04T01:03:25Z")

</div>

Thanks for checking @cassella. It works -- I can download the .chpl file that you uploaded.

---

<div class="post-metadata">

**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\
**Post date:** [March 11, 2023, 2:11am UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/10 "2023-03-11T02:11:13Z")

</div>

I'm coming very late to the meat of this topic after a December in which I couldn't keep up and am wondering if the original issue Damian asks about could be explained by:

> <https://github.com/chapel-lang/chapel/issues/13147>
>
> \[This is a longstanding issue that I'm capturing on GitHub since I couldn't find… it here\]
> 
> When iterating over multiple multidimensional arrays using zippered iteration, performance currently falls off a steep cliff. For example:
> 
> \`\`\`chapel
> var A, B: \[1..n, 1..n\] real;
> forall (a, b) in zip(A, B) do
> a = 2\*b;
> \`\`\`
> 
> This can also happen when calling a scalar function in a promoted manner with multidimensional arrays, since it is equivalent to a zippered iteration:
> 
> \`\`\`chapel
> foo(A, B);
> proc foo(x: real, y: real) { ... }
> \`\`\`
> 
> The reason for this is that the compiler currently doesn't know how to combine the parallel iterators for multidimensional arrays into a single compact loop nest, so it falls back to the catch-all implementation of creating an iterator object for each array and iterating over those objects in a zippered manner. While this result in a correct and very general-purpose implementation for very arbitrary array iterators, it is also very expensive.
> 
> Until this situation is improved, users can often work around the issue for cases that share the same domain (like above) by looping over the domain and indexing into the arrays to avoid the zippered iteration:
> 
> \`\`\`chapel
> forall ind in A.domain do
> A\[ind\] = 2\*B\[ind\];
> \`\`\`
> or:
> \`\`\`chapel
> forall ind in A.domain do
> foo(A\[ind\], B\[ind\]);
> \`\`\`

If the:

```Chapel
const e = forall (i,j,k) in eD do expression;

```

expression resulted in a zippered loop over `e` and `eD` then it might. But since this is an initialization context, I'm not sure whether that will happen, or whether the RHS will create a virtual array and then `e` will steal it. The expertise on that question lies somewhere between @vass and @mppf.

At the same time, it's taken me so long to catch up with this question that it may be moot or not of concern anymore. If it is (even a little bit), I think it'd be interesting to capture the two code idioms and some timings on a github issue.

-Brad

---

<div class="post-metadata">

**Author:** ![damianmoz](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/damianmoz/32/319_2.png) [@damianmoz](https://chapel.discourse.group/u/damianmoz)\
**Post date:** [March 12, 2023, 8:24pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/11 "2023-03-12T20:24:17Z")

</div>

Thanks for the insight. Yes it is still an issue. Not sure I have totally wrapped my head around the detail of #13147. Should we move this discussion into that issue or should are you suggest that we create a new issue? It needs some input from someone with more understanding of the internals than I have.

---

<div class="post-metadata">

**Author:** ![mppf](https://avatars.discourse-cdn.com/v4/letter/m/94ad74/32.png) [@mppf](https://chapel.discourse.group/u/mppf)\
**Post date:** [March 13, 2023, 1:30pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/12 "2023-03-13T13:30:00Z")

</div>

> [@bradcray](#):
>
> expression resulted in a zippered loop over `e` and `eD` then it might. But since this is an initialization context, I'm not sure whether that will happen, or whether the RHS will create a virtual array and then `e` will steal it. The expertise on that question lies somewhere between @vass and @mppf.

When initializing an array from a loop expression / iterator, indeed it ends up using a zippered loop. I think the relevant code for Damian's case is here: [https://github.com/chapel-lang/chapel/blob/main/modules/internal/ChapelArray.chpl#L2635](https://github.com/chapel-lang/chapel/blob/main/modules/internal/ChapelArray.chpl#L2635) but that should be confirmed before we rely on it.

---

<div class="post-metadata">

**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\
**Post date:** [March 13, 2023, 5:41pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/13 "2023-03-13T17:41:35Z")

</div>

> [@damianmoz](#):
>
> Should we move this discussion into that issue or should are you suggest that we create a new issue?

I was suggesting moving your observations into a new issue and then if we validate that it's an instance of #13147, we can have them refer to one another.

The gist of #13147 is that zippered iteration over multidimensional domains or arrays has known performance problems at present. Iteration over a single multidimensional domain or array does not share this problem.

-Brad

---

<div class="post-metadata">

**Author:** ![damianmoz](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/damianmoz/32/319_2.png) [@damianmoz](https://chapel.discourse.group/u/damianmoz)\
**Post date:** [March 13, 2023, 9:27pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/14 "2023-03-13T21:27:02Z")

</div>

So is

```Chapel
Slow zippered iteration over a multidimensional domain

```

a sufficiently concise title for the new issue?

---

<div class="post-metadata">

**Author:** ![bradcray](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/bradcray/32/72_2.png) [@bradcray](https://chapel.discourse.group/u/bradcray)\
**Post date:** [March 13, 2023, 9:33pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/15 "2023-03-13T21:33:49Z")

</div>

No, I think that's essentially what #13147 already says. For your issue I'd focus on the observation you're making rather than the cause (or presumed cause). So, for example, maybe something like "Multidimensional array initialization is much slower than multidimensional array assignment"

-Brad

---

<div class="post-metadata">

**Author:** ![damianmoz](https://yyz2.discourse-cdn.com/free1/user_avatar/chapel.discourse.group/damianmoz/32/319_2.png) [@damianmoz](https://chapel.discourse.group/u/damianmoz)\
**Post date:** [March 13, 2023, 9:50pm UTC](https://chapel.discourse.group/t/array-initialization-by-expression/18059/16 "2023-03-13T21:50:47Z")

</div>

Spot on. That will teach me to **not** hit reply before my early morning coffee..

Thanks