Parallelization of Locales

Hi Chapel Community,

I am quite confused with parallelization in Chapel. I have a big program, and I want to run it concurrently on Locales (Linux systems). I have been using coforall loc in Locales on loc do ..., but it runs the program sequentially. First on the first node, then when it is over, it starts it on the second node. I'm reading about begin and cobegin statement now to find a way to run the program concurrently. Is it possible to have

cobegin {
         begin{
                    on Loc[1] ...
                  }
         begin{
                   on Loc[2]...
                  }
             ....
}

The problem arises when we have an unknown number of Locales.

I appreciate any help.
Marjan

Hi Marjan —

This should definitely result in parallel execution. When you say "it runs the program sequentially", how are you determining that?

It is definitely legal to nest coforalls/cobegins/begins/foralls/foreachs arbitrarily. But as you note, if you have a variable number of tasks you need to create, the coforall is really the right way to do it. Also note that having begins within cobegins isn't a particularly useful pattern because you're essentially creating two tasks per on-clause in the code above.

-Brad

Hi Brad,
I have set a bunch of writeln() in some specific points of the program to report the locale's hostname and the values of functions. I see all the results for the Locale #2 come exactly after Locale #1. Even variables' initializations take place after the jobs are done on Locale #1. For example, I want both machines to start at the same time and get their initial values before any function call takes place on them since these initial values can get changed after functions call and I don't want it.

Hi Marjan —

Neither of these mean that the tasks aren't running concurrently. It could simply mean that it's faster to start up on locale 1 than locale 2 (e.g., if it's local) or that locale 1 wins the race to print out its contribution or initialize its variables because it's spun up epsilon time before locale 2 is.

Nothing about Chapel's tasks ensures that they will start simultaneously or run in lockstep. Each task is running independently, and will only synchronize with others if you ask them to in the code (e.g., through atomic variables, synchronization variables, barriers, or the implicit joins at the end of coforall/forall/cobegin/sync statements). So, for example, you could do something like:

use AllLocalesBarriers;

coforall loc in Locales {
  on loc {
    writeln("Locale ", loc.id, " is starting");
    allLocalesBarrier.barrier();
    writeln("Locale ", loc.id, " is continuing);
  }
}

if you needed them to all do one thing before any of them did another.

In any case, I strongly suspect that your tasks are running concurrently, but just not doing enough work that you're seeing any indication of that.

-Brad

Thank you a lot Brad for this great answer and for confirmation that coforall actually creates independent and parallel tasks. It was worrisome to me since it is my Ph.D. project, but now I am confident about it. Also, many thanks for suggesting the AllLocalesBarries, I absolutely need it for the initialization step of tasks.

@bradcray Hi Brad, this is the result of running coforall on two nodes. As you see the execution is sequential. One node after the other (Start and end time of the Model execution).I have used

coforall loc in Locales{
on loc {
run model
}
}

Is there any way to stop this way of execution?

Marjan

Is it just because of nodes' "power"? I am working on Compute Canada's cluster, so I am thinking the nodes should have the same power. The nodes I have used had the same core number and same memory capacity

Definitely


node power! It is proved!

Thank you!

Hi Marjan —

Sorry for the slow response (due to Thanksgiving). That output doesn't mean anything to me, but if you've found something that makes you happy, I'm happy as well.

Best wishes,
-Brad