Branch: refs/heads/main
Revision: 2b789ea368cfe30673ecdab2d3373b6f2185a51b
Author: DanilaFe
Link: Fix chapel-py build parallelism by using Make. by DanilaFe · Pull Request #28593 · chapel-lang/chapel · GitHub
Log Message:
Fix chapel-py build parallelism by using Make. (#28593)
I expect this to close (though we should wait and see)
https://github.com/Cray/chapel-private/issues/7733.
Ninja (the default build engine for scikit-build) uses parallelism by
default, and doesn't seem to understand the constraints of the top-level
Makefile. Thus, even running make chapel-py-venv with -j1 caused
Ninja to parallelize the build. This has led to out-of-memory issues on
machines with constrained memory, where we normally serialize
compilation to avoid OOM.
Newer versions of Ninja support the Make jobserver, which is the feature
that makes Make pass along parallelism to subjobs. However, I don't feel
like we should commit to using Ninja specifically, since we don't
generally use Ninja, and since chapel-py is small enough not to
benefit that much from it.
So, explicitly specify the builder as Unix Makefiles, which properly
inherits parallelism flags from the outer Make invocation. Seems like
the default make on macOS is failing to pass parallelism through to
the pip install -> CMake -> inner make pipeline, but using gmake
works fine. The consequence of this PR is that if you use plain make (a
20 year old version of it on macOS) you will build chapel-py serially.
Since you can use gmake to avoid this, I find this an acceptable
tradeoff.
Reviewed by @jabraham17 and @arifthpe -- thanks!
Testing
-
gmake -j1buildschapel-pyserially -
gmake -j32buildschapel-pyin parallel
Diff:
M third-party/chpl-venv/Makefile
https://github.com/chapel-lang/chapel/pull/28593.diff