29014, "bonachea", "[Bug]: Parallel Chapel build can invoke incompatible sub-makes, leading to fatal errors", "2026-06-24T02:29:52Z"
As discussed on Slack and in #29007, GNU Make assumes/requires that any nested GNU Make invocations within a process tree performing a parallel build are using a compatible version of GNU Make. When this requirement is violated, internal errors from Make ensue. Here is a simple demonstration of the failure mode in a minimal Makefile, where I've invoked an outer Make v4.4 and then arranged for a nested invocation of Make v4.3:
cgpu1$ cat Makefile
bar:
@echo executing sub-make...
@set -x ; \
$$CHPL_MAKE foo
foo:
@env | grep MAKE
cgpu1$ env CHPL_MAKE=/usr/local/pkg/gmake/4.3/bin/make /usr/local/pkg/gmake/4.4/bin/make -j 4
executing sub-make...
+ /usr/local/pkg/gmake/4.3/bin/make foo
make[1]: *** internal error: invalid --jobserver-auth string 'fifo:/tmp/GMfifo2445453'. Stop.
make: *** [Makefile:4: bar] Error 2
Within a Makefile, this requirement is usually handled by using the make-provided $(MAKE) Makefile variable for nested invocations of make, which (usually) automatically ensures "the same" make is used for the nested invocation. (GNU make documentation on $(MAKE))
Unfortunately the $(MAKE) Makefile variable is not automatically exported to subshell environments (GNU make documentation on automatic exports). So if a subshell (say, a Python script) within a build wants to invoke a nested make command, it needs some mechanism to discover a make executable that is "compatible" with the ongoing outer parallel make, in order to avoid internal errors.
This same failure mode can and does impact parallel builds of Chapel under certain system conditions. This is the root cause for issue #25891.
Here is a simple demonstration on NERSC Perlmutter using main @ e229facefeb, with a deliberately minimal build configuration:
+ ./configure
Compiling Python scripts in util/
Currently selected Chapel configuration:
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: llvm *
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: native *
CHPL_LOCALE_MODEL: flat
CHPL_COMM: none *
CHPL_TASKS: fifo *
CHPL_LAUNCHER: none
CHPL_TIMERS: generic
CHPL_UNWIND: none *
CHPL_TARGET_MEM: cstdlib *
CHPL_ATOMICS: cstdlib
CHPL_GMP: none *
CHPL_HWLOC: none *
CHPL_RE2: none *
CHPL_LLVM: system *
CHPL_AUX_FILESYS: none
...
+ which gmake
/usr/bin/gmake
+ gmake --version
GNU Make 4.2.1
Built for x86_64-suse-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ /global/cfs/cdirs/m5060/rgayatri/install/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spac/gmake/4.4.1/bin/make --version
GNU Make 4.4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ /global/cfs/cdirs/m5060/rgayatri/install/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spac/gmake/4.4.1/bin/make -j16 VERBOSE=1
Making the compiler...
...
Making the modules...
mkdir -p standard/gen/linux64-x86_64-llvm
mkdir -p /<redacted>/chapel/modules/internal/gen
cd standard/gen/linux64-x86_64-llvm && CHPL_HOST_COMPILER='gnu' CHPL_HOST_CC='gcc' CHPL_HOST_CXX='g++' CHPL_TARGET_COMPILER='llvm' CHPL_TARGET_CC='/global/common/software/nersc9/pe-llvm/zen3-ampere80/llvm/21.1.4/bin/clang' CHPL_TARGET_CXX='/global/common/software/nersc9/pe-llvm/zen3-ampere80/llvm/21.1.4/bin/clang++' python3 /<redacted>/chapel/util/config/make_sys_basic_types.py ChapelSysCTypes.chpl
cd /<redacted>/chapel/modules/internal/gen && `/<redacted>/chapel/util/config/compileline --compile-c++` /<redacted>/chapel/util/config/gen-set-program-data-module.cpp && (./a.out ChapelSetProgramInfoDataEntries.chpl; rm a.out)
Error: command failed: ['gmake', '--no-print-directory', '-f', '/<redacted>/chapel/runtime/etc/Makefile.include', 'printcxxcompileline']
output was:
gmake[3]: *** internal error: invalid --jobserver-auth string 'fifo:/dev/shm/GMfifo153845'. Stop.
/bin/sh: /<redacted>/chapel/util/config/gen-set-program-data-module.cpp: Permission denied
make[2]: *** [Makefile:87: /<redacted>/chapel/modules/internal/gen/ChapelSetProgramInfoDataEntries.chpl] Error 126
make[2]: *** Waiting for unfinished jobs....
Error: command failed: ['gmake', '--no-print-directory', '-f', '/<redacted>/chapel/runtime/etc/Makefile.include', 'printcompileline']
output was:
gmake[3]: *** internal error: invalid --jobserver-auth string 'fifo:/dev/shm/GMfifo153845'. Stop.
/bin/sh: -E: command not found
ERROR: Found 0 max values, but 17 types were expected.
make[2]: *** [Makefile:75: standard/gen/linux64-x86_64-llvm/ChapelSysCTypes.chpl] Error 1
make[1]: *** [Makefile:106: modules] Error 2
make: *** [Makefile:69: comprt] Error 2
In this example I've deliberately invoked the top-level Makefile with a system install of GNU Make 4.4.1, but without adding it to my PATH. The first gmake in my PATH is /usr/bin/gmake which has incompatible version 4.2.1. This is the most straightforward demonstration I've found, but is just one way in which a nested sub-Make invocation might end up being an incompatible version that leads to this Make internal error. The problem was originally seen in a Spack build of Chapel using an external install from GNU Make from source, where it's especially likely because such installs might not include the gmake symlink (the gmake symlink is often added by a distro or other packaging, for example the gmake spackage explicitly adds the symlink after install, but is not part of the official GNU Make install).
In a nutshell, the problem shown above is that util/config/compileline.py invokes chpl_make.py which on Linux effectively returns $CHPL_MAKE || "gmake" .
When $CHPL_MAKE is unset, this results in a PATH search for gmake that (in the example above and original problem) turned up a different (in this case older) version of GNU Make than the one the ongoing outer build is using. This leads to a communication failure between the incompatible make processes of different versions (which precipitates the fatal error).
To state it another way, chpl_make.py should probably never be permitted to fall back to a $PATH search, because on a system with more than one GNU Make version installed it might result in the invocation of an incompatible version (and there's no guarantee that the compatible Make is anywhere in PATH!). IOW just because there's a gmake symlink somewhere in PATH doesn't imply it's safe to use as a nested make invocation within an ongoing parallel build, where a nested make invocation needs to be the same version as the outer make (or possibly at least a newer one).
The suggested resolution is to ensure $CHPL_MAKE is always set to a version compatible with the outer Make invocation, so that chpl_make.py selects a compatible Make.