New Issue: Linker errors with Random while using the GPU Locale Model

Notifications Issues
1

19891, "ShreyasKhandekar", "Linker errors with Random while using the GPU Locale Model", "2022-05-26T23:27:37Z"

Summary of Problem

Compiling a simple program that uses fillRandom gives me a linker error for pow(double, double) and ldexp(double, int)

Steps to Reproduce

Source Code:

import Random;
config const size = 10;
var X : [0..#size] real;
Random.fillRandom(X);

When I compile using the Flat Locale Model the executable compiles and runs with no issues.
Compile command:

chpl random.chpl --print-commands --ldflags=--verbose --locale-model=flat

Gives me the following output:

<internal clang code generation> --gcc-toolchain=/opt/gcc/11.2.0/snos -I/lus/scratch/khandshr/chapel/modules/standard -I/lus/scratch/khandshr/chapel/modules/packages -I/lus/scratch/khandshr/chapel/runtime/include/localeModels/flat -I/lus/scratch/khandshr/chapel/runtime/include/localeModels -I/lus/scratch/khandshr/chapel/runtime/include/comm/none -I/lus/scratch/khandshr/chapel/runtime/include/comm -I/lus/scratch/khandshr/chapel/runtime/include/tasks/qthreads -I/lus/scratch/khandshr/chapel/runtime/include -I/lus/scratch/khandshr/chapel/runtime/include/qio -I/lus/scratch/khandshr/chapel/runtime/include/atomics/cstdlib -I/lus/scratch/khandshr/chapel/runtime/include/mem/jemalloc -I/lus/scratch/khandshr/chapel/third-party/utf8-decoder -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/include -I/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-flat/include -I/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-flat-jemalloc-bundled/include -I/lus/scratch/khandshr/chapel/third-party/jemalloc/install/target/cray-xc-x86_64-native-llvm-none/include -I/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/include -I. -I/tmp/chpl-khandshr.deleteme-egPowQ -I/lus/scratch/khandshr/chapel/modules/internal -DCHPL_GEN_CODE -pthread -I/opt/cray/pe/libsci/20.09.1/GNU/8.1/x86_64/include -I/opt/cray/pe/mpt/7.7.20/gni/mpich-gnu/8.2/include -I/opt/cray/pe/pmi/5.0.17/include -I/opt/cray/rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/nvvm/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/CUPTI/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/Debugger/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/math_libs/11.3/include -I/opt/cray/alps/6.6.67-7.0.4.0_33.26__gb91cd181.ari/include -I/opt/cray/wlm_detect/1.3.3-7.0.4.0_50.7__g7109084.ari/include -I/opt/cray/alps/6.6.67-7.0.4.0_33.26__gb91cd181.ari/include -I/opt/cray/krca/2.2.8-7.0.4.0_37.16__g59af36e.ari/include -I/opt/cray-hss-devel/9.0.0/include -I/opt/cray/xpmem/default/include -I/opt/cray/ugni/default/include -I/opt/cray/gni-headers/default/include -I/opt/cray/udreg/default/include -I/opt/cray/pe/atp/3.14.11/include -include sys_basic.h -include ctype.h -include wctype.h -include llvm/chapel_libc_wrapper.h 

# Make Binary - Linking
/cray/css/users/chapelu/llvm/13/linux64-x86_64/bin/clang++ --verbose /tmp/chpl-khandshr.deleteme-egPowQ/chpl__module.o /lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -o /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp --gcc-toolchain=/opt/gcc/11.2.0/snos -L/lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -lchpl -L/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -lgmp -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-flat/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-flat/lib -lhwloc -L/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -lqthread -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-flat/lib -lhwloc -lchpl -L/lus/scratch/khandshr/chapel/third-party/jemalloc/install/target/cray-xc-x86_64-native-llvm-none/lib -ljemalloc -L/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -lre2 -lnuma -lm -lpthread -L/opt/cray/pe/libsci/20.09.1/GNU/8.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.20/gni/mpich-gnu/8.2/lib -L/opt/cray/rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/nvvm/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/CUPTI/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/Debugger/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/math_libs/11.3/lib64 -L/opt/cray/pe/atp/3.14.11/lib -lAtpSigHandler -Wl,-rpath,/opt/cray/pe/atp/3.14.11/lib -Wl,--undefined=__libAtpSigHandler_ctor -lrca -Wl,--as-needed,-lcupti,-lcudart,--no-as-needed -lcuda -Wl,--as-needed,-lsci_gnu_82_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_82,--no-as-needed -Wl,--as-needed,-lmpich_gnu_82,--no-as-needed
clang version 13.0.0 (https://github.com/daviditen/chapel.git a1d85baf29fb91dbdf477da0d9929c3571bec968)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cray/css/users/chapelu/llvm/13/linux64-x86_64/bin
Found candidate GCC installation: /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0
Selected GCC installation: /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3, version 11.2
 "/usr/bin/ld" --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/crtbegin.o -L/lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -L/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-flat/lib -L/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-flat/lib -L/lus/scratch/khandshr/chapel/third-party/jemalloc/install/target/cray-xc-x86_64-native-llvm-none/lib -L/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -L/opt/cray/pe/libsci/20.09.1/GNU/8.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.20/gni/mpich-gnu/8.2/lib -L/opt/cray/rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/nvvm/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/CUPTI/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/Debugger/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/math_libs/11.3/lib64 -L/opt/cray/pe/atp/3.14.11/lib -L/opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0 -L/opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/cray/css/users/chapelu/llvm/13/linux64-x86_64/bin/../lib -L/lib -L/usr/lib /tmp/chpl-khandshr.deleteme-egPowQ/chpl__module.o /lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -lchpl -rpath /lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -lgmp -rpath /lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-flat/lib -lhwloc -rpath /lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -lqthread -lhwloc -lchpl -ljemalloc -rpath /lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -lre2 -lnuma -lm -lpthread -lAtpSigHandler -rpath /opt/cray/pe/atp/3.14.11/lib --undefined=__libAtpSigHandler_ctor -lrca --as-needed -lcupti -lcudart --no-as-needed -lcuda --as-needed -lsci_gnu_82_mpi --no-as-needed --as-needed -lsci_gnu_82 --no-as-needed --as-needed -lmpich_gnu_82 --no-as-needed -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/crtend.o /usr/lib/../lib64/crtn.o

# Make Binary - Building Launcher and Copying
gmake -f /tmp/chpl-khandshr.deleteme-egPowQ/Makefile
gmake -f /lus/scratch/khandshr/chapel/runtime/etc/Makefile.launcher all CHPL_MAKE_HOME=/lus/scratch/khandshr/chapel TMPBINNAME=/tmp/chpl-khandshr.deleteme-egPowQ/random.tmp BINNAME=random TMPDIRNAME=/tmp/chpl-khandshr.deleteme-egPowQ CHPL_MAKE_RUNTIME_LIB=/lus/scratch/khandshr/chapel/lib CHPL_MAKE_RUNTIME_INCL=/lus/scratch/khandshr/chapel/runtime/include CHPL_MAKE_THIRD_PARTY=/lus/scratch/khandshr/chapel/third-party
rm -f random_real
mv /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp random_real
echo "#include \"chplcgfns.h\"" > /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
echo "#include \"config.h\"" >> /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
echo "#include \"_config.c\"" >> /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
echo "#include \"chpl_compilation_config.c\"" >> /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
echo "const char launcher_real_suffix[] = \"_real\";" >> /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
echo "const char launcher_exe_suffix[] = \"\";" >> /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
# TODO would be nice to provide ABS path to REAL_BINARY_NAME instead of ./
echo "const char* launcher_mli_real_name = \"./random_real\";" >> /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
echo "const int launcher_is_mli = 0;" >> /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
gcc        \
	-c -o /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp_launcher.o \
	-DLAUNCHER -I/lus/scratch/khandshr/chapel/runtime/include/cray-xc -I/lus/scratch/khandshr/chapel/runtime/include -I. /tmp/chpl-khandshr.deleteme-egPowQ/launcher_support.c
g++    -o /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp_launcher -L/lus/scratch/khandshr/chapel/lib/cray-xc/gnu/x86_64/loc-flat/comm-none/tasks-qthreads/launch-slurm-srun/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/lib_pic-none/san-none /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp_launcher.o /lus/scratch/khandshr/chapel/lib/cray-xc/gnu/x86_64/loc-flat/comm-none/tasks-qthreads/launch-slurm-srun/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/lib_pic-none/san-none/main_launcher.o -lchpllaunch -lm 
cp /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp_launcher /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp
rm -f random
mv /tmp/chpl-khandshr.deleteme-egPowQ/random.tmp random

Running the executable works just fine in this case.

But when I have use the GPU Locale Model:

chpl random.chpl --print-commands --ldflags=--verbose

It gives me this output with the linker error at the end:

warning: The prototype GPU support implies --no-checks. This may impact debuggability. To suppress this warning, compile with --no-checks explicitly
<internal clang code generation> --gcc-toolchain=/opt/gcc/11.2.0/snos -I/lus/scratch/khandshr/chapel/modules/standard -I/lus/scratch/khandshr/chapel/modules/packages -I/lus/scratch/khandshr/chapel/runtime/include/localeModels/gpu -I/lus/scratch/khandshr/chapel/runtime/include/localeModels -I/lus/scratch/khandshr/chapel/runtime/include/comm/none -I/lus/scratch/khandshr/chapel/runtime/include/comm -I/lus/scratch/khandshr/chapel/runtime/include/tasks/qthreads -I/lus/scratch/khandshr/chapel/runtime/include -I/lus/scratch/khandshr/chapel/runtime/include/qio -I/lus/scratch/khandshr/chapel/runtime/include/atomics/cstdlib -I/lus/scratch/khandshr/chapel/runtime/include/mem/jemalloc -I/lus/scratch/khandshr/chapel/third-party/utf8-decoder -DHAS_GPU_LOCALE -I/lus/scratch/khandshr/chapel/runtime/include/gpu/cuda -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/include -I/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/include -I/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-gpu-jemalloc-bundled/include -I/lus/scratch/khandshr/chapel/third-party/jemalloc/install/target/cray-xc-x86_64-native-llvm-none/include -I/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/include -I. -I/tmp/chpl-khandshr.deleteme-OfBTn1 -I/lus/scratch/khandshr/chapel/modules/internal -DCHPL_GEN_CODE -pthread -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/include -I/opt/cray/pe/libsci/20.09.1/GNU/8.1/x86_64/include -I/opt/cray/pe/mpt/7.7.20/gni/mpich-gnu/8.2/include -I/opt/cray/pe/pmi/5.0.17/include -I/opt/cray/rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/nvvm/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/CUPTI/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/Debugger/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/math_libs/11.3/include -I/opt/cray/alps/6.6.67-7.0.4.0_33.26__gb91cd181.ari/include -I/opt/cray/wlm_detect/1.3.3-7.0.4.0_50.7__g7109084.ari/include -I/opt/cray/alps/6.6.67-7.0.4.0_33.26__gb91cd181.ari/include -I/opt/cray/krca/2.2.8-7.0.4.0_37.16__g59af36e.ari/include -I/opt/cray-hss-devel/9.0.0/include -I/opt/cray/xpmem/default/include -I/opt/cray/ugni/default/include -I/opt/cray/gni-headers/default/include -I/opt/cray/udreg/default/include -I/opt/cray/pe/atp/3.14.11/include --std=c++11 -x cuda --cuda-gpu-arch=sm_60 -include sys_basic.h -include /tmp/chpl-khandshr.deleteme-OfBTn1/command-line-includes.h -include llvm/chapel_libc_wrapper.h 
warning: Unknown CUDA version. cuda.h: CUDA_VERSION=11030. Assuming the latest supported version 10.1

# Check to see if ptxas command can be found
which ptxas > /dev/null 2>&1

# Check to see if fatbinary command can be found
which fatbinary > /dev/null 2>&1

# PTX to  object file
ptxas -m64 --gpu-name sm_60 --output-file /tmp/chpl-khandshr.deleteme-OfBTn1/chpl__gpu_ptx.o /tmp/chpl-khandshr.deleteme-OfBTn1/chpl__gpu_ptx.s
ptxas warning : Unresolved extern variable 'chpl_nodeID' in whole program compilation, ignoring extern qualifier

# object file to fatbinary
fatbinary -64 --create /tmp/chpl-khandshr.deleteme-OfBTn1/chpl__gpu.fatbin --image=profile=sm_60,file=/tmp/chpl-khandshr.deleteme-OfBTn1/chpl__gpu_ptx.o --image=profile=compute_60,file=/tmp/chpl-khandshr.deleteme-OfBTn1/chpl__gpu_ptx.s
<internal clang code generation> --gcc-toolchain=/opt/gcc/11.2.0/snos -I/lus/scratch/khandshr/chapel/modules/standard -I/lus/scratch/khandshr/chapel/modules/packages -I/lus/scratch/khandshr/chapel/runtime/include/localeModels/gpu -I/lus/scratch/khandshr/chapel/runtime/include/localeModels -I/lus/scratch/khandshr/chapel/runtime/include/comm/none -I/lus/scratch/khandshr/chapel/runtime/include/comm -I/lus/scratch/khandshr/chapel/runtime/include/tasks/qthreads -I/lus/scratch/khandshr/chapel/runtime/include -I/lus/scratch/khandshr/chapel/runtime/include/qio -I/lus/scratch/khandshr/chapel/runtime/include/atomics/cstdlib -I/lus/scratch/khandshr/chapel/runtime/include/mem/jemalloc -I/lus/scratch/khandshr/chapel/third-party/utf8-decoder -DHAS_GPU_LOCALE -I/lus/scratch/khandshr/chapel/runtime/include/gpu/cuda -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/include -I/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/include -I/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-gpu-jemalloc-bundled/include -I/lus/scratch/khandshr/chapel/third-party/jemalloc/install/target/cray-xc-x86_64-native-llvm-none/include -I/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/include -I. -I/tmp/chpl-khandshr.deleteme-OfBTn1 -I/lus/scratch/khandshr/chapel/modules/internal -DCHPL_GEN_CODE -pthread -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/include -I/opt/cray/pe/libsci/20.09.1/GNU/8.1/x86_64/include -I/opt/cray/pe/mpt/7.7.20/gni/mpich-gnu/8.2/include -I/opt/cray/pe/pmi/5.0.17/include -I/opt/cray/rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/nvvm/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/CUPTI/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/Debugger/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/math_libs/11.3/include -I/opt/cray/alps/6.6.67-7.0.4.0_33.26__gb91cd181.ari/include -I/opt/cray/wlm_detect/1.3.3-7.0.4.0_50.7__g7109084.ari/include -I/opt/cray/alps/6.6.67-7.0.4.0_33.26__gb91cd181.ari/include -I/opt/cray/krca/2.2.8-7.0.4.0_37.16__g59af36e.ari/include -I/opt/cray-hss-devel/9.0.0/include -I/opt/cray/xpmem/default/include -I/opt/cray/ugni/default/include -I/opt/cray/gni-headers/default/include -I/opt/cray/udreg/default/include -I/opt/cray/pe/atp/3.14.11/include --std=c++11 -x cuda --cuda-gpu-arch=sm_60 -include sys_basic.h -include /tmp/chpl-khandshr.deleteme-OfBTn1/command-line-includes.h -include llvm/chapel_libc_wrapper.h 
warning: Unknown CUDA version. cuda.h: CUDA_VERSION=11030. Assuming the latest supported version 10.1

# Make Binary - Linking
/cray/css/users/chapelu/llvm/13/linux64-x86_64/bin/clang++ --verbose /tmp/chpl-khandshr.deleteme-OfBTn1/chpl__module.o /lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -o /tmp/chpl-khandshr.deleteme-OfBTn1/random.tmp --gcc-toolchain=/opt/gcc/11.2.0/snos -L/lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -lchpl -L/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -lgmp -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/lib -lhwloc -L/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -lqthread -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/lib -lhwloc -lchpl -L/lus/scratch/khandshr/chapel/third-party/jemalloc/install/target/cray-xc-x86_64-native-llvm-none/lib -ljemalloc -L/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -Wl,-rpath,/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -lre2 -lcudart -lnuma -lm -lpthread -L/opt/cray/pe/libsci/20.09.1/GNU/8.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.20/gni/mpich-gnu/8.2/lib -L/opt/cray/rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/nvvm/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/CUPTI/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/Debugger/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/math_libs/11.3/lib64 -L/opt/cray/pe/atp/3.14.11/lib -lAtpSigHandler -Wl,-rpath,/opt/cray/pe/atp/3.14.11/lib -Wl,--undefined=__libAtpSigHandler_ctor -lrca -Wl,--as-needed,-lcupti,-lcudart,--no-as-needed -lcuda -Wl,--as-needed,-lsci_gnu_82_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_82,--no-as-needed -Wl,--as-needed,-lmpich_gnu_82,--no-as-needed
clang version 13.0.0 (https://github.com/daviditen/chapel.git a1d85baf29fb91dbdf477da0d9929c3571bec968)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cray/css/users/chapelu/llvm/13/linux64-x86_64/bin
Found candidate GCC installation: /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0
Selected GCC installation: /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3, version 11.2
 "/usr/bin/ld" --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/chpl-khandshr.deleteme-OfBTn1/random.tmp /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/crtbegin.o -L/lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -L/lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/lib -L/lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -L/lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/lib -L/lus/scratch/khandshr/chapel/third-party/jemalloc/install/target/cray-xc-x86_64-native-llvm-none/lib -L/lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -L/opt/cray/pe/libsci/20.09.1/GNU/8.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.20/gni/mpich-gnu/8.2/lib -L/opt/cray/rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/nvvm/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/CUPTI/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/cuda/11.3/extras/Debugger/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.5/math_libs/11.3/lib64 -L/opt/cray/pe/atp/3.14.11/lib -L/opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0 -L/opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/cray/css/users/chapelu/llvm/13/linux64-x86_64/bin/../lib -L/lib -L/usr/lib /tmp/chpl-khandshr.deleteme-OfBTn1/chpl__module.o /lus/scratch/khandshr/chapel/lib/cray-xc/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -lchpl -rpath /lus/scratch/khandshr/chapel/third-party/gmp/install/cray-xc-x86_64-native-llvm-none/lib -lgmp -rpath /lus/scratch/khandshr/chapel/third-party/hwloc/install/cray-xc-x86_64-native-llvm-none-gpu/lib -lhwloc -rpath /lus/scratch/khandshr/chapel/third-party/qthread/install/cray-xc-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -lqthread -lhwloc -lchpl -ljemalloc -rpath /lus/scratch/khandshr/chapel/third-party/re2/install/cray-xc-x86_64-native-llvm-none/lib -lre2 -lcudart -lnuma -lm -lpthread -lAtpSigHandler -rpath /opt/cray/pe/atp/3.14.11/lib --undefined=__libAtpSigHandler_ctor -lrca --as-needed -lcupti -lcudart --no-as-needed -lcuda --as-needed -lsci_gnu_82_mpi --no-as-needed --as-needed -lsci_gnu_82 --no-as-needed --as-needed -lmpich_gnu_82 --no-as-needed -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /opt/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/crtend.o /usr/lib/../lib64/crtn.o
/usr/bin/ld: /tmp/chpl-khandshr.deleteme-OfBTn1/chpl__module.o: in function `chpl__init_NPBRandom':
root:(.text+0xbb782): undefined reference to `pow(double, double)'
/usr/bin/ld: root:(.text+0xbb7b2): undefined reference to `pow(double, double)'
/usr/bin/ld: root:(.text+0xbb7e2): undefined reference to `pow(double, double)'
/usr/bin/ld: root:(.text+0xbb812): undefined reference to `pow(double, double)'
/usr/bin/ld: /tmp/chpl-khandshr.deleteme-OfBTn1/chpl__module.o: in function `coforall_fn_chpl12':
root:(.text+0xbe374): undefined reference to `ldexp(double, int)'
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
error: Make Binary - Linking

Configuration Information

  • Output of chpl --version:
chpl version 1.27.0 pre-release (7a36fe9151)
  built with LLVM version 13.0.0
  • Output of $CHPL_HOME/util/printchplenv --anonymize:
CHPL_TARGET_PLATFORM: cray-xc
CHPL_TARGET_COMPILER: llvm *
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: native *
CHPL_LOCALE_MODEL: gpu *
CHPL_COMM: none *
CHPL_TASKS: qthreads
CHPL_LAUNCHER: slurm-srun *
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: cstdlib
CHPL_GMP: bundled
CHPL_HWLOC: bundled
CHPL_RE2: bundled
CHPL_LLVM: system *
CHPL_AUX_FILESYS: none
  • Back-end compiler and version, e.g. gcc --version or clang --version:
❯ clang --version
clang version 13.0.0 (https://github.com/daviditen/chapel.git a1d85baf29fb91dbdf477da0d9929c3571bec968)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cray/css/users/chapelu/llvm/13/linux64-x86_64//bin
❯ clang++ --version
clang version 13.0.0 (https://github.com/daviditen/chapel.git a1d85baf29fb91dbdf477da0d9929c3571bec968)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cray/css/users/chapelu/llvm/13/linux64-x86_64//bin
  • (For Cray systems only) Output of module list:
Currently Loaded Modulefiles:
  1) modules/3.2.11.4                                    14) Base-opts/2.4.142-7.0.4.0_43.7__g8f27585.ari
  2) craype-network-aries                                15) cray-mpich/7.7.20
  3) nodestat/2.3.89-7.0.4.0_34.11__g8645157.ari         16) dws/3.0.37-7.0.4.0_67.26__g5d83a9b.ari
  4) sdb/3.3.821-7.0.4.0_28.24__g8c59c9d.ari             17) cudatoolkit/21.5_11.3
  5) udreg/2.3.2-7.0.4.0_37.17__g5f0d670.ari             18) rca/2.2.21-7.0.4.0_26.1__gb0ce89b.ari
  6) ugni/6.0.14.0-7.0.4.0_28.4__ge0d449e.ari            19) gcc/11.2.0
  7) gni-headers/5.0.12.0-7.0.4.0_38.10__gd0d73fe.ari    20) craype/2.7.16.3
  8) dmapp/7.1.1-7.0.4.0_40.8__gcec52bc.ari              21) cray-libsci/20.09.1
  9) xpmem/2.2.29-7.0.4.0_50.1__g35859a4.ari             22) pmi/5.0.17
 10) llm/21.4.635-7.0.4.0_46.3__g33a55bc.ari             23) atp/3.14.11
 11) nodehealth/5.6.32-7.0.4.0_81.1__g66010cb.ari        24) perftools-base/22.04.0
 12) system-config/3.6.3187-7.0.4.0_53.8__gd6312243.ari  25) PrgEnv-gnu/6.0.11
 13) slurm/20.11.5-1