External Issue: linker error in GPU code generation

chapelu · May 3, 2022, 12:12am

19740, "milthorpe", "linker error in GPU code generation", "2022-05-03T00:10:49Z"

linker error in GPU code generation

opened 12:10AM - 03 May 22 UTC

### Summary of Problem Compiling example GPU code on some platforms results in …linker errors against `memcpy` and `memset` on some platforms ### Steps to Reproduce When I compile the [Jacobi example code](https://chapel-lang.org/docs/technotes/gpu.html?highlight=gpu#example) without GPU support, it compiles and links fine as expected: ``` milthorpe@EmptyCup:~/git/chapel$ chpl jacobi-gpu.chpl --ldflags -v --print-commands <internal clang code generation> -I/home/milthorpe/git/chapel/modules/standard -I/home/milthorpe/git/chapel/modules/packages -I/home/milthorpe/git/chapel/runtime/include/localeModels/flat -I/home/milthorpe/git/chapel/runtime/include/localeModels -I/home/milthorpe/git/chapel/runtime/include/comm/none -I/home/milthorpe/git/chapel/runtime/include/comm -I/home/milthorpe/git/chapel/runtime/include/tasks/qthreads -I/home/milthorpe/git/chapel/runtime/include -I/home/milthorpe/git/chapel/runtime/include/qio -I/home/milthorpe/git/chapel/runtime/include/atomics/cstdlib -I/home/milthorpe/git/chapel/runtime/include/mem/jemalloc -I/home/milthorpe/git/chapel/third-party/utf8-decoder -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/include -I/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/include -I/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/include -I/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/include -I. -I/tmp/chpl-milthorpe.deleteme-hgWLDf -I/home/milthorpe/git/chapel/modules/internal -DCHPL_GEN_CODE -pthread -include sys_basic.h -include ctype.h -include wctype.h -include llvm/chapel_libc_wrapper.h # Make Binary - Linking /usr/lib/llvm-13/bin/clang++ -v /tmp/chpl-milthorpe.deleteme-hgWLDf/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -o /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -lchpl -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -lhwloc -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -lqthread -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -lhwloc -lchpl -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -ljemalloc -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -lnuma -lm -lpthread Ubuntu clang version 13.0.1-++20220120110924+75e33f71c2da-1~exp1~20220120231001.58 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/lib/llvm-13/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Candidate multilib: .;@m64 Selected multilib: .;@m64 Found CUDA installation: /usr/local/cuda-11.4, version 11.2 "/usr/bin/ld" -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbegin.o -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/llvm-13/bin/../lib -L/lib -L/usr/lib /tmp/chpl-milthorpe.deleteme-hgWLDf/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -lchpl -rpath /home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -lhwloc -rpath /home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -lqthread -lhwloc -lchpl -ljemalloc -rpath /home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -lnuma -lm -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/11/crtend.o /lib/x86_64-linux-gnu/crtn.o rm -f jacobi-gpu mv /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp jacobi-gpu # mv /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp jacobi-gpu mv /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp jacobi-gpu ``` (Running the binary produces `error: halt reached - requesting a child from a flat LocaleModel locale`, however, that is expected.) However, when I compile with GPU, I get a linker error on the `memcpy` and `memset` symbols from the C standard library: ``` milthorpe@EmptyCup:~/git/chapel$ CHPL_CODEGEN_GPU=true CHPL_CUDA_PATH=/usr/local/cuda-11.4 CHPL_LOCALE_MODEL=gpu chpl --gpu-arch sm_60 jacobi-gpu.chpl --ldflags -v --print-commands warning: The prototype GPU support implies --no-checks. This may impact debuggability. To suppress this warning, compile with --no-checks explicitly <internal clang code generation> -I/home/milthorpe/git/chapel/modules/standard -I/home/milthorpe/git/chapel/modules/packages -I/home/milthorpe/git/chapel/runtime/include/localeModels/gpu -I/home/milthorpe/git/chapel/runtime/include/localeModels -I/home/milthorpe/git/chapel/runtime/include/comm/none -I/home/milthorpe/git/chapel/runtime/include/comm -I/home/milthorpe/git/chapel/runtime/include/tasks/qthreads -I/home/milthorpe/git/chapel/runtime/include -I/home/milthorpe/git/chapel/runtime/include/qio -I/home/milthorpe/git/chapel/runtime/include/atomics/cstdlib -I/home/milthorpe/git/chapel/runtime/include/mem/jemalloc -I/home/milthorpe/git/chapel/third-party/utf8-decoder -DHAS_GPU_LOCALE -I/home/milthorpe/git/chapel/runtime/include/gpu/cuda -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/include -I/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/include -I/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/include -I/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/include -I. -I/tmp/chpl-milthorpe.deleteme-cBZ74w -I/home/milthorpe/git/chapel/modules/internal -DCHPL_GEN_CODE -pthread -I/usr/local/cuda-11.4/include --std=c++11 -x cuda --cuda-gpu-arch=sm_60 -include sys_basic.h -include /tmp/chpl-milthorpe.deleteme-cBZ74w/command-line-includes.h -include llvm/chapel_libc_wrapper.h warning: Unknown CUDA version. cuda.h: CUDA_VERSION=11040. Assuming the latest supported version 10.1 # Check to see if ptxas command can be found which ptxas > /dev/null 2>&1 # Check to see if fatbinary command can be found which fatbinary > /dev/null 2>&1 # PTX to object file ptxas -m64 --gpu-name sm_60 --output-file /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.o /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.s ptxas warning : Unresolved extern variable 'chpl_nodeID' in whole program compilation, ignoring extern qualifier # object file to fatbinary fatbinary -64 --create /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu.fatbin --image=profile=sm_60,file=/tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.o --image=profile=compute_60,file=/tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.s <internal clang code generation> -I/home/milthorpe/git/chapel/modules/standard -I/home/milthorpe/git/chapel/modules/packages -I/home/milthorpe/git/chapel/runtime/include/localeModels/gpu -I/home/milthorpe/git/chapel/runtime/include/localeModels -I/home/milthorpe/git/chapel/runtime/include/comm/none -I/home/milthorpe/git/chapel/runtime/include/comm -I/home/milthorpe/git/chapel/runtime/include/tasks/qthreads -I/home/milthorpe/git/chapel/runtime/include -I/home/milthorpe/git/chapel/runtime/include/qio -I/home/milthorpe/git/chapel/runtime/include/atomics/cstdlib -I/home/milthorpe/git/chapel/runtime/include/mem/jemalloc -I/home/milthorpe/git/chapel/third-party/utf8-decoder -DHAS_GPU_LOCALE -I/home/milthorpe/git/chapel/runtime/include/gpu/cuda -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/include -I/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/include -I/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/include -I/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/include -I. -I/tmp/chpl-milthorpe.deleteme-cBZ74w -I/home/milthorpe/git/chapel/modules/internal -DCHPL_GEN_CODE -pthread -I/usr/local/cuda-11.4/include --std=c++11 -x cuda --cuda-gpu-arch=sm_60 -include sys_basic.h -include /tmp/chpl-milthorpe.deleteme-cBZ74w/command-line-includes.h -include llvm/chapel_libc_wrapper.h warning: Unknown CUDA version. cuda.h: CUDA_VERSION=11040. Assuming the latest supported version 10.1 # Make Binary - Linking /usr/lib/llvm-13/bin/clang++ -v /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -o /tmp/chpl-milthorpe.deleteme-cBZ74w/jacobi-gpu.tmp -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -lchpl -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -lhwloc -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -lqthread -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -lhwloc -lchpl -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -ljemalloc -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -L/usr/local/cuda-11.4/lib64 -lcuda -lcudart -lnuma -lm -lpthread Ubuntu clang version 13.0.1-++20220120110924+75e33f71c2da-1~exp1~20220120231001.58 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/lib/llvm-13/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Candidate multilib: .;@m64 Selected multilib: .;@m64 Found CUDA installation: /usr/local/cuda-11.4, version 11.2 "/usr/bin/ld" -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/chpl-milthorpe.deleteme-cBZ74w/jacobi-gpu.tmp /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbegin.o -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -L/usr/local/cuda-11.4/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/llvm-13/bin/../lib -L/lib -L/usr/lib /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -lchpl -rpath /home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -lhwloc -rpath /home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -lqthread -lhwloc -lchpl -ljemalloc -rpath /home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -lcuda -lcudart -lnuma -lm -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/11/crtend.o /lib/x86_64-linux-gnu/crtn.o /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `decodeByteBuffer_chpl': root:(.text+0x411c): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `initWithNewBuffer_chpl': root:(.text+0x4ac4): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `getSlice_chpl': root:(.text+0x70bd): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `getSlice_chpl2': root:(.text+0x7760): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `getSlice_chpl3': root:(.text+0x7df0): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o:root:(.text+0x8480): more undefined references to `memcpy(void*, void const*, unsigned long)' follow /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl2': root:(.text+0x59901): undefined reference to `memset(void*, int, unsigned long)' /usr/bin/ld: root:(.text+0x5a0bc): undefined reference to `memset(void*, int, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl3': root:(.text+0x5cca6): undefined reference to `memset(void*, int, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl4': root:(.text+0x5ff3b): undefined reference to `memset(void*, int, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl5': root:(.text+0x63436): undefined reference to `memset(void*, int, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o:root:(.text+0x66456): more undefined references to `memset(void*, int, unsigned long)' follow /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `chpl_createStringWithLiteral': root:(.text+0x8d835): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `init_ASSIGN__chpl21': root:(.text+0x8dd2c): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `item_chpl': root:(.text+0x912fd): undefined reference to `memcpy(void*, void const*, unsigned long)' /usr/bin/ld: root:(.text+0x9187a): undefined reference to `memcpy(void*, void const*, unsigned long)' clang: error: linker command failed with exit code 1 (use -v to see invocation) error: Make Binary - Linking ``` The undefined symbols that are generated for `memcpy` look different between the non-GPU version: ``` milthorpe@EmptyCup:~/git/chapel$ readelf -VWs /tmp/chpl-milthorpe.deleteme-hgWLDf/chpl__module.o | grep memcpy 1368: 0000000000089b50 121 FUNC LOCAL DEFAULT 2 chpl_memcpy 1482: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND memcpy ``` and the GPU version: ``` milthorpe@EmptyCup:~/git/chapel$ readelf -VWs /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o | grep memcpy 1540: 00000000000e8960 142 FUNC LOCAL DEFAULT 2 _ZL11chpl_memcpyPvPKvm 1680: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZL6memcpyPvPKvm 1949: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND memcpy ``` I've observed this behaviour on two different Nvidia platforms: GTX1050/CUDA 11.6 and V100/CUDA 11.2. However, it's working fine on another platform: P100/CUDA11.6. All platforms are Ubuntu 20.04 with Clang 13.0.1. On the working platform, the symbols for the GPU build look different - the undefined `_ZL6memcpyPvPKvm` is missing: ``` (base) milthorpe@whale:~/chapel$ readelf -VWs /tmp/chpl-milthorpe.deleteme-SiZd5S/chpl__module.o | grep memcpy 1540: 00000000000e8960 142 FUNC LOCAL DEFAULT 2 _ZL11chpl_memcpyPvPKvm 1680: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND memcpy ``` I haven't been able to work out why the GPU code generation is creating different symbols on the two malfunctioning platforms. Can anyone suggest which bits of the compiler I should be poking at? ### Configuration Information - Output of `chpl --version`: ``` chpl version 1.27.0 pre-release (298724cbcb) built with LLVM version 13.0.1 ``` - Output of `$CHPL_HOME/util/printchplenv --anonymize`: ``` CHPL_TARGET_PLATFORM: linux64 CHPL_TARGET_COMPILER: llvm CHPL_TARGET_ARCH: x86_64 CHPL_TARGET_CPU: native CHPL_LOCALE_MODEL: gpu * CHPL_COMM: none * CHPL_TASKS: qthreads CHPL_LAUNCHER: none * CHPL_TIMERS: generic CHPL_UNWIND: none CHPL_MEM: jemalloc CHPL_ATOMICS: cstdlib CHPL_GMP: none CHPL_HWLOC: bundled CHPL_RE2: bundled CHPL_LLVM: system CHPL_AUX_FILESYS: none ``` - Back-end compiler and version, e.g. `gcc --version` or `clang --version`: ``` clang version 13.0.1 Target: x86_64-unknown-linux-gnu Thread model: posix ```

Summary of Problem

Compiling example GPU code on some platforms results in linker errors against memcpy and memset on some platforms

Steps to Reproduce

When I compile the Jacobi example code without GPU support, it compiles and links fine as expected:

milthorpe@EmptyCup:~/git/chapel$ chpl jacobi-gpu.chpl --ldflags -v --print-commands
<internal clang code generation> -I/home/milthorpe/git/chapel/modules/standard -I/home/milthorpe/git/chapel/modules/packages -I/home/milthorpe/git/chapel/runtime/include/localeModels/flat -I/home/milthorpe/git/chapel/runtime/include/localeModels -I/home/milthorpe/git/chapel/runtime/include/comm/none -I/home/milthorpe/git/chapel/runtime/include/comm -I/home/milthorpe/git/chapel/runtime/include/tasks/qthreads -I/home/milthorpe/git/chapel/runtime/include -I/home/milthorpe/git/chapel/runtime/include/qio -I/home/milthorpe/git/chapel/runtime/include/atomics/cstdlib -I/home/milthorpe/git/chapel/runtime/include/mem/jemalloc -I/home/milthorpe/git/chapel/third-party/utf8-decoder -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/include -I/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/include -I/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/include -I/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/include -I. -I/tmp/chpl-milthorpe.deleteme-hgWLDf -I/home/milthorpe/git/chapel/modules/internal -DCHPL_GEN_CODE -pthread -include sys_basic.h -include ctype.h -include wctype.h -include llvm/chapel_libc_wrapper.h

# Make Binary - Linking
/usr/lib/llvm-13/bin/clang++ -v /tmp/chpl-milthorpe.deleteme-hgWLDf/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -o /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -lchpl -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -lhwloc -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -lqthread -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -lhwloc -lchpl -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -ljemalloc -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -lnuma -lm -lpthread
Ubuntu clang version 13.0.1-++20220120110924+75e33f71c2da-1~exp1~20220120231001.58
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-13/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-11.4, version 11.2
 "/usr/bin/ld" -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbegin.o -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/llvm-13/bin/../lib -L/lib -L/usr/lib /tmp/chpl-milthorpe.deleteme-hgWLDf/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-flat/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -lchpl -rpath /home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib -lhwloc -rpath /home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled/lib -lqthread -lhwloc -lchpl -ljemalloc -rpath /home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -lnuma -lm -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/11/crtend.o /lib/x86_64-linux-gnu/crtn.o
rm -f jacobi-gpu
mv /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp jacobi-gpu

# mv /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp jacobi-gpu
mv /tmp/chpl-milthorpe.deleteme-hgWLDf/jacobi-gpu.tmp jacobi-gpu

(Running the binary produces error: halt reached - requesting a child from a flat LocaleModel locale, however, that is expected.)

However, when I compile with GPU, I get a linker error on the memcpy and memset symbols from the C standard library:

milthorpe@EmptyCup:~/git/chapel$ CHPL_CODEGEN_GPU=true CHPL_CUDA_PATH=/usr/local/cuda-11.4 CHPL_LOCALE_MODEL=gpu chpl --gpu-arch sm_60 jacobi-gpu.chpl --ldflags -v --print-commands
warning: The prototype GPU support implies --no-checks. This may impact debuggability. To suppress this warning, compile with --no-checks explicitly
<internal clang code generation> -I/home/milthorpe/git/chapel/modules/standard -I/home/milthorpe/git/chapel/modules/packages -I/home/milthorpe/git/chapel/runtime/include/localeModels/gpu -I/home/milthorpe/git/chapel/runtime/include/localeModels -I/home/milthorpe/git/chapel/runtime/include/comm/none -I/home/milthorpe/git/chapel/runtime/include/comm -I/home/milthorpe/git/chapel/runtime/include/tasks/qthreads -I/home/milthorpe/git/chapel/runtime/include -I/home/milthorpe/git/chapel/runtime/include/qio -I/home/milthorpe/git/chapel/runtime/include/atomics/cstdlib -I/home/milthorpe/git/chapel/runtime/include/mem/jemalloc -I/home/milthorpe/git/chapel/third-party/utf8-decoder -DHAS_GPU_LOCALE -I/home/milthorpe/git/chapel/runtime/include/gpu/cuda -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/include -I/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/include -I/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/include -I/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/include -I. -I/tmp/chpl-milthorpe.deleteme-cBZ74w -I/home/milthorpe/git/chapel/modules/internal -DCHPL_GEN_CODE -pthread -I/usr/local/cuda-11.4/include --std=c++11 -x cuda --cuda-gpu-arch=sm_60 -include sys_basic.h -include /tmp/chpl-milthorpe.deleteme-cBZ74w/command-line-includes.h -include llvm/chapel_libc_wrapper.h
warning: Unknown CUDA version. cuda.h: CUDA_VERSION=11040. Assuming the latest supported version 10.1

# Check to see if ptxas command can be found
which ptxas > /dev/null 2>&1

# Check to see if fatbinary command can be found
which fatbinary > /dev/null 2>&1

# PTX to  object file
ptxas -m64 --gpu-name sm_60 --output-file /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.o /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.s
ptxas warning : Unresolved extern variable 'chpl_nodeID' in whole program compilation, ignoring extern qualifier

# object file to fatbinary
fatbinary -64 --create /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu.fatbin --image=profile=sm_60,file=/tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.o --image=profile=compute_60,file=/tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__gpu_ptx.s
<internal clang code generation> -I/home/milthorpe/git/chapel/modules/standard -I/home/milthorpe/git/chapel/modules/packages -I/home/milthorpe/git/chapel/runtime/include/localeModels/gpu -I/home/milthorpe/git/chapel/runtime/include/localeModels -I/home/milthorpe/git/chapel/runtime/include/comm/none -I/home/milthorpe/git/chapel/runtime/include/comm -I/home/milthorpe/git/chapel/runtime/include/tasks/qthreads -I/home/milthorpe/git/chapel/runtime/include -I/home/milthorpe/git/chapel/runtime/include/qio -I/home/milthorpe/git/chapel/runtime/include/atomics/cstdlib -I/home/milthorpe/git/chapel/runtime/include/mem/jemalloc -I/home/milthorpe/git/chapel/third-party/utf8-decoder -DHAS_GPU_LOCALE -I/home/milthorpe/git/chapel/runtime/include/gpu/cuda -DCHPL_JEMALLOC_PREFIX=chpl_je_ -I/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/include -I/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/include -I/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/include -I/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/include -I. -I/tmp/chpl-milthorpe.deleteme-cBZ74w -I/home/milthorpe/git/chapel/modules/internal -DCHPL_GEN_CODE -pthread -I/usr/local/cuda-11.4/include --std=c++11 -x cuda --cuda-gpu-arch=sm_60 -include sys_basic.h -include /tmp/chpl-milthorpe.deleteme-cBZ74w/command-line-includes.h -include llvm/chapel_libc_wrapper.h
warning: Unknown CUDA version. cuda.h: CUDA_VERSION=11040. Assuming the latest supported version 10.1

# Make Binary - Linking
/usr/lib/llvm-13/bin/clang++ -v /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -o /tmp/chpl-milthorpe.deleteme-cBZ74w/jacobi-gpu.tmp -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -lchpl -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -lhwloc -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -lqthread -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -lhwloc -lchpl -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -ljemalloc -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -Wl,-rpath,/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -L/usr/local/cuda-11.4/lib64 -lcuda -lcudart -lnuma -lm -lpthread
Ubuntu clang version 13.0.1-++20220120110924+75e33f71c2da-1~exp1~20220120231001.58
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-13/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-11.4, version 11.2
 "/usr/bin/ld" -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/chpl-milthorpe.deleteme-cBZ74w/jacobi-gpu.tmp /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbegin.o -L/home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -L/home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -L/home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -L/home/milthorpe/git/chapel/third-party/jemalloc/install/target/linux64-x86_64-native-llvm-none/lib -L/home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -L/usr/local/cuda-11.4/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/llvm-13/bin/../lib -L/lib -L/usr/lib /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o /home/milthorpe/git/chapel/lib/linux64/llvm/x86_64/cpu-native/loc-gpu/comm-none/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-cstdlib/hwloc-bundled/re2-bundled/fs-none/lib_pic-none/san-none/main.o -lchpl -rpath /home/milthorpe/git/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-gpu/lib -lhwloc -rpath /home/milthorpe/git/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-gpu-jemalloc-bundled/lib -lqthread -lhwloc -lchpl -ljemalloc -rpath /home/milthorpe/git/chapel/third-party/re2/install/linux64-x86_64-native-llvm-none/lib -lre2 -lcuda -lcudart -lnuma -lm -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/11/crtend.o /lib/x86_64-linux-gnu/crtn.o
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `decodeByteBuffer_chpl':
root:(.text+0x411c): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `initWithNewBuffer_chpl':
root:(.text+0x4ac4): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `getSlice_chpl':
root:(.text+0x70bd): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `getSlice_chpl2':
root:(.text+0x7760): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `getSlice_chpl3':
root:(.text+0x7df0): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o:root:(.text+0x8480): more undefined references to `memcpy(void*, void const*, unsigned long)' follow
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl2':
root:(.text+0x59901): undefined reference to `memset(void*, int, unsigned long)'
/usr/bin/ld: root:(.text+0x5a0bc): undefined reference to `memset(void*, int, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl3':
root:(.text+0x5cca6): undefined reference to `memset(void*, int, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl4':
root:(.text+0x5ff3b): undefined reference to `memset(void*, int, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `dsiReallocate_chpl5':
root:(.text+0x63436): undefined reference to `memset(void*, int, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o:root:(.text+0x66456): more undefined references to `memset(void*, int, unsigned long)' follow
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `chpl_createStringWithLiteral':
root:(.text+0x8d835): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `init_ASSIGN__chpl21':
root:(.text+0x8dd2c): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o: in function `item_chpl':
root:(.text+0x912fd): undefined reference to `memcpy(void*, void const*, unsigned long)'
/usr/bin/ld: root:(.text+0x9187a): undefined reference to `memcpy(void*, void const*, unsigned long)'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: Make Binary - Linking

The undefined symbols that are generated for memcpy look different between the non-GPU version:

milthorpe@EmptyCup:~/git/chapel$ readelf -VWs /tmp/chpl-milthorpe.deleteme-hgWLDf/chpl__module.o | grep memcpy
  1368: 0000000000089b50   121 FUNC    LOCAL  DEFAULT    2 chpl_memcpy
  1482: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND memcpy

and the GPU version:

milthorpe@EmptyCup:~/git/chapel$ readelf -VWs /tmp/chpl-milthorpe.deleteme-cBZ74w/chpl__module.o | grep memcpy
  1540: 00000000000e8960   142 FUNC    LOCAL  DEFAULT    2 _ZL11chpl_memcpyPvPKvm
  1680: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _ZL6memcpyPvPKvm
  1949: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND memcpy

I've observed this behaviour on two different Nvidia platforms: GTX1050/CUDA 11.6 and V100/CUDA 11.2. However, it's working fine on another platform: P100/CUDA11.6. All platforms are Ubuntu 20.04 with Clang 13.0.1. On the working platform, the symbols for the GPU build look different - the undefined _ZL6memcpyPvPKvm is missing:

(base) milthorpe@whale:~/chapel$ readelf -VWs /tmp/chpl-milthorpe.deleteme-SiZd5S/chpl__module.o | grep memcpy
  1540: 00000000000e8960   142 FUNC    LOCAL  DEFAULT    2 _ZL11chpl_memcpyPvPKvm
  1680: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND memcpy

I haven't been able to work out why the GPU code generation is creating different symbols on the two malfunctioning platforms. Can anyone suggest which bits of the compiler I should be poking at?

Configuration Information

Output of chpl --version:

chpl version 1.27.0 pre-release (298724cbcb)
  built with LLVM version 13.0.1

Output of $CHPL_HOME/util/printchplenv --anonymize:

CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: llvm
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: native
CHPL_LOCALE_MODEL: gpu *
CHPL_COMM: none *
CHPL_TASKS: qthreads
CHPL_LAUNCHER: none *
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: cstdlib
CHPL_GMP: none
CHPL_HWLOC: bundled
CHPL_RE2: bundled
CHPL_LLVM: system
CHPL_AUX_FILESYS: none

Back-end compiler and version, e.g. gcc --version or clang --version:

clang version 13.0.1
Target: x86_64-unknown-linux-gnu
Thread model: posix

Topic		Replies	Views	Activity
How to use the prototype GPU codegen feature in 1.24? Users	11	316	April 28, 2021
Announcing Chapel 1.25.0! Announcements	0	403	September 24, 2021

External Issue: linker error in GPU code generation

Summary of Problem

Steps to Reproduce

Configuration Information

Related Topics