28837, "jabraham17", "Investigating LLVM 22 performance regression: nbody", "2026-05-15T17:54:05Z"
When upgrading our perf infrastructure from LLVM 20 to LLVM 22, we noticed a pretty significant performance regression to nbody. I am going to use this issue to catalog what I have found as we search for a fix/reason.
Perf graph in question: Chapel Performance Graphs
I am focusing on test/studies/shootout/nbody/mppf/nbody3-no-cube.chpl
Compiled as chpl --fast --savec gen test/studies/shootout/nbody/mppf/nbody3-no-cube.chpl with LLVM 20 and LLVM 22
The problem comes from the advance function, which gets inlined into main. I grabbed IR for that with llvm-extract --func=chpl_gen_main gen/chpl__module-opt1.bc -S -o chpl_gen_main.ll. The differences are immediately clear, LLVM 20 is using 4 calls to a 2 wide vector sqrt (tail call contract <2 x double> @llvm.sqrt.v2f64) and 2 calls to plain sqrt, while LLVM 22 uses 4 calls to a 4 wide vector sqrt (tail call contract <4 x double> @llvm.sqrt.v4f64). Both resulting loops process 10 elements per trip, but LLVM 22 uses all vector calls with extractelement, whereas LLVM 20 uses some scalar calls.
There are quite a few other differences, but this is the most glaring one.
I tracked this change the SLP vectorizer. Comparing the output of chpl --fast nbody3-no-cube.chpl --mllvm --print-after=slp-vectorizer --mllvm --print-before=slp-vectorizer --mllvm --filter-print-funcs=chpl_gen_main --mllvm --print-loop-func-scope shows that with LLVM 22 the SLP vectorizer is much more aggressive.
I am still investigating possible mitigation stratagies, but this seems to be a flaw in the LLVM slp vectorizer cost model