For v1.24.1, I'm trying to understand what steps are taken when these environment variables are changed:
CHPL_LLVM={bundled, system}
CHPL_HOST_COMPILER={gnu, clang}
CHPL_TARGET_COMPILER={gnu, clang, clang-included}
I'll illustrate a few scenarios below to work through an intuition for how LLVM support behaves. Please correct me along the way.
Variant 1: Base Case
This is how I usually build because it works almost all of the time. Main disadvantage is the amount of time it takes to build.
CHPL_LLVM=bundled
CHPL_HOST_COMPILER=gnu
CHPL_TARGET_COMPILER=gnu
I think this means (respectively):
Use the bundled LLVM
Build bundled LLVM and Chapel tooling (chpl, ...) with the HOST system GCC
Build runtime libraries and have chpl generate code with the TARGET system GCC
chpl --llvm uses the bundled LLVM to codegen
chpl --no-llvm uses TARGET system GCC
Variant 2: System LLVM
In the documentation for CHPL_*_COMPILER, it states:
Note that builds with LLVM Support (i.e. when CHPL_LLVM=bundled) will build the runtime twice: once with the compiler as described above and once with clang-included. We do this in order to avoid issues in linking objects built by different compilers.
This sounds like a two-stage compiler, but what happens when CHPL_LLVM=system? Does it use the system LLVM as stage 1 to build the bundled LLVM? (Is the bundled LLVM always used, but we skip the stage 1 build?)
What happens with ...
CHPL_LLVM=system
CHPL_HOST_COMPILER={gnu, clang}
during the build step? (How is CHPL_HOST_COMPILER used with CHPL_LLVM=system?)
Variant 3: CHPL_TARGET_COMPILER when CHPL_LLVM={bundled, system}
What happens when I set ...
CHPL_LLVM=bundled
CHPL_TARGET_COMPILER={gnu, clang}
when a user compiles with chpl --[no-]llvm?
Does this behavior when CHPL_TARGET_COMPILER=clang-included (other than using the included, bundled compiler)?
Does anything change for the questions above when CHPL_LLVM=system? I don't have an intuition for how this case behaves:
Good questions, and I'm realizing belatedly that I think I led you astray
in my [off-discourse] review of your build environment. Specifically, I forgot
about the "two runtime builds" when LLVM is enabled, and the implicit use of
clang-included. I'm going to run your questions past others on the team
who know this better than I do, but my quick guesses in the meantime:
I'm fairly certain you're correct about the role of CHPL_HOST_COMPILER
I believe that the compiler used for the generated code is always:
clang-included when compiling with --llvm and CHPL_LLVM==bundled
CHPL_TARGET_COMPILER otherwise (but am less confident about this
when CHPL_LLVM=system and --llvm is used, so let me check on that)
Brad is correct about CHPL_LLVM=bundled. In that case we compile the runtime with both the clang-included that was built in third-party and the CHPL_TARGET_COMPILER. If using --llvm we link with the clang-included runtime using clang-included. With --no-llvm we use the CHPL_TARGET_COMPILER.
For CHPL_LLVM=system we again build two versions of the runtime - one with CHPL_TARGET_COMPILER and one with the clang that is associated with the system install of llvm. If --llvm is used we link with the clang version of the runtime using the system clang. For --no-llvm we link with the CHPL_TARGET_COMPILER version of the runtime using the CHPL_TARGET_COMPILER.
If the bundled LLVM isn't built, this configuration looks illegal to me. Does CHPL_TARGET_COMPILER=clang-included magically behave like CHPL_TARGET_COMPILER=clang?
I don't think it ought to. It seems to me that we should either have clang-included not be permitted (error) when there is no clang-included available; or that we should retire clang-included in favor of just using clang, as the identifier, getting it from the third-party/llvm build when CHPL_LLVM=bundled and its available, or from the system otherwise. I'd be inclined to make it an error for simplicity / lack of confusion. But I have to admit I've never been a big fan of clang-included as an identifier`, which is why I mention the second option.
# Build
CHPL_LLVM=system
# Build again
CHPL_LLVM=bundled
# Run `chpl --llvm` or `chpl --no-llvm`
CHPL_LLVM=system
CHPL_TARGET_COMPILER=clang-included
From David's comment here, I think this would always use the system clang and never the clang-included. In a way, this is one of the worst configurations (in terms of truthfulness) because clang-included would exist.
If so, I would agree with retiring clang-included.
FWIW I'm not sure clang-included is really meant to be something that's requested by users at this point. It's the backend compiler that the system automatically selects when working with / preparing for --llvm compilation. While it might be useful to advertise and document it, for now I would recommend you consider it an implementation detail it.
The reason that we have clang-included and don't just use clang instead in all cases is that on some systems (and commonly on Mac OS X) there is a system clang but it's not the right version to work with the bundled LLVM (and the right version of clang may or may not be installed). So, the implementation uses clang-included to refer to the bundled one.
Specifically for CHPL_LLVM=system, it would be reasonable to call it CHPL_TARGET_COMPILER=clang. However there is a wrinkle. With CHPL_LLVM=system there can be multiple versions of LLVM and clang installed and we're supposed to be selecting the one that matches the version we expect (which is currently 11). I do not know for sure but I would think that clang-included would do this today but clang would not. Put another way, CHPL_TARGET_COMPILER=clang would seem to request whatever version of clang is available with the clang command. But with --llvm we need to be able to find clang-11. For CHPL_LLVM=system it might be more reasonable to call it something like CHPL_TARGET_COMPILER=clang-11.
Today, which values do these environment variables take to cause this behavior?
This is definitely some of the information I was looking for when starting this thread. There's a lot of behaviors going on that is not apparent for what a packager has to do to correctly build Chapel for their specific system.
In this (make-believe) world, is the thought here to get rid of clang-included by making the behaviors more explicit?
Also, doesn't the same problem occur with any implicitly defined variables? Definitely gcc; maybe re.
What would this do?
CHPL_LLVM=bundled
CHPL_TARGET_COMPILER=clang-11
Maybe we need to build out a complete combinatorial profile for these three environment variables.
We might continue to have something that says "use the bundled version of clang" for CHPL_LLVM=bundled.
I guess I'm not sure what you mean. Are you asking if we should be able to say something like CHPL_TARGET_COMPILER=gcc10 ? I would imagine that would be useful but we haven't gotten to implementing such things (or seen a strong need for it).
When using CHPL_LLVM=bundled, the CHPL_TARGET_COMPILER setting should be set automatically by the compiler, when you do chpl --llvm (or use the LLVM backend any way at all including it being the default). The compiler might need to set CHPL_TARGET_COMPILER to something that indicates "the bundled version of clang".
Oh right. I got confused. I thought you were talking about a way to mitigate the Mac OS X issue of not having a recent enough system clang by pointing to a different system clang. But you actually meant "use bundled LLVM" to mitigate this issue on Mac OS X.
I think I understand now. It boils down to (in make-believe world for a user compiling with chpl --llvm):
System LLVM governed by CHPL_TARGET_COMPILER={clang, clang-11, etc.} and compiler is chosen by where the command resolves. Feature request(?): set an absolute path.
Bundled LLVM governed by bundled clang-included. The CHPL_TARGET_COMPILER setting doesn't matter.
Which begs the question. Today, when does a user ever set clang-included?