New Issue: Add setting that can be used in SKIPIFs to ensure test is always run in default settings

19443, "lydia-duncan", "Add setting that can be used in SKIPIFs to ensure test is always run in default settings", "2022-03-14T19:59:11Z"

We recently ran across a case where a test was being skipped with the intention of only running it in our "default" build, but the definition of our default build had changed so it was not tested there. (The test was still run in the old setting but that is no longer our default). This makes it more likely that bugs will only be noticed in nightly testing as opposed to prior to merging.

It seems plausible that there are other such cases that have moved to non-default settings and we haven't noticed. Looking for them and updating them seems like a pain, and their inclusion seems fragile. I think it would be ideal if such "default-only" tests had a way of marking themselves so that they will always be run in our default set up without having to update them.

I'm thinking something along the lines of:

CHPL_HOST_COMPILER=default
CHPL_LLVM=default

Where "default" wouldn't have meaning as an actual setting for the environment variable, but start_test would know to replace it with whatever the default value is. This might add confusion if someone was using the .skipif file to determine a setting to run the test in, but I think that can be solved by always commenting it in a way that indicates the information is for start_test only or something.