[Chapel Merge] Use python3/python/python2 instead of just python

Notifications Commits
1

Branch: refs/heads/master
Revision: 49b7834
Author: lydia-duncan
Log Message:

Merge pull request #16560 from mppf/use-python3-python-python2

Use python3/python/python2 instead of just python
[developed by @mppf, reviewed and merged by @lydia-duncan]

This PR adjusts core parts of Chapel to work with whatever python it can find - looking for python3, then python, then python2. This should enable the compiler and runtime to build and operate in a variety of systems.

Features that need maximum portability are adapted to use a new sh script util/config/find-python.sh which returns the version of python to run. It chooses (in this order) python3 python python2. In each case the thing calling the python script in normal usage computes the name of the interpreter to run.

  • Makefiles set CHPL_MAKE_PYTHON from util/config/find-python.sh and then run Python support scripts like $(CHPL_MAKE_PYTHON) some-script.py (which translates to e.g. python3 some-script.py)
  • updated checkChplInstall similarly (called from make check)
  • printchplenv already included an sh wrapper so updated it to run util/config/find-python.sh
  • added an sh wrapper for compileline that uses util/config/find-python.sh
  • setchplenv scripts now call util/config/find-python.sh to invoke helpers
  • mason test also uses find-python.sh

This PR then adjusts many other uses of python to now require python3. Things in this category are:

  • make docs, chpldoc and related support
  • start_test and related support
  • python scripts in test/ e.g. .prediff files
  • tests that run python scripts by naming the interpreter
  • [x] (with #16561) quickstart make / make check on bento-centos-6 with Python 2.6 and no Python 3
  • [x] (with #16561) quickstart make / make check / make docs on Ubuntu with python3 but no python
  • [x] (with #16561) standard config make / make check / make docs / start_test hello on Ubuntu with python3 but no python
  • [x] (with #16561) full local testing - with source /data/cf/chapel/setup_python35-byhost.bash in my .bashrc

Future work – revisit PR #3571 if necessary

  • Useful trick for finding issues here is make SHELL='sh -x'

Modified Files:
A util/config/compileline.py
A util/config/find-python.sh
M Makefile
M Makefile.devel
M compiler/Makefile
M compiler/util/files.cpp
M doc/rst/developer/bestPractices/TestSystem.rst
M doc/util/chpl2rst.py
M doc/util/extract-rst-tests.py
M make/Makefile.base
M man/Makefile
M modules/Makefile
M runtime/Makefile
M runtime/Makefile.config
M test/analysis/alias/PREDIFF
M test/compflags/bradc/mungeUserIdents/testmunge-export.prediff
M test/compflags/bradc/mungeUserIdents/testmunge-extern.prediff
M test/compflags/bradc/mungeUserIdents/testmunge.prediff
M test/compflags/ferguson/default-binary-name/sub_test
M test/compflags/link/sungeun/static_dynamic.prediff
M test/compflags/sungeun/configs/basic/PREDIFF
M test/compflags/sungeun/configs/type_variables/PREDIFF
M test/distributions/robust/arithmetic/PREDIFF
M test/distributions/robust/arithmetic/basics/test_scan1.prediff
M test/domains/sungeun/assoc/index_not_in_domain_1.prediff
M test/domains/sungeun/assoc/index_not_in_domain_2.prediff
M test/domains/sungeun/sparse/index_not_in_domain_1.prediff
M test/domains/sungeun/sparse/index_not_in_domain_2.prediff
M test/execflags/bradc/errors/PREDIFF
M test/execflags/bradc/gdbddash/gdbSetConfig.skipif
M test/execflags/dinan/PREDIFF
M test/execflags/ferguson/help2.prediff
M test/execflags/shannon/configs/configVarDashSVarEquals.prediff
M test/execflags/shannon/configs/help/PREDIFF
M test/execflags/shannon/help.prediff
M test/execflags/sungeun/about.prediff
M test/execflags/thomasvandoren/exceedMemInts.skipif
M test/functions/ferguson/main/sub_test
M test/interop/C/multilocale.skipif
M test/interop/python/errorMessages/abiIncompatible.skipif
M test/interop/python/multilocale.skipif
M test/io/ferguson/ctests/qio_bits_test.skipif
M test/io/ferguson/ctests/qio_test.compopts
M test/io/ferguson/ctests/qio_test.skipif
M test/io/ferguson/ctests/skip_non_fifo_atomic_locks.py
M test/io/ferguson/stdin-is-directory/stdin-is-directory.prediff
M test/io/thomasvandoren/lustre/SKIPIF
M test/io/vass/time-write.skipif
M test/library/draft/Vector/PREDIFF
M test/library/packages/Crypto/saru.skipif
M test/library/packages/Crypto/saru/COMPOPTS
M test/library/packages/FFTW/SKIPIF
M test/library/packages/LAPACK.skipif
M test/library/packages/LinearAlgebra/performance/csrmatmatMult-perf.py
M test/library/packages/MPI/multilocale.skipif
M test/library/packages/MPI/spmd.skipif
M test/library/packages/NetCDF.skipif
M test/library/packages/ProtobufProtocolSupport/anyRunner.skipif
M test/library/packages/ProtobufProtocolSupport/enumsRunner.skipif
M test/library/packages/ProtobufProtocolSupport/mapsRunner.skipif
M test/library/packages/ProtobufProtocolSupport/messagefieldRunner.skipif
M test/library/packages/ProtobufProtocolSupport/oneofsRunner.skipif
M test/library/packages/ProtobufProtocolSupport/repeatedfieldRunner.skipif
M test/library/packages/ProtobufProtocolSupport/typesRunner.skipif
M test/library/packages/TOML/BurntSushi/compare.py
M test/library/packages/TensorFlow.skipif
M test/library/packages/UnitTest/Launcher_Test.prediff
M test/library/packages/UnitTest/Launcher_Test.skipif
M test/library/packages/ZMQ.skipif
M test/library/packages/ZMQ/SKIPIF
M test/library/packages/ZMQ/multilocale/SKIPIF
M test/library/standard/DataFrames/diten/readHDF5Df.skipif
M test/library/standard/FileSystem/chmod/chmod.prediff
M test/library/standard/FileSystem/lydia/copy/copying.prediff
M test/library/standard/FileSystem/lydia/copy/copying.preexec
M test/library/standard/FileSystem/lydia/copyFile/copyingContents.prediff
M test/library/standard/FileSystem/lydia/copyFile/copyingContents.preexec
M test/library/standard/FileSystem/lydia/copyMode/copyingMode.prediff
M test/library/standard/FileSystem/lydia/copyMode/copyingMode.preexec
M test/library/standard/FileSystem/lydia/getIDs/getting.prediff
M test/library/standard/FileSystem/lydia/getMode/basic.preexec
M test/library/standard/FileSystem/lydia/isMount/checkMount.skipif
M test/library/standard/FileSystem/lydia/isMount/checkRoot.skipif
M test/library/standard/FileSystem/lydia/isMount/symlinkToMount.skipif
M test/library/standard/FileSystem/lydia/mkdir/makeWithMode.prediff
M test/library/standard/FileSystem/lydia/parallelDirState/chdirAllLocales.prediff
M test/library/standard/FileSystem/lydia/parallelDirState/chdirMultiTasks.prediff
M test/library/standard/Map/mapOfArray.skipif
M test/library/standard/Math/promotion/generate_tests/make_promotion_tests.py
M test/library/standard/Path/dlongnecke/absPath/absPath.preexec
M test/library/standard/Path/dlongnecke/absPath/fileAbsPath.preexec
M test/library/standard/Path/dlongnecke/normPath/normPath.preexec
M test/library/standard/Path/relPath/fileRelPath.preexec
M test/library/standard/Path/relPath/relPath.preexec
M test/library/standard/Path/saru/getParentName/parentName.preexec
M test/library/standard/Path/victor-ludorum/testcases_isAbsPath.chpl
M test/library/standard/machine/numPUs.prediff
M test/llvm/llvmDebug/llvmDebug_test.py
M test/localeModels/numa/basics/coforall_on.prediff
M test/localeModels/numa/dataParallel/forall.prediff
M test/man/checkManPages
M test/mason/SKIPIF
M test/mason/filterDigits
M test/mason/mason-example-with-opts/SKIPIF
M test/mason/mason-example/SKIPIF
M test/mason/mason-external/SKIPIF
M test/mason/mason-external/libtomlc99/mason-external.prediff
M test/mason/mason-external/masonExternalRanges/mason-external.prediff
M test/mason/mason-help-scope/masonHelpScope.skipif
M test/mason/mason-test/mason-test.prediff
M test/mason/mason-update-toml-error/SKIPIF
M test/mason/masonTestSome/mason-test-some.prediff
M test/mason/masonTestSubString/mason-test-sub-string.prediff
M test/mason/search/cacheTest/SKIPIF
M test/mason/spec-parser/test-spec-parser.prediff
M test/multilocale/bradc/needMultiLocales/writeThisUsingOn.prediff
M test/multilocale/local/diten/test_local2.prediff
M test/multilocale/numLocales/bradc/printNumLocalesMissingArg.prediff
M test/npb/ft/npadmana/ft_transposed.skipif
M test/optimizations/bulkcomm/block/exchange.skipif
M test/optimizations/localon/wrong-local-on.skipif
M test/optimizations/sungeun/scalar-replace/PREDIFF
M test/parallel/cobegin/gbt/cobegin-stacksize.prediff
M test/parallel/coforall/gbt/time-sync-incs.prediff
M test/parallel/forall/task-private-vars/throwing-init-exprs/int-tpvs.prediff
M test/performance/comm/barrier/empty-chpl-barrier.ml-compopts
M test/performance/comm/low-level/arrayTransfer.execopts
M test/performance/compiler/elliot/passCheck.prediff
M test/performance/ferguson/MYCOMPOPTS
M test/performance/vectorization/vectorPragmas/PREDIFF
M test/release/examples/benchmarks/hpcc/ra-atomics.ml-compopts
M test/release/examples/benchmarks/hpcc/ra-atomics.skipif
M test/release/examples/benchmarks/hpcc/ra.ml-compopts
M test/release/examples/benchmarks/hpcc/variants/ra-unordered-atomics.ml-compopts
M test/release/examples/benchmarks/hpcc/variants/ra-unordered-atomics.skipif
M test/release/examples/benchmarks/miniMD/miniMD.skipif
M test/release/examples/benchmarks/shootout/mandelbrot.prediff
M test/release/examples/benchmarks/shootout/threadring.suppressif
M test/release/examples/benchmarks/ssca2/SSCA2_main.ml-execopts
M test/release/examples/benchmarks/ssca2/SSCA2_main.prediff
M test/release/examples/primers/FFTWlib.skipif
M test/release/examples/primers/LAPACKlib.skipif
M test/release/examples/primers/learnChapelInYMinutes.prediff
M test/release/examples/primers/learnChapelInYMinutes.skipif
M test/release/examples/primers/taskParallel.prediff
M test/runtime/configMatters/comm/atomics-api-on.compopts
M test/runtime/configMatters/comm/cache-remote/miniUnorderedCopyStress.execopts
M test/runtime/configMatters/comm/oversubscribedArrayAlloc.execopts
M test/runtime/configMatters/comm/ugni/SIGBUS-array-alloc.skipif
M test/runtime/configMatters/comm/ugni/SIGBUS-heap-extend.execenv
M test/runtime/configMatters/comm/ugni/SIGBUS-heap-extend.skipif
M test/runtime/configMatters/comm/ugni/overflow-mem-region-table.skipif
M test/runtime/configMatters/comm/unordered/oneLocaleParallelAtomicPerf.ml-execopts
M test/runtime/configMatters/comm/unordered/oneLocaleSerialAtomicPerf.ml-execopts
M test/runtime/configMatters/comm/unordered/unorderedAtomicsApiOn.compopts
M test/runtime/configMatters/comm/unordered/unorderedAtomicsStress.execopts
M test/runtime/configMatters/comm/unordered/unorderedCopyStress.execopts
M test/runtime/configMatters/comm/unordered/unorderedCopyStress.ml-execopts
M test/runtime/configMatters/mem/goodAllocSize.prediff
M test/runtime/sungeun/chpl-env-gen.precomp
M test/setchplenv/verify_setchplenv_scripts.py
M test/sparse/CS/multiplication/performance.py
M test/studies/bale/histogram/histo-atomics-forall-opt.ml-execopts
M test/studies/bale/histogram/histo-atomics.ml-execopts
M test/studies/bale/indexgather/ig-forall-opt.ml-execopts
M test/studies/bale/indexgather/ig-variants.ml-execopts
M test/studies/bale/indexgather/ig.ml-execopts
M test/studies/comd/elegant/arrayOfStructs/CoMD.prediff
M test/studies/comd/elegant/arrayOfStructs/CoMD.skipif
M test/studies/comd/llnl/CoMD.prediff
M test/studies/compOcean/compOcean.skipif
M test/studies/dedup/PREDIFF
M test/studies/hpcc/CommDiags/PRECOMP
M test/studies/hpcc/HPL/stonea/serial/hplExample5.prediff
M test/studies/prk/DGEMM/SUMMA/dgemm.skipif
M test/studies/shootout/submitted/archive/threadring.suppressif
M test/studies/shootout/submitted/clbg-diff.py
M test/studies/shootout/thread-ring/lydia/thread-ring-coforall-begin.suppressif
M test/studies/shootout/thread-ring/lydia/thread-ring-coforall.suppressif
M test/studies/shootout/thread-ring/lydia/thread-ring-for-begin.suppressif
M test/studies/twitter/strshuffle.prediff
M test/studies/twitter/survey.prediff
M test/trivial/figueroa/UnmangledSymbols.skipif
M test/types/cptr/func_ptr_as_c_fn_ptr.skipif
M test/types/cptr/func_ptr_as_void_ptr.skipif
M test/types/range/elliot/anonymousRangeIter.prediff
M test/types/string/StringImpl/memLeaks/begin.prediff
M test/users/npadmana/fftw/SKIPIF
M test/users/npadmana/gsl_demonstration/SKIPIF
M test/users/npadmana/mpi/test_mpi.skipif
M test/users/vass/isX/src/isX.byBlankArgs-compWarns.py
M test/users/vass/isX/src/isX.module-writeln.py
M third-party/chpl-venv/Makefile
M third-party/chpl-venv/Makefile.include
M third-party/chpl-venv/venv-use-sys-python.py
M third-party/llvm/Makefile
M tools/c2chapel/c2chapel.py
M tools/mason/Makefile
M tools/mason/MasonTest.chpl
M util/buildRelease/add_license_to_sources.py
M util/buildRelease/chpl-make-cpu_count
M util/buildRelease/install.sh
M util/build_configs/build_configs.py
M util/chplenv/chpl_arch.py
M util/chplenv/chpl_atomics.py
M util/chplenv/chpl_aux_filesys.py
M util/chplenv/chpl_bin_subdir.py
M util/chplenv/chpl_comm.py
M util/chplenv/chpl_comm_debug.py
M util/chplenv/chpl_comm_segment.py
M util/chplenv/chpl_comm_substrate.py
M util/chplenv/chpl_compiler.py
M util/chplenv/chpl_cpu.py
M util/chplenv/chpl_gmp.py
M util/chplenv/chpl_hwloc.py
M util/chplenv/chpl_jemalloc.py
M util/chplenv/chpl_launcher.py
M util/chplenv/chpl_lib_pic.py
M util/chplenv/chpl_libfabric.py
M util/chplenv/chpl_llvm.py
M util/chplenv/chpl_locale_model.py
M util/chplenv/chpl_make.py
M util/chplenv/chpl_mem.py
M util/chplenv/chpl_platform.py
M util/chplenv/chpl_python_version.py
M util/chplenv/chpl_regexp.py
M util/chplenv/chpl_sanitizers.py
M util/chplenv/chpl_tasks.py
M util/chplenv/chpl_timers.py
M util/chplenv/chpl_unwind.py
M util/chplenv/overrides.py
M util/chplenv/printchplenv.py
M util/chplenv/third-party-pkgs
M util/chpltags
M util/config/compileline
M util/config/fixpath.py
M util/config/make_sys_basic_types.py
M util/config/replace-paths.py
M util/config/update-if-different
M util/cron/syncPerfGraphs.py
M util/cron/verify_config_names.py
M util/devel/check_paths
M util/devel/check_scripts
M util/devel/gen-chpl-bash-completion.py
M util/devel/lookForBadRTCalls
M util/devel/look_for_calls.py
M util/devel/test/comment_perfdat
M util/devel/test/spliceDat
M util/devel/test/vagrant/re2-supported.py
M util/devel/updateDatFiles.py
M util/printchplenv
M util/quickstart/setchplenv.bash
M util/quickstart/setchplenv.csh
M util/quickstart/setchplenv.fish
M util/quickstart/setchplenv.sh
M util/run-in-venv.bash
M util/setchplenv.bash
M util/setchplenv.csh
M util/setchplenv.fish
M util/setchplenv.sh
M util/start_test
M util/test/activate_chpl_test_venv.py
M util/test/checkChplInstall
M util/test/check_annotations.py
M util/test/chpl_launchcmd.py
M util/test/combineCompPerfData
M util/test/computePerfStats
M util/test/convert_start_test_log_to_junit_xml.py
M util/test/fileReadHelp.py
M util/test/genGraphs
M util/test/paratest.chapcs
M util/test/paratest.local
M util/test/prediff-for-incremental-warning
M util/test/prediff-for-mpirun4ofi
M util/test/prediff-for-slurm-srun
M util/test/prediff-for-stacktrace
M util/test/re2_supports_valgrind.py
M util/test/send_email.py
M util/test/sub_clean
M util/test/sub_test
M util/test/timers/highPrecisionTimer

Compare: https://github.com/chapel-lang/chapel/compare/e5584b14ca06...49b7834b19d3