[Chapel Merge] Use python3 -m venv rather than virtualenv

Branch: refs/heads/master
Revision: 2c971ab
Author: mppf
Log Message:

Merge pull request #16644 from mppf/use-venv-not-virtualenv

Use python3 -m venv rather than virtualenv

Follow-up to PR #16560 and PR #16614

This PR addresses a problem with make docs on Ubuntu 18.04 where we
were getting an error like

ERROR: Can not combine '--user' and '--prefix' as they imply different installation locations

in a make docs (when doing make chpldoc-venv).

To do so, the PR takes these general steps:

  • uses python3 -m venv rather than installing and running
    virtualenv, and as a result adds a dependency on python3-venv for
    make docs
  • changes the strategy for handling start_test venv dependencies.

Note that the change to python3 -m venv from virtualenv also changes
the order in which pip is used:

  • Before, we needed to make sure pip was available, and we downloaded
    it if not. Then, we used pip to install virtualenv and then from
    there used pip to install dependencies in the virtualenv.
  • Now, we run python3 -m venv to create the venv, and then we use
    pip to install dependencies in the venv. Since python3 -m venv
    will fail if pip is not available, from that point we can assume
    that python3 -m pip will work in the venv even if pip is not
    installed systemwide.

In more detail, the PR:

  • stops doing --force-reinstall on the python3 -m pip install
    commands. As far as I can tell, this was always a workaround (see
    51cace8e57cea1582bb509b4ee163da6942ce37a) and doing a touch
    afterwards is a better workaround.
  • stops running virtualenv --relocateable since venv has no analogue
    and the venv-use-sys-python.py script should handle it anyway
  • installs wheel in the venv since I needed that to get the chpldocs
    dependencies to install for some reason
  • removes version numbers in
    third-party/chpl-venv/sphinxcontrib-chapeldomain/setup.py - this is
    parallel to a change in
    https://github.com/chapel-lang/sphinxcontrib-chapeldomain/pull/38 and
    solves a problem with make docs
  • removes util/test/activate_chpl_test_venv.py in favor of making
    start_test a wrapper script that runs start_test.py in the venv. We
    had to change something here since moving away from virtualenv meant
    that activate_this.py is no longer installed. Instead of trying to
    add Python venv paths after the Python interpreter is running, the
    approach here is to run the Python code for start_test in a venv in
    the first place. This required some care in order to preserve #16614 -
    that is, the tested code and its prediffs need to use the system python
    rather than running in the venv. In particular, this PR takes the
    approach of removing all PATH components starting with CHPL_HOME in
    start_test and sub_test. This prevents the venv from interfering
    with tests and also makes the test environment more consistent. I have
    observed cases in the past where a paratest would fail but start_test
    would not for tests that assumed CHPL_HOME/bin/linux64-x86 (e.g.) was
    in the PATH. In practice, the testing infrastructure uses absolute
    paths for things like the chpl command to invoke.
  • Due to the PATH adjustments above, it was also necessary to make
    sub_test a wrapper script that runs sub_test.py in the venv.

Reviewed by @ben-albrecht and @ronawho - thanks!

  • [x] full local futures testing
  • [x] make && make docs && make check-chpldoc succeeds on:
    • [x] Ubuntu 20.04
    • [x] Ubuntu 18.04
    • [x] Fedora 32
    • [x] on a Cray XC with the system python
    • [x] on a Cray XC with a custom python

Future Work:

  • Since we python3 -m venv will fail if it could not install a pip
    to the virtualenv, we can remove getpip support and the check for pip.
    We can also simplify the pip install commands to run in the
    activated venv rather than adjusting environment variables in
    third-party/chpl-venv/Makefile targets.
  • Consider having start_test / sub_test call deactivate for each
    subprocess as an alternative to manipulating the PATH explicitly

Modified Files:
A util/test/start_test.py
A util/test/sub_test.py
R third-party/chpl-venv/virtualenv.txt
R util/test/activate_chpl_test_venv.py
M .travis.yml
M doc/rst/usingchapel/prereqs.rst
M man/Makefile
M test/library/packages/ZMQ/interop-py/PREEXEC
M third-party/chpl-venv/Makefile
M third-party/chpl-venv/sphinxcontrib-chapeldomain/setup.py
M tools/c2chapel/Makefile
M util/config/fixpath.py
M util/devel/test/vagrant/provision-scripts/apt-get-deps.sh
M util/start_test
M util/test/annotate.py
M util/test/checkChplDoc
M util/test/execution_limiter.py
M util/test/paratest.server
M util/test/sub_test

Compare: https://github.com/chapel-lang/chapel/compare/6e5534e4df12...2c971ab11852