Branch: refs/heads/master
Revision: 2c971ab
Author: mppf
Log Message:
Merge pull request #16644 from mppf/use-venv-not-virtualenv
Use python3 -m venv rather than virtualenv
Follow-up to PR #16560 and PR #16614
This PR addresses a problem with make docs
on Ubuntu 18.04 where we
were getting an error like
ERROR: Can not combine '--user' and '--prefix' as they imply different installation locations
in a make docs
(when doing make chpldoc-venv
).
To do so, the PR takes these general steps:
- uses
python3 -m venv
rather than installing and running
virtualenv
, and as a result adds a dependency on python3-venv for
make docs
- changes the strategy for handling
start_test
venv dependencies.
Note that the change to python3 -m venv
from virtualenv
also changes
the order in which pip
is used:
- Before, we needed to make sure
pip
was available, and we downloaded
it if not. Then, we usedpip
to installvirtualenv
and then from
there usedpip
to install dependencies in the virtualenv. - Now, we run
python3 -m venv
to create the venv, and then we use
pip
to install dependencies in the venv. Sincepython3 -m venv
will fail ifpip
is not available, from that point we can assume
thatpython3 -m pip
will work in the venv even ifpip
is not
installed systemwide.
In more detail, the PR:
- stops doing
--force-reinstall
on thepython3 -m pip
install
commands. As far as I can tell, this was always a workaround (see
51cace8e57cea1582bb509b4ee163da6942ce37a) and doing atouch
afterwards is a better workaround. - stops running
virtualenv --relocateable
sincevenv
has no analogue
and thevenv-use-sys-python.py
script should handle it anyway - installs
wheel
in the venv since I needed that to get the chpldocs
dependencies to install for some reason - removes version numbers in
third-party/chpl-venv/sphinxcontrib-chapeldomain/setup.py - this is
parallel to a change in
https://github.com/chapel-lang/sphinxcontrib-chapeldomain/pull/38 and
solves a problem withmake docs
- removes
util/test/activate_chpl_test_venv.py
in favor of making
start_test
a wrapper script that runsstart_test.py
in the venv. We
had to change something here since moving away fromvirtualenv
meant
thatactivate_this.py
is no longer installed. Instead of trying to
add Python venv paths after the Python interpreter is running, the
approach here is to run the Python code forstart_test
in a venv in
the first place. This required some care in order to preserve #16614 -
that is, the tested code and its prediffs need to use the system python
rather than running in the venv. In particular, this PR takes the
approach of removing allPATH
components starting withCHPL_HOME
in
start_test
andsub_test
. This prevents the venv from interfering
with tests and also makes the test environment more consistent. I have
observed cases in the past where a paratest would fail butstart_test
would not for tests that assumedCHPL_HOME/bin/linux64-x86
(e.g.) was
in thePATH
. In practice, the testing infrastructure uses absolute
paths for things like thechpl
command to invoke. - Due to the
PATH
adjustments above, it was also necessary to make
sub_test
a wrapper script that runssub_test.py
in the venv.
Reviewed by @ben-albrecht and @ronawho - thanks!
- [x] full local futures testing
- [x]
make && make docs && make check-chpldoc
succeeds on:- [x] Ubuntu 20.04
- [x] Ubuntu 18.04
- [x] Fedora 32
- [x] on a Cray XC with the system python
- [x] on a Cray XC with a custom python
Future Work:
- Since we
python3 -m venv
will fail if it could not install apip
to the virtualenv, we can remove getpip support and the check for pip.
We can also simplify thepip install
commands to run in the
activated venv rather than adjusting environment variables in
third-party/chpl-venv/Makefile
targets. - Consider having
start_test
/sub_test
calldeactivate
for each
subprocess as an alternative to manipulating thePATH
explicitly
Modified Files:
A util/test/start_test.py
A util/test/sub_test.py
R third-party/chpl-venv/virtualenv.txt
R util/test/activate_chpl_test_venv.py
M .travis.yml
M doc/rst/usingchapel/prereqs.rst
M man/Makefile
M test/library/packages/ZMQ/interop-py/PREEXEC
M third-party/chpl-venv/Makefile
M third-party/chpl-venv/sphinxcontrib-chapeldomain/setup.py
M tools/c2chapel/Makefile
M util/config/fixpath.py
M util/devel/test/vagrant/provision-scripts/apt-get-deps.sh
M util/start_test
M util/test/annotate.py
M util/test/checkChplDoc
M util/test/execution_limiter.py
M util/test/paratest.server
M util/test/sub_test
Compare: https://github.com/chapel-lang/chapel/compare/6e5534e4df12...2c971ab11852