[Chapel GA Tests] 2026-10-13

Notifications Nightly Testing
1

The following GitHub Actions jobs failed:

Ran 877 total jobs at commit a6741b8d86a075af6210afad51821975bb0529be "Try adding a '.gpu-timeout' file for the SHOC Triad test as well (#29514)".

GitHub Actions run info:
2

ASAN failure is: [Error: Timed out executing program library/packages/UnitTest/doc-examples/Runner (execopts: 5)]
*
Arkouda timed out installing deps
*
The sample of apptainer failures I checked are Failed to create mount namespace: mount namespace requires privileges, check Apptainer installation. Almost certainly from updating to Ubuntu 26.04 runners (a week ahead of when the ubuntu-latest alias will change to it) in Update `ubuntu-latest` runners to `ubuntu-26.04` now by arifthpe · Pull Request #29452 · chapel-lang/chapel · GitHub.

3

Oh, and I missed this one:

test-make-install-check failed with Disk quota exceeded. I suspect this is probably also due to the runner change.

4

I thought I saw a number of "disk full" errors as well?

-Brad

5

Jinx!

anna

1 Like
6 
-- stdout output is:
  -- verifying file...
       file='/__w/chapel/chapel/arkouda/dep/build/arrow_dependencies/zlib-1.3.1.tar.gz'
  
  -- stderr output is:
  CMake Error at zlib_ep-stamp/verify-zlib_ep.cmake:29 (message):
    error: SHA256 hash of
  
      /__w/chapel/chapel/arkouda/dep/build/arrow_dependencies/zlib-1.3.1.tar.gz
  
    does not match expected value
  
      expected: '9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23'
        actual: '5df433cfa0be756ef7eb2bc0973514fa7b0b7724762c62fd732e1cab53a4d330'
  
  
  
  CMake Error at zlib_ep-stamp/zlib_ep-download-RELEASE-impl.cmake:9 (message):
    Command failed (1):
  
     '/usr/bin/cmake' '-P' '/__w/chapel/chapel/arkouda/dep/build/arrow-apache-arrow-19.0.1/cpp/build-release/zlib_ep-prefix/src/zlib_ep-stamp/verify-zlib_ep.cmake'

Just being super cautious here - is this what we normally expect to see when Arkouda times out installing deps?

7

It was a bit inaccurate of me to call it a timeout, but yes this is expected for a network issue.

IIUC the download is getting interrupted so the resulting partially-downloaded tarball differs from expected. Or it could be that the download attempt is getting served an error page instead of a tarball, obliviously downloading that and then ofc has a different hash.

1 Like
8

I went through all the failures just to make sure there wasn't one that we missed and I only see the three failure modes that Anna captured in her first post(s).

9

The make install errors are certainly due to the tmp file system shrinking

26:

+ df -h /tmp /home/runner/work/_temp /home/runner
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           7.9G   38M  7.8G   1% /tmp
/dev/root       145G   54G   91G  38% /
/dev/root       145G   54G   91G  38% /

24:

+ df -h /tmp /home/runner/work/_temp /home/runner
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       145G   60G   85G  42% /
/dev/root       145G   60G   85G  42% /
/dev/root       145G   60G   85G  42% /

I have a simple fix to just use a different tmp location which I will PR shortly

The apptainer issue seems to be caused by an older apptainer being used. I have a few potential workarounds I am looking at

-Jade

10

The apptainer issue seems to be caused by an older apptainer being used. I have a few potential workarounds I am looking at

One option (which may have already occurred to you) is to pin to 24.04, since it's pretty much irrelevant as the actual chpl testing is containerized. The downside is still that ubuntu-24.04 will eventually go away, and then someone will be forced to do a proper fix in the future.

11

The fix I added in Attempt to fix apptainer and make-install CI failures with Ubuntu 26 by jabraham17 · Pull Request #29522 · chapel-lang/chapel · GitHub seems to work, so no worries there

-Jade