New Issue: Make sure library documentation is testable

Notifications Issues
1

27184, "jabraham17", "Make sure library documentation is testable", "2025-04-29T22:21:43Z"

Recently, a user reported a bug in our Sort docs examples. When I went to fix it, I found at least two more bugs in the Sort module docs.

To my knowledge, this is not the first time we have had these kinds of issues. I personally have hit this with the Image module docs. I learned from that experience and took great pains with the Python module docs to have the code samples be testable.

We currently have a few strategies for dealing with this

  • write tests in a special format and use doc/util/extract-rst-tests.py
    • This is the approach the spec takes. It technically only works by extracting Chapel tests from *.rst files. But I see no reason it can't extract Chapel tests from Chapel files.
  • Use .. literalinclude
    • This is the approach I used when resolving the sort module issues.
    • Its also used by other stuff like chplvis, the users guide, some of the primers, ZMQ, and Futures
  • Some other "custom solution"
    • This is the approach the Python module takes, using test/library/packages/Python/doc-examples/parse_docs.py.
    • This is a bad option, we should have more standardized tooling. Whether that is making this script used more widely or replacing it with one of the other options.
    • This script was created naively by me without knowledge of doc/util/extract-rst-tests.py

I think we should spend the time to make sure all of our documentation/examples are testable (or at a minimum the frequently used modules have testable docs). Additionally, we should consolidate on the first two ways of doing things today.

A bonus here would be a linter/smoketest to enforce testable examples in docs. Essentially disallowing .. code-block:: chapel. This feels like an extreme option that probably needs an opt-out, but untestable docs can and will becomes stale which is not useful for users.