CI and Testing¶

Five workflows under .github/workflows/, plus a pre-commit config that mirrors the fast half of them locally.

Workflow	Runner	Trigger	Gates
`lint.yml`	GitHub-hosted	push to `main`, every PR	black, isort, ruff, the mypy ratchet, `cargo fmt`, `cargo clippy -D warnings`, `cargo test`
`test-cpu.yml`	GitHub-hosted	push to `main`, every PR	`pytest tests/` on Python 3.10 + 3.12 with no GPU, behind a `--min-passed` coverage floor
`test-gpu.yml`	self-hosted `oasr-gpu`	nightly + manual	the full suite with `--strict-assets`, split per family, plus the `slow` and `concurrent` markers
`test-gpu-modal.yml`	GitHub-hosted → Modal GPU	manual only	the same per-family split, on one rented accelerator per architecture (sm_80 / 89 / 90 / 100 / 120)
`docs.yml`	GitHub-hosted	PRs and pushes touching `docs/`, `mkdocs.yml` or the hook	`mkdocs build --strict` (broken links and anchors, pages missing from the nav); deploys the site to GitHub Pages from `main`

Running the gates locally¶

pip install -r requirements-dev.txt

black --check oasr/ tests/ benchmarks/ scripts/ ci/
isort --check-only oasr/ tests/ benchmarks/ scripts/ ci/
ruff check oasr/ tests/ benchmarks/ scripts/ ci/
python scripts/mypy_ratchet.py

cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test
cd rust && cargo clippy -p oasr-core --lib -- -D warnings

Or install the hooks and let them run on commit:

pip install pre-commit && pre-commit install
pre-commit run --all-files

Tool versions are pinned in requirements-dev.txt and .pre-commit-config.yaml, and the two files must agree: black and ruff reformat differently across releases, so an unpinned toolchain turns a green PR red on an upstream release with no change to this repo.

A full local GPU run additionally needs the external assets exported:

set -a; source .env; set +a                      # CKPT_DIR, AUDIO_DIR
export WAV_DIR="$AUDIO_DIR"
export AUDIO_PATH=$(ls "$AUDIO_DIR"/*.wav | head -1)
export ZIPFORMER_CKPT=/path/to/icefall-asr-librispeech-zipformer-large-cr-ctc-...

pytest tests/ -m "not slow and not concurrent" -q -rs

Read the asset table at the end before believing the count.

The asset gate¶

pytest tests/ used to report a fully green suite while silently skipping every real-checkpoint test. Several environment variables gated those suites, each read ad hoc in its own module with its own default path, and pytest's summary does not distinguish "passed" from "skipped because the checkpoint was not on this box". That is how the audio_scale defect — which drops the leading token of every transcript — shipped and survived a green suite.

Two mechanisms close it.

Every gated input is declared once, in tests/assets.py, together with the marker file that proves it is really there. An HF snapshot present as dangling LFS symlinks with no payload is not a usable checkpoint, and the probe says so. assets.require(...) is the single skip site, so every skip is counted and every reason string is uniform.

Every run prints what it did not check. Not behind a flag — always:

external assets:
  CKPT_DIR                 ok       /.../20210610_u2pp_conformer_exp_librispeech
  WAV_DIR                  ok       /.../wavs  [3 test(s) skipped: incomplete]
  WHISPER_CKPT             MISSING  /.../whisper-tiny  [18 test(s) skipped]
  ...
  -> 21 test(s) skipped for missing or incomplete assets. A green run does not
     cover them; use --strict-assets to make this fatal.

An asset can be present and still too small for a test: a WAV_DIR with fewer clips than the test decodes, a release without the sub-path the test reads, an audio file too short to chunk. That is the same gap as a missing asset, so it goes through the same gate, is counted against the asset (incomplete) and fails under --strict-assets.

--strict-assets makes it fatal, which is what the GPU workflows run. An asset a runner genuinely cannot have is named one by one, so the gap sits in the workflow file where it can be read rather than inside a skip nobody reads:

pytest tests/ --strict-assets --allow-missing-asset WENET_REF_DIR

An unknown name is a usage error, not a silent no-op.

--min-passed N guards the passed count. A CUDA guard added at the wrong scope, or an import that quietly turns a module into skips, otherwise shrinks coverage without turning anything red. Raise the floor to just under the observed count after a green run; never lower it to make a red run pass.

Adding a gated input¶

Add an Asset to tests/assets.py; nothing else changes. Then, in the test:

@pytest.mark.requires_assets("MY_CKPT")      # class or function

or, inside a fixture or body:

path = assets.require("MY_CKPT")
wavs = assets.require_wavs(4)                # also gates on the file count
assets.require_contents("MY_CKPT", (Path(path) / "exp").is_dir(), "no exp/ subdir")

The tests/helpers/audio.py readers (waveforms(wav_dir, n), speech_corpus) gate their own count the same way.

Never read the environment variable directly, and never pytest.skip on what an asset holds. Both are exactly the pattern that produced the trap: the variable is unset when the asset resolves through $OASR_ASSET_ROOT, and a bare skip is not counted and stays green under --strict-assets. Taking [:n] of a listing without the count gate is the quiet version, because the test then runs on however many files there are. A skip that splits valid assets into variants is different: of a causal and a non-causal release, one test of a pair must skip whichever is supplied, and that skip stays a plain one.

The compiled extension is an asset too¶

oasr._C (the CPU decoders, the word-alignment pass, the streaming feature append) is declared in tests/assets.py as OASR_C, kind build. It is present when it imports with its alignment, decoder and streaming submodules. Tests that need it carry @pytest.mark.requires_assets("OASR_C"), never an importorskip. The two behave the same in a skip run, but only the declared asset is counted, and under --strict-assets it fails.

The self-hosted nightly is where that matters. actions/checkout runs git clean -ffdx, which deletes every gitignored build output, so a job with no build step tests a checkout with no _C. With importorskip, those tests skipped and --strict-assets stayed green. Every test-gpu.yml job now rebuilds the extensions (pip install --no-deps --no-build-isolation -e .) and runs python -c "import oasr._C, oasr._core" before pytest, so a broken build fails that step and no test runs. The cargo target directory is kept outside the workspace, so the clean does not discard the Rust build.

The accuracy gate¶

Everything else in this repo compares tensors. That is necessary and it is not sufficient: a parity oracle feeds identical features to both sides, so a bug in how audio becomes features cancels on both and the suite stays green.

tests/accuracy/test_accuracy.py measures WER on a fixed 200-utterance LJSpeech subset and fails when a rate exceeds its recorded value in ci/wer-reference.json plus a tolerance of 0.3 absolute.

Those rates are for regression detection, not publication. LJSpeech is out-of-domain for every architecture and the subset is 200 utterances. Do not quote them as OASR's WER.

The tolerance is slack for a different GPU or torch build, not for noise: the offline rates are bit-identical across consecutive runs and across max_batch_size 1 / 8 / 16 / 32, which also makes "batching does not change the answer" a measured property rather than an assumption. Where that does not hold the entry says so and records the spread instead of widening the tolerance — see the transducer and nemotron_streaming comments.

An entry may pin decode_options, which is the decode configuration the rate was measured under. The llm row does: for a speech-LLM the prompt is part of what is being asked, and a different prompt moves the WER with no defect behind it. That row is also the family's only end-to-end check — its decode path is the one with per-row KV offsets, group merging, paged KV and step CUDA graphs behind it, and a graph that captured the wrong thing returns a plausible transcript, which is exactly what a tensor oracle cannot see.

Manifests ship; audio does not¶

A manifest is JSON Lines — {"id", "audio", "text"} — with audio resolved against --audio-root, so 200 utterances is 31 KB of checked-in text. Build one for a corpus you have:

python benchmarks/run.py --family accuracy --build-manifest out.jsonl \
    --audio-root $WAV_DIR --transcripts .../metadata.csv --limit 200

One trap. Whisper's EnglishTextNormalizer deletes (...) and [...] spans, because in many corpora they annotate non-speech events. In LJSpeech the parenthetical is read aloud, so leaving brackets in the manifest drops those words from the reference while the hypothesis keeps them — scoring a correct transcription as a run of insertions. --build-manifest unwraps brackets by default (--keep-brackets to opt out) and TestManifest::test_no_bracketed_spans_survive keeps it that way.

oasr/testing/wer.py reports the corpus rate — total edits over total reference words — not the mean of per-utterance rates. Averaging weights a three-word utterance like a thirty-word one and is not comparable to any published figure.

Sweeping rather than gating is benchmarks/run.py --family accuracy — see benchmarks.md.

Two GPU backends¶

The GPU suite runs in two places from one family split, ci/gpu_suites.py. A split maintained twice drifts, and the failure mode is a test file that runs nowhere — so --check is both a pre-commit hook and a lint.yml step. Since the families are directories under tests/, it checks two things: every tests/**/test_*.py sits under exactly one family directory, and no two test modules share a basename (pytest imports them by basename, so a clash is an import error rather than a merge).

	self-hosted (`test-gpu.yml`)	Modal (`test-gpu-modal.yml`)
GPU	the reference box — one SM	one accelerator per architecture, `ARCH_SWEEP`
checkpoints	already on disk	a Volume the container fills itself
cost	none	GPU-minutes per run
when	nightly sweep	on demand only — per PR, or when the box is busy or wedged

Use the self-hosted box as the primary. The assets are local and it is a machine somebody can look at. Modal exists because that box is also the benchmarking machine (CI competing for clocks is exactly the noise the perf work fights), because its GPU has fallen off the bus in a way only a host reset recovers, and — the reason that grew — because one SM is not a test matrix. Kernel selection reads the compute capability in half a dozen places, _default_cuda_cflags emits a different -gencode per arch, and a kernel that compiles for an architecture can still take CUTE_INVALID_CONTROL_PATH on the silicon — compiling is cross-targetable, running is not. Code that assumed one architecture is invisible until a second one runs it.

What a matrix does not buy is coverage of the shape-aware rule tables. Those are pure functions of (op, M, N, K, dtype, sm) — test_gemm_heuristic.py used to skip its whole rule-table suite unless the running SM matched the rules, which meant an SM120 table nobody could validate anywhere else. It now checks every registered architecture's table from any box, and the matrix is left to do what only it can: run the kernels.

Nothing triggers the Modal workflow automatically — no schedule, no pull_request — because every run is rented GPU-minutes. Run it from Actions → test-gpu-modal → Run workflow, or:

gh workflow run test-gpu-modal.yml -f pr=123                  # test PR #123
gh workflow run test-gpu-modal.yml -f pr=123 -f gpus=H100 -f suites=kernels
gh workflow run test-gpu-modal.yml --ref my-branch            # a branch, no PR

With pr set, the job resolves the PR's head commit once, at the start (a push during the run is not what gets tested), checks that commit out, and posts the outcome on it as the test-gpu-modal commit status, so the result shows on the PR. A PR from a fork is refused unless allow_fork is ticked — see Security. The same entry points run from a developer box:

modal run ci/modal_app.py::main --gpus all                    # one per architecture
modal run ci/modal_app.py::main --gpus H100,B200              # a subset
modal run ci/modal_app.py::main --gpus L40S --suites kernels  # one arch, one family

--gpus all expands to ARCH_SWEEP in ci/modal_app.py: one accelerator per distinct SM rather than one per product, because two L40S-class cards run the same kernels. Every (gpu, suite) pair is a separate container and they all run concurrently.

accelerator	SM	notes
`A100-40GB` / `A100-80GB`	sm_80	Ampere; matches the A30 dev box's arch
`A10G`	sm_86
`L4` / `L40S`	sm_89	Ada
`H100` / `H200`	sm_90	Hopper — the `sm_90a` arch-conditional path
`B200`	sm_100	Blackwell datacentre
`RTX-PRO-6000`	sm_120	GB202, the SM the reference box ships on

The assets Volume fills itself¶

ci/modal_assets.py names every external asset by (source, pinned revision) — a Hub repo id, a URL, a handful of raw files out of a git tree — and

modal run ci/modal_app.py::fetch_assets            # ~20 GiB, once
modal run ci/modal_app.py::fetch_assets --dry-run  # the plan, first
modal run ci/modal_app.py::fetch_assets --skip SPEECH_LLM_CKPT

downloads them inside the container, straight into the Volume they will be read from. It is idempotent: an asset whose marker file is already in place costs one stat. The workflow runs it on every dispatch so a new source reaches CI without anybody remembering to.

This replaced a 29 GiB upload from whichever developer machine had .env set. That upload was slow, it was not reproducible — whatever happened to be on that box, at whatever revision — and it made a CI whose job is to check the code depend on one person's filesystem. Revisions are pinned for the same reason ci/wer-reference.json records a checkpoint: an asset that moves under CI turns a WER regression into an unattributable one.

Not everything has a public source. Four assets are listed in modal_assets.NO_PUBLIC_SOURCE with the reason — WeNet's librispeech u2pp_conformer release (wenet.org.cn/downloads now 404s for it, and the only Hub mirror ships an exported TorchScript final.zip the converter cannot read), the generated tiny Qwen2-Audio fixture, and a locally built k2 HLG.pt. Those are uploaded once, per asset:

set -a; source .env; set +a
modal run ci/modal_app.py::seed_assets --dry-run
modal run ci/modal_app.py::seed_assets --only CKPT_DIR,SPEECH_LLM_TINY

ALLOW_MISSING is derived from that same dict rather than written out again, so an asset that gains a public source drops off the allow-list by itself — an allow-list that outlives the gap it documented is the exact failure --strict-assets exists to prevent.

A second Volume caches the JIT artifacts, partitioned per architecture under /jit/sm<cc>. Without it every run recompiles the kernels from cold, because they JIT on first call. The partition is not a correctness fix — the cache key already covers the target, since _default_cuda_cflags puts -gencode=…sm_XX and -DOASR_TARGET_SM into the flags JitSpec._content_hash hashes — it is so that five suites on five GPUs are not five concurrent writers to one Volume prefix.

The image, and what is not in it¶

One image serves every architecture. The CMake half of the build compiles no CUDA (OASR_SOURCES is six .cc files), so CUDA_ARCHITECTURES is the whole list 80;86;89;90;100;120 and costs nothing; the kernels that actually differ per arch are JIT-compiled on the GPU that will run them. An image per arch would mean an image build per arch for no coverage.

The base is nvidia/cuda:*-devel, not -runtime, because the JIT shells out to nvcc at run time. Torch comes from the cu128 index — the first one with sm_100 and sm_120 wheels.

3rdparty/cutlass is not uploaded. It is 26 069 files and 2.0 GiB, and it was going into the image context on every modal run, dwarfing the code change that prompted the run. The container fetches the pinned commit from GitHub as a source tarball in its own cached layer, keeps the two include roots oasr/jit/env.py collects, and symlinks it to /repo/3rdparty/cutlass — which is the only place that function looks. The sha comes from git ls-tree HEAD 3rdparty/cutlass, i.e. from the index, so it is correct whether or not the submodule is checked out. That is why the workflow's actions/checkout no longer asks for submodules.

Security — this repo is public¶

test-gpu.yml triggers on schedule and workflow_dispatch only. Never add pull_request — a fork PR would execute arbitrary code on the maintainer's machine. The Modal workflow does not have that problem: the GitHub-hosted job only holds a token and the GPU work happens in Modal's sandbox. It is dispatch-only, so only someone with write access can start it. A dispatch with pr=<n> for a PR from a fork is refused unless allow_fork is ticked: the GitHub-hosted job imports the checked-out ci/modal_app.py with the Modal token in its environment, so a fork's code could read that token. Review the diff first.

Why `test-cpu.yml` builds nothing¶

oasr/_C*.so (CMake + CUDA) and oasr/_core*.so (setuptools-rust) are both absent on a GitHub-hosted runner, and pip install -e . would need the CUDA toolkit for a job that cannot use a GPU anyway. It does not need to: import oasr works without either, because the CUDA kernels JIT-compile on first call and _C is loaded lazily. The job puts the repo on PYTHONPATH and runs pytest directly.

Files whose every test allocates on device="cuda" carry a module-level pytestmark skip, the tests that need the compiled _C gate on the OASR_C asset (see below), and @pytest.mark.cuda actually gates.

Three rules follow, each of which cost a red run to learn:

"No GPU" and "no CUDA toolkit" are different conditions, and a developer box has both. Reproducing a GitHub runner locally therefore needs four things removed, not one — the GPU, the compiled extensions, the checkpoints, and nvcc plus a warm ~/.cache/oasr/jit. Anything that JIT-compiles will otherwise pass locally off the cache and fail in CI.
No pipes in a step that is supposed to gate. GitHub runs steps under bash -e, so pytest ... --collect-only | tail -5 reports tail's exit status. A collection error then looks like a passing step, and the real failure surfaces later as a bare exit code 2 — pytest's INTERRUPTED, i.e. "a module failed to import", not "a test failed". A test module that imports a third-party package at module scope without importorskip can take down the entire session.
No source file may be gitignored. A bare checkpoints/ pattern matches a directory of that name at any depth, which once kept the whole oasr/checkpoints/ package out of every clone while the local worktree had it — and silently excluded those files from black, isort and ruff, all three of which respect .gitignore. Patterns are anchored now (/checkpoints/) and scripts/check_no_ignored_sources.py runs as a pre-commit hook. It only works locally: on CI the files are already absent, so there is nothing left to find.

Python 3.10 is in the matrix because pyproject.toml declares requires-python = ">=3.10", and a floor nobody tests is a guess. That floor is real: oasr/decoder/wfst/graph_image.py annotates defaults with PEP 604 unions (np.ndarray | None) and has no from __future__ import annotations, so importing it on 3.9 raises TypeError at def time.

Why mypy is a ratchet, not a gate¶

mypy oasr/ reports several hundred errors. Nearly all are untyped-torch noise (no-any-return, attr-defined on tensors) rather than defects, and the CuteDSL kernels under oasr/kernels/cute/ are excluded outright because the cutlass.cute decorators rewrite function bodies and mypy sees the DSL's injected names as undefined.

Gating on zero would mean the job is red forever and nobody reads it. Gating on no file got worse is a check that passes today and still fails on a real regression:

python scripts/mypy_ratchet.py            # check against ci/mypy-baseline.json
python scripts/mypy_ratchet.py --update   # after a cleanup, lower the numbers

A file whose count drops is reported as slack to reclaim, not an error, so a cleanup commit does not have to touch the baseline to stay green. A file whose count rises fails the job and names the file. Fix the new error or narrow it with a targeted # type: ignore[code] — do not raise the baseline.

Because the baseline counts errors coming out of third-party stubs, the environment is part of it and is pinned in ci/mypy-requirements.txt:

pip install --index-url https://download.pytorch.org/whl/cpu torch
pip install -r ci/mypy-requirements.txt -r requirements-dev.txt

Two settings there are load-bearing:

[tool.mypy] sets no python_version. It applies to third-party stubs too, so pinning it below the interpreter aborts the run before a line of this repo is checked ("3.8" dies on torch's match statements; "3.10" dies on numpy's PEP 695 type statements). mypy runs at the interpreter's version; CI pins that to 3.12. Real 3.10 compatibility is covered by test-cpu.yml actually running the suite on 3.10.
oasr/decoder/wfst/lattice.py is excluded. mypy hits an INTERNAL ERROR on its np.lexsort against recent numpy stubs, which kills the whole run; a per-module ignore_errors does not help, because the crash happens during checking rather than reporting.

Known lint deviations¶

B905 (zip(..., strict=)) is in ruff's ignore list. It is a real bug class and enabling it is a worthwhile follow-up, but deciding strict=True per call site requires knowing the lengths always match, and there are dozens of them across the engine and the kernels. Turning it on blind would be a guess, not a fix.
oasr-core allows clippy::useless_conversion at crate scope — it fires inside #[pyfunction]'s own expansion, where a function-level #[allow] does not reach.
oasr-server-grpc allows clippy::result_large_err — tonic::Status is 176 bytes and is tonic's type, not ours.

Rust: never `cargo --workspace`¶

oasr-core enables pyo3/extension-module while oasr-server enables pyo3/auto-initialize; the two are mutually exclusive and Cargo unifies features per build, so one invocation covering both fails to compile. oasr-core is excluded from default-members and linted in its own step. Run cargo from rust/, not the repo root, so the target-dir redirect applies — see AGENTS.md.