Nothing runs weblib-ci's own tests #9

New Issue

claude · 2026-09-08T07:55:20Z

claude commented

2026-09-08 07:55:20 +00:00

weblib-ci has no .gitea/workflows at all, so nothing here is ever run by
the runner. That was harmless while the repo held three scripts that were only
executed by other repos' jobs; #8 adds test_report_job_log.py, 21 tests that
currently only run if someone remembers to run them by hand.

The failure mode is quiet and specific: a broken report_job_log.py on main
is consumed immediately by cfbypass, weblib-archive, weblib-fs and
weblib-viewer, and the thing it breaks is the mechanism those repos use to
report breakage. A reporter that crashes reports nothing, so the symptom is
four repos going silent rather than four repos going red.

What makes it non-obvious

with-nixpkgs.sh — the pinning wrapper this org uses to get python3 on a
runner that has none — reads the consuming repo's flake.lock relative to
the working directory. weblib-ci has no flake.lock, so it cannot pin
against itself, and a workflow here would have to fall back to
nix shell nixpkgs#python3, which re-resolves the registry and refetches a
channel tarball whenever the branch moves. That is the exact cost
with-nixpkgs.sh exists to avoid, which is why this is worth a decision rather
than a quick commit:

add a minimal flake.lock (and the flake.nix to justify it) purely so the
repo can pin its own tooling — the README currently says, deliberately, that
this repo has no flake; or
accept the unpinned nix shell nixpkgs#python3 for a suite that takes 1.2s
and imports nothing outside the standard library; or
teach with-nixpkgs.sh a documented fallback when no flake.lock is found,
which would also cover any future consumer in the same position.

Whichever way it goes, the workflow needs runs-on: nix and, for the reporter
to comment on failures, the if: failure() step the other four already carry.

weblib-ci has **no `.gitea/workflows` at all**, so nothing here is ever run by the runner. That was harmless while the repo held three scripts that were only executed by other repos' jobs; #8 adds `test_report_job_log.py`, 21 tests that currently only run if someone remembers to run them by hand. The failure mode is quiet and specific: a broken `report_job_log.py` on `main` is consumed immediately by cfbypass, weblib-archive, weblib-fs and weblib-viewer, and the thing it breaks is *the mechanism those repos use to report breakage*. A reporter that crashes reports nothing, so the symptom is four repos going silent rather than four repos going red. ## What makes it non-obvious `with-nixpkgs.sh` — the pinning wrapper this org uses to get `python3` on a runner that has none — reads the **consuming** repo's `flake.lock` relative to the working directory. **weblib-ci has no `flake.lock`**, so it cannot pin against itself, and a workflow here would have to fall back to `nix shell nixpkgs#python3`, which re-resolves the registry and refetches a channel tarball whenever the branch moves. That is the exact cost `with-nixpkgs.sh` exists to avoid, which is why this is worth a decision rather than a quick commit: - add a minimal `flake.lock` (and the `flake.nix` to justify it) purely so the repo can pin its own tooling — the README currently says, deliberately, that this repo has no flake; or - accept the unpinned `nix shell nixpkgs#python3` for a suite that takes 1.2s and imports nothing outside the standard library; or - teach `with-nixpkgs.sh` a documented fallback when no `flake.lock` is found, which would also cover any future consumer in the same position. Whichever way it goes, the workflow needs `runs-on: nix` and, for the reporter to comment on failures, the `if: failure()` step the other four already carry.

claude referenced this issue

2026-09-08 07:55:30 +00:00

Take the job's outcome as an argument, not as an assumption #10

claude referenced this issue

2026-09-08 08:01:27 +00:00

Take the job's outcome as an argument, not as an assumption #10

claude added the

labels 2026-09-08 08:01:57 +00:00

claude commented

2026-09-08 08:01:58 +00:00

Priority/Medium rather than Priority/Low, on this issue's own argument: the thing an unrun change breaks is the mechanism the other four repos use to report breakage, so the symptom is four repos going quiet rather than four repos going red. That is worth more than a low.

Count correction: the suite is 22 tests, not 21 — the cold re-read of #10 added one for a backtick escaping its code span in the verbatim-status path.

`Priority/Medium` rather than `Priority/Low`, on this issue's own argument: the thing an unrun change breaks is *the mechanism the other four repos use to report breakage*, so the symptom is four repos going quiet rather than four repos going red. That is worth more than a low. Count correction: the suite is **22** tests, not 21 — the cold re-read of #10 added one for a backtick escaping its code span in the verbatim-status path.

claude referenced this issue

2026-09-08 08:34:22 +00:00

Reconciler can never clear Status/Blocked when every blocker is in another repo #11

bit commented

2026-09-08 09:09:24 +00:00

Which of the three options do you recommend?

claude commented

2026-09-08 09:11:30 +00:00

The third, but not quite as this issue words it — and the reason is something I only saw after re-reading with-nixpkgs.sh rather than my own summary of it.

What the script actually needs

It never evaluates a flake. The whole of its use of flake.lock is one string:

nodes.nixpkgs.locked.rev

and then nix shell github:nixos/nixpkgs/${rev}#${attr}. So option 1 would add a flake.nix that builds nothing, and a flake.lock to maintain, in order to hold a single 40-character rev — and it would contradict the README's deliberate "they are run, not built, so this repo has no flake". That is a lot of structure for one string.

Verified the rev is genuinely all it takes:

$ nix shell github:nixos/nixpkgs/56c02bc00adcf003215cc4bd996d6efaf4cff188#python3 \
    --command python3 -c 'import sys; print(sys.version.split()[0])'
3.14.7          real 0m0.219s

Why not option 2

It pays exactly the cost this script exists to avoid, in the repo that owns the script. The README warns against nix shell nixpkgs#python3 and the job log caught it unpacking a channel tarball; weblib-ci's own workflow would then be the one place demonstrating the anti-pattern the repo documents.

There is a second reason I did not know until measuring just now. A registry reference does not resolve the same way everywhere. On this workstation:

system flake:nixpkgs path:/nix/store/snzg7...-source?...&rev=6b316287bae2ee04c9b93c8c858d930fd07d7338

— a pinned system entry, and a different nixpkgs from the 56c02bc the repos' locks use. So nix shell nixpkgs#python3 means one nixpkgs on the runner and a different, frozen one here, with nothing saying so. That is the same trap CLAUDE.md records for nix develop cfbypass silently running old code. For a suite meant to gate a shared script, "works here, different there" is the wrong property to design in.

Option 3, with one change to what the fallback is

As worded, the fallback is "when no flake.lock is found" — but if the fallback then degrades to the registry, it is option 2 wrapped in a function: same cost, now centralised and harder to notice. The fallback should still pin:

./flake.lock present → today's behaviour, byte-identical for all four consumers.
else ./nixpkgs-pin present (one line, a rev) → use that rev.
else → fail loudly, saying how to pin.

Step 3 is the load-bearing one. A shared tool that silently degrades is how the channel-refetch cost comes back without anyone seeing it; better to refuse and say so.

weblib-ci then commits a one-line nixpkgs-pin, stays flake-free, and any future consumer in the same position is covered — which is the part of option 3 that made it attractive to begin with.

The cost, stated plainly

A nixpkgs-pin file has no nix flake update. A flake.lock can be refreshed by a standard command; a bare rev is bumped by hand and will go stale. I think that is acceptable here and not elsewhere: the pin exists to run a stdlib-only suite in 1.2 s, so a stale nixpkgs costs an older CPython and nothing else, and the suite would notice a genuinely broken interpreter immediately. If you would rather not carry a hand-bumped pin at all, that is the honest argument for option 1 and I would not object to it — it buys nix flake update at the price of a flake that builds nothing.

If you take this

It is a change to a script four repos consume, so it wants the same treatment #10 just got: prove the existing call shape is byte-for-byte unchanged for a caller that has a flake.lock, rather than assuming an added elif is inert. That check is what made #10 safe to land and it is cheap to repeat.

Sequencing: #10 is still open and touches this repo, so this should follow it rather than race it.

**The third**, but not quite as this issue words it — and the reason is something I only saw after re-reading `with-nixpkgs.sh` rather than my own summary of it. ## What the script actually needs It never evaluates a flake. The whole of its use of `flake.lock` is one string: ``` nodes.nixpkgs.locked.rev ``` and then `nix shell github:nixos/nixpkgs/${rev}#${attr}`. So **option 1 would add a `flake.nix` that builds nothing, and a `flake.lock` to maintain, in order to hold a single 40-character rev** — and it would contradict the README's deliberate "they are *run*, not built, so this repo has no flake". That is a lot of structure for one string. Verified the rev is genuinely all it takes: ``` $ nix shell github:nixos/nixpkgs/56c02bc00adcf003215cc4bd996d6efaf4cff188#python3 \ --command python3 -c 'import sys; print(sys.version.split()[0])' 3.14.7 real 0m0.219s ``` ## Why not option 2 It pays exactly the cost this script exists to avoid, **in the repo that owns the script**. The README warns against `nix shell nixpkgs#python3` and the job log caught it unpacking a channel tarball; weblib-ci's own workflow would then be the one place demonstrating the anti-pattern the repo documents. There is a second reason I did not know until measuring just now. A registry reference does not resolve the same way everywhere. On this workstation: ``` system flake:nixpkgs path:/nix/store/snzg7...-source?...&rev=6b316287bae2ee04c9b93c8c858d930fd07d7338 ``` — a **pinned system entry**, and a different nixpkgs from the `56c02bc` the repos' locks use. So `nix shell nixpkgs#python3` means one nixpkgs on the runner and a different, frozen one here, with nothing saying so. That is the same trap `CLAUDE.md` records for `nix develop cfbypass` silently running old code. For a suite meant to gate a shared script, "works here, different there" is the wrong property to design in. ## Option 3, with one change to what the fallback *is* As worded, the fallback is "when no `flake.lock` is found" — but if the fallback then degrades to the registry, it is option 2 wrapped in a function: same cost, now centralised and harder to notice. The fallback should still **pin**: 1. `./flake.lock` present → today's behaviour, **byte-identical for all four consumers**. 2. else `./nixpkgs-pin` present (one line, a rev) → use that rev. 3. else → **fail loudly**, saying how to pin. Step 3 is the load-bearing one. A shared tool that silently degrades is how the channel-refetch cost comes back without anyone seeing it; better to refuse and say so. weblib-ci then commits a one-line `nixpkgs-pin`, stays flake-free, and any future consumer in the same position is covered — which is the part of option 3 that made it attractive to begin with. ## The cost, stated plainly **A `nixpkgs-pin` file has no `nix flake update`.** A `flake.lock` can be refreshed by a standard command; a bare rev is bumped by hand and will go stale. I think that is acceptable here and not elsewhere: the pin exists to run a **stdlib-only suite in 1.2 s**, so a stale nixpkgs costs an older CPython and nothing else, and the suite would notice a genuinely broken interpreter immediately. If you would rather not carry a hand-bumped pin at all, that is the honest argument for option 1 and I would not object to it — it buys `nix flake update` at the price of a flake that builds nothing. ## If you take this It is a change to a script four repos consume, so it wants the same treatment #10 just got: prove the existing call shape is **byte-for-byte unchanged** for a caller that has a `flake.lock`, rather than assuming an added `elif` is inert. That check is what made #10 safe to land and it is cheap to repeat. Sequencing: #10 is still open and touches this repo, so this should follow it rather than race it.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Reference: weblib/weblib-ci#9