Mitigate the stuck/reasserted cross-repo Status/Blocked (does not fully fix #11) #13

Merged
bit merged 2 commits from fix/cross-repo-blocker-state into main 2026-09-09 16:35:13 +00:00
Member

Relates to weblib-ci#11 -- deliberately not Closes, see "What this
actually is" below: this does not fix the title's claim
("Reconciler can never clear Status/Blocked when every blocker is in another
repo") -- that is still true after this merges. It fixes the worse half: the
reconciler no longer lies about it.

What this actually is

Before: a cross-repo Status/Blocked was stuck, silently. Worse, found
live on 2026-09-09: it was being reasserted over a human's hand-removal
every 15 minutes, also silently. Nothing in the log said why, or that it was
happening at all.

After: the label is still stuck in exactly the same cases (the token
genuinely cannot resolve the blocker -- see below), but now the run says so
by issue number every time, and it stops fighting a human's correction. That
is "loudly unknown" replacing "silently wrong" -- a real improvement, worth
merging on its own, but not the fix the issue title describes. Do not close
#11 on merge.
Leave it open, pointing at the decision below, which is what
would actually let a cross-repo state be positively confirmed.

Decision needed (on #11, not blocking this PR)

Should a broader-scoped credential be added so cross-repo blocker state can
be positively confirmed, instead of only ever reported as unknown?

That is bit's call -- a new credential has a real blast radius, so I have not
added one unilaterally. Evidence for the decision is below.

What this changes for you

Once merged, a cross-repo Status/Blocked will stop being silently stuck or
silently reasserted -- the reconciler logs unknown <repo>#<n>: N blocker(s) unresolvable with this token -- leaving 'Status/Blocked' as-is (present|absent) and leaves the label exactly where it was. A human still has
to look and decide; this only stops the automation from fighting that
decision.

Evidence: two credentials, same instant, opposite results

This is what rules out "the endpoint is broken" -- it is a permission
substitution, not a general limitation. Same call
(GET repos/weblib/weblib-archive/issues/<n>/dependencies), same issues, same
moment, only the token differs.

Under a personal token with read access to the whole org (not the
reconciler's token -- a human's), weblib-archive#30 and #29's dependencies
come back fully populated:

#30 -> id=155 #33 'closed' cfbypass ; id=51 #1 'closed' cfbypass
#29 -> id=45 #27 'closed' weblib-archive ; id=43 #25 'closed' weblib-archive ;
       id=32 #14 'closed' weblib-archive ; id=48 #17 'open' weblib-fs

Under weblib-archive's own scoped Actions token -- the credential the
reconciler actually runs with -- the identical call for #30 returns two
entries that are Go's zero-value Issue struct instead, verbatim from the
live run:

DEBUG weblib/weblib-archive#30: len(deps)=2 slim=[{"repo": null, "number": 0, "state": "", "keys": [...26 keys, all present, all blank...]}, {"repo": null, "number": 0, "state": "", "keys": [...same...]}]
DEBUG-RAW weblib/weblib-archive#30: [{"id": 0, "url": "", "html_url": "", "number": 0, "user": null, "original_author": "", "original_author_id": 0, "title": "", "body": "", "ref": "", "assets": null, "labels": null, "milestone": null, "assignee": null, "assignees": null, "state": "", "is_locked": false, "comments": 0, "created_at": "0001-01-01T00:00:00Z", "updated_at": "0001-01-01T00:00:00Z", "closed_at": null, "due_date": null, "time_estimate": 0, "pull_request": null, "repository": null, "pin_order": 0}, {"id": 0, "url": "", "html_url": "", "number": 0, "user": null, "original_author": "", "original_author_id": 0, "title": "", "body": "", "ref": "", "assets": null, "labels": null, "milestone": null, "assignee": null, "assignees": null, "state": "", "is_locked": false, "comments": 0, "created_at": "0001-01-01T00:00:00Z", "updated_at": "0001-01-01T00:00:00Z", "closed_at": null, "due_date": null, "time_estimate": 0, "pull_request": null, "repository": null, "pin_order": 0}]

Entries are never omitted (len(deps) is 2, matching the two real
blockers) -- each one is a placeholder with id: 0, state: "",
repository: null, and every other field blank or zero. d.get("state") != "closed" on the old code read that blank as "still open", forever.

The wall is hard, not a permissions nuance. Same scoped token, direct
GET instead of via /dependencies:

repos/weblib/cfbypass/issues/1: HTTPError 404 b'{"errors":null,"message":"not found","url":"https://git.chaosbit.de/api/swagger"}\n'
repos/weblib/cfbypass: HTTPError 404 b'{"errors":null,"message":"not found","url":"https://git.chaosbit.de/api/swagger"}\n'
repos/weblib/weblib-archive/issues/68: HTTP 200, 1653 bytes, starts b'{"id":192,...

The token cannot see cfbypass at all -- not "cannot see this issue", cannot
see the repository. Making cross-repo state resolvable needs a token that can
read across repos; that is the decision above.

(All three snippets above are one measurement each, by me, not yet
independently re-run by anyone else -- flagging that for the cold re-read
rather than presenting single live-instance output as settled fact.)

The fix

is_resolved() keys on id == 0, not on the empty state -- id cannot
legitimately be zero for a real issue, and a future Gitea populating state
while still stubbing the rest of the entry would silently revive the bug if
the check were on state instead.

reconcile() is now a three-way split instead of two:

  • any blocker positively open -> label stays/goes on (decisive on its
    own, even alongside unresolvable entries -- this direction must never
    soften)
  • every blocker positively closed -> label comes off (unchanged; same-repo
    dependencies always resolve fully, so this path is untouched in practice --
    TestSameRepoUnchanged pins that so a cross-repo fix cannot regress it)
  • any blocker unresolvable, with no known-open one to decide it anyway ->
    change nothing, log the issue number

This keeps the asymmetry from cfbypass#19 (a false "unblocked" sends someone
at work that cannot be done; a false "blocked" only hides work) -- it adds a
third outcome for the case the old two-way split had no honest way to
express, rather than inverting either existing one.

Acceptance test, run for real

Not the throwaway -- the actual weblib-archive#30, #57, #54, #29, which
carried Status/Blocked going in for four different reasons (checked with a
personal token beforehand): #30 both blockers closed-but-cross-repo, #29 one
same-repo-closed set plus one cross-repo open (weblib-fs#17), #57 and #54
same-repo blockers that are genuinely still open.

Ran this fix branch for real against weblib/weblib-archive, under its own
Actions token (a throwaway branch off development that only swapped the
composite action's ref, workflow_dispatched, then deleted). First
non-dry-run, to prove no label moved; then --dry-run again to capture the
log without depending on the (broken, 500s-for-every-id) job-log viewer:

weblib/weblib-archive:
  hands off weblib/weblib-archive#62  (Status/On Hold)
  hands off weblib/weblib-archive#61  (Status/On Hold)
  unknown weblib/weblib-archive#30: 2 blocker(s) unresolvable with this token -- leaving 'Status/Blocked' as-is (present)
  unknown weblib/weblib-archive#29: 1 blocker(s) unresolvable with this token -- leaving 'Status/Blocked' as-is (present)
--> 0 change(s) (dry run, nothing applied)
  • #30 and #29 are both named unknown, not silently skipped and not
    wrongly reported as resolved. #29 matters here specifically because its
    cross-repo blocker (weblib-fs#17) happens to still be genuinely open under a
    personal token -- a label that is correct by luck is not evidence the
    reconciler resolved it, and this confirms it did not: it is unknown to this
    token exactly like #30 is.
  • #57 and #54 do not appear at all -- meaning their (same-repo) blockers
    resolved normally, were found genuinely open, and the label was already
    correct, so nothing was logged (matches TestSameRepoUnchanged's "already
    correct is a no-op" case). Confirmed directly: their Status/Blocked labels
    were unchanged by the earlier non-dry-run pass.
  • The real, non-dry-run pass left all four labels exactly as they started --
    verified by reading each issue's labels via the API before and after, not
    by trusting the run's exit code.

Nothing on #30/#57/#54/#29 was hand-edited at any point; every state change
(none, in this case) came only from the reconciler's own calls.

Tests

test_sync_blocked_label.py, new, 12 tests, offline against a local
http.server fake (same pattern as test_report_job_log.py):

python3 test_sync_blocked_label.py   # 12 tests, OK
python3 test_report_job_log.py       # 22 tests, OK (unaffected)

Proved by fault injection. Stashed the fix (kept the new tests), leaving
sync_blocked_label.py at its pre-#11-fix state:

FAILED (failures=4, errors=1)
FAIL   test_a_known_open_blocker_still_wins_over_an_unresolvable_one
FAIL   test_an_unresolvable_blocker_blocks_removal_even_with_other_closed
FAIL   test_never_reasserts_a_label_a_human_removed
FAIL   test_never_removes_a_label_it_cannot_confirm_is_clear
ERROR  test_is_resolved_keys_on_id_not_on_state  (AttributeError: no is_resolved)

test_never_reasserts_a_label_a_human_removed reproduced the live bug
verbatim against the old code: changed == [(30, 'add', 'blocked by #0, #0')]
-- it really does read the two zero-value stubs' blank #0 as things to
report as blockers and put the label back on. Every other test (same-repo,
hands-off, no-deps, no-label-in-repo, dry-run) still passed, so the fault was
isolated to exactly the new behaviour. Restored the fix -- diffed
byte-identical against the pre-revert copy -- and the suite is green again.

Not done here, on purpose

  • weblib-archive#68, the throwaway issue this reproduction started from,
    is closed with its dependency and label removed. Its comment history still
    has the original probe output (superseded by the quotes above, which are
    copied verbatim from it so the evidence does not depend on a throwaway
    issue staying around).
  • Separately filed, not fixed here: main in both weblib-archive
    (weblib-archive#70) and cfbypass (cfbypass#35) still runs each repo's own
    pre-migration local copy of this script, which can no longer find
    Status/Blocked at all since labels moved to the org
    (repos/<repo>/labels now returns []) -- every scheduled run on main
    currently prints skipped: no 'Status/Blocked' label in this repo and does
    nothing, for any issue, in either repo. development in weblib-archive
    already migrated to consuming this repo via the composite action
    (weblib-archive#44); main has not been promoted since. This is live and
    currently worse than #11 -- both filed and assigned to bit, since promoting
    main is each repo's release process to decide, not something to fold into
    this script fix.
Relates to weblib-ci#11 -- deliberately **not** `Closes`, see "What this actually is" below: this does not fix the title's claim ("Reconciler can never clear Status/Blocked when every blocker is in another repo") -- that is still true after this merges. It fixes the worse half: the reconciler no longer lies about it. ## What this actually is **Before:** a cross-repo `Status/Blocked` was stuck, silently. Worse, found live on 2026-09-09: it was being *reasserted* over a human's hand-removal every 15 minutes, also silently. Nothing in the log said why, or that it was happening at all. **After:** the label is still stuck in exactly the same cases (the token genuinely cannot resolve the blocker -- see below), but now the run says so by issue number every time, and it stops fighting a human's correction. That is "loudly unknown" replacing "silently wrong" -- a real improvement, worth merging on its own, but not the fix the issue title describes. **Do not close #11 on merge.** Leave it open, pointing at the decision below, which is what would actually let a cross-repo state be positively confirmed. ## Decision needed (on #11, not blocking this PR) **Should a broader-scoped credential be added so cross-repo blocker state can be positively confirmed, instead of only ever reported as unknown?** That is bit's call -- a new credential has a real blast radius, so I have not added one unilaterally. Evidence for the decision is below. ## What this changes for you Once merged, a cross-repo `Status/Blocked` will stop being silently stuck or silently reasserted -- the reconciler logs `unknown <repo>#<n>: N blocker(s) unresolvable with this token -- leaving 'Status/Blocked' as-is (present|absent)` and leaves the label exactly where it was. A human still has to look and decide; this only stops the automation from fighting that decision. ## Evidence: two credentials, same instant, opposite results This is what rules out "the endpoint is broken" -- it is a permission substitution, not a general limitation. Same call (`GET repos/weblib/weblib-archive/issues/<n>/dependencies`), same issues, same moment, only the token differs. **Under a personal token with read access to the whole org** (not the reconciler's token -- a human's), weblib-archive#30 and #29's dependencies come back fully populated: ``` #30 -> id=155 #33 'closed' cfbypass ; id=51 #1 'closed' cfbypass #29 -> id=45 #27 'closed' weblib-archive ; id=43 #25 'closed' weblib-archive ; id=32 #14 'closed' weblib-archive ; id=48 #17 'open' weblib-fs ``` **Under weblib-archive's own scoped Actions token** -- the credential the reconciler actually runs with -- the identical call for #30 returns two entries that are Go's zero-value `Issue` struct instead, verbatim from the live run: ``` DEBUG weblib/weblib-archive#30: len(deps)=2 slim=[{"repo": null, "number": 0, "state": "", "keys": [...26 keys, all present, all blank...]}, {"repo": null, "number": 0, "state": "", "keys": [...same...]}] DEBUG-RAW weblib/weblib-archive#30: [{"id": 0, "url": "", "html_url": "", "number": 0, "user": null, "original_author": "", "original_author_id": 0, "title": "", "body": "", "ref": "", "assets": null, "labels": null, "milestone": null, "assignee": null, "assignees": null, "state": "", "is_locked": false, "comments": 0, "created_at": "0001-01-01T00:00:00Z", "updated_at": "0001-01-01T00:00:00Z", "closed_at": null, "due_date": null, "time_estimate": 0, "pull_request": null, "repository": null, "pin_order": 0}, {"id": 0, "url": "", "html_url": "", "number": 0, "user": null, "original_author": "", "original_author_id": 0, "title": "", "body": "", "ref": "", "assets": null, "labels": null, "milestone": null, "assignee": null, "assignees": null, "state": "", "is_locked": false, "comments": 0, "created_at": "0001-01-01T00:00:00Z", "updated_at": "0001-01-01T00:00:00Z", "closed_at": null, "due_date": null, "time_estimate": 0, "pull_request": null, "repository": null, "pin_order": 0}] ``` Entries are never *omitted* (`len(deps)` is 2, matching the two real blockers) -- each one is a placeholder with `id: 0`, `state: ""`, `repository: null`, and every other field blank or zero. `d.get("state") != "closed"` on the old code read that blank as "still open", forever. **The wall is hard, not a permissions nuance.** Same scoped token, direct `GET` instead of via `/dependencies`: ``` repos/weblib/cfbypass/issues/1: HTTPError 404 b'{"errors":null,"message":"not found","url":"https://git.chaosbit.de/api/swagger"}\n' repos/weblib/cfbypass: HTTPError 404 b'{"errors":null,"message":"not found","url":"https://git.chaosbit.de/api/swagger"}\n' repos/weblib/weblib-archive/issues/68: HTTP 200, 1653 bytes, starts b'{"id":192,... ``` The token cannot see `cfbypass` at all -- not "cannot see this issue", cannot see the repository. Making cross-repo state resolvable needs a token that can read across repos; that is the decision above. *(All three snippets above are one measurement each, by me, not yet independently re-run by anyone else -- flagging that for the cold re-read rather than presenting single live-instance output as settled fact.)* ## The fix `is_resolved()` keys on `id == 0`, not on the empty `state` -- `id` cannot legitimately be zero for a real issue, and a future Gitea populating `state` while still stubbing the rest of the entry would silently revive the bug if the check were on `state` instead. `reconcile()` is now a three-way split instead of two: - any blocker **positively open** -> label stays/goes on (decisive on its own, even alongside unresolvable entries -- this direction must never soften) - every blocker **positively closed** -> label comes off (unchanged; same-repo dependencies always resolve fully, so this path is untouched in practice -- `TestSameRepoUnchanged` pins that so a cross-repo fix cannot regress it) - any blocker **unresolvable, with no known-open one to decide it anyway** -> change nothing, log the issue number This keeps the asymmetry from cfbypass#19 (a false "unblocked" sends someone at work that cannot be done; a false "blocked" only hides work) -- it adds a third outcome for the case the old two-way split had no honest way to express, rather than inverting either existing one. ## Acceptance test, run for real Not the throwaway -- the actual weblib-archive#30, #57, #54, #29, which carried `Status/Blocked` going in for four different reasons (checked with a personal token beforehand): #30 both blockers closed-but-cross-repo, #29 one same-repo-closed set plus one cross-repo *open* (weblib-fs#17), #57 and #54 same-repo blockers that are genuinely still open. Ran this fix branch for real against `weblib/weblib-archive`, under its own Actions token (a throwaway branch off `development` that only swapped the composite action's ref, `workflow_dispatch`ed, then deleted). First non-dry-run, to prove no label moved; then `--dry-run` again to capture the log without depending on the (broken, 500s-for-every-id) job-log viewer: ``` weblib/weblib-archive: hands off weblib/weblib-archive#62 (Status/On Hold) hands off weblib/weblib-archive#61 (Status/On Hold) unknown weblib/weblib-archive#30: 2 blocker(s) unresolvable with this token -- leaving 'Status/Blocked' as-is (present) unknown weblib/weblib-archive#29: 1 blocker(s) unresolvable with this token -- leaving 'Status/Blocked' as-is (present) --> 0 change(s) (dry run, nothing applied) ``` - **#30 and #29 are both named `unknown`**, not silently skipped and not wrongly reported as resolved. #29 matters here specifically because its cross-repo blocker (weblib-fs#17) happens to still be genuinely open under a personal token -- a label that is correct *by luck* is not evidence the reconciler resolved it, and this confirms it did not: it is unknown to this token exactly like #30 is. - **#57 and #54 do not appear at all** -- meaning their (same-repo) blockers resolved normally, were found genuinely open, and the label was already correct, so nothing was logged (matches `TestSameRepoUnchanged`'s "already correct is a no-op" case). Confirmed directly: their `Status/Blocked` labels were unchanged by the earlier non-dry-run pass. - The real, non-dry-run pass left all four labels exactly as they started -- verified by reading each issue's labels via the API before and after, not by trusting the run's exit code. Nothing on #30/#57/#54/#29 was hand-edited at any point; every state change (none, in this case) came only from the reconciler's own calls. ## Tests `test_sync_blocked_label.py`, new, 12 tests, offline against a local `http.server` fake (same pattern as `test_report_job_log.py`): python3 test_sync_blocked_label.py # 12 tests, OK python3 test_report_job_log.py # 22 tests, OK (unaffected) **Proved by fault injection.** Stashed the fix (kept the new tests), leaving `sync_blocked_label.py` at its pre-#11-fix state: FAILED (failures=4, errors=1) FAIL test_a_known_open_blocker_still_wins_over_an_unresolvable_one FAIL test_an_unresolvable_blocker_blocks_removal_even_with_other_closed FAIL test_never_reasserts_a_label_a_human_removed FAIL test_never_removes_a_label_it_cannot_confirm_is_clear ERROR test_is_resolved_keys_on_id_not_on_state (AttributeError: no is_resolved) `test_never_reasserts_a_label_a_human_removed` reproduced the live bug verbatim against the old code: `changed == [(30, 'add', 'blocked by #0, #0')]` -- it really does read the two zero-value stubs' blank `#0` as things to report as blockers and put the label back on. Every other test (same-repo, hands-off, no-deps, no-label-in-repo, dry-run) still passed, so the fault was isolated to exactly the new behaviour. Restored the fix -- diffed byte-identical against the pre-revert copy -- and the suite is green again. ## Not done here, on purpose - **weblib-archive#68**, the throwaway issue this reproduction started from, is closed with its dependency and label removed. Its comment history still has the original probe output (superseded by the quotes above, which are copied verbatim from it so the evidence does not depend on a throwaway issue staying around). - **Separately filed, not fixed here:** `main` in both weblib-archive (weblib-archive#70) and cfbypass (cfbypass#35) still runs each repo's own pre-migration local copy of this script, which can no longer find `Status/Blocked` at all since labels moved to the org (`repos/<repo>/labels` now returns `[]`) -- every scheduled run on `main` currently prints `skipped: no 'Status/Blocked' label in this repo` and does *nothing*, for any issue, in either repo. `development` in weblib-archive already migrated to consuming this repo via the composite action (weblib-archive#44); `main` has not been promoted since. This is live and currently worse than #11 -- both filed and assigned to bit, since promoting `main` is each repo's release process to decide, not something to fold into this script fix.
claude added 1 commit 2026-09-09 15:49:39 +00:00
Closes weblib-ci#11 in part: the reconciler could add Status/Blocked but
never remove it when every blocker lived in another repo, and, found live
on weblib-archive#30 (2026-09-09), it went further and *reasserted* the
label over a human's hand-removal every 15 minutes.

Established which of the two candidate mechanisms in #11 was real by
instrumenting the script and running it under a real Gitea Actions
token scoped to weblib-archive, against weblib-archive#68 (a throwaway
issue depending on the already-closed weblib/cfbypass#1) and the live
weblib-archive#30: `deps` is never empty for a cross-repo edge (rules out
"Gitea omits the entry"), but every cross-repo entry comes back as Go's
zero-value Issue struct -- id 0, state "", repository null -- rather than
being omitted or erroring. The old `d.get("state") != "closed"` read that
blank state as "still open", which is why the label could never come off,
and why it kept getting silently reapplied.

`is_resolved()` keys on `id == 0` rather than on the empty `state`, since
`id` cannot legitimately be zero for a real issue and a future Gitea could
start populating `state` while still stubbing the rest of the entry.
`reconcile()` now keeps the existing add/remove behaviour whenever every
blocker resolves (same-repo dependencies always do; pinned by
TestSameRepoUnchanged so this cannot regress), but when a blocker cannot
be resolved and none of the resolvable ones are positively open, it
changes nothing and logs the issue number instead of guessing -- keeping
the asymmetry from cfbypass#19 (a false "unblocked" sends someone at work
that cannot be done; a false "blocked" only hides work) rather than
inverting it.

Also probed whether the repo-scoped token could resolve a cross-repo
blocker by querying the other repo directly instead of via
`/dependencies`: `GET repos/weblib/cfbypass/issues/1` and
`GET repos/weblib/cfbypass` both 404 under weblib-archive's own Actions
token, while the same call against its own repo succeeds. So this token
cannot see cfbypass at all; making cross-repo state resolvable would need
a broader-scoped credential, which is a real blast-radius decision left
for bit rather than assumed here. The "unknown -> leave alone, log it"
behaviour in this PR fixes the stuck/reasserted label either way, with or
without that credential ever existing.

12 new tests in test_sync_blocked_label.py, offline against a local
http.server fake. Proved by fault injection: reverting sync_blocked_label.py
to its pre-fix form (keeping the new tests) fails exactly the 5 tests in
TestUnresolvableCrossRepoBlocker -- one of them reproducing the live
reassert bug verbatim (`[(30, 'add', 'blocked by #0, #0')]`) -- while every
other test still passes; restoring the fix (diffed byte-identical against
the pre-revert copy) turns the suite green again.

  python3 test_sync_blocked_label.py   # 12 tests, OK
  python3 test_report_job_log.py       # 22 tests, OK (unaffected)

Co-authored-by: bit <bit@das-labor.org>
claude changed title from Treat an unresolvable cross-repo blocker as unknown, not as open forever to Mitigate the stuck/reasserted cross-repo Status/Blocked (does not fully fix #11) 2026-09-09 15:59:47 +00:00
claude added 1 commit 2026-09-09 16:04:25 +00:00
The docstring credited the every-15-minutes cron with putting Status/Blocked
back after a hand-removal. It cannot have: schedule and `issues` events run the
workflow from the default branch, weblib-archive's default branch is `main`, and
`main` still carries a stale pre-org-label-migration copy at
tools/sync_blocked_label.py. That copy looks the label up with
`repos/{repo}/labels`; both repos now have zero repo-level labels (all 25 are
org-level), so it returns None and exits "skipped: no 'Status/Blocked' label in
this repo" without writing anything. Measured today: 377 scheduled and 54
`issues` runs, every one a no-op.

The reassertions came from pull_request and workflow_dispatch runs on branches
cut from `development`, which consume the shared script. The symptom tracks PR
activity, not a timer -- which also explains why it looked intermittent.

Mechanism and fix are unchanged; only the attribution was wrong, and a wrong
cause in a permanent comment is what sends the next reader to the wrong file.

Co-authored-by: bit <bit@das-labor.org>
bit approved these changes 2026-09-09 16:32:04 +00:00
bit scheduled this pull request to auto merge when all checks succeed 2026-09-09 16:35:11 +00:00
bit merged commit 8c0088987b into main 2026-09-09 16:35:13 +00:00
bit deleted branch fix/cross-repo-blocker-state 2026-09-09 16:35:13 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Reference: weblib/weblib-ci#13