weblib/fdroid-repo-ci
2
0
Fork 0
Code Issues Pull Requests Actions Activity

publish.sh: verify credentials and upload the index last; add next-version-code.sh

Browse Source 
Closes #5
Closes #6

An F-Droid repo is a signed index plus the files it names, and a partial
upload could previously leave them disagreeing: aws s3 sync walks the tree
in its own order, so an interrupted publish could land the index before the
APKs it names, breaking the repo for every client until the upload finished
or was rolled back. publish.sh now uploads in two passes, unconditionally --
every non-index file first, the index (index-v1.json(.asc), index-v2.json,
entry.json, entry.jar) last -- so an interruption anywhere in pass 1 simply
leaves the repo describing the old state a little longer, never a mixture.
Credentials are also verified with one real authenticated call
(head-bucket) before either pass, rather than merely checking the
environment variables are non-empty, since an unusable-but-present
credential failing halfway through is exactly how a partial upload happens.

next-version-code.sh is the one implementation of the versionCode rule
(candidate = minutes since 2026-01-01Z, clamped to strictly exceed whatever
is already published for that applicationId), shared by CI and manual
builds so there is exactly one place this can drift. bit pointed out
(appended to #6) that the clamp is monotonic against *observed* state, not
monotonic by construction: two jobs starting in the same minute can both
read the same published value and race to publish the same versionCode on
different APKs. Nothing can prevent that wasted build, so publish.sh now
also refuses to overwrite an already-existing APK object before pass 1 --
turning the collision from a silent, permanent wrong state into a loud
failure at the last moment the information is reliable.

Tested offline (83 assertions: pure-stdlib suites for both scripts, the
real CLIs through with-nixpkgs.sh, and a small local HEAD-only S3 stub
proving the credential check is a real authenticated call and the
APK-collision refusal fires only when it should) and live against
fdroid-weblib-dev (7 more: a real publish observed mid-flight via a
same-process synchronization hook to prove the index is genuinely absent
while the file is already present, both objects reading back
byte-identical, and a second publish refused against a real
already-published APK with its bytes left unchanged).

Co-authored-by: bit <bit@das-labor.org>
This commit is contained in:
claude
2026-09-09 08:00:45 +00:00
co-authored by bit
parent fcc6a741ef
commit 19e55c4a53
10 changed files with 1045 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+116 -15
View File
@@ -21,11 +21,13 @@ here.
| script | what it does |
|---|---|
| `next-version-code.sh` | print the next monotonic `versionCode` for an application id, read from a repo index -- called before the build, CI and a manual build alike (see "One versionCode rule" below) |
| `sign-apk.sh` | check alignment, then sign a prebuilt unsigned release APK with a keystore the caller supplies |
| `fdroid-update.sh` | run `fdroid update` over a `repo/` directory of already-signed APKs and produce the signed index |
| `publish.sh` | upload the resulting repo tree to an S3-compatible bucket |
| `publish.sh` | upload the resulting repo tree to an S3-compatible bucket -- files first, index last, and refuses to overwrite an already-published APK (see "Publish ordering and safety" below) |
| `with-nixpkgs.sh` | run a command with one or more nixpkgs packages on PATH, pinned to the revision in `nixpkgs-pin` |
| `check_zip_alignment.py` | the alignment check `sign-apk.sh` runs before signing (see "Align before sign" below) |
| `next_version_code.py` | the stdlib-only implementation `next-version-code.sh` wraps (index read, arithmetic) |
| `action.yml` | the composite action a consumer includes to get all of the above |
Deliberately **not** in scope: fetching the APK. That needs an upstream
@@ -42,6 +44,13 @@ below for what that costs and why it is still the right call here.
- uses: actions/checkout@v4
- id: ci
uses: https://git.chaosbit.de/weblib/fdroid-repo-ci@main
- run: |
version_code=$(bash "${{ steps.ci.outputs.path }}/next-version-code.sh" \
"$FDROID_REPO_INDEX_URL" "$APPLICATION_ID")
echo "version_code=$version_code" >> "$GITHUB_ENV"
# ... feed $version_code into whatever builds app-release-unsigned.apk
- env:
APK_KEYSTORE_PASSWORD: ${{ secrets.APK_KEYSTORE_PASSWORD }}
run: |
@@ -72,13 +81,73 @@ only one that works while a referenced repo is private -- irrelevant for
consuming it, since that one holds the keys.
`$APK_KEYSTORE`/`$INDEX_KEYSTORE`/`$FDROID_REPO_ROOT`/`$BUCKET`/
`$S3_ENDPOINT_URL` above are the consumer's own paths and values -- nothing
in this repo defaults or hints at them. Env var *names* like
`APK_KEYSTORE_PASSWORD` are the consumer's choice too, not a convention this
repo imposes: `sign-apk.sh` and `fdroid-update.sh` each take the name of the
environment variable to read a password from as an argument, specifically so
two different keystores in the same job (the APK key and the separate repo
index key -- see below) never have to share one variable name.
`$S3_ENDPOINT_URL`/`$FDROID_REPO_INDEX_URL`/`$APPLICATION_ID` above are the
consumer's own paths and values -- nothing in this repo defaults or hints at
them. Env var *names* like `APK_KEYSTORE_PASSWORD` are the consumer's choice
too, not a convention this repo imposes: `sign-apk.sh` and `fdroid-update.sh`
each take the name of the environment variable to read a password from as an
argument, specifically so two different keystores in the same job (the APK
key and the separate repo index key -- see below) never have to share one
variable name.
## One `versionCode` rule, shared by CI and a manual build (fdroid-repo-ci#6)
Two implementations of "the next version code" is the exact failure mode
`next-version-code.sh` exists to avoid, so both CI and a manual build call
this one script rather than each computing their own:
```
candidate = whole minutes since 2026-01-01T00:00:00Z
published = highest versionCode already in the repo index for this applicationId
result = max(candidate, published + 1)
```
Minutes, not seconds or raw Unix time: `versionCode` is a signed 32-bit int,
and this scheme does not exhaust it for 4,082 years, against 67 years for
seconds and 11 for Unix time. The clamp against `published` exists because
minute granularity means two builds started in the same minute would
otherwise compute the same number -- and it also covers a clock stepping
backwards or an old commit being rebuilt, both of which would otherwise
produce a number lower than one a device has already seen.
**The clamp is monotonic against *observed* state, not monotonic by
construction.** It does not close the race where two jobs start in the same
minute, both read the same `published` value, both compute the same next
versionCode, and both build and publish -- the window is the whole build,
not just the clock read, since the version code has to be baked into the APK
before it exists. Nothing in `next-version-code.sh` can prevent that.
`publish.sh`'s APK-existence check (below) is the other half of this: it
cannot stop the wasted build, but it turns the collision from a silent,
permanent wrong state into a loud failure at the last moment, when the
information is finally reliable.
Takes an index-v2.json URL or local file path, and an application id;
prints the integer and nothing else, so a caller does
`versionCode=$(next-version-code.sh "$URL" "$APPLICATION_ID")`. The URL form
needs no credentials, since both this repo and the F-Droid repo it reads are
public -- the point of weblib-viewer#16's "buildable when CI is down". Fails
loudly (non-zero exit, nothing on stdout) if the index cannot be read; an
application id with no entry at all in the index is not an error -- that is
simply the first-ever publish for a channel, and yields the bare candidate.
## Publish ordering and safety (fdroid-repo-ci#5, fdroid-repo-ci#6)
An F-Droid repo is a signed index plus the files it names. `publish.sh`
uploads in two passes, unconditionally: every non-index file first, the
index (`index-v1.json(.asc)`, `index-v2.json`, `entry.json`, `entry.jar`,
matched only at the top level of the tree) last. A publish interrupted
anywhere in the first pass leaves the repo describing the old state a little
longer; it can never leave an index naming files that 404. Credentials are
verified with a real authenticated call (`head-bucket`) before either pass,
rather than merely checking that the environment variables are non-empty --
an unusable-but-present credential failing halfway through is exactly how a
partial (and potentially index-first) upload happens.
Before pass 1, `publish.sh` also checks every local `*.apk`'s destination
key with `head-object` and refuses the whole publish if any of them already
exists -- the safeguard `next-version-code.sh`'s clamp cannot provide on its
own (see above). This does not prevent the wasted build behind a same-minute
collision; it prevents the collision from silently corrupting the repo.
## Align before sign, and how "no `zipalign`" was verified
@@ -188,12 +257,21 @@ here needs, resolved through the same pin.
bash run_tests.sh
```
42 assertions across five suites, ~20s on a warm nix store:
83 assertions across seven suites offline (~20s on a warm nix store), plus 5
more that only run against a real bucket (see below):
* `test_check_zip_alignment.py` -- pure standard library, no nix, no
network at all; builds an aligned and a misaligned fixture zip by hand and
checks both directions, including the 4096-byte boundary specific to `.so`
entries.
* `test_next_version_code.py` -- pure standard library too, testing
`next_version_code.py` as a subprocess against fixture index-v2.json files
written to a temp dir: no prior build, a lower/equal/higher prior (the
clock-went-backwards case), several versions at once, a version entry with
no versionCode, an application id missing from the index entirely (not an
error), and an unreadable/malformed index (fails loudly, nothing on
stdout). "now" is pinned via an undocumented env var seam so the expected
candidate is a fixed number.
* `test_with_nixpkgs.sh` -- offline; every way the pinning wrapper is
supposed to refuse rather than silently degrade (missing pin, malformed
pin, too few arguments). Does not exercise the happy path of actually
@@ -205,14 +283,37 @@ bash run_tests.sh
against a throwaway keystore generated fresh per run, real `apksigner`
and `fdroid update`. Needs the pinned toolchain to be resolvable (locally
cached, or fetched once over the network); nothing beyond that.
* `test_publish.sh` -- argument/credential validation and `--dry-run`
command construction, fully offline, plus one real invocation against a
closed local port (`127.0.0.1:1`, nothing listens there) to prove the
built command genuinely attempts a network call rather than failing on its
own arguments. The actual upload path is out of scope for an offline
suite, matching the constraint that only `publish.sh` needs network in the
* `test_next_version_code.sh` -- the real `next-version-code.sh` CLI, through
`with-nixpkgs.sh`: argument count, a local index file end to end, a
missing application id, a missing/malformed local file, and an
unreachable URL (`http://127.0.0.1:1`, the same closed-port trick
`test_publish.sh` uses) -- proving the wrapper's argv/exit-code/stdout
contract, not just the Python module's arithmetic.
* `test_publish.sh` -- argument/credential validation, `--dry-run` command
construction (now two commands, files then index, and asserted in that
order), and a real invocation against a closed local port
(`127.0.0.1:1`), fully offline. Also runs `test_s3_stub.py`, a
same-repo, HEAD-only local HTTP stub, to prove the credential check is a
real authenticated call (a wrong bucket is rejected) and that an
already-published APK is refused before any upload, with a negative
control proving a non-colliding APK does not trip the same check. The
actual upload path -- and the two-pass *ordering* of real uploads, which
nothing offline can see -- is exercised only by the opt-in live round trip
below, matching the constraint that only `publish.sh` needs network in the
first place.
`test_publish.sh`'s live section (skipped by default; needs
`FDROID_CI_LIVE_BUCKET`, `FDROID_CI_LIVE_ENDPOINT` and real credentials) adds
5 more assertions against a real bucket: the round trip itself; the index
object provably absent while the file object is already present at the one
point in the middle of a real publish that the ordering guarantee depends on
(observed via a same-process synchronization hook `publish.sh` exposes only
for this, `PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES` -- deliberately not a poll
racing a background upload from outside, since that is exactly the
flakiness fdroid-repo-ci#5 warned against); both objects reading back
byte-identical; a second publish refused when it would overwrite a
real, already-published APK; and that APK's bytes unchanged afterwards.
No `.gitea/workflows` runs this, on the same reasoning weblib-ci#9 recorded:
run it by hand before pushing. If that changes for weblib-ci it should
probably change here too, at the same time.