weblib/weblib-viewer#95: every version in the published index claimed it was added at the moment of the *last* publish, so an F-Droid client's version list showed five separate builds all added within seconds of each other. Measured on the live dev repo: index-v2.json gave versionCodes 365139/365516/365666/365710/366099 the added dates 05:50:06/08/10/12/13 on 2026-09-12, while the bucket's own Last-Modified for those objects says they were actually published on 2026-09-11 at 13:53, 20:09, 22:37, 23:23 and 2026-09-12 05:50. It is not an mtime problem, and --use-date-from-apk does not fix it. fdroidserver takes the date from common.KnownApks, whose __init__ loads it out of an already existing repo/index-v2.json (and archive/index-v2.json), and whose recordapk falls back to datetime.now() for any filename it has not seen; --use-date-from-apk only changes that fallback to the local file's mtime, which for a re-downloaded APK is when it was downloaded. KnownApks' own docstring says there is nowhere else to look: the index is "currently the only way to permanently store the 'updated' date of APKs". A CI job that rebuilds repo/ on a fresh runner -- which is exactly what weblib-viewer#32's retention scheme does when it re-downloads the previous APKs -- therefore starts with an empty KnownApks, and every build in it counts as new. So hand fdroid-update.sh the previously published index, optionally, as FDROID_PREVIOUS_INDEX -- a local path, because fetching it needs to know the bucket and per fdroid-repo-ci#1 that stays on the consumer's side; this script still needs no network. preserve_added.py stages it where KnownApks looks, and afterwards *checks* that every build present in both indexes really did keep its date, failing the run if not. That is not belt-and-braces: fdroidserver's own read_added_date_from_all_apks carries a TODO about reading these dates differently, so a future version could drop the behaviour this depends on, and a silent return to flattened history is the failure mode that took a human noticing to catch the first time. Set-but-missing is a hard failure rather than a shrug, and so is a file that is not a parseable index-v2.json (a truncated download, an S3 NoSuchKey error document): KnownApks has no error handling of its own and would abort mid-library with a traceback that never names the file, and a consumer whose fetch silently produced nothing would otherwise flatten its whole history on every publish while looking healthy. A first-ever publish expresses "there is no previous index" by leaving the variable unset. Verified end to end without publishing anything, against the five real APKs and a copy of the live dev index-v2.json pulled over HTTPS: - no previous index (today's behaviour), all five re-stamped: 366099 06:23:19, 365710 06:23:18, 365666 06:23:17, 365516 06:23:15, 365139 06:23:14 -- a five-second spread, all of it "now". - previous index minus 366099, i.e. the real "four retained plus one genuinely new" case: 365139/365516/365666/365710 keep 05:50:06/08/10/12 exactly, 366099 gets 06:24:01, and lastUpdated follows the new build. - a second run on a fresh repo/ with the APK mtimes deliberately touched to now: all five unchanged, "5 already-published build(s) kept their original 'added' timestamp, 0 new". test_preserve_added.py (12 assertions, stdlib only) covers the drift detection in both directions, including the negative control that a drifted date fails; test_fdroid_update.sh covers the shell contract. run_tests.sh: 123 passed, 4 skipped, 0 failed. A weblib-viewer change is needed too -- its workflows have to fetch last publish's index and set FDROID_PREVIOUS_INDEX -- and is a separate PR in that repo. Co-authored-by: bit <bit@das-labor.org>
640 lines
36 KiB
Markdown
640 lines
36 KiB
Markdown
# fdroid-repo-ci
|
|
|
|
Shared tooling to build a signed F-Droid repo index over already-signed APKs
|
|
and publish the result to an S3-compatible bucket -- consumed as a composite
|
|
action by whichever repository currently holds the (index) signing key and
|
|
runs the publish. Which repository that is has changed more than once, so
|
|
nothing here names it; see "Not hard-coded" below.
|
|
|
|
**The builder signs; this repo only distributes.** Settled on
|
|
weblib-viewer#19: an APK can reach a device via a git release *and* via
|
|
F-Droid, and updates from either route have to install over each other,
|
|
which needs the *same* app signature on both -- so signing has to happen
|
|
once, at the build, not again here. This is also simply how F-Droid's
|
|
binary-repo mode works: `fdroid update` **reads** each APK's existing
|
|
signature and never creates one (see "The repo index key is not the APK
|
|
signing key" below). `sign-apk.sh` exists only for the exceptional consumer
|
|
whose build genuinely emits an unsigned artifact -- it is not part of the
|
|
default pipeline and `action.yml` does not call it.
|
|
|
|
**This repository is never handed an app signing key.** The only key it
|
|
ever receives is the **repo index key**, which signs `index-v1.jar`/
|
|
`entry.jar` and identifies the repo itself to a client (pinned via
|
|
`?fingerprint=` in the repo URL) -- never an individual app. A workflow
|
|
change that starts passing an APK keystore or its password into this repo's
|
|
scripts (other than as an explicit, opt-in `sign-apk.sh` argument for the
|
|
unsigned-artifact exception above) is a sign the invariant has been broken,
|
|
not a new normal.
|
|
|
|
Same shape as [weblib-ci], and public for the same measured reason: a
|
|
private repository cannot be consumed cross-repo via `uses:` on this Gitea
|
|
instance (see weblib-ci's README) -- every cross-repo form fails while the
|
|
repo being referenced is private, and only the full-URL form works once it
|
|
is public.
|
|
|
|
**Nothing secret belongs here, ever.** Keys, keystores, credentials, bucket
|
|
names and hostnames all live in the private repo that consumes this one and
|
|
arrive as inputs -- arguments and environment variables, never committed
|
|
here.
|
|
|
|
## What is here
|
|
|
|
| script | what it does |
|
|
|---|---|
|
|
| `next-version-code.sh` | print the next monotonic `versionCode` for an application id and a commit's committer date, read from a repo index -- called before the build, CI and a manual build alike (see "One versionCode rule" below) |
|
|
| `sign-apk.sh` | **the exception, not the default flow** (see above) -- check alignment, then sign a prebuilt unsigned release APK with a keystore the caller supplies, for a consumer whose build does not already produce a signed APK |
|
|
| `fdroid-update.sh` | run `fdroid update` over a `repo/` directory of already-signed APKs and produce the signed index |
|
|
| `publish.sh` | upload the resulting repo tree to an S3-compatible bucket -- files first, index last, and refuses to overwrite an already-published APK (see "Publish ordering and safety" below) |
|
|
| `with-nixpkgs.sh` | run a command with one or more nixpkgs packages on PATH, pinned to the revision in `nixpkgs-pin` |
|
|
| `check_zip_alignment.py` | the alignment check `sign-apk.sh` runs before signing (see "Align before sign" below) |
|
|
| `next_version_code.py` | the stdlib-only implementation `next-version-code.sh` wraps (index read, arithmetic) |
|
|
| `preserve_added.py` | stage last publish's `index-v2.json` so `fdroid update` reuses its per-build `added` dates, and check afterwards that it did (see "Per-build `added` timestamps" below) |
|
|
| `action.yml` | the composite action a consumer includes to get all of the above |
|
|
|
|
Deliberately **not** in scope: fetching the APK. That needs an upstream
|
|
credential and is the consumer's business, not the tooling's -- which also
|
|
keeps this repo from knowing which upstreams exist.
|
|
|
|
All of it is standard library / plain bash plus the pinned tools below. They
|
|
are *run*, not built, so this repo has no flake -- see "Pinning nixpkgs"
|
|
below for what that costs and why it is still the right call here.
|
|
|
|
## Using it
|
|
|
|
This is the default flow: the APK arrives at this action **already signed**
|
|
by the builder (weblib-viewer#19), so the pipeline only computes the next
|
|
`versionCode`, indexes and publishes. It never touches an app signing key --
|
|
only the repo index key, in the `fdroid-update.sh` step.
|
|
|
|
```yaml
|
|
- uses: actions/checkout@v4
|
|
- id: ci
|
|
uses: https://git.chaosbit.de/weblib/fdroid-repo-ci@main
|
|
|
|
- run: |
|
|
version_code=$(bash "${{ steps.ci.outputs.path }}/next-version-code.sh" \
|
|
"$FDROID_REPO_INDEX_URL" "$APPLICATION_ID" "$(git log -1 --format=%ct)")
|
|
echo "version_code=$version_code" >> "$GITHUB_ENV"
|
|
# ... feed $version_code into whatever builds the already-signed
|
|
# app-release.apk
|
|
|
|
- env:
|
|
INDEX_KEYSTORE_PASSWORD: ${{ secrets.INDEX_KEYSTORE_PASSWORD }}
|
|
FDROID_REPO_URL: https://fdroid-weblib.s3w.dev.chaosbit.de/repo
|
|
FDROID_REPO_NAME: weblib
|
|
FDROID_REPO_DESCRIPTION: "weblib's F-Droid repository"
|
|
# Last publish's index, fetched by the consumer (only it knows the
|
|
# bucket). Leave UNSET for a channel's first-ever publish; see
|
|
# "Per-build `added` timestamps" below.
|
|
FDROID_PREVIOUS_INDEX: /tmp/previous-index-v2.json
|
|
run: |
|
|
bash "${{ steps.ci.outputs.path }}/fdroid-update.sh" \
|
|
"$FDROID_REPO_ROOT" "$INDEX_KEYSTORE" repo-key INDEX_KEYSTORE_PASSWORD
|
|
|
|
- env:
|
|
AWS_ACCESS_KEY_ID: ${{ secrets.PUBLISH_ACCESS_KEY_ID }}
|
|
AWS_SECRET_ACCESS_KEY: ${{ secrets.PUBLISH_SECRET_ACCESS_KEY }}
|
|
run: |
|
|
bash "${{ steps.ci.outputs.path }}/publish.sh" \
|
|
"$FDROID_REPO_ROOT/repo" "$BUCKET" "$S3_ENDPOINT_URL" repo
|
|
```
|
|
|
|
The trailing `repo` is `publish.sh`'s `<prefix>` argument -- it has to put
|
|
the tree under a URL path ending in `/repo` to match `FDROID_REPO_URL` above
|
|
(see "`FDROID_REPO_URL` must end in the literal suffix `/repo`" below).
|
|
|
|
**Use the full URL, `https://git.chaosbit.de/weblib/fdroid-repo-ci@main`, not
|
|
`weblib/fdroid-repo-ci@main`.** Measured on weblib-ci (weblib-archive#44):
|
|
the bare `owner/repo` form resolves against the instance's default actions
|
|
URL rather than this host and fails outright. The full-URL form is also the
|
|
only one that works while a referenced repo is private -- irrelevant for
|
|
*this* repo, which is public on purpose, but relevant for whatever repo is
|
|
consuming it, since that one holds the index key.
|
|
|
|
`$INDEX_KEYSTORE`/`$FDROID_REPO_ROOT`/`$BUCKET`/`$S3_ENDPOINT_URL`/
|
|
`$FDROID_REPO_INDEX_URL`/`$APPLICATION_ID` above are the consumer's own
|
|
paths and values -- nothing in this repo defaults or hints at them. The env
|
|
var *name* `INDEX_KEYSTORE_PASSWORD` is the consumer's choice too, not a
|
|
convention this repo imposes: `fdroid-update.sh` takes the name of the
|
|
environment variable to read the password from as an argument. Name it
|
|
distinctly from whatever holds the APK key's password even if, today, both
|
|
happen to resolve to the same underlying keystore/secret (see "The repo
|
|
index key is not the APK signing key" below) -- that way separating the two
|
|
keys later is a value change on the consumer's side, not a rename of this
|
|
convention.
|
|
|
|
**`FDROID_REPO_URL` and `FDROID_REPO_NAME` are required environment
|
|
variables, not arguments** -- unlike the password env var *names* above,
|
|
these are the actual values (not secrets), so they are passed directly
|
|
rather than through an indirection. `fdroid-update.sh` refuses to run
|
|
without them, naming the missing one (fdroid-repo-ci#11): a repo that
|
|
`fdroid update` builds without `repo_url`/`repo_name` set publishes
|
|
*successfully* while advertising fdroidserver's own placeholder template
|
|
(`MyFirstFDroidRepo.org`, "My First F-Droid Repo Demo") -- silently wrong,
|
|
which is worse than a failed job. `FDROID_REPO_DESCRIPTION` is optional and
|
|
has no fallback if omitted (fdroidserver's own default, which is the same
|
|
placeholder problem, one field narrower).
|
|
|
|
**`FDROID_REPO_URL` must end in the literal suffix `/repo`.** This is not a
|
|
convention of this repo -- fdroidserver's own `common.py` refuses to run
|
|
*any* command once `config.yml` sets `repo_url` at all, unless it ends in
|
|
`/repo` (`repo_url needs to end with /repo`), and there is no flag to turn
|
|
that check off. Whatever value is set here is also embedded verbatim as
|
|
`repo.address` in `index-v2.json`, so it has to be **the actual URL path
|
|
`index-v2.json` is served under**, not merely a plausible-looking one:
|
|
confirm with `<repo_url>/index-v2.json` resolving, don't guess.
|
|
|
|
Concretely, that means `publish.sh`'s own `<prefix>` argument has to be used
|
|
to put the tree under a URL path ending in `/repo` -- e.g. publish with
|
|
prefix `repo` so the bucket serves `.../repo/index-v2.json`, and set
|
|
`FDROID_REPO_URL` to `https://<bucket>.<domain>/repo` to match. Publishing
|
|
to the bucket root with no prefix (as this repo's consumer did before this
|
|
was noticed) cannot satisfy fdroidserver's own requirement at all: any
|
|
`repo_url` set there either lacks the suffix (and `fdroid update` refuses to
|
|
run) or has the suffix and points at a path the bucket never actually
|
|
serves (and a client's "Add to F-Droid" 404s on `index-v2.json`) -- the
|
|
`/repo` prefix is not optional decoration, it is what makes `repo_url`
|
|
and the published tree agree at all.
|
|
|
|
### Per-build `added` timestamps (`FDROID_PREVIOUS_INDEX`, weblib/weblib-viewer#95)
|
|
|
|
**A repo that rebuilds `repo/` from scratch on every run must hand
|
|
`fdroid-update.sh` the index it published last time, or every build in it
|
|
gets re-stamped with the current wall clock.** That is
|
|
weblib/weblib-viewer#95: an F-Droid client's version list showed five
|
|
separate builds all "added" within seconds of each other, all at the moment
|
|
of the most recent publish.
|
|
|
|
It is not an mtime problem and `--use-date-from-apk` does not fix it.
|
|
fdroidserver takes the date from `common.KnownApks`, which loads it out of
|
|
an *already existing index*:
|
|
|
|
```python
|
|
for part in ('repo', 'archive'):
|
|
path = os.path.join(part, 'index-v2.json')
|
|
if os.path.isfile(path):
|
|
...
|
|
filename = version["file"]["name"][1:]
|
|
date = datetime.fromtimestamp(version["added"] // 1000, tz=timezone.utc)
|
|
self.apks[filename] = date
|
|
```
|
|
|
|
and falls back to the wall clock for any filename it has not seen:
|
|
|
|
```python
|
|
if apkName not in self.apks:
|
|
if default_date is None:
|
|
default_date = datetime.now(timezone.utc)
|
|
self.apks[apkName] = default_date
|
|
return self.apks[apkName]
|
|
```
|
|
|
|
`KnownApks`' own docstring says why there is nowhere else to look: the index
|
|
is "currently the only way to permanently store the 'updated' date of APKs".
|
|
A CI job on a fresh runner starts with no `repo/index-v2.json`, so
|
|
`KnownApks` starts empty and *everything*, including APKs re-downloaded by a
|
|
retention scheme, counts as new. (`--use-date-from-apk` only changes the
|
|
fallback from `now()` to the local file's mtime, which for a re-downloaded
|
|
file is the time it was downloaded -- no better.)
|
|
|
|
So: fetch the previously published `index-v2.json` and point
|
|
`FDROID_PREVIOUS_INDEX` at the local copy. `fdroid-update.sh` stages it
|
|
where `KnownApks` looks, runs the update, and then **checks** that every
|
|
build present in both indexes really did keep its date, failing the job if
|
|
not -- fdroidserver's own `read_added_date_from_all_apks` carries a TODO
|
|
about reading these dates differently, so this is verified rather than
|
|
assumed.
|
|
|
|
- **Unset it for a first-ever publish.** A path that is set but does not
|
|
exist is a hard failure, deliberately: a consumer whose fetch silently
|
|
produced nothing would otherwise flatten its entire history on every
|
|
publish and look perfectly healthy doing it.
|
|
- A file that is not a parseable `index-v2.json` (a truncated download, an
|
|
S3 `NoSuchKey` error document) is refused **before** `fdroid update`
|
|
starts, naming the path -- `KnownApks` has no error handling of its own
|
|
and would abort mid-library with a traceback that never mentions the file.
|
|
- A genuinely new build is untouched and still gets "now"; that is the one
|
|
date that *should* be the current time.
|
|
- `lastUpdated` keeps meaning the real last update: fdroidserver derives it
|
|
as the newest `added` across the app's builds.
|
|
- Fetching it is the consumer's job, not this repo's, for the same reason
|
|
fetching the APK is: only the consumer knows the bucket, and per
|
|
fdroid-repo-ci#1 this repo must not come to know. `fdroid-update.sh`
|
|
itself still needs no network.
|
|
|
|
One caveat worth stating rather than discovering: the *app-level*
|
|
`metadata.added` is derived by fdroidserver as the oldest `added` among the
|
|
builds currently in the index, so it moves forward as old APKs age out of a
|
|
retention window. That is inherent to retention -- the index cannot
|
|
reference an APK that is no longer published -- and is not what #95 is
|
|
about.
|
|
|
|
### The exception: a consumer whose build does not already sign
|
|
|
|
`sign-apk.sh` is **not part of the flow above** and `action.yml` does not
|
|
call it. It exists only for a consumer whose build genuinely emits an
|
|
unsigned APK and has nowhere else to sign it -- an app signing key cannot be
|
|
rotated, so a build that produces an unsigned artifact and relies on this
|
|
step is a real (if narrow) case, not a shortcut around the settled model.
|
|
Insert it between the build and the `fdroid-update.sh` step above:
|
|
|
|
```yaml
|
|
- env:
|
|
APK_KEYSTORE_PASSWORD: ${{ secrets.APK_KEYSTORE_PASSWORD }}
|
|
run: |
|
|
bash "${{ steps.ci.outputs.path }}/sign-apk.sh" \
|
|
app-release-unsigned.apk app-release.apk \
|
|
"$APK_KEYSTORE" release-key APK_KEYSTORE_PASSWORD
|
|
```
|
|
|
|
Using it at all means this repo's job **is** handed an app signing key for
|
|
that run, which is exactly the thing "This repository is never handed an
|
|
app signing key" (above) warns against as the default. Treat a workflow
|
|
that includes this step as the documented exception, and prefer fixing the
|
|
build to emit a signed APK over reaching for it.
|
|
|
|
## One `versionCode` rule, shared by CI and a manual build (fdroid-repo-ci#6)
|
|
|
|
Two implementations of "the next version code" is the exact failure mode
|
|
`next-version-code.sh` exists to avoid, so both CI and a manual build call
|
|
this one script rather than each computing their own. The candidate is
|
|
derived from the **committer date of the commit being built**, not the
|
|
build clock -- deterministic per commit, rather than depending on when a
|
|
runner happened to pick the job up -- and clamped on **both** sides:
|
|
|
|
```
|
|
candidate = whole minutes since 2026-01-01T00:00:00Z, of the commit's committer date
|
|
candidate = min(candidate, minutes(now) + TOLERANCE) # forwards
|
|
published = highest versionCode already in the repo index for this applicationId
|
|
result = max(candidate, published + 1) # backwards
|
|
```
|
|
|
|
Minutes, not seconds or raw Unix time: `versionCode` is a signed 32-bit int,
|
|
and this scheme does not exhaust it for 4,082 years, against 67 years for
|
|
seconds and 11 for Unix time. Granularity is no longer what keeps
|
|
same-commit rebuilds from colliding, though -- that is a concurrency-group
|
|
plus refuse-on-existing-APK problem now (see below) -- so minutes stand
|
|
purely as a wide, cheap headroom choice, not a load-bearing part of the
|
|
concurrency story.
|
|
|
|
**Backwards.** Rebase, amend and cherry-pick can all produce a commit older
|
|
than one already published. The rule uses the commit's **committer** date,
|
|
not its author date, specifically because of this: author date survives a
|
|
rebase unchanged, so a rebased branch would keep re-emitting its original
|
|
(now stale) timestamp and the backwards clamp would never see it move.
|
|
Committer date changes on every rewrite, which is what lets `published + 1`
|
|
actually catch it.
|
|
|
|
**Forwards.** A commit's timestamp is attacker- and typo-controlled, and the
|
|
number this rule produces only ever rises -- so a bad one can never be
|
|
walked back down. A commit dated a century ahead burns decades of headroom
|
|
permanently; worse, a value placed just *below* the int32 ceiling is still
|
|
accepted by Android, still publishes, and then **no later build can ever
|
|
exceed it** -- the app becomes permanently un-updatable, recoverable only by
|
|
shipping a new `applicationId` and asking every user to reinstall by hand.
|
|
`TOLERANCE` only has to absorb honest clock skew between the committer's
|
|
machine and the runner -- a few hours is generous, a day would still be
|
|
harmless, since its only job is keeping the number near the present. A
|
|
commit that lands beyond it is **clamped and a warning is printed, never
|
|
rejected outright**: rejecting would block a legitimate build over a few
|
|
minutes of innocent skew, and the clamp already removes the actual harm (a
|
|
permanently reserved version-code range), so a hard failure would buy
|
|
nothing further.
|
|
|
|
**The clamp is monotonic against *observed* state, not monotonic by
|
|
construction.** It does not close the race where two jobs start in the same
|
|
minute, both read the same `published` value, both compute the same next
|
|
versionCode, and both build and publish -- the window is the whole build,
|
|
not just the clock read, since the version code has to be baked into the APK
|
|
before it exists. Nothing in `next-version-code.sh` can prevent that.
|
|
`publish.sh`'s APK-existence check (below) is the other half of this: it
|
|
cannot stop the wasted build, but it turns the collision from a silent,
|
|
permanent wrong state into a loud failure at the last moment, when the
|
|
information is finally reliable. Concurrent same-commit builds are now also
|
|
addressed at the workflow level with a concurrency group
|
|
(cancel-in-progress) -- measured working on this Gitea instance -- which is
|
|
an optimization on top of the refusal, not a replacement for it.
|
|
|
|
Takes an index-v2.json URL or local file path, an application id, and the
|
|
committer timestamp of the commit being built as Unix seconds (exactly what
|
|
`git log -1 --format=%ct` prints); prints the integer and nothing else, so a
|
|
caller does
|
|
`versionCode=$(next-version-code.sh "$URL" "$APPLICATION_ID" "$(git log -1 --format=%ct)")`.
|
|
The script takes the timestamp as an argument rather than shelling out to
|
|
git itself, which keeps it pure and testable -- the caller already knows
|
|
its own commit. The URL form needs no credentials, since both this repo and
|
|
the F-Droid repo it reads are public -- the point of weblib-viewer#16's
|
|
"buildable when CI is down". Fails loudly (non-zero exit, nothing on
|
|
stdout) if the index cannot be read; an application id with no entry at all
|
|
in the index is not an error -- that is simply the first-ever publish for a
|
|
channel, and yields the bare (clamped) candidate.
|
|
|
|
## Publish ordering and safety (fdroid-repo-ci#5, fdroid-repo-ci#6)
|
|
|
|
An F-Droid repo is a signed index plus the files it names. `publish.sh`
|
|
uploads in two passes, unconditionally: every non-index file first, the
|
|
index (`index-v1.json(.asc)`, `index-v2.json`, `entry.json`, `entry.jar`,
|
|
matched only at the top level of the tree) last. A publish interrupted
|
|
anywhere in the first pass leaves the repo describing the old state a little
|
|
longer; it can never leave an index naming files that 404. Credentials are
|
|
verified with a real authenticated call (`head-bucket`) before either pass,
|
|
rather than merely checking that the environment variables are non-empty --
|
|
an unusable-but-present credential failing halfway through is exactly how a
|
|
partial (and potentially index-first) upload happens.
|
|
|
|
Before pass 1, `publish.sh` also checks every local `*.apk`'s destination
|
|
key with `head-object` and refuses the whole publish if any of them already
|
|
exists -- the safeguard `next-version-code.sh`'s clamp cannot provide on its
|
|
own (see above). This does not prevent the wasted build behind a same-minute
|
|
collision; it prevents the collision from silently corrupting the repo.
|
|
Skipped entirely for a plain `--index-only` invocation (below), since that
|
|
pass can never upload a `*.apk` in the first place.
|
|
|
|
### `--files-only` and `--index-only`
|
|
|
|
The two passes can also be run as two separate invocations:
|
|
|
|
```bash
|
|
bash publish.sh --files-only "$FDROID_REPO_ROOT/repo" "$BUCKET" "$S3_ENDPOINT_URL"
|
|
# ... build finishes, or a later job resumes here ...
|
|
bash publish.sh --index-only "$FDROID_REPO_ROOT/repo" "$BUCKET" "$S3_ENDPOINT_URL"
|
|
```
|
|
|
|
With neither flag, both passes run in one call, files then index -- the
|
|
default, and the only way to reach "index before files" is to invoke
|
|
`--index-only` on its own, which is opt-in misuse, never the default. This
|
|
is real, independently useful behaviour, not just a testing seam: an
|
|
interrupted publish can resume at the index step without re-uploading every
|
|
APK, and a caller can stage files now and publish the index later.
|
|
|
|
It is also what makes the ordering guarantee testable **by construction**:
|
|
run `--files-only`, assert the bucket has the file and not the index yet,
|
|
run `--index-only`, assert the index then appears -- two genuinely separate
|
|
invocations, with the bucket inspected for real in between. An earlier
|
|
version of this script proved the same property with a same-process
|
|
synchronization hook that `eval`'d an environment variable between the two
|
|
passes; that was removed in favour of these flags; a script that runs with
|
|
a release signing key and S3 credentials in its environment, in a public
|
|
repository, should not carry a documented path from env-var influence to
|
|
arbitrary code execution, however unlikely, when a real feature closes the
|
|
same gap for free.
|
|
|
|
## Align before sign, and how "no `zipalign`" was verified
|
|
|
|
`zipalign` then `apksigner`. Signing first and aligning after is the old
|
|
`jarsigner`-era order and invalidates an `apksigner`-era (v2/v3) signature,
|
|
because that scheme signs the exact bytes of the packaged zip and alignment
|
|
padding changes those bytes.
|
|
|
|
**`apksigner` is a standalone nixpkgs attribute** (`nixpkgs#apksigner`,
|
|
verified at 35.0.6). **`zipalign` is not** -- it exists only inside
|
|
`androidenv`'s composed build-tools, which pulls a full versioned Android SDK
|
|
and its licence machinery for one binary. Before accepting that as a
|
|
dependency, three things were measured rather than assumed:
|
|
|
|
1. **Google's own docs** (developer.android.com/tools/zipalign): "If you
|
|
build using Android Studio, which uses the Android Gradle plugin (AGP),
|
|
this is done automatically. In this case, you should still use zipalign
|
|
to verify that the APK is aligned, but you don't need to align it." AGP is
|
|
what `flutter build apk` (or any Gradle-based Android build) drives
|
|
underneath, so this applies to any such build's output, not just
|
|
Android-Studio-initiated ones.
|
|
2. **`fdroidserver` 2.4.3's own source never calls `zipalign`, anywhere** --
|
|
grepped across all 38 of its modules. Its APK-signature-copying code
|
|
(`apksigcopier.py`, used for reproducible-build verification, a different
|
|
flow than binary-repo mode) implements realignment itself in pure Python
|
|
against the zip format directly, rather than shelling out to the
|
|
`zipalign` binary.
|
|
3. **`apksigner` does not check or enforce alignment at all**, in either
|
|
direction -- measured directly: a deliberately misaligned fixture zip
|
|
(including a `.so` misaligned to the stricter 4096-byte boundary) was
|
|
signed and then verified with `apksigner verify --verbose`, and the
|
|
output carried no warning about alignment whatsoever.
|
|
|
|
**Confidence: high** that a Gradle/AGP-built unsigned release APK is already
|
|
aligned, resting on (1) and (2) above. **Not independently confirmed**
|
|
against a real `flutter build apk` output run in this environment -- that
|
|
build was judged too expensive to attempt for this measurement. A sibling
|
|
repo's own CI proposal for building the Flutter app was rejected on the same
|
|
"the toolchain is too large to pull on every push" grounds this pin sidesteps
|
|
(composing a multi-platform Android SDK plus NDK pulls tens of gigabytes),
|
|
and the host available for this measurement had well under that in free
|
|
disk space.
|
|
|
|
Point (3) is why this repo does not simply trust point (1) and move on:
|
|
since nothing downstream complains about misalignment either, a build that
|
|
stopped emitting aligned output would fail silently -- an APK that signs and
|
|
verifies fine but behaves badly (or refuses to install) on a real device.
|
|
`check_zip_alignment.py`, a few lines against the standard library `zipfile`
|
|
module, checks every uncompressed entry of the APK `sign-apk.sh` is asked to
|
|
sign, and refuses to sign one that fails it. If that check ever legitimately
|
|
trips, the fix is to add `androidenv`'s `zipalign` as a dependency here, not
|
|
to weaken or remove the check.
|
|
|
|
## The repo index key is not the APK signing key
|
|
|
|
Binary-repo mode (every APK arrives already signed) means `fdroid update`
|
|
reads each APK's existing signature and never touches an app's own signing
|
|
key. It signs the **index** with a separate key of its own -- `fdroid-update.sh`
|
|
takes its own keystore/alias/password arguments, deliberately unrelated to
|
|
`sign-apk.sh`'s. Conflating the two would mean one compromised key
|
|
undermines both the apps and the index that vouches for them.
|
|
|
|
This is also, concretely, the boundary of what this repo ever holds: it
|
|
receives the **repo index key**, and nothing here is the *app* signing key
|
|
except through the narrow, opt-in `sign-apk.sh` exception documented above.
|
|
The index key identifies this repo to a client, which pins it via
|
|
`?fingerprint=` in the repo URL; the app key identifies each app to Android
|
|
and, unlike the index key, can never be rotated once a device has installed
|
|
against it.
|
|
|
|
As of 2026-09-09 (bit), the index key is *for now* the same keystore as the
|
|
APK key -- they are expected to be **split later**. That is exactly why
|
|
`fdroid-update.sh` and `sign-apk.sh` take independently-named keystore/alias/
|
|
password arguments rather than sharing one: a consumer's workflow should
|
|
already name the index key's secrets (e.g. `INDEX_KEYSTORE`,
|
|
`INDEX_KEYSTORE_PASSWORD`) distinctly from the APK key's (e.g.
|
|
`APK_KEYSTORE`, `APK_KEYSTORE_PASSWORD`), even while both names resolve to
|
|
the same underlying secret today. Splitting the keys later is then a value
|
|
change on the consumer's side -- a new keystore, a new secret -- not a
|
|
rename of any argument or config key here.
|
|
|
|
**Generate the index key as RSA 4096, never 2048.** F-Droid's own
|
|
documentation suggests 2048 for a repo signing key; that is not followed
|
|
here on purpose (bit, 2026-09-09) -- use 4096 regardless of what a `keytool
|
|
-genkeypair`/`fdroid init` example elsewhere shows. Recorded here so the
|
|
choice is not "corrected" back to F-Droid's default by someone going only
|
|
off those docs.
|
|
|
|
`fdroid-update.sh` writes `<repo-root>/config.yml` itself, on every run,
|
|
using fdroidserver's own `{env: VARNAME}` indirection for both passwords --
|
|
the exact mechanism its `common.py` implements *specifically so config.yml
|
|
can be safely git-managed*: "it is a good idea to manage config.yml (WITHOUT
|
|
PASSWORDS!) in git." A caller never has to know or hand-write fdroidserver's
|
|
config format. It also writes `repo_url`, `repo_name` and (if set)
|
|
`repo_description` from the environment variables of the same name, single-
|
|
quoted YAML scalars so an arbitrary value never needs the caller to think
|
|
about YAML escaping.
|
|
|
|
## Pinning nixpkgs
|
|
|
|
`fdroidserver`'s output feeds a *signed index*, so an unpinned or
|
|
channel-drifting nixpkgs is a worse trade here than for a disposable test
|
|
job. This repo carries a `nixpkgs-pin` file -- one line, a 40-character
|
|
nixpkgs revision -- and `with-nixpkgs.sh` refuses to run at all if that file
|
|
is missing or malformed, rather than silently falling back to an unpinned
|
|
`nix shell nixpkgs#foo`, which re-resolves the registry and fetches whatever
|
|
the channel currently points at.
|
|
|
|
This is deliberately **not** the same mechanism as [weblib-ci]'s
|
|
`with-nixpkgs.sh`, which reads the *consuming* repo's `flake.lock`. This repo
|
|
has no flake (see weblib-ci#9 for the same tension in that repo: a
|
|
`flake.lock` existing only to hold one revision string), and per this repo's
|
|
own originating issue, the repo that consumes it holds signing keys and is
|
|
not guaranteed to be a flake either -- reading a caller's `flake.lock` the
|
|
way weblib-ci's script does would not even find one here. So the pin travels
|
|
with *this* repo instead, in a file named `nixpkgs-pin` on purpose: it is the
|
|
exact fallback weblib-ci#9 proposes adding to the shared script (flake.lock
|
|
present -> today's behaviour there; else `nixpkgs-pin` -> a bare revision;
|
|
else fail loudly). If that lands, this repo already speaks its format and
|
|
nothing here needs to change.
|
|
|
|
A `nixpkgs-pin` file has no `nix flake update` -- it is bumped by hand.
|
|
Accepted here for the same reason weblib-ci#9 accepted it for a
|
|
stdlib-only test suite, generalised: the alternative (a flake that builds
|
|
nothing, just to hold a lockfile) is a lot of structure for one string, and a
|
|
genuinely broken pinned nixpkgs would be caught immediately by this repo's
|
|
own test suite.
|
|
|
|
`fdroidserver` is in nixpkgs as `fdroidserver`, verified at 2.4.3.
|
|
`publish.sh` uses `awscli2` for the same S3-compatible upload every consumer
|
|
here needs, resolved through the same pin.
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
bash run_tests.sh
|
|
```
|
|
|
|
123 assertions across eight suites offline (~20s on a warm nix store, 4
|
|
deliberately skipped -- see below), plus 13 more that only run against a
|
|
real bucket:
|
|
|
|
* `test_check_zip_alignment.py` -- pure standard library, no nix, no
|
|
network at all; builds an aligned and a misaligned fixture zip by hand and
|
|
checks both directions, including the 4096-byte boundary specific to `.so`
|
|
entries.
|
|
* `test_next_version_code.py` -- pure standard library too, testing
|
|
`next_version_code.py` as a subprocess against fixture index-v2.json files
|
|
written to a temp dir: a normal commit yields its own minute, a commit
|
|
older than the published maximum still yields published+1 (backwards --
|
|
rebase/amend/cherry-pick), a future-dated commit within tolerance passes
|
|
through untouched, one beyond tolerance is clamped to roughly now *and
|
|
warns* (forwards -- a bad clock or a malicious/typo'd date), a commit
|
|
dated near the int32 ceiling (year 2999) does not produce a near-int32
|
|
result, several published versions at once, a version entry with no
|
|
versionCode, an application id missing from the index entirely (not an
|
|
error), and an unreadable/malformed index (fails loudly, nothing on
|
|
stdout). A dedicated property test also checks, across every one of those
|
|
scenarios in one place, that the result always strictly exceeds every
|
|
versionCode already in the fixture -- the property that actually matters,
|
|
which a future refactor could break while each individual case still
|
|
looks right. Both the commit's committer date and "now" (used only for
|
|
the forwards clamp) are pinned independently, the latter via an
|
|
undocumented env var seam, so every expected number is fixed rather than
|
|
depending on whatever minute the suite happens to run in.
|
|
* `test_preserve_added.py` -- standard library too, running
|
|
`preserve_added.py` as a subprocess against synthetic index-v2.json
|
|
fixtures: staging puts the file at exactly `repo/index-v2.json` (the only
|
|
path `common.KnownApks` looks at), a missing/truncated/not-an-index
|
|
previous index is refused with the path named and nothing staged, and
|
|
verification passes when every shared build kept its date while **failing**
|
|
when one drifted -- the negative control, since a checker that only ever
|
|
passes is worth nothing. Also: a build that aged out of the retention
|
|
window is not drift, a leading `/` on `file.name` (present in the index,
|
|
stripped by `KnownApks`) must not make every build look new, a version
|
|
entry with no `added` at all is skipped rather than defaulted, and the
|
|
sub-second tolerance is exactly one second wide -- fdroidserver only ever
|
|
emits whole seconds, but a whole second of drift still fails.
|
|
`test_fdroid_update.sh` covers the same feature's shell-level contract
|
|
(`FDROID_PREVIOUS_INDEX` set but missing, or set to garbage, aborts before
|
|
`fdroid update` runs); the preservation *itself*, end to end against real
|
|
APKs, was measured against a copy of the live dev index rather than in the
|
|
suite, because this repo's fixture APK is deliberately not parseable by
|
|
androguard and so never reaches the index at all.
|
|
* `test_with_nixpkgs.sh` -- offline; every way the pinning wrapper is
|
|
supposed to refuse rather than silently degrade (missing pin, malformed
|
|
pin, too few arguments). Does not exercise the happy path of actually
|
|
resolving a `github:nixos/nixpkgs/...` reference automatically, since that
|
|
needs network (or an already-warm local cache) the same way weblib-ci's
|
|
own `with-nixpkgs.sh` does, and that repo does not automate a test for it
|
|
either. It was checked by hand instead.
|
|
* `test_sign_apk.sh` and `test_fdroid_update.sh` -- full end-to-end runs
|
|
against a throwaway keystore generated fresh per run, real `apksigner`
|
|
and `fdroid update`. Needs the pinned toolchain to be resolvable (locally
|
|
cached, or fetched once over the network); nothing beyond that.
|
|
* `test_next_version_code.sh` -- the real `next-version-code.sh` CLI, through
|
|
`with-nixpkgs.sh`: argument count (now three: index, application id,
|
|
committer timestamp), a local index file end to end, a first-ever publish
|
|
for an application id, a far-future committer date clamped with a warning
|
|
on stderr, a non-numeric committer timestamp refused rather than coerced,
|
|
a missing/malformed local file, and an unreachable URL
|
|
(`http://127.0.0.1:1`, the same closed-port trick `test_publish.sh` uses)
|
|
-- proving the wrapper's argv/exit-code/stdout contract, not just the
|
|
Python module's arithmetic.
|
|
* `test_publish.sh` -- argument/credential validation, `--dry-run` command
|
|
construction (both the default two-command form, files then index, and
|
|
each of `--files-only`/`--index-only` printing exactly one command, plus
|
|
their mutual exclusivity), and a real invocation against a closed local
|
|
port (`127.0.0.1:1`), fully offline. Also runs `test_s3_stub.py`, a
|
|
same-repo, HEAD-only local HTTP stub, to prove the credential check is a
|
|
real authenticated call (a wrong bucket is rejected), that an
|
|
already-published APK is refused before any upload, that `--index-only`
|
|
alone never runs that check at all (it cannot upload an APK), and a
|
|
negative control proving a non-colliding APK does not trip the check. The
|
|
actual upload path -- and the two-pass *ordering* of real uploads, which
|
|
nothing offline can see -- is exercised only by the opt-in live round trip
|
|
below, matching the constraint that only `publish.sh` needs network in the
|
|
first place.
|
|
|
|
`test_publish.sh`'s live section (skipped by default; needs
|
|
`FDROID_CI_LIVE_BUCKET`, `FDROID_CI_LIVE_ENDPOINT` and real credentials) adds
|
|
13 more assertions against a real bucket:
|
|
|
|
* the round trip itself, both objects reading back byte-identical;
|
|
* upload ordering proven **by construction**: `--files-only` run alone,
|
|
then the bucket inspected for real (file present, index absent), then
|
|
`--index-only` run alone, then the bucket inspected again (index now
|
|
present) -- two genuinely separate invocations, nothing racing, no
|
|
synchronization hook, no `eval` (see "`--files-only` and `--index-only`"
|
|
above for why that matters in a public repo);
|
|
* a fault-injection contrast proving the flags have real teeth: invoking
|
|
`--index-only` **alone**, against a prefix `--files-only` never touched,
|
|
really does publish an index naming a file that is not there -- the exact
|
|
state the whole ordering guarantee exists to prevent -- while the
|
|
*default* path (neither flag, one call) never leaves that state, which
|
|
holds by construction (the default's first phase is the identical command
|
|
array `--files-only` runs) and is checked here by its final state;
|
|
* a second publish refused when it would overwrite a real, already-published
|
|
APK, and that APK's bytes unchanged afterwards.
|
|
|
|
No `.gitea/workflows` runs this, on the same reasoning weblib-ci#9 recorded:
|
|
run it by hand before pushing. If that changes for weblib-ci it should
|
|
probably change here too, at the same time.
|
|
|
|
## Not hard-coded
|
|
|
|
No bucket name, repo URL, app id, keystore path, or channel name or count --
|
|
a consumer may have one publish target or several, under whatever names it
|
|
likes; nothing here assumes "release" and "dev", or that there are exactly
|
|
two. No consumer is named anywhere either: the repository that runs the
|
|
publish has already changed twice during design, so naming one would be
|
|
wrong again shortly.
|
|
|
|
The one exception is deliberate and unavoidable: **this repository's own
|
|
address** appears in the `uses:` examples above, because that is what a
|
|
consumer has to type. Those lines are what will need editing if this repo
|
|
moves organisation -- nothing else should.
|
|
|
|
[weblib-ci]: https://git.chaosbit.de/weblib/weblib-ci
|