fdroid-update.sh: carry each build's "added" date across a rebuild #15

2026-09-12T06:33:17Z

claude commented

2026-09-12 06:33:17 +00:00

The symptom

Every version in the published index claimed it was added at the moment of the last publish. Measured on the live dev repo, index-v2.json against the bucket's own Last-Modified for the very same objects:

versionCode	index said `added`	actually published (S3 `Last-Modified`)
366099	2026-09-12 05:50:13	2026-09-12 05:50:25
365710	2026-09-12 05:50:12	2026-09-11 23:23:13
365666	2026-09-12 05:50:10	2026-09-11 22:37:25
365516	2026-09-12 05:50:08	2026-09-11 20:09:32
365139	2026-09-12 05:50:06	2026-09-11 13:53:19

Sixteen hours of history collapsed into seven seconds.

The mechanism

It is not an mtime problem, and --use-date-from-apk does not fix it. fdroidserver takes the date from common.KnownApks, whose __init__ loads it out of an already existing index:

for part in ('repo', 'archive'):
    path = os.path.join(part, 'index-v2.json')
    if os.path.isfile(path):
        ...
        filename = version["file"]["name"][1:]
        date = datetime.fromtimestamp(version["added"] // 1000, tz=timezone.utc)
        self.apks[filename] = date

and whose recordapk falls back to the wall clock for anything it has not seen:

if apkName not in self.apks:
    if default_date is None:
        default_date = datetime.now(timezone.utc)
    self.apks[apkName] = default_date
return self.apks[apkName]

(fdroidserver 2.4.3, common.py:2771-2810; update.py:2026 is the caller, update.py:2596 the --use-date-from-apk flag, which only swaps now() for the local file's mtime -- no better for a file that was just re-downloaded.)

KnownApks' own docstring says why there is nowhere else to look: the index is "currently the only way to permanently store the 'updated' date of APKs".

weblib-viewer's jobs rebuild REPO_ROOT/repo from scratch on a fresh runner, so there is no previous repo/index-v2.json for KnownApks to load -- and every APK re-downloaded by weblib-viewer#32's retention scheme counts as new. Hence "all apps show the date of the last update".

The fix

Optional FDROID_PREVIOUS_INDEX: a local path to the previously published index-v2.json. Fetching it needs to know the bucket, and per #1 that stays on the consumer's side, so this script still needs no network.

preserve_added.py stages it where KnownApks looks and, after the run, checks that every build present in both indexes really kept its date, failing the run if not. That check is not decoration: fdroidserver's own read_added_date_from_all_apks carries a TODO about reading these dates differently, so a future version could drop the behaviour this depends on -- and "the history quietly flattened again" is a failure mode that took a human noticing to catch the first time.

Refused loudly rather than shrugged at: a path that is set but missing, and a file that is not a parseable index-v2.json (truncated download, S3 NoSuchKey error document). KnownApks has no error handling of its own and would abort mid-library with a traceback that never names the file; and a consumer whose fetch silently produced nothing would otherwise flatten its entire history on every publish while looking perfectly healthy. A first-ever publish says "there is no previous index" by leaving the variable unset.

Verified end to end, without publishing anything

Against the five real APKs and a copy of the live dev index-v2.json, both pulled over HTTPS from fdroid-weblib-dev (read-only). Nothing was written to any bucket.

Before -- no previous index, i.e. today's behaviour:

versionCode	`added`
366099	2026-09-12 06:23:19
365710	2026-09-12 06:23:18
365666	2026-09-12 06:23:17
365516	2026-09-12 06:23:15
365139	2026-09-12 06:23:14

All five re-stamped with the run's clock: the bug, reproduced.

After -- previous index = the live one minus 366099, i.e. the real "four retained, one genuinely new" case:

versionCode	`added`
366099	2026-09-12 06:24:01	new build, correctly "now"
365710	2026-09-12 05:50:12	unchanged
365666	2026-09-12 05:50:10	unchanged
365516	2026-09-12 05:50:08	unchanged
365139	2026-09-12 05:50:06	unchanged

app-level: added=1789192206000 lastUpdated=1789194241000 -- lastUpdated still tracks the real last update.

Across two runs -- a second run in a fresh repo/, APK mtimes deliberately touched to now, previous index = the first run's output:

preserve_added.py: staged ...; fdroid update will carry the 'added' date of 5 known build(s)
preserve_added.py: 5 already-published build(s) kept their original 'added' timestamp, 0 new

All five byte-identical to the table above, including 366099 keeping 06:24:01 rather than being re-stamped. The touch matters: it proves the fix does not secretly depend on mtimes.

Tests

test_preserve_added.py, 12 assertions, stdlib only -- including the negative control that a drifted date fails verification (a checker that only ever passes is worth nothing), a build aged out of retention not counting as drift, the leading / on file.name that KnownApks strips, and a sub-second tolerance exactly one second wide.

test_fdroid_update.sh gains 7: set-but-missing refused, truncated JSON refused, an S3 error document refused, and a valid previous index staged, surviving the run, and verified afterwards.

bash run_tests.sh: passed=123 skipped=4 failed=0.

Also needed

weblib-viewer#97 -- the workflows have to fetch last publish's index and set FDROID_PREVIOUS_INDEX. Separate repo, separate PR, as it should be.

Fixes weblib/weblib-viewer#95 ## The symptom Every version in the published index claimed it was added at the moment of the **last** publish. Measured on the live dev repo, `index-v2.json` against the bucket's own `Last-Modified` for the very same objects: | versionCode | index said `added` | actually published (S3 `Last-Modified`) | |---|---|---| | 366099 | 2026-09-12 05:50:13 | 2026-09-12 05:50:25 | | 365710 | 2026-09-12 05:50:12 | 2026-09-11 23:23:13 | | 365666 | 2026-09-12 05:50:10 | 2026-09-11 22:37:25 | | 365516 | 2026-09-12 05:50:08 | 2026-09-11 20:09:32 | | 365139 | 2026-09-12 05:50:06 | 2026-09-11 13:53:19 | Sixteen hours of history collapsed into seven seconds. ## The mechanism It is **not** an mtime problem, and `--use-date-from-apk` does not fix it. fdroidserver takes the date from `common.KnownApks`, whose `__init__` loads it out of an *already existing* index: ```python for part in ('repo', 'archive'): path = os.path.join(part, 'index-v2.json') if os.path.isfile(path): ... filename = version["file"]["name"][1:] date = datetime.fromtimestamp(version["added"] // 1000, tz=timezone.utc) self.apks[filename] = date ``` and whose `recordapk` falls back to the wall clock for anything it has not seen: ```python if apkName not in self.apks: if default_date is None: default_date = datetime.now(timezone.utc) self.apks[apkName] = default_date return self.apks[apkName] ``` (fdroidserver 2.4.3, `common.py:2771-2810`; `update.py:2026` is the caller, `update.py:2596` the `--use-date-from-apk` flag, which only swaps `now()` for the local file's mtime -- no better for a file that was just re-downloaded.) `KnownApks`' own docstring says why there is nowhere else to look: the index is *"currently the only way to permanently store the 'updated' date of APKs"*. weblib-viewer's jobs rebuild `REPO_ROOT/repo` from scratch on a fresh runner, so there is no previous `repo/index-v2.json` for `KnownApks` to load -- and every APK re-downloaded by weblib-viewer#32's retention scheme counts as new. Hence "all apps show the date of the last update". ## The fix Optional `FDROID_PREVIOUS_INDEX`: a **local path** to the previously published `index-v2.json`. Fetching it needs to know the bucket, and per #1 that stays on the consumer's side, so this script still needs no network. `preserve_added.py` stages it where `KnownApks` looks and, after the run, **checks** that every build present in both indexes really kept its date, failing the run if not. That check is not decoration: fdroidserver's own `read_added_date_from_all_apks` carries a TODO about reading these dates differently, so a future version could drop the behaviour this depends on -- and "the history quietly flattened again" is a failure mode that took a human noticing to catch the first time. Refused loudly rather than shrugged at: a path that is **set but missing**, and a file that is not a parseable `index-v2.json` (truncated download, S3 `NoSuchKey` error document). `KnownApks` has no error handling of its own and would abort mid-library with a traceback that never names the file; and a consumer whose fetch silently produced nothing would otherwise flatten its entire history on every publish while looking perfectly healthy. A first-ever publish says "there is no previous index" by leaving the variable unset. ## Verified end to end, without publishing anything Against the five real APKs and a copy of the live dev `index-v2.json`, both pulled over HTTPS from `fdroid-weblib-dev` (read-only). **Nothing was written to any bucket.** **Before** -- no previous index, i.e. today's behaviour: | versionCode | `added` | |---|---| | 366099 | 2026-09-12 06:23:19 | | 365710 | 2026-09-12 06:23:18 | | 365666 | 2026-09-12 06:23:17 | | 365516 | 2026-09-12 06:23:15 | | 365139 | 2026-09-12 06:23:14 | All five re-stamped with the run's clock: the bug, reproduced. **After** -- previous index = the live one minus 366099, i.e. the real "four retained, one genuinely new" case: | versionCode | `added` | | |---|---|---| | 366099 | 2026-09-12 06:24:01 | new build, correctly "now" | | 365710 | 2026-09-12 05:50:12 | unchanged | | 365666 | 2026-09-12 05:50:10 | unchanged | | 365516 | 2026-09-12 05:50:08 | unchanged | | 365139 | 2026-09-12 05:50:06 | unchanged | `app-level: added=1789192206000 lastUpdated=1789194241000` -- `lastUpdated` still tracks the real last update. **Across two runs** -- a second run in a *fresh* `repo/`, APK mtimes deliberately `touch`ed to now, previous index = the first run's output: ``` preserve_added.py: staged ...; fdroid update will carry the 'added' date of 5 known build(s) preserve_added.py: 5 already-published build(s) kept their original 'added' timestamp, 0 new ``` All five byte-identical to the table above, including 366099 keeping `06:24:01` rather than being re-stamped. The `touch` matters: it proves the fix does not secretly depend on mtimes. ## Tests `test_preserve_added.py`, 12 assertions, stdlib only -- including the negative control that a drifted date **fails** verification (a checker that only ever passes is worth nothing), a build aged out of retention not counting as drift, the leading `/` on `file.name` that `KnownApks` strips, and a sub-second tolerance exactly one second wide. `test_fdroid_update.sh` gains 7: set-but-missing refused, truncated JSON refused, an S3 error document refused, and a valid previous index staged, surviving the run, and verified afterwards. `bash run_tests.sh`: **passed=123 skipped=4 failed=0.** ## Also needed weblib-viewer#97 -- the workflows have to fetch last publish's index and set `FDROID_PREVIOUS_INDEX`. Separate repo, separate PR, as it should be.

claude added 1 commit 2026-09-12 06:33:18 +00:00

fdroid-update.sh: carry each build's "added" date across a rebuild 2906f00564

weblib/weblib-viewer#95: every version in the published index claimed it
was added at the moment of the *last* publish, so an F-Droid client's
version list showed five separate builds all added within seconds of each
other. Measured on the live dev repo: index-v2.json gave versionCodes
365139/365516/365666/365710/366099 the added dates 05:50:06/08/10/12/13 on
2026-09-12, while the bucket's own Last-Modified for those objects says
they were actually published on 2026-09-11 at 13:53, 20:09, 22:37, 23:23
and 2026-09-12 05:50.

It is not an mtime problem, and --use-date-from-apk does not fix it.
fdroidserver takes the date from common.KnownApks, whose __init__ loads it
out of an already existing repo/index-v2.json (and archive/index-v2.json),
and whose recordapk falls back to datetime.now() for any filename it has
not seen; --use-date-from-apk only changes that fallback to the local
file's mtime, which for a re-downloaded APK is when it was downloaded.
KnownApks' own docstring says there is nowhere else to look: the index is
"currently the only way to permanently store the 'updated' date of APKs".
A CI job that rebuilds repo/ on a fresh runner -- which is exactly what
weblib-viewer#32's retention scheme does when it re-downloads the previous
APKs -- therefore starts with an empty KnownApks, and every build in it
counts as new.

So hand fdroid-update.sh the previously published index, optionally, as
FDROID_PREVIOUS_INDEX -- a local path, because fetching it needs to know
the bucket and per fdroid-repo-ci#1 that stays on the consumer's side;
this script still needs no network. preserve_added.py stages it where
KnownApks looks, and afterwards *checks* that every build present in both
indexes really did keep its date, failing the run if not. That is not
belt-and-braces: fdroidserver's own read_added_date_from_all_apks carries
a TODO about reading these dates differently, so a future version could
drop the behaviour this depends on, and a silent return to flattened
history is the failure mode that took a human noticing to catch the first
time.

Set-but-missing is a hard failure rather than a shrug, and so is a file
that is not a parseable index-v2.json (a truncated download, an S3
NoSuchKey error document): KnownApks has no error handling of its own and
would abort mid-library with a traceback that never names the file, and a
consumer whose fetch silently produced nothing would otherwise flatten its
whole history on every publish while looking healthy. A first-ever publish
expresses "there is no previous index" by leaving the variable unset.

Verified end to end without publishing anything, against the five real
APKs and a copy of the live dev index-v2.json pulled over HTTPS:

  - no previous index (today's behaviour), all five re-stamped:
    366099 06:23:19, 365710 06:23:18, 365666 06:23:17, 365516 06:23:15,
    365139 06:23:14 -- a five-second spread, all of it "now".
  - previous index minus 366099, i.e. the real "four retained plus one
    genuinely new" case: 365139/365516/365666/365710 keep 05:50:06/08/10/12
    exactly, 366099 gets 06:24:01, and lastUpdated follows the new build.
  - a second run on a fresh repo/ with the APK mtimes deliberately touched
    to now: all five unchanged, "5 already-published build(s) kept their
    original 'added' timestamp, 0 new".

test_preserve_added.py (12 assertions, stdlib only) covers the drift
detection in both directions, including the negative control that a
drifted date fails; test_fdroid_update.sh covers the shell contract.
run_tests.sh: 123 passed, 4 skipped, 0 failed.

A weblib-viewer change is needed too -- its workflows have to fetch last
publish's index and set FDROID_PREVIOUS_INDEX -- and is a separate PR in
that repo.

Co-authored-by: bit <bit@das-labor.org>

bit scheduled this pull request to auto merge when all checks succeed 2026-09-12 06:43:27 +00:00

bit merged commit 0db9e3d3d4 into main

2026-09-12 06:43:30 +00:00

bit deleted branch fix/95-preserve-added-timestamps

2026-09-12 06:43:30 +00:00

bit referenced this issue from a commit

2026-09-12 06:43:31 +00:00

Merge pull request 'fdroid-update.sh: carry each build's "added" date across a rebuild' (#15) from fix/95-preserve-added-timestamps into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: weblib/fdroid-repo-ci#15