weblib/fdroid-repo-ci
2
0
Fork 0
Code Issues Pull Requests Actions Activity

fdroid-update.sh: carry each build's "added" date across a rebuild

Browse Source 
weblib/weblib-viewer#95: every version in the published index claimed it
was added at the moment of the *last* publish, so an F-Droid client's
version list showed five separate builds all added within seconds of each
other. Measured on the live dev repo: index-v2.json gave versionCodes
365139/365516/365666/365710/366099 the added dates 05:50:06/08/10/12/13 on
2026-09-12, while the bucket's own Last-Modified for those objects says
they were actually published on 2026-09-11 at 13:53, 20:09, 22:37, 23:23
and 2026-09-12 05:50.

It is not an mtime problem, and --use-date-from-apk does not fix it.
fdroidserver takes the date from common.KnownApks, whose __init__ loads it
out of an already existing repo/index-v2.json (and archive/index-v2.json),
and whose recordapk falls back to datetime.now() for any filename it has
not seen; --use-date-from-apk only changes that fallback to the local
file's mtime, which for a re-downloaded APK is when it was downloaded.
KnownApks' own docstring says there is nowhere else to look: the index is
"currently the only way to permanently store the 'updated' date of APKs".
A CI job that rebuilds repo/ on a fresh runner -- which is exactly what
weblib-viewer#32's retention scheme does when it re-downloads the previous
APKs -- therefore starts with an empty KnownApks, and every build in it
counts as new.

So hand fdroid-update.sh the previously published index, optionally, as
FDROID_PREVIOUS_INDEX -- a local path, because fetching it needs to know
the bucket and per fdroid-repo-ci#1 that stays on the consumer's side;
this script still needs no network. preserve_added.py stages it where
KnownApks looks, and afterwards *checks* that every build present in both
indexes really did keep its date, failing the run if not. That is not
belt-and-braces: fdroidserver's own read_added_date_from_all_apks carries
a TODO about reading these dates differently, so a future version could
drop the behaviour this depends on, and a silent return to flattened
history is the failure mode that took a human noticing to catch the first
time.

Set-but-missing is a hard failure rather than a shrug, and so is a file
that is not a parseable index-v2.json (a truncated download, an S3
NoSuchKey error document): KnownApks has no error handling of its own and
would abort mid-library with a traceback that never names the file, and a
consumer whose fetch silently produced nothing would otherwise flatten its
whole history on every publish while looking healthy. A first-ever publish
expresses "there is no previous index" by leaving the variable unset.

Verified end to end without publishing anything, against the five real
APKs and a copy of the live dev index-v2.json pulled over HTTPS:

  - no previous index (today's behaviour), all five re-stamped:
    366099 06:23:19, 365710 06:23:18, 365666 06:23:17, 365516 06:23:15,
    365139 06:23:14 -- a five-second spread, all of it "now".
  - previous index minus 366099, i.e. the real "four retained plus one
    genuinely new" case: 365139/365516/365666/365710 keep 05:50:06/08/10/12
    exactly, 366099 gets 06:24:01, and lastUpdated follows the new build.
  - a second run on a fresh repo/ with the APK mtimes deliberately touched
    to now: all five unchanged, "5 already-published build(s) kept their
    original 'added' timestamp, 0 new".

test_preserve_added.py (12 assertions, stdlib only) covers the drift
detection in both directions, including the negative control that a
drifted date fails; test_fdroid_update.sh covers the shell contract.
run_tests.sh: 123 passed, 4 skipped, 0 failed.

A weblib-viewer change is needed too -- its workflows have to fetch last
publish's index and set FDROID_PREVIOUS_INDEX -- and is a separate PR in
that repo.

Co-authored-by: bit <bit@das-labor.org>
This commit is contained in:
claude
2026-09-12 06:29:20 +00:00
co-authored by bit
parent 760583b7e3
commit 2906f00564
7 changed files with 670 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
preserve_added.py
Executable
+208
View File
@@ -0,0 +1,208 @@
#!/usr/bin/env python3
"""Carry an F-Droid repo's per-build "added" timestamps across a rebuild.
weblib/weblib-viewer#95: every version in the published index claimed it was
added at the moment of the *last* publish, so a client's version list showed
five builds all "added" within seconds of each other.
## Why that happens (fdroidserver 2.4.3, verified in the source)
`fdroid update` does not read an APK's mtime, and does not invent the date
either. It asks `common.KnownApks`, whose `__init__` loads the dates out of
an *already existing* index:
for part in ('repo', 'archive'):
path = os.path.join(part, 'index-v2.json')
if os.path.isfile(path):
...
filename = version["file"]["name"][1:]
date = datetime.fromtimestamp(version["added"] // 1000, tz=timezone.utc)
self.apks[filename] = date
and whose `recordapk` falls back to the wall clock for anything it has not
seen before:
if apkName not in self.apks:
if default_date is None:
default_date = datetime.now(timezone.utc)
self.apks[apkName] = default_date
return self.apks[apkName]
(`default_date` is only non-None under `--use-date-from-apk`, which uses the
local file's mtime -- no better here, since a re-downloaded file's mtime is
the time it was downloaded.)
So the per-build `added` date is stored *in the published index and nowhere
else* -- `KnownApks`' own docstring says as much: "This is currently the only
way to permanently store the 'updated' date of APKs."
A CI job that rebuilds `repo/` from scratch on a fresh runner -- which is
exactly what a retention scheme re-downloading the previous APKs does --
starts with no `repo/index-v2.json`, so `KnownApks` starts empty and *every*
build, old ones included, gets `datetime.now()`. Hence "all apps show the
date of the last update".
## The fix
Put last publish's `index-v2.json` back into `repo/` before `fdroid update`
runs, and let fdroidserver's own mechanism do the rest. Nothing here writes
a date itself; `install` stages the previous index and `verify` afterwards
*measures* that the dates actually survived, so a future fdroidserver that
drops this behaviour (its own `read_added_date_from_all_apks` carries a TODO
about reading the dates differently) fails the publish loudly instead of
silently flattening the history again.
preserve_added.py install <previous-index-v2.json> <repo-dir>
preserve_added.py verify <previous-index-v2.json> <new-index-v2.json>
Builds are keyed on the index's own `file.name`, because that is the key
`KnownApks` itself uses -- not the versionCode, which is one level deeper in
the JSON and is not what fdroidserver matches on.
"""
import argparse
import json
import os
import shutil
import sys
def load_index(path, what):
"""Parse an index-v2.json, rejecting anything KnownApks would choke on.
KnownApks indexes straight into `index["packages"]` and into each
version's `file.name`/`added` with no guards at all, so a truncated or
wrong-shaped file does not degrade -- it aborts `fdroid update` from
inside a library, with a traceback that says nothing about where the
file came from. Checking the shape here costs nothing and turns that
into one sentence naming the actual path.
"""
try:
with open(path, "r", encoding="utf-8") as f:
index = json.load(f)
except FileNotFoundError:
sys.exit(f"preserve_added.py: no such {what}: {path}")
except json.JSONDecodeError as e:
sys.exit(f"preserve_added.py: {what} {path} is not valid JSON: {e}")
if not isinstance(index, dict) or not isinstance(index.get("packages"), dict):
sys.exit(
f"preserve_added.py: {what} {path} has no top-level 'packages' object "
"-- it does not look like an index-v2.json"
)
return index
def added_by_file(index):
"""Map {apk file name -> added (ms)} exactly as KnownApks keys it.
Entries with no `added`, or no `file.name`, are skipped rather than
defaulted: a missing date is not a date, and inventing one here would be
the very bug this script exists to stop.
"""
out = {}
for pkg in index["packages"].values():
if not isinstance(pkg, dict):
continue
for version in (pkg.get("versions") or {}).values():
if not isinstance(version, dict):
continue
name = (version.get("file") or {}).get("name")
added = version.get("added")
if isinstance(name, str) and isinstance(added, int):
out[name.lstrip("/")] = added
return out
def cmd_install(args):
previous = load_index(args.previous_index, "previous index")
dates = added_by_file(previous)
if not os.path.isdir(args.repo_dir):
sys.exit(f"preserve_added.py: no such repo directory: {args.repo_dir}")
target = os.path.join(args.repo_dir, "index-v2.json")
shutil.copyfile(args.previous_index, target)
print(
f"preserve_added.py: staged {args.previous_index} at {target}; "
f"fdroid update will carry the 'added' date of {len(dates)} known build(s)"
)
return 0
def cmd_verify(args):
previous = added_by_file(load_index(args.previous_index, "previous index"))
new = added_by_file(load_index(args.new_index, "new index"))
carried, drifted = [], []
for name, was in previous.items():
if name not in new:
# Aged out of the retention window, or otherwise no longer
# published. Not this script's business.
continue
# Whole seconds, deliberately. fdroidserver stores `added` in
# milliseconds but only ever *produces* whole seconds
# (index.py: `int(calendar.timegm(obj.timetuple()) * 1000)`), and
# KnownApks reads it back as `version["added"] // 1000`. A
# hand-made previous index carrying a sub-second value would
# therefore come back rounded, through no fault of the mechanism
# being checked -- failing a publish over that would be a false
# alarm, while every real regression here is seconds to hours.
if new[name] // 1000 == was // 1000:
carried.append(name)
else:
drifted.append((name, was, new[name]))
for name, was, now in sorted(drifted):
print(
f"preserve_added.py: {name} was added at {was}, "
f"the new index says {now}",
file=sys.stderr,
)
fresh = sorted(set(new) - set(previous))
for name in fresh:
print(f"preserve_added.py: {name} is new in this index (added {new[name]})")
if drifted:
print(
f"preserve_added.py: {len(drifted)} already-published build(s) lost their "
"original 'added' timestamp (weblib/weblib-viewer#95)",
file=sys.stderr,
)
return 1
print(
f"preserve_added.py: {len(carried)} already-published build(s) kept their "
f"original 'added' timestamp, {len(fresh)} new"
)
return 0
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = parser.add_subparsers(dest="command", required=True)
p_install = sub.add_parser(
"install",
help="stage a previously published index-v2.json in repo/ so "
"`fdroid update` reuses its per-build 'added' dates",
)
p_install.add_argument("previous_index")
p_install.add_argument("repo_dir")
p_install.set_defaults(func=cmd_install)
p_verify = sub.add_parser(
"verify",
help="fail if any build present in both indexes changed its "
"'added' timestamp",
)
p_verify.add_argument("previous_index")
p_verify.add_argument("new_index")
p_verify.set_defaults(func=cmd_verify)
args = parser.parse_args(argv)
return args.func(args)
if __name__ == "__main__":
sys.exit(main())