weblib/fdroid-repo-ci
2
0
Fork 0
Code Issues Pull Requests Actions Activity

fdroid-update.sh: carry each build's "added" date across a rebuild

Browse Source 
weblib/weblib-viewer#95: every version in the published index claimed it
was added at the moment of the *last* publish, so an F-Droid client's
version list showed five separate builds all added within seconds of each
other. Measured on the live dev repo: index-v2.json gave versionCodes
365139/365516/365666/365710/366099 the added dates 05:50:06/08/10/12/13 on
2026-09-12, while the bucket's own Last-Modified for those objects says
they were actually published on 2026-09-11 at 13:53, 20:09, 22:37, 23:23
and 2026-09-12 05:50.

It is not an mtime problem, and --use-date-from-apk does not fix it.
fdroidserver takes the date from common.KnownApks, whose __init__ loads it
out of an already existing repo/index-v2.json (and archive/index-v2.json),
and whose recordapk falls back to datetime.now() for any filename it has
not seen; --use-date-from-apk only changes that fallback to the local
file's mtime, which for a re-downloaded APK is when it was downloaded.
KnownApks' own docstring says there is nowhere else to look: the index is
"currently the only way to permanently store the 'updated' date of APKs".
A CI job that rebuilds repo/ on a fresh runner -- which is exactly what
weblib-viewer#32's retention scheme does when it re-downloads the previous
APKs -- therefore starts with an empty KnownApks, and every build in it
counts as new.

So hand fdroid-update.sh the previously published index, optionally, as
FDROID_PREVIOUS_INDEX -- a local path, because fetching it needs to know
the bucket and per fdroid-repo-ci#1 that stays on the consumer's side;
this script still needs no network. preserve_added.py stages it where
KnownApks looks, and afterwards *checks* that every build present in both
indexes really did keep its date, failing the run if not. That is not
belt-and-braces: fdroidserver's own read_added_date_from_all_apks carries
a TODO about reading these dates differently, so a future version could
drop the behaviour this depends on, and a silent return to flattened
history is the failure mode that took a human noticing to catch the first
time.

Set-but-missing is a hard failure rather than a shrug, and so is a file
that is not a parseable index-v2.json (a truncated download, an S3
NoSuchKey error document): KnownApks has no error handling of its own and
would abort mid-library with a traceback that never names the file, and a
consumer whose fetch silently produced nothing would otherwise flatten its
whole history on every publish while looking healthy. A first-ever publish
expresses "there is no previous index" by leaving the variable unset.

Verified end to end without publishing anything, against the five real
APKs and a copy of the live dev index-v2.json pulled over HTTPS:

  - no previous index (today's behaviour), all five re-stamped:
    366099 06:23:19, 365710 06:23:18, 365666 06:23:17, 365516 06:23:15,
    365139 06:23:14 -- a five-second spread, all of it "now".
  - previous index minus 366099, i.e. the real "four retained plus one
    genuinely new" case: 365139/365516/365666/365710 keep 05:50:06/08/10/12
    exactly, 366099 gets 06:24:01, and lastUpdated follows the new build.
  - a second run on a fresh repo/ with the APK mtimes deliberately touched
    to now: all five unchanged, "5 already-published build(s) kept their
    original 'added' timestamp, 0 new".

test_preserve_added.py (12 assertions, stdlib only) covers the drift
detection in both directions, including the negative control that a
drifted date fails; test_fdroid_update.sh covers the shell contract.
run_tests.sh: 123 passed, 4 skipped, 0 failed.

A weblib-viewer change is needed too -- its workflows have to fetch last
publish's index and set FDROID_PREVIOUS_INDEX -- and is a separate PR in
that repo.

Co-authored-by: bit <bit@das-labor.org>
This commit is contained in:
claude
2026-09-12 06:29:20 +00:00
co-authored by bit
parent 760583b7e3
commit 2906f00564
7 changed files with 670 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test_preserve_added.py
Executable
+228
View File
@@ -0,0 +1,228 @@
#!/usr/bin/env python3
"""Offline tests for preserve_added.py (weblib/weblib-viewer#95).
Standard library only, no network, no nix -- like test_next_version_code.py
and test_check_zip_alignment.py, this is the piece of the change that needs
none of the pinned toolchain. `fdroid update` itself is exercised by
test_fdroid_update.sh, which needs fdroidserver; what is tested here is the
part that has to be right *before* fdroidserver is ever reached and the part
that judges what fdroidserver produced.
Run as a *subprocess*, same reasoning the neighbouring suites give: the
contract includes the exit code (fdroid-update.sh aborts a publish on a
non-zero `verify`) and the messages, not just the return value of a function.
The one property that matters, and the one a refactor could quietly break
while still looking correct: `verify` must FAIL when a build that exists in
both indexes changed its date. A checker that only ever passes is worth
nothing, so every "carried" assertion below is paired with a drift case.
"""
import json
import os
import subprocess
import sys
import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
SCRIPT = os.path.join(HERE, "preserve_added.py")
APP_ID = "de.example.app"
def index(entries, app_id=APP_ID):
"""Build a minimal index-v2.json shaped exactly like fdroidserver's.
`entries` is {versionCode: added_ms}. The map key under `versions` is a
sha256 in a real index; nothing here reads it, and using the versionCode
keeps the fixtures readable -- what both sides key on is `file.name`,
which is what `common.KnownApks` itself keys on.
"""
versions = {}
for code, added in entries.items():
versions["%064x" % code] = {
"added": added,
"file": {"name": "/%s_%d.apk" % (app_id, code), "size": 1},
"manifest": {"versionCode": code, "versionName": "1.0"},
}
return {
"repo": {"timestamp": 0, "address": "https://example.invalid/repo"},
"packages": {app_id: {"metadata": {"added": 0}, "versions": versions}},
}
class PreserveAddedTestCase(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
def write(self, name, obj):
path = os.path.join(self.tmp.name, name)
with open(path, "w", encoding="utf-8") as f:
if isinstance(obj, str):
f.write(obj)
else:
json.dump(obj, f)
return path
def run_script(self, *args):
return subprocess.run(
[sys.executable, SCRIPT, *args],
capture_output=True,
text=True,
)
# --- install ---------------------------------------------------------
def test_install_puts_the_index_where_knownapks_looks_for_it(self):
"""fdroidserver reads repo/index-v2.json and nothing else.
common.KnownApks.__init__ globs exactly os.path.join(part,
'index-v2.json') for part in ('repo', 'archive'), relative to the
repo root. A file staged anywhere else, or under any other name, is
invisible to it -- so the filename is the whole contract here.
"""
prev = self.write("prev.json", index({1: 1000, 2: 2000}))
repo = os.path.join(self.tmp.name, "repo")
os.mkdir(repo)
result = self.run_script("install", prev, repo)
self.assertEqual(result.returncode, 0, result.stderr)
staged = os.path.join(repo, "index-v2.json")
self.assertTrue(os.path.isfile(staged))
with open(staged, encoding="utf-8") as f:
self.assertEqual(json.load(f), index({1: 1000, 2: 2000}))
self.assertIn("2 known build(s)", result.stdout)
def test_install_refuses_a_missing_repo_directory(self):
prev = self.write("prev.json", index({1: 1000}))
result = self.run_script("install", prev, os.path.join(self.tmp.name, "nope"))
self.assertNotEqual(result.returncode, 0)
self.assertIn("no such repo directory", result.stderr)
def test_install_refuses_a_missing_previous_index(self):
repo = os.path.join(self.tmp.name, "repo")
os.mkdir(repo)
result = self.run_script(
"install", os.path.join(self.tmp.name, "gone.json"), repo
)
self.assertNotEqual(result.returncode, 0)
self.assertIn("no such previous index", result.stderr)
def test_install_refuses_a_truncated_index(self):
"""KnownApks has no error handling at all around json.load().
A half-written download would abort `fdroid update` from inside a
library, with a traceback that never names the file. Refuse it here,
naming the path, before the run starts.
"""
prev = self.write("prev.json", '{"packages": {"a":')
repo = os.path.join(self.tmp.name, "repo")
os.mkdir(repo)
result = self.run_script("install", prev, repo)
self.assertNotEqual(result.returncode, 0)
self.assertIn("not valid JSON", result.stderr)
self.assertFalse(os.path.exists(os.path.join(repo, "index-v2.json")))
def test_install_refuses_valid_json_that_is_not_an_index(self):
"""An S3 error document, or entry.json fetched by mistake, parses fine."""
for payload in ('{"error": "NoSuchKey"}', "[]", '{"packages": []}'):
with self.subTest(payload=payload):
prev = self.write("prev.json", payload)
repo = os.path.join(self.tmp.name, "repo")
os.makedirs(repo, exist_ok=True)
result = self.run_script("install", prev, repo)
self.assertNotEqual(result.returncode, 0)
self.assertIn("does not look like an index-v2.json", result.stderr)
# --- verify ----------------------------------------------------------
def test_verify_passes_when_every_shared_build_kept_its_date(self):
prev = self.write("prev.json", index({1: 1000, 2: 2000}))
new = self.write("new.json", index({1: 1000, 2: 2000, 3: 9000}))
result = self.run_script("verify", prev, new)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("2 already-published build(s) kept", result.stdout)
self.assertIn("1 new", result.stdout)
self.assertIn("_3.apk is new", result.stdout)
def test_verify_fails_when_a_shared_build_drifted(self):
"""The negative control: this is exactly weblib-viewer#95.
Both builds re-stamped with the current wall clock, which is what a
run with no previous index produces.
"""
prev = self.write("prev.json", index({1: 1000, 2: 2000}))
new = self.write("new.json", index({1: 1789194194000, 2: 1789194195000}))
result = self.run_script("verify", prev, new)
self.assertEqual(result.returncode, 1)
self.assertIn("lost their original 'added' timestamp", result.stderr)
self.assertIn("_1.apk was added at 1000", result.stderr)
self.assertIn("_2.apk was added at 2000", result.stderr)
def test_verify_ignores_a_build_that_aged_out_of_retention(self):
"""Retention deletes old APKs on purpose; that is not drift.
Only builds present in BOTH indexes are judged.
"""
prev = self.write("prev.json", index({1: 1000, 2: 2000, 3: 3000}))
new = self.write("new.json", index({3: 3000, 4: 4000}))
result = self.run_script("verify", prev, new)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("1 already-published build(s) kept", result.stdout)
def test_verify_accepts_a_first_publish_shaped_empty_index(self):
prev = self.write("prev.json", {"packages": {}})
new = self.write("new.json", index({1: 1000}))
result = self.run_script("verify", prev, new)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("0 already-published build(s) kept", result.stdout)
def test_verify_tolerates_sub_second_rounding_only(self):
"""fdroidserver stores ms but only ever emits whole seconds.
index.py serialises a datetime as
`int(calendar.timegm(obj.timetuple()) * 1000)` and KnownApks reads it
back as `version["added"] // 1000`, so a hand-made previous index
carrying a sub-second value comes back rounded through no fault of
the mechanism. That must not fail a publish -- but one whole second
of drift still must.
"""
prev = self.write("prev.json", index({1: 1789194194678}))
result = self.run_script(
"verify", prev, self.write("new.json", index({1: 1789194194000}))
)
self.assertEqual(result.returncode, 0, result.stderr)
result = self.run_script(
"verify", prev, self.write("new2.json", index({1: 1789194195000}))
)
self.assertEqual(result.returncode, 1)
def test_verify_ignores_a_version_entry_with_no_added(self):
"""A missing date is not a date; do not default it to anything."""
broken = index({1: 1000})
del list(broken["packages"][APP_ID]["versions"].values())[0]["added"]
prev = self.write("prev.json", broken)
new = self.write("new.json", index({1: 5000}))
result = self.run_script("verify", prev, new)
self.assertEqual(result.returncode, 0, result.stderr)
def test_leading_slash_on_file_name_is_stripped_consistently(self):
"""index-v2 writes "/name.apk"; KnownApks strips the slash ([1:]).
Both sides of verify have to agree, or every build looks new.
"""
prev_obj = index({1: 1000})
for v in prev_obj["packages"][APP_ID]["versions"].values():
v["file"]["name"] = v["file"]["name"].lstrip("/") # no leading slash
prev = self.write("prev.json", prev_obj)
new = self.write("new.json", index({1: 1000})) # with leading slash
result = self.run_script("verify", prev, new)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("1 already-published build(s) kept", result.stdout)
self.assertNotIn("is new", result.stdout)
if __name__ == "__main__":
unittest.main()