publish.sh: verify credentials and upload the index last; add next-version-code.sh

Closes #5 Closes #6 An F-Droid repo is a signed index plus the files it names, and a partial upload could previously leave them disagreeing: aws s3 sync walks the tree in its own order, so an interrupted publish could land the index before the APKs it names, breaking the repo for every client until the upload finished or was rolled back. publish.sh now uploads in two passes, unconditionally -- every non-index file first, the index (index-v1.json(.asc), index-v2.json, entry.json, entry.jar) last -- so an interruption anywhere in pass 1 simply leaves the repo describing the old state a little longer, never a mixture. Credentials are also verified with one real authenticated call (head-bucket) before either pass, rather than merely checking the environment variables are non-empty, since an unusable-but-present credential failing halfway through is exactly how a partial upload happens. next-version-code.sh is the one implementation of the versionCode rule (candidate = minutes since 2026-01-01Z, clamped to strictly exceed whatever is already published for that applicationId), shared by CI and manual builds so there is exactly one place this can drift. bit pointed out (appended to #6) that the clamp is monotonic against *observed* state, not monotonic by construction: two jobs starting in the same minute can both read the same published value and race to publish the same versionCode on different APKs. Nothing can prevent that wasted build, so publish.sh now also refuses to overwrite an already-existing APK object before pass 1 -- turning the collision from a silent, permanent wrong state into a loud failure at the last moment the information is reliable. Tested offline (83 assertions: pure-stdlib suites for both scripts, the real CLIs through with-nixpkgs.sh, and a small local HEAD-only S3 stub proving the credential check is a real authenticated call and the APK-collision refusal fires only when it should) and live against fdroid-weblib-dev (7 more: a real publish observed mid-flight via a same-process synchronization hook to prove the index is genuinely absent while the file is already present, both objects reading back byte-identical, and a second publish refused against a real already-published APK with its bytes left unchanged). Co-authored-by: bit <bit@das-labor.org>
2026-09-09 08:00:45 +00:00
parent fcc6a741ef
commit 19e55c4a53
10 changed files with 1045 additions and 64 deletions
@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+"""Offline tests for next_version_code.py (fdroid-repo-ci#6).
+
+Standard library only, no network, no nix -- like test_check_zip_alignment.py,
+this is the piece of the change that needs none of the pinned toolchain.
+
+Runs next_version_code.py as a *subprocess* against fixture index-v2.json
+files written to a temp dir, the same reasoning test_check_zip_alignment.py
+gives: a test that only imports the function would miss an argv/exit-code/
+stdout-pollution regression, and "prints the result; do nothing else" is
+itself part of the contract (README/issue #6: a caller does
+`versionCode=$(next-version-code.sh ...)`, so stray stdout output would be
+silently captured as part of the number).
+
+"now" is pinned via the NEXT_VERSION_CODE_NOW testing seam (see the script's
+main()) so `candidate` is a fixed, known integer instead of whatever minute
+the suite happens to run in.
+
+The property every case after the first two actually defends -- and the one
+a future refactor could quietly break while still "looking" correct -- is
+that the result is *strictly greater than every versionCode already in the
+fixture*, not merely equal to some formula. Each clamp case asserts that
+directly, not just the arithmetic.
+"""
+import datetime
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import unittest
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+SCRIPT = os.path.join(HERE, "next_version_code.py")
+APP_ID = "de.example.app"
+
+EPOCH = datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc)
+# An arbitrary, fixed "now": 100 minutes after the epoch this script uses,
+# so the expected candidate (100) is known without re-deriving the formula
+# under test.
+FIXED_NOW = (EPOCH + datetime.timedelta(minutes=100)).isoformat()
+CANDIDATE_AT_FIXED_NOW = 100
+
+
+def index_with_versions(application_id, version_codes):
+    """Build an index-v2.json fixture with one package with each versionCode
+    in a separate version entry, matching fdroidserver's own shape:
+    packages.<id>.versions.<sha256>.manifest.versionCode."""
+    versions = {}
+    for i, vc in enumerate(version_codes):
+        versions[f"fakehash{i}"] = {"manifest": {"versionCode": vc}}
+    return {"packages": {application_id: {"versions": versions}}}
+
+
+class NextVersionCodeTest(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+
+    def _write_index(self, name, obj_or_text):
+        path = os.path.join(self.tmp.name, name)
+        with open(path, "w") as f:
+            if isinstance(obj_or_text, str):
+                f.write(obj_or_text)
+            else:
+                json.dump(obj_or_text, f)
+        return path
+
+    def _run(self, index_path, application_id=APP_ID, now=FIXED_NOW):
+        env = dict(os.environ)
+        if now is not None:
+            env["NEXT_VERSION_CODE_NOW"] = now
+        else:
+            env.pop("NEXT_VERSION_CODE_NOW", None)
+        return subprocess.run(
+            [sys.executable, SCRIPT, index_path, application_id],
+            capture_output=True, text=True, env=env,
+        )
+
+    # -- no prior build: the first-ever publish for a channel -------------
+
+    def test_no_prior_build_yields_bare_candidate(self):
+        path = self._write_index("empty.json", {"packages": {}})
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(result.stdout, f"{CANDIDATE_AT_FIXED_NOW}\n")
+
+    def test_application_id_missing_entirely_is_not_an_error(self):
+        path = self._write_index(
+            "other-app.json", index_with_versions("some.other.app", [5]))
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(result.stdout, f"{CANDIDATE_AT_FIXED_NOW}\n")
+
+    # -- the clamp: result must exceed every published code ----------------
+
+    def test_lower_prior_is_ignored_in_favour_of_the_clock(self):
+        path = self._write_index("lower.json", index_with_versions(APP_ID, [1, 2, 3]))
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        got = int(result.stdout)
+        self.assertEqual(got, CANDIDATE_AT_FIXED_NOW)
+        self.assertGreater(got, 3)
+
+    def test_equal_prior_is_bumped_by_one(self):
+        # published == candidate: the same-minute collision case. The clamp
+        # must still move strictly past it.
+        path = self._write_index(
+            "equal.json", index_with_versions(APP_ID, [CANDIDATE_AT_FIXED_NOW]))
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        got = int(result.stdout)
+        self.assertGreater(got, CANDIDATE_AT_FIXED_NOW)
+        self.assertEqual(got, CANDIDATE_AT_FIXED_NOW + 1)
+
+    def test_higher_prior_wins_the_clamp_clock_went_backwards(self):
+        higher = CANDIDATE_AT_FIXED_NOW + 500
+        path = self._write_index("higher.json", index_with_versions(APP_ID, [higher]))
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        got = int(result.stdout)
+        self.assertGreater(got, higher)
+        self.assertEqual(got, higher + 1)
+
+    def test_highest_of_several_versions_is_the_one_that_matters(self):
+        path = self._write_index(
+            "several.json", index_with_versions(APP_ID, [10, 999, 50]))
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        got = int(result.stdout)
+        self.assertGreater(got, 999)
+
+    def test_version_entry_without_a_versionCode_is_ignored_not_fatal(self):
+        # A version whose manifest lacks versionCode (or has no manifest at
+        # all) must not crash the read -- it is simply not a candidate for
+        # "published".
+        obj = index_with_versions(APP_ID, [7])
+        obj["packages"][APP_ID]["versions"]["nomanifest"] = {}
+        obj["packages"][APP_ID]["versions"]["emptymanifest"] = {"manifest": {}}
+        path = self._write_index("partial.json", obj)
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        got = int(result.stdout)
+        self.assertGreater(got, 7)
+
+    # -- failure modes: fail loudly, never guess ---------------------------
+
+    def test_unreadable_index_fails_loudly(self):
+        missing = os.path.join(self.tmp.name, "does-not-exist.json")
+        result = self._run(missing)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertEqual(result.stdout, "")  # never a guessed number on stdout
+        self.assertIn("could not read the index", result.stderr)
+
+    def test_malformed_index_fails_loudly(self):
+        path = self._write_index("malformed.json", "{not valid json")
+        result = self._run(path)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertEqual(result.stdout, "")
+        self.assertIn("could not read the index", result.stderr)
+
+    # -- output contract: nothing but the integer on stdout -----------------
+
+    def test_stdout_is_only_the_integer(self):
+        path = self._write_index("clean.json", {"packages": {}})
+        result = self._run(path)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        # Exactly one line, and it parses as an int with no surrounding text.
+        self.assertEqual(int(result.stdout.strip()), CANDIDATE_AT_FIXED_NOW)
+        self.assertEqual(result.stdout.count("\n"), 1)
+
+
+if __name__ == "__main__":
+    unittest.main()