fdroid-update.sh: carry each build's "added" date across a rebuild

weblib/weblib-viewer#95: every version in the published index claimed it
was added at the moment of the *last* publish, so an F-Droid client's
version list showed five separate builds all added within seconds of each
other. Measured on the live dev repo: index-v2.json gave versionCodes
365139/365516/365666/365710/366099 the added dates 05:50:06/08/10/12/13 on
2026-09-12, while the bucket's own Last-Modified for those objects says
they were actually published on 2026-09-11 at 13:53, 20:09, 22:37, 23:23
and 2026-09-12 05:50.

It is not an mtime problem, and --use-date-from-apk does not fix it.
fdroidserver takes the date from common.KnownApks, whose __init__ loads it
out of an already existing repo/index-v2.json (and archive/index-v2.json),
and whose recordapk falls back to datetime.now() for any filename it has
not seen; --use-date-from-apk only changes that fallback to the local
file's mtime, which for a re-downloaded APK is when it was downloaded.
KnownApks' own docstring says there is nowhere else to look: the index is
"currently the only way to permanently store the 'updated' date of APKs".
A CI job that rebuilds repo/ on a fresh runner -- which is exactly what
weblib-viewer#32's retention scheme does when it re-downloads the previous
APKs -- therefore starts with an empty KnownApks, and every build in it
counts as new.

So hand fdroid-update.sh the previously published index, optionally, as
FDROID_PREVIOUS_INDEX -- a local path, because fetching it needs to know
the bucket and per fdroid-repo-ci#1 that stays on the consumer's side;
this script still needs no network. preserve_added.py stages it where
KnownApks looks, and afterwards *checks* that every build present in both
indexes really did keep its date, failing the run if not. That is not
belt-and-braces: fdroidserver's own read_added_date_from_all_apks carries
a TODO about reading these dates differently, so a future version could
drop the behaviour this depends on, and a silent return to flattened
history is the failure mode that took a human noticing to catch the first
time.

Set-but-missing is a hard failure rather than a shrug, and so is a file
that is not a parseable index-v2.json (a truncated download, an S3
NoSuchKey error document): KnownApks has no error handling of its own and
would abort mid-library with a traceback that never names the file, and a
consumer whose fetch silently produced nothing would otherwise flatten its
whole history on every publish while looking healthy. A first-ever publish
expresses "there is no previous index" by leaving the variable unset.

Verified end to end without publishing anything, against the five real
APKs and a copy of the live dev index-v2.json pulled over HTTPS:

  - no previous index (today's behaviour), all five re-stamped:
    366099 06:23:19, 365710 06:23:18, 365666 06:23:17, 365516 06:23:15,
    365139 06:23:14 -- a five-second spread, all of it "now".
  - previous index minus 366099, i.e. the real "four retained plus one
    genuinely new" case: 365139/365516/365666/365710 keep 05:50:06/08/10/12
    exactly, 366099 gets 06:24:01, and lastUpdated follows the new build.
  - a second run on a fresh repo/ with the APK mtimes deliberately touched
    to now: all five unchanged, "5 already-published build(s) kept their
    original 'added' timestamp, 0 new".

test_preserve_added.py (12 assertions, stdlib only) covers the drift
detection in both directions, including the negative control that a
drifted date fails; test_fdroid_update.sh covers the shell contract.
run_tests.sh: 123 passed, 4 skipped, 0 failed.

A weblib-viewer change is needed too -- its workflows have to fetch last
publish's index and set FDROID_PREVIOUS_INDEX -- and is a separate PR in
that repo.

Co-authored-by: bit <bit@das-labor.org>

fdroid-update.sh

@@ -45,7 +45,36 @@
 # argument -- or the value embedded here (verbatim, as repo.address in
 # index-v2.json) will not be where the index actually is.
 #
+# FDROID_PREVIOUS_INDEX (optional) is a **local path** to the
+# `index-v2.json` this repo published last time. Give it and every build
+# that is still present keeps the date it was first published; omit it and
+# `fdroid update` stamps *every* APK in repo/ with the current wall clock,
+# which is weblib/weblib-viewer#95 -- a version list where five builds all
+# claim to have been added within seconds of each other.
+#
+# The dates live in the published index and nowhere else. fdroidserver's
+# `common.KnownApks` loads them from `repo/index-v2.json` (and
+# `archive/index-v2.json`) relative to the repo root, and falls back to
+# `datetime.now()` for any APK filename it has not seen -- its own docstring
+# calls the index "currently the only way to permanently store the 'updated'
+# date of APKs". A CI job that rebuilds repo/ on a fresh runner, as a
+# retention scheme re-downloading the previous APKs does, therefore starts
+# with an empty KnownApks. `preserve_added.py` (next to this script, and
+# where the full citation lives) puts the previous index back before the
+# run and checks afterwards that the dates really did survive; a build that
+# is genuinely new is untouched and still gets "now". Nothing here invents
+# or writes a date.
+#
+# A path that is set but does not exist is a hard failure, not a shrug: a
+# consumer whose download silently produced nothing would otherwise flatten
+# its whole history on every publish and look fine doing it. A genuine
+# first-ever publish has no previous index, and expresses that by leaving
+# the variable unset.
+#
 # Needs no network (fdroid-repo-ci#1) -- it only reads repo/ and signs.
+# FDROID_PREVIOUS_INDEX is a local file for that reason: fetching last
+# publish's index is the caller's job, since only the caller knows the
+# bucket (and, per fdroid-repo-ci#1, must remain the only one that does).
 # Pins fdroidserver, the JDK (fdroidserver shells out to jarsigner/keytool
 # for the index signature) and apksigner (for the per-entry v1 signature in
 # newer index formats) via with-nixpkgs.sh/nixpkgs-pin, next to this script.
@@ -59,7 +88,11 @@ usage: fdroid-update.sh <repo-root> <index-keystore> <key-alias> \
                          <keystore-pass-env> [<key-pass-env>] [extra fdroid update args...]
 
 required environment variables: FDROID_REPO_URL, FDROID_REPO_NAME
-optional environment variable:  FDROID_REPO_DESCRIPTION
+optional environment variables: FDROID_REPO_DESCRIPTION,
+                                FDROID_PREVIOUS_INDEX (local path to the
+                                previously published index-v2.json, so
+                                already-published builds keep the date they
+                                were first added)
 USAGE
   exit 2
 fi
@@ -90,6 +123,13 @@ extra_args=("$@")
 : "${FDROID_REPO_URL:?fdroid-update.sh: environment variable FDROID_REPO_URL (canonical repo URL, no trailing slash) is not set}"
 : "${FDROID_REPO_NAME:?fdroid-update.sh: environment variable FDROID_REPO_NAME (repo display name) is not set}"
 
+previous_index=${FDROID_PREVIOUS_INDEX:-}
+if [ -n "$previous_index" ] && [ ! -f "$previous_index" ]; then
+  echo "fdroid-update.sh: FDROID_PREVIOUS_INDEX is set to '$previous_index', which does not exist." >&2
+  echo "fdroid-update.sh: leave it unset for a first-ever publish; do not point it at a file that was never fetched." >&2
+  exit 2
+fi
+
 # YAML single-quoted scalar: doubling an embedded single quote is the only
 # escape it needs, and (unlike double-quoted) it does not interpret
 # backslashes -- so this is safe for an arbitrary URL/name/description
@@ -124,16 +164,44 @@ EOF
 } > "$root/config.yml"
 chmod 600 "$root/config.yml"
 
+# Stash the previous index somewhere `fdroid update` cannot reach. install
+# copies it *into* repo/, where the run then overwrites it -- so the "before"
+# side of the verification has to be read from a copy taken beforehand. It
+# also makes FDROID_PREVIOUS_INDEX=<root>/repo/index-v2.json (a caller
+# re-running in place) work rather than being a copy onto itself.
+previous_stash=
+if [ -n "$previous_index" ]; then
+  previous_stash=$(mktemp "${TMPDIR:-/tmp}/fdroid-previous-index.XXXXXX")
+  trap 'rm -f "$previous_stash"' EXIT
+  cp "$previous_index" "$previous_stash"
+fi
+
 echo "fdroid-update.sh: running fdroid update in $root"
 # fdroidserver locates a JDK by globbing standard system install locations
 # (or $JAVA_HOME) -- a nix store path matches neither, so JAVA_HOME has to be
 # set explicitly from whichever jarsigner this nix shell actually put on
 # PATH, rather than assumed.
-bash "$here/with-nixpkgs.sh" "fdroidserver jdk21 apksigner" bash -c '
+#
+# python3 is on the list for preserve_added.py: fdroidserver is a python
+# application but its nix output only exposes `fdroid` on PATH, not the
+# interpreter it runs on.
+bash "$here/with-nixpkgs.sh" "fdroidserver jdk21 apksigner python3" bash -c '
   set -euo pipefail
   export JAVA_HOME="$(dirname "$(dirname "$(command -v jarsigner)")")"
+  tools=$1; prev=$2; shift 2
   cd "$1"; shift
-  exec fdroid update "$@"
-' bash "$root" "${extra_args[@]}"
+  if [ -n "$prev" ]; then
+    python3 "$tools/preserve_added.py" install "$prev" repo
+  fi
+  fdroid update "$@"
+  # After, not instead of: install only makes the dates *available* to
+  # fdroidserver. This measures that they actually came out the other side,
+  # so a future fdroidserver that stops consulting the old index fails the
+  # publish here rather than silently flattening the history again
+  # (weblib/weblib-viewer#95).
+  if [ -n "$prev" ]; then
+    python3 "$tools/preserve_added.py" verify "$prev" repo/index-v2.json
+  fi
+' bash "$here" "$previous_stash" "$root" "${extra_args[@]}"
 
 echo "fdroid-update.sh: done; index at $root/repo/index-v2.json"