weblib/fdroid-repo-ci
2
0
Fork 0
Code Issues Pull Requests Actions Activity

publish.sh: replace the test-only eval hook with --files-only/--index-only

Browse Source 
bit's review on PR #7 (comment on the PR, appended to this reasoning):
run_test_hook() eval'd an environment variable in publish.sh, which runs
with the release signing key and S3 credentials in its environment, in a
public repo. "Never set by a real caller" was true, but it was still a
documented, discoverable path from any env-var influence to arbitrary code
execution at the worst possible moment -- unnecessary, which is the
stronger reason to remove it than any claim about likelihood.

The replacement is a feature rather than a seam: --files-only and
--index-only split the two passes into separately invokable steps. With
neither flag the default is unchanged (both passes, files first). This
makes the ordering guarantee testable by construction -- run --files-only,
assert the bucket state, run --index-only, assert again -- with no hook, no
eval, and no test-only branch in production code. It is also independently
useful: an interrupted publish can resume at the index step without
re-uploading every APK, and a caller can stage files now and publish the
index later. The APK-existence check is skipped entirely for a plain
--index-only invocation, since that pass can never upload a *.apk.

Re-tested end to end: offline 91 passed / 4 skipped / 0 failed (up from 83/
3/0, mostly new flag-construction and stub assertions); live against
fdroid-weblib-dev 104 passed / 0 skipped / 0 failed, bucket verified empty
before and after. Added a live fault-injection test that --index-only alone
against a prefix --files-only never touched really does publish an index
naming an absent file -- the exact state the ordering guarantee exists to
prevent -- contrasted with the default path, which never reaches it.
Offline, removing the run_files/run_index gating (simulating the flags
being ignored) was confirmed to fail 5 of the new assertions before being
reverted.

Left the versionCode granularity as minutes, per instruction: that is being
settled separately on weblib-viewer#15 and is a one-constant change for
whenever it lands, not something to pre-empt here.

Co-authored-by: bit <bit@das-labor.org>
This commit is contained in:
claude
2026-09-09 08:17:18 +00:00
co-authored by bit
parent 19e55c4a53
commit ccb8cb616e
3 changed files with 346 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+60 -17
View File
@@ -148,6 +148,37 @@ key with `head-object` and refuses the whole publish if any of them already
exists -- the safeguard `next-version-code.sh`'s clamp cannot provide on its
own (see above). This does not prevent the wasted build behind a same-minute
collision; it prevents the collision from silently corrupting the repo.
Skipped entirely for a plain `--index-only` invocation (below), since that
pass can never upload a `*.apk` in the first place.
### `--files-only` and `--index-only`
The two passes can also be run as two separate invocations:
```bash
bash publish.sh --files-only "$FDROID_REPO_ROOT/repo" "$BUCKET" "$S3_ENDPOINT_URL"
# ... build finishes, or a later job resumes here ...
bash publish.sh --index-only "$FDROID_REPO_ROOT/repo" "$BUCKET" "$S3_ENDPOINT_URL"
```
With neither flag, both passes run in one call, files then index -- the
default, and the only way to reach "index before files" is to invoke
`--index-only` on its own, which is opt-in misuse, never the default. This
is real, independently useful behaviour, not just a testing seam: an
interrupted publish can resume at the index step without re-uploading every
APK, and a caller can stage files now and publish the index later.
It is also what makes the ordering guarantee testable **by construction**:
run `--files-only`, assert the bucket has the file and not the index yet,
run `--index-only`, assert the index then appears -- two genuinely separate
invocations, with the bucket inspected for real in between. An earlier
version of this script proved the same property with a same-process
synchronization hook that `eval`'d an environment variable between the two
passes; that was removed in favour of these flags; a script that runs with
a release signing key and S3 credentials in its environment, in a public
repository, should not carry a documented path from env-var influence to
arbitrary code execution, however unlikely, when a real feature closes the
same gap for free.
## Align before sign, and how "no `zipalign`" was verified
@@ -257,8 +288,9 @@ here needs, resolved through the same pin.
bash run_tests.sh
```
83 assertions across seven suites offline (~20s on a warm nix store), plus 5
more that only run against a real bucket (see below):
91 assertions across seven suites offline (~20s on a warm nix store, 4
deliberately skipped -- see below), plus 13 more that only run against a
real bucket:
* `test_check_zip_alignment.py` -- pure standard library, no nix, no
network at all; builds an aligned and a misaligned fixture zip by hand and
@@ -290,13 +322,15 @@ more that only run against a real bucket (see below):
`test_publish.sh` uses) -- proving the wrapper's argv/exit-code/stdout
contract, not just the Python module's arithmetic.
* `test_publish.sh` -- argument/credential validation, `--dry-run` command
construction (now two commands, files then index, and asserted in that
order), and a real invocation against a closed local port
(`127.0.0.1:1`), fully offline. Also runs `test_s3_stub.py`, a
construction (both the default two-command form, files then index, and
each of `--files-only`/`--index-only` printing exactly one command, plus
their mutual exclusivity), and a real invocation against a closed local
port (`127.0.0.1:1`), fully offline. Also runs `test_s3_stub.py`, a
same-repo, HEAD-only local HTTP stub, to prove the credential check is a
real authenticated call (a wrong bucket is rejected) and that an
already-published APK is refused before any upload, with a negative
control proving a non-colliding APK does not trip the same check. The
real authenticated call (a wrong bucket is rejected), that an
already-published APK is refused before any upload, that `--index-only`
alone never runs that check at all (it cannot upload an APK), and a
negative control proving a non-colliding APK does not trip the check. The
actual upload path -- and the two-pass *ordering* of real uploads, which
nothing offline can see -- is exercised only by the opt-in live round trip
below, matching the constraint that only `publish.sh` needs network in the
@@ -304,15 +338,24 @@ more that only run against a real bucket (see below):
`test_publish.sh`'s live section (skipped by default; needs
`FDROID_CI_LIVE_BUCKET`, `FDROID_CI_LIVE_ENDPOINT` and real credentials) adds
5 more assertions against a real bucket: the round trip itself; the index
object provably absent while the file object is already present at the one
point in the middle of a real publish that the ordering guarantee depends on
(observed via a same-process synchronization hook `publish.sh` exposes only
for this, `PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES` -- deliberately not a poll
racing a background upload from outside, since that is exactly the
flakiness fdroid-repo-ci#5 warned against); both objects reading back
byte-identical; a second publish refused when it would overwrite a
real, already-published APK; and that APK's bytes unchanged afterwards.
13 more assertions against a real bucket:
* the round trip itself, both objects reading back byte-identical;
* upload ordering proven **by construction**: `--files-only` run alone,
then the bucket inspected for real (file present, index absent), then
`--index-only` run alone, then the bucket inspected again (index now
present) -- two genuinely separate invocations, nothing racing, no
synchronization hook, no `eval` (see "`--files-only` and `--index-only`"
above for why that matters in a public repo);
* a fault-injection contrast proving the flags have real teeth: invoking
`--index-only` **alone**, against a prefix `--files-only` never touched,
really does publish an index naming a file that is not there -- the exact
state the whole ordering guarantee exists to prevent -- while the
*default* path (neither flag, one call) never leaves that state, which
holds by construction (the default's first phase is the identical command
array `--files-only` runs) and is checked here by its final state;
* a second publish refused when it would overwrite a real, already-published
APK, and that APK's bytes unchanged afterwards.
No `.gitea/workflows` runs this, on the same reasoning weblib-ci#9 recorded:
run it by hand before pushing. If that changes for weblib-ci it should
publish.sh
+90 -47
View File
@@ -39,6 +39,23 @@
# it is only ever reached by a client that already has last publish's index
# and is willing to look for it).
#
# The two passes are also separately invokable, with `--files-only` and
# `--index-only`. This is not a test seam -- it is real, useful on its own:
# an interrupted publish can resume at the index step without re-uploading
# every APK, and a caller can stage files now and publish the index later.
# It is also what makes the ordering guarantee testable *by construction*:
# run `--files-only`, assert the bucket has the files and not the index, run
# `--index-only`, assert the index appears -- no synchronization hook and no
# `eval` of anything env-supplied needed in a script that runs with a
# signing key and S3 credentials in its environment, in a public repo. (An
# earlier version of this script had exactly such a hook; it was removed
# once these flags made it unnecessary.) With neither flag, both passes run,
# files then index, which is the safe default and is unconditional -- there
# is no way to select "index then files" short of invoking `--index-only`
# by hand before ever running `--files-only`, which is the deliberate
# misuse the two flags exist to make possible for cases that want it
# (resuming, staging) while never being the default.
#
# ## Credentials are verified before anything is uploaded (fdroid-repo-ci#5)
#
# Checking that AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are merely *set* does
@@ -94,40 +111,64 @@
# `fdroid update` needs no network; this is the one script in this repo that
# does (fdroid-repo-ci#1), and is accordingly the one script whose happy path
# the test suite cannot exercise against a real bucket. `--dry-run` prints
# the exact two commands this script would run, in order -- including the
# `with-nixpkgs.sh` pin -- and exits without touching the network at all: no
# credential check, no APK-existence check, no sync. That is what the test
# suite asserts against for command construction and ordering. (A
# PATH-stubbed `aws` cannot do this job: `nix shell` puts the packages it was
# asked for at the *front* of PATH, ahead of anything already there, so a
# stub earlier on the caller's PATH would simply be shadowed by the real
# `aws` from awscli2.) The credential check, the APK-existence check and the
# two-pass ordering are each proven separately: a real invocation against a
# closed local port for "fails on the network, and fails during the
# credential check specifically, before any upload is attempted"; a tiny
# local HEAD-only HTTP stub for "an existing APK is refused, and only an
# existing APK", offline; and the opt-in live round trip against a real
# bucket for the actual upload path plus a real head-to-head APK collision.
# the command(s) this script would run for whichever pass(es) are selected,
# in order -- including the `with-nixpkgs.sh` pin -- and exits without
# touching the network at all: no credential check, no APK-existence check,
# no sync. That is what the test suite asserts against for command
# construction and ordering. (A PATH-stubbed `aws` cannot do this job: `nix
# shell` puts the packages it was asked for at the *front* of PATH, ahead of
# anything already there, so a stub earlier on the caller's PATH would
# simply be shadowed by the real `aws` from awscli2.) The credential check,
# the APK-existence check and the two-pass ordering are each proven
# separately: a real invocation against a closed local port for "fails on
# the network, and fails during the credential check specifically, before
# any upload is attempted"; a tiny local HEAD-only HTTP stub for "an
# existing APK is refused, and only an existing APK", offline; and the
# opt-in live round trip against a real bucket, run once as `--files-only`
# then `--index-only` to observe the ordering by construction, and once
# more as a real head-to-head APK collision.
#
# Pins awscli2 via with-nixpkgs.sh/nixpkgs-pin, next to this script.
set -euo pipefail
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
if [ "$#" -lt 3 ]; then
usage() {
cat >&2 <<'USAGE'
usage: publish.sh [--dry-run] <local-repo-dir> <bucket> <endpoint-url> [<prefix>] [extra aws s3 sync args...]
usage: publish.sh [--dry-run] [--files-only | --index-only] \
<local-repo-dir> <bucket> <endpoint-url> [<prefix>] [extra aws s3 sync args...]
With neither --files-only nor --index-only, both passes run: every
non-index file first, then the index -- the safe default, and the only way
to select "index before files" is to invoke --index-only by hand ahead of
--files-only, never the default. --files-only and --index-only are mutually
exclusive.
Credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and optionally
AWS_SESSION_TOKEN) must already be exported -- never passed as arguments.
USAGE
}
dry_run=false
files_only=false
index_only=false
while [ "$#" -gt 0 ]; do
case "$1" in
--dry-run) dry_run=true; shift ;;
--files-only) files_only=true; shift ;;
--index-only) index_only=true; shift ;;
*) break ;;
esac
done
if [ "$files_only" = true ] && [ "$index_only" = true ]; then
echo "publish.sh: --files-only and --index-only are mutually exclusive" >&2
exit 2
fi
dry_run=false
if [ "$1" = "--dry-run" ]; then
dry_run=true
shift
if [ "$#" -lt 3 ]; then
usage
exit 2
fi
local_dir=$1
@@ -183,11 +224,23 @@ cmd_files=("${aws_base[@]}" s3 sync "$local_dir" "$dest"
cmd_index=("${aws_base[@]}" s3 sync "$local_dir" "$dest"
"${index_only_filters[@]}" "${extra_args[@]}")
# Which pass(es) this invocation runs. Unconditional and exactly one of
# three shapes: both (the default), files alone, or index alone -- the
# mutual-exclusivity check above already ruled out both flags together.
run_files=true
run_index=true
[ "$index_only" = true ] && run_files=false
[ "$files_only" = true ] && run_index=false
if [ "$dry_run" = true ]; then
if [ "$run_files" = true ]; then
printf '%q ' "${cmd_files[@]}"
printf '\n'
fi
if [ "$run_index" = true ]; then
printf '%q ' "${cmd_index[@]}"
printf '\n'
fi
exit 0
fi
@@ -198,11 +251,15 @@ if ! verify_out=$("${aws_base[@]}" s3api head-bucket --bucket "$bucket" 2>&1); t
exit 1
fi
# Refuse to overwrite an existing APK -- see the comment block above for why
# this exists and what it does and does not close. Every local *.apk is
# checked, regardless of nesting, since fdroidserver's binary-repo layout
# keeps them under repo/ directly but nothing here should assume that.
while IFS= read -r -d '' apk; do
if [ "$run_files" = true ]; then
# Refuse to overwrite an existing APK -- see the comment block above for
# why this exists and what it does and does not close. Every local *.apk
# is checked, regardless of nesting, since fdroidserver's binary-repo
# layout keeps them under repo/ directly but nothing here should assume
# that. Skipped entirely for a plain --index-only invocation: the index
# pass never uploads a *.apk (index_only_filters excludes everything but
# the index/entry names), so there is nothing this check could catch.
while IFS= read -r -d '' apk; do
rel=${apk#"$local_dir"/}
key="$rel"
if [ -n "$prefix" ]; then
@@ -218,29 +275,15 @@ while IFS= read -r -d '' apk; do
echo "$head_out" >&2
exit 1
fi
done < <(find "$local_dir" -type f -name '*.apk' -print0)
done < <(find "$local_dir" -type f -name '*.apk' -print0)
# Test-only synchronization point: if set, run it after the APK-existence
# check and before pass 1, and again between the two passes. Lets the test
# suite observe/assert real bucket state at the one moment that actually
# matters for the ordering guarantee (fdroid-repo-ci#5) -- files present,
# index not yet -- without racing a real background upload from outside.
# Never set by a real caller; not part of the documented interface above.
run_test_hook() {
local hook_var=$1
local hook_value="${!hook_var:-}"
[ -n "$hook_value" ] && eval "$hook_value"
return 0
}
echo "publish.sh: pass 1/2: uploading files (index deferred) $local_dir -> $dest"
"${cmd_files[@]}"
fi
run_test_hook PUBLISH_SH_TEST_HOOK_BEFORE_UPLOAD
echo "publish.sh: pass 1/2: uploading files (index deferred) $local_dir -> $dest"
"${cmd_files[@]}"
run_test_hook PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES
echo "publish.sh: pass 2/2: uploading the index $local_dir -> $dest"
"${cmd_index[@]}"
if [ "$run_index" = true ]; then
echo "publish.sh: pass 2/2: uploading the index $local_dir -> $dest"
"${cmd_index[@]}"
fi
echo "publish.sh: done"
test_publish.sh
+174 -53
View File
@@ -5,8 +5,9 @@
# a genuine bucket is out of scope for most of this file. What this asserts:
#
# * argument/credential validation, offline;
# * `--dry-run` builds the exact two commands (files, then index) a real run
# would use, in that order, offline;
# * `--dry-run` builds the exact command(s) a real run would use for
# whichever of --files-only/--index-only (or neither -> both, files then
# index) is selected, offline;
# * a real invocation against a closed local port fails on the network call
# itself -- during the credential check specifically, before any upload is
# attempted -- proving both that the constructed command is well-formed
@@ -14,12 +15,19 @@
# * a real invocation against a tiny local HEAD-only S3 stub (test_s3_stub.py)
# proves the credential check is a genuine authenticated call (a wrong
# bucket is rejected, not waved through because a variable was merely
# non-empty) and that an already-published APK is refused before any
# upload -- and, as a negative control, that a *non*-colliding APK does
# NOT trip the same check (fdroid-repo-ci#6);
# non-empty), that an already-published APK is refused before any upload
# (and that --index-only alone never runs that check at all, since it
# cannot upload an APK) -- and, as a negative control, that a
# *non*-colliding APK does NOT trip the same check (fdroid-repo-ci#6);
# * the opt-in live round trip against a real bucket, extended to prove the
# two-pass ordering with real uploads (not just command construction) and
# the APK-collision refusal against a real object.
# two-pass ordering **by construction**: `--files-only` then `--index-only`
# as two genuinely separate invocations, with the bucket state asserted in
# between -- no synchronization hook, no `eval` of anything in this public
# repo's production script -- plus the APK-collision refusal against a
# real object, and a deliberate-misuse case proving `--index-only` alone
# really does produce the exact state the ordering guarantee exists to
# prevent (an index naming a file that is not there), which the *default*
# path never reaches.
set -euo pipefail
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
@@ -102,6 +110,33 @@ echo "--- --dry-run with no prefix omits the trailing slash entirely ---"
out_noprefix=$(bash "$here/publish.sh" --dry-run "$work/repo" mybucket https://s3.example.invalid)
assert_contains "no-prefix dry-run targets the bucket root" "$out_noprefix" "s3://mybucket "
echo "--- --files-only and --index-only each print exactly one command, and are mutually exclusive ---"
out_files_only=$(bash "$here/publish.sh" --dry-run --files-only "$work/repo" mybucket https://s3.example.invalid myprefix)
n_files_only=$(echo "$out_files_only" | grep -c 's3 sync' || true)
if [ "$n_files_only" -eq 1 ]; then
pass "--files-only prints exactly one sync command"
else
fail "--files-only prints exactly one sync command (got $n_files_only)"
fi
assert_contains "the --files-only command still excludes index-shaped names" "$out_files_only" "--exclude index"
if echo "$out_files_only" | grep -q -- "--include"; then
fail "--files-only has no --include (it did!)"
else
pass "--files-only has no --include"
fi
out_index_only=$(bash "$here/publish.sh" --dry-run --index-only "$work/repo" mybucket https://s3.example.invalid myprefix)
n_index_only=$(echo "$out_index_only" | grep -c 's3 sync' || true)
if [ "$n_index_only" -eq 1 ]; then
pass "--index-only prints exactly one sync command"
else
fail "--index-only prints exactly one sync command (got $n_index_only)"
fi
assert_contains "the --index-only command still includes index-shaped names" "$out_index_only" "--include index"
assert_failure "publish.sh refuses --files-only together with --index-only" \
bash "$here/publish.sh" --dry-run --files-only --index-only "$work/repo" mybucket https://s3.example.invalid
echo "--- a real invocation fails on the network call, not on arguments -- and fails during credential verification, before any upload ---"
# port 1 is a reserved, unassigned TCP port -- nothing listens there, so this
# never leaves the loopback interface and needs no real network access.
@@ -188,9 +223,29 @@ if [ "$have_python" = true ]; then
# once it gets there -- the only thing this proves is that the collision
# check itself did not false-positive and let the script proceed past it.
assert_contains "with nothing pre-existing, publish.sh proceeds to pass 1" "$out" "pass 1/2"
echo "--- --index-only alone never runs the APK-collision check at all (it cannot upload one) ---"
# Same colliding key pre-seeded as the refusal test above, but this time
# asking only for the index pass: the check must not even run, since
# index_only_filters can never select a *.apk, so there is nothing it
# could refuse. Proceeding to "pass 2/2" (and failing there, since this
# stub has no sync support) is exactly the proof: a version of publish.sh
# that ran the check regardless of the flags would instead refuse here,
# before ever printing "pass 2/2".
start_stub mybucket "myprefix/de.example.app_1.apk"
out=$(bash "$here/publish.sh" --index-only "$collide_dir" mybucket "http://127.0.0.1:$stub_port" myprefix 2>&1) && rc=0 || rc=$?
kill "$stub_pid" 2>/dev/null || true
wait "$stub_pid" 2>/dev/null || true
assert_contains "--index-only proceeds straight to pass 2/2, skipping the APK check" "$out" "pass 2/2"
if echo "$out" | grep -q "already exists"; then
fail "--index-only must not run the APK-collision check (it refused anyway!)"
else
pass "--index-only does not run the APK-collision check"
fi
else
skip "credential-check-is-real / APK-collision-refusal (no python3 available, even via with-nixpkgs.sh)"
skip "negative control: non-colliding APK does not trip the check"
skip "--index-only never runs the APK-collision check"
fi
# --- live round trip, opt-in ------------------------------------------------
@@ -223,63 +278,64 @@ if [ -n "${FDROID_CI_LIVE_BUCKET:-}" ] && [ -n "${FDROID_CI_LIVE_ENDPOINT:-}" ]
rm -rf "$live_dir"
}
echo "--- upload ordering: index must not exist while the file already does (the hard part -- see below) ---"
object_exists() {
# object_exists <key> -- a real head-object against the live bucket.
bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \
--region "${LIVE_REGION:-us-east-1}" \
s3api head-object --bucket "$FDROID_CI_LIVE_BUCKET" --key "$1" >/dev/null 2>&1
}
echo "--- upload ordering by construction: --files-only, assert, then --index-only, assert (the hard part -- see below) ---"
# This is the one property the offline suite structurally cannot see: it
# is about the order two *real* uploads land in, which only a real bucket
# can answer. Racing the check in from outside against a real network call
# is exactly the flakiness fdroid-repo-ci#5 warned off ("worth building the
# check into the script's own output rather than trying to catch a race
# from outside"), so instead this uses the synchronization hook publish.sh
# exposes for tests (PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES): a real HEAD
# against the real bucket runs from *inside* publish.sh's own process,
# at the one moment guaranteed to be after pass 1 (files) finished and
# before pass 2 (index) started -- not a poll racing a background upload.
order_result=$(mktemp)
hook_script=$(mktemp)
cat > "$hook_script" <<'HOOK'
#!/usr/bin/env bash
set -uo pipefail
aws=(bash "$FDROID_HOOK_HERE/with-nixpkgs.sh" awscli2 aws \
--endpoint-url "$FDROID_HOOK_ENDPOINT" --region "${FDROID_HOOK_REGION:-us-east-1}")
file_present=0
index_absent=0
if "${aws[@]}" s3api head-object --bucket "$FDROID_HOOK_BUCKET" \
--key "$FDROID_HOOK_STAMP/probe.txt" >/dev/null 2>&1; then
file_present=1
fi
if ! "${aws[@]}" s3api head-object --bucket "$FDROID_HOOK_BUCKET" \
--key "$FDROID_HOOK_STAMP/index-v2.json" >/dev/null 2>&1; then
index_absent=1
fi
echo "file_present=$file_present index_absent=$index_absent" > "$FDROID_HOOK_RESULT_FILE"
HOOK
FDROID_HOOK_HERE="$here" FDROID_HOOK_ENDPOINT="$FDROID_CI_LIVE_ENDPOINT" \
FDROID_HOOK_REGION="${LIVE_REGION:-us-east-1}" FDROID_HOOK_BUCKET="$FDROID_CI_LIVE_BUCKET" \
FDROID_HOOK_STAMP="$stamp" FDROID_HOOK_RESULT_FILE="$order_result" \
PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES="bash '$hook_script'" \
bash "$here/publish.sh" "$live_dir" "$FDROID_CI_LIVE_BUCKET" \
# can answer. Racing a check in from outside against one real invocation
# doing both passes is exactly the flakiness fdroid-repo-ci#5 warned off
# ("worth building the check into the script's own output rather than
# trying to catch a race from outside") -- and, per review on #7, doing
# that with an env-var-triggered `eval` synchronization hook inside
# publish.sh was its own problem in a public script that runs with a
# signing key and S3 credentials in its environment. --files-only and
# --index-only remove the need for either: they are two genuinely separate
# invocations, so the bucket can be inspected for real, in between, with
# nothing racing and nothing evaluated.
bash "$here/publish.sh" --files-only "$live_dir" "$FDROID_CI_LIVE_BUCKET" \
"$FDROID_CI_LIVE_ENDPOINT" "$stamp" >/tmp/publish_live_out.$$ 2>&1
publish_rc=$?
if [ "$publish_rc" -eq 0 ]; then
pass "publish.sh uploaded to a real bucket"
files_only_rc=$?
if [ "$files_only_rc" -eq 0 ]; then
pass "--files-only uploads to a real bucket"
else
fail "publish.sh uploaded to a real bucket (exit $publish_rc)"
fail "--files-only uploads to a real bucket (exit $files_only_rc)"
cat /tmp/publish_live_out.$$ >&2
fi
rm -f /tmp/publish_live_out.$$
if [ -s "$order_result" ]; then
order_line=$(cat "$order_result")
if echo "$order_line" | grep -q "file_present=1 index_absent=1"; then
pass "mid-publish (real bucket): the file was already there and the index was not ($order_line)"
if object_exists "$stamp/probe.txt"; then
pass "after --files-only alone: the file object exists"
else
fail "mid-publish (real bucket): the file was already there and the index was not (got: $order_line)"
fail "after --files-only alone: the file object exists"
fi
if object_exists "$stamp/index-v2.json"; then
fail "after --files-only alone: the index object must not exist yet (it does!)"
else
fail "the between-passes hook never ran against the real bucket -- ordering was not actually observed"
pass "after --files-only alone: the index object does not exist yet"
fi
bash "$here/publish.sh" --index-only "$live_dir" "$FDROID_CI_LIVE_BUCKET" \
"$FDROID_CI_LIVE_ENDPOINT" "$stamp" >/tmp/publish_live_out2.$$ 2>&1
index_only_rc=$?
if [ "$index_only_rc" -eq 0 ]; then
pass "--index-only uploads to a real bucket"
else
fail "--index-only uploads to a real bucket (exit $index_only_rc)"
cat /tmp/publish_live_out2.$$ >&2
fi
rm -f /tmp/publish_live_out2.$$
if object_exists "$stamp/index-v2.json"; then
pass "after --index-only resumes: the index object now exists too"
else
fail "after --index-only resumes: the index object now exists too"
fi
rm -f "$order_result" "$hook_script"
got=$(bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \
s3 cp "s3://$FDROID_CI_LIVE_BUCKET/$stamp/probe.txt" - 2>/dev/null || true)
@@ -296,6 +352,70 @@ HOOK
fail "the uploaded index object (pass 2) reads back byte-identical"
fi
echo "--- fault injection: --index-only alone (misuse) really does produce the dangerous state; the default never does ---"
# The property that actually matters: --index-only in isolation, against a
# prefix that has never had --files-only run against it, must produce
# exactly the state the whole ordering guarantee exists to prevent -- an
# index naming a file that 404s. If it did not, the two flags would not be
# a faithful decomposition of the default and the "by construction" claim
# above would be empty.
danger_dir=$(mktemp -d)
danger_stamp="live-danger-$(date -u +%s)-$$"
printf 'publish.sh live probe %s\n' "$danger_stamp" > "$danger_dir/probe.txt"
printf '{"probe":"%s"}\n' "$danger_stamp" > "$danger_dir/index-v2.json"
bash "$here/publish.sh" --index-only "$danger_dir" "$FDROID_CI_LIVE_BUCKET" \
"$FDROID_CI_LIVE_ENDPOINT" "$danger_stamp" >/tmp/publish_danger.$$ 2>&1
danger_rc=$?
if [ "$danger_rc" -ne 0 ]; then
fail "misuse setup: --index-only alone uploads to a real bucket (exit $danger_rc)"
cat /tmp/publish_danger.$$ >&2
fi
rm -f /tmp/publish_danger.$$
if object_exists "$danger_stamp/index-v2.json"; then
pass "misuse (--index-only alone) really does publish an index"
else
fail "misuse (--index-only alone) really does publish an index"
fi
if object_exists "$danger_stamp/probe.txt"; then
fail "misuse (--index-only alone) must leave the file it names absent (it does not!)"
else
pass "misuse (--index-only alone) leaves an index naming a file that 404s -- exactly the state the ordering guarantee exists to prevent"
fi
bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \
s3 rm --recursive "s3://$FDROID_CI_LIVE_BUCKET/$danger_stamp/" >/dev/null 2>&1 || true
rm -rf "$danger_dir"
# Contrast: the default (neither flag, one invocation) must never reach
# that state. There is deliberately no mid-flight observation point for a
# single default invocation any more (that was the hook this PR removes),
# so this is not re-proven by racing a check against it -- it holds by
# construction instead: the default's first phase runs the *same command
# array*, built the same way, as --files-only above, so its effect on the
# bucket is identical -- files land, and only then does the index command
# run. There is no code path through publish.sh that reaches "index
# present, file absent" without --index-only having been invoked alone,
# which is opt-in and never the default. What is left to check here is
# only the default's own final state.
safe_dir=$(mktemp -d)
safe_stamp="live-safe-$(date -u +%s)-$$"
printf 'publish.sh live probe %s\n' "$safe_stamp" > "$safe_dir/probe.txt"
printf '{"probe":"%s"}\n' "$safe_stamp" > "$safe_dir/index-v2.json"
bash "$here/publish.sh" "$safe_dir" "$FDROID_CI_LIVE_BUCKET" \
"$FDROID_CI_LIVE_ENDPOINT" "$safe_stamp" >/tmp/publish_safe.$$ 2>&1
safe_rc=$?
rm -f /tmp/publish_safe.$$
if [ "$safe_rc" -eq 0 ] && object_exists "$safe_stamp/probe.txt" && object_exists "$safe_stamp/index-v2.json"; then
pass "the default path (neither flag) never leaves an index without its file: both are present after one call"
else
fail "the default path (neither flag) never leaves an index without its file"
fi
bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \
s3 rm --recursive "s3://$FDROID_CI_LIVE_BUCKET/$safe_stamp/" >/dev/null 2>&1 || true
rm -rf "$safe_dir"
echo "--- APK-collision refusal against a real, already-published object ---"
apk_stamp="live-apk-$(date -u +%s)-$$"
apk_key="$apk_stamp/de.example.app_1.apk"
@@ -332,7 +452,8 @@ HOOK
cleanup_live
else
skip "live round trip (set FDROID_CI_LIVE_BUCKET, FDROID_CI_LIVE_ENDPOINT and AWS credentials)"
skip "live upload ordering (file present, index absent mid-publish)"
skip "live upload ordering by construction (--files-only then --index-only)"
skip "live fault injection: --index-only misuse vs. the default"
skip "live APK-collision refusal against a real object"
fi