diff --git a/README.md b/README.md index 6da4bc0..1258aa3 100644 --- a/README.md +++ b/README.md @@ -148,6 +148,37 @@ key with `head-object` and refuses the whole publish if any of them already exists -- the safeguard `next-version-code.sh`'s clamp cannot provide on its own (see above). This does not prevent the wasted build behind a same-minute collision; it prevents the collision from silently corrupting the repo. +Skipped entirely for a plain `--index-only` invocation (below), since that +pass can never upload a `*.apk` in the first place. + +### `--files-only` and `--index-only` + +The two passes can also be run as two separate invocations: + +```bash +bash publish.sh --files-only "$FDROID_REPO_ROOT/repo" "$BUCKET" "$S3_ENDPOINT_URL" +# ... build finishes, or a later job resumes here ... +bash publish.sh --index-only "$FDROID_REPO_ROOT/repo" "$BUCKET" "$S3_ENDPOINT_URL" +``` + +With neither flag, both passes run in one call, files then index -- the +default, and the only way to reach "index before files" is to invoke +`--index-only` on its own, which is opt-in misuse, never the default. This +is real, independently useful behaviour, not just a testing seam: an +interrupted publish can resume at the index step without re-uploading every +APK, and a caller can stage files now and publish the index later. + +It is also what makes the ordering guarantee testable **by construction**: +run `--files-only`, assert the bucket has the file and not the index yet, +run `--index-only`, assert the index then appears -- two genuinely separate +invocations, with the bucket inspected for real in between. An earlier +version of this script proved the same property with a same-process +synchronization hook that `eval`'d an environment variable between the two +passes; that was removed in favour of these flags; a script that runs with +a release signing key and S3 credentials in its environment, in a public +repository, should not carry a documented path from env-var influence to +arbitrary code execution, however unlikely, when a real feature closes the +same gap for free. ## Align before sign, and how "no `zipalign`" was verified @@ -257,8 +288,9 @@ here needs, resolved through the same pin. bash run_tests.sh ``` -83 assertions across seven suites offline (~20s on a warm nix store), plus 5 -more that only run against a real bucket (see below): +91 assertions across seven suites offline (~20s on a warm nix store, 4 +deliberately skipped -- see below), plus 13 more that only run against a +real bucket: * `test_check_zip_alignment.py` -- pure standard library, no nix, no network at all; builds an aligned and a misaligned fixture zip by hand and @@ -290,13 +322,15 @@ more that only run against a real bucket (see below): `test_publish.sh` uses) -- proving the wrapper's argv/exit-code/stdout contract, not just the Python module's arithmetic. * `test_publish.sh` -- argument/credential validation, `--dry-run` command - construction (now two commands, files then index, and asserted in that - order), and a real invocation against a closed local port - (`127.0.0.1:1`), fully offline. Also runs `test_s3_stub.py`, a + construction (both the default two-command form, files then index, and + each of `--files-only`/`--index-only` printing exactly one command, plus + their mutual exclusivity), and a real invocation against a closed local + port (`127.0.0.1:1`), fully offline. Also runs `test_s3_stub.py`, a same-repo, HEAD-only local HTTP stub, to prove the credential check is a - real authenticated call (a wrong bucket is rejected) and that an - already-published APK is refused before any upload, with a negative - control proving a non-colliding APK does not trip the same check. The + real authenticated call (a wrong bucket is rejected), that an + already-published APK is refused before any upload, that `--index-only` + alone never runs that check at all (it cannot upload an APK), and a + negative control proving a non-colliding APK does not trip the check. The actual upload path -- and the two-pass *ordering* of real uploads, which nothing offline can see -- is exercised only by the opt-in live round trip below, matching the constraint that only `publish.sh` needs network in the @@ -304,15 +338,24 @@ more that only run against a real bucket (see below): `test_publish.sh`'s live section (skipped by default; needs `FDROID_CI_LIVE_BUCKET`, `FDROID_CI_LIVE_ENDPOINT` and real credentials) adds -5 more assertions against a real bucket: the round trip itself; the index -object provably absent while the file object is already present at the one -point in the middle of a real publish that the ordering guarantee depends on -(observed via a same-process synchronization hook `publish.sh` exposes only -for this, `PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES` -- deliberately not a poll -racing a background upload from outside, since that is exactly the -flakiness fdroid-repo-ci#5 warned against); both objects reading back -byte-identical; a second publish refused when it would overwrite a -real, already-published APK; and that APK's bytes unchanged afterwards. +13 more assertions against a real bucket: + +* the round trip itself, both objects reading back byte-identical; +* upload ordering proven **by construction**: `--files-only` run alone, + then the bucket inspected for real (file present, index absent), then + `--index-only` run alone, then the bucket inspected again (index now + present) -- two genuinely separate invocations, nothing racing, no + synchronization hook, no `eval` (see "`--files-only` and `--index-only`" + above for why that matters in a public repo); +* a fault-injection contrast proving the flags have real teeth: invoking + `--index-only` **alone**, against a prefix `--files-only` never touched, + really does publish an index naming a file that is not there -- the exact + state the whole ordering guarantee exists to prevent -- while the + *default* path (neither flag, one call) never leaves that state, which + holds by construction (the default's first phase is the identical command + array `--files-only` runs) and is checked here by its final state; +* a second publish refused when it would overwrite a real, already-published + APK, and that APK's bytes unchanged afterwards. No `.gitea/workflows` runs this, on the same reasoning weblib-ci#9 recorded: run it by hand before pushing. If that changes for weblib-ci it should diff --git a/publish.sh b/publish.sh index db91404..e4d866e 100755 --- a/publish.sh +++ b/publish.sh @@ -39,6 +39,23 @@ # it is only ever reached by a client that already has last publish's index # and is willing to look for it). # +# The two passes are also separately invokable, with `--files-only` and +# `--index-only`. This is not a test seam -- it is real, useful on its own: +# an interrupted publish can resume at the index step without re-uploading +# every APK, and a caller can stage files now and publish the index later. +# It is also what makes the ordering guarantee testable *by construction*: +# run `--files-only`, assert the bucket has the files and not the index, run +# `--index-only`, assert the index appears -- no synchronization hook and no +# `eval` of anything env-supplied needed in a script that runs with a +# signing key and S3 credentials in its environment, in a public repo. (An +# earlier version of this script had exactly such a hook; it was removed +# once these flags made it unnecessary.) With neither flag, both passes run, +# files then index, which is the safe default and is unconditional -- there +# is no way to select "index then files" short of invoking `--index-only` +# by hand before ever running `--files-only`, which is the deliberate +# misuse the two flags exist to make possible for cases that want it +# (resuming, staging) while never being the default. +# # ## Credentials are verified before anything is uploaded (fdroid-repo-ci#5) # # Checking that AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are merely *set* does @@ -94,40 +111,64 @@ # `fdroid update` needs no network; this is the one script in this repo that # does (fdroid-repo-ci#1), and is accordingly the one script whose happy path # the test suite cannot exercise against a real bucket. `--dry-run` prints -# the exact two commands this script would run, in order -- including the -# `with-nixpkgs.sh` pin -- and exits without touching the network at all: no -# credential check, no APK-existence check, no sync. That is what the test -# suite asserts against for command construction and ordering. (A -# PATH-stubbed `aws` cannot do this job: `nix shell` puts the packages it was -# asked for at the *front* of PATH, ahead of anything already there, so a -# stub earlier on the caller's PATH would simply be shadowed by the real -# `aws` from awscli2.) The credential check, the APK-existence check and the -# two-pass ordering are each proven separately: a real invocation against a -# closed local port for "fails on the network, and fails during the -# credential check specifically, before any upload is attempted"; a tiny -# local HEAD-only HTTP stub for "an existing APK is refused, and only an -# existing APK", offline; and the opt-in live round trip against a real -# bucket for the actual upload path plus a real head-to-head APK collision. +# the command(s) this script would run for whichever pass(es) are selected, +# in order -- including the `with-nixpkgs.sh` pin -- and exits without +# touching the network at all: no credential check, no APK-existence check, +# no sync. That is what the test suite asserts against for command +# construction and ordering. (A PATH-stubbed `aws` cannot do this job: `nix +# shell` puts the packages it was asked for at the *front* of PATH, ahead of +# anything already there, so a stub earlier on the caller's PATH would +# simply be shadowed by the real `aws` from awscli2.) The credential check, +# the APK-existence check and the two-pass ordering are each proven +# separately: a real invocation against a closed local port for "fails on +# the network, and fails during the credential check specifically, before +# any upload is attempted"; a tiny local HEAD-only HTTP stub for "an +# existing APK is refused, and only an existing APK", offline; and the +# opt-in live round trip against a real bucket, run once as `--files-only` +# then `--index-only` to observe the ordering by construction, and once +# more as a real head-to-head APK collision. # # Pins awscli2 via with-nixpkgs.sh/nixpkgs-pin, next to this script. set -euo pipefail here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -if [ "$#" -lt 3 ]; then +usage() { cat >&2 <<'USAGE' -usage: publish.sh [--dry-run] [] [extra aws s3 sync args...] +usage: publish.sh [--dry-run] [--files-only | --index-only] \ + [] [extra aws s3 sync args...] + +With neither --files-only nor --index-only, both passes run: every +non-index file first, then the index -- the safe default, and the only way +to select "index before files" is to invoke --index-only by hand ahead of +--files-only, never the default. --files-only and --index-only are mutually +exclusive. Credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and optionally AWS_SESSION_TOKEN) must already be exported -- never passed as arguments. USAGE +} + +dry_run=false +files_only=false +index_only=false +while [ "$#" -gt 0 ]; do + case "$1" in + --dry-run) dry_run=true; shift ;; + --files-only) files_only=true; shift ;; + --index-only) index_only=true; shift ;; + *) break ;; + esac +done + +if [ "$files_only" = true ] && [ "$index_only" = true ]; then + echo "publish.sh: --files-only and --index-only are mutually exclusive" >&2 exit 2 fi -dry_run=false -if [ "$1" = "--dry-run" ]; then - dry_run=true - shift +if [ "$#" -lt 3 ]; then + usage + exit 2 fi local_dir=$1 @@ -183,11 +224,23 @@ cmd_files=("${aws_base[@]}" s3 sync "$local_dir" "$dest" cmd_index=("${aws_base[@]}" s3 sync "$local_dir" "$dest" "${index_only_filters[@]}" "${extra_args[@]}") +# Which pass(es) this invocation runs. Unconditional and exactly one of +# three shapes: both (the default), files alone, or index alone -- the +# mutual-exclusivity check above already ruled out both flags together. +run_files=true +run_index=true +[ "$index_only" = true ] && run_files=false +[ "$files_only" = true ] && run_index=false + if [ "$dry_run" = true ]; then - printf '%q ' "${cmd_files[@]}" - printf '\n' - printf '%q ' "${cmd_index[@]}" - printf '\n' + if [ "$run_files" = true ]; then + printf '%q ' "${cmd_files[@]}" + printf '\n' + fi + if [ "$run_index" = true ]; then + printf '%q ' "${cmd_index[@]}" + printf '\n' + fi exit 0 fi @@ -198,49 +251,39 @@ if ! verify_out=$("${aws_base[@]}" s3api head-bucket --bucket "$bucket" 2>&1); t exit 1 fi -# Refuse to overwrite an existing APK -- see the comment block above for why -# this exists and what it does and does not close. Every local *.apk is -# checked, regardless of nesting, since fdroidserver's binary-repo layout -# keeps them under repo/ directly but nothing here should assume that. -while IFS= read -r -d '' apk; do - rel=${apk#"$local_dir"/} - key="$rel" - if [ -n "$prefix" ]; then - key="${prefix#/}/$rel" - fi - if head_out=$("${aws_base[@]}" s3api head-object --bucket "$bucket" --key "$key" 2>&1); then - echo "publish.sh: refusing to publish -- $key already exists in s3://$bucket (would overwrite an existing APK, fdroid-repo-ci#6)" >&2 - exit 1 - elif echo "$head_out" | grep -qi '404\|not found\|NoSuchKey'; then - : # does not exist yet -- fine, this is the common case - else - echo "publish.sh: could not check whether $key already exists -- refusing to guess:" >&2 - echo "$head_out" >&2 - exit 1 - fi -done < <(find "$local_dir" -type f -name '*.apk' -print0) +if [ "$run_files" = true ]; then + # Refuse to overwrite an existing APK -- see the comment block above for + # why this exists and what it does and does not close. Every local *.apk + # is checked, regardless of nesting, since fdroidserver's binary-repo + # layout keeps them under repo/ directly but nothing here should assume + # that. Skipped entirely for a plain --index-only invocation: the index + # pass never uploads a *.apk (index_only_filters excludes everything but + # the index/entry names), so there is nothing this check could catch. + while IFS= read -r -d '' apk; do + rel=${apk#"$local_dir"/} + key="$rel" + if [ -n "$prefix" ]; then + key="${prefix#/}/$rel" + fi + if head_out=$("${aws_base[@]}" s3api head-object --bucket "$bucket" --key "$key" 2>&1); then + echo "publish.sh: refusing to publish -- $key already exists in s3://$bucket (would overwrite an existing APK, fdroid-repo-ci#6)" >&2 + exit 1 + elif echo "$head_out" | grep -qi '404\|not found\|NoSuchKey'; then + : # does not exist yet -- fine, this is the common case + else + echo "publish.sh: could not check whether $key already exists -- refusing to guess:" >&2 + echo "$head_out" >&2 + exit 1 + fi + done < <(find "$local_dir" -type f -name '*.apk' -print0) -# Test-only synchronization point: if set, run it after the APK-existence -# check and before pass 1, and again between the two passes. Lets the test -# suite observe/assert real bucket state at the one moment that actually -# matters for the ordering guarantee (fdroid-repo-ci#5) -- files present, -# index not yet -- without racing a real background upload from outside. -# Never set by a real caller; not part of the documented interface above. -run_test_hook() { - local hook_var=$1 - local hook_value="${!hook_var:-}" - [ -n "$hook_value" ] && eval "$hook_value" - return 0 -} + echo "publish.sh: pass 1/2: uploading files (index deferred) $local_dir -> $dest" + "${cmd_files[@]}" +fi -run_test_hook PUBLISH_SH_TEST_HOOK_BEFORE_UPLOAD - -echo "publish.sh: pass 1/2: uploading files (index deferred) $local_dir -> $dest" -"${cmd_files[@]}" - -run_test_hook PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES - -echo "publish.sh: pass 2/2: uploading the index $local_dir -> $dest" -"${cmd_index[@]}" +if [ "$run_index" = true ]; then + echo "publish.sh: pass 2/2: uploading the index $local_dir -> $dest" + "${cmd_index[@]}" +fi echo "publish.sh: done" diff --git a/test_publish.sh b/test_publish.sh index 3afcebd..a2b7d6f 100755 --- a/test_publish.sh +++ b/test_publish.sh @@ -5,8 +5,9 @@ # a genuine bucket is out of scope for most of this file. What this asserts: # # * argument/credential validation, offline; -# * `--dry-run` builds the exact two commands (files, then index) a real run -# would use, in that order, offline; +# * `--dry-run` builds the exact command(s) a real run would use for +# whichever of --files-only/--index-only (or neither -> both, files then +# index) is selected, offline; # * a real invocation against a closed local port fails on the network call # itself -- during the credential check specifically, before any upload is # attempted -- proving both that the constructed command is well-formed @@ -14,12 +15,19 @@ # * a real invocation against a tiny local HEAD-only S3 stub (test_s3_stub.py) # proves the credential check is a genuine authenticated call (a wrong # bucket is rejected, not waved through because a variable was merely -# non-empty) and that an already-published APK is refused before any -# upload -- and, as a negative control, that a *non*-colliding APK does -# NOT trip the same check (fdroid-repo-ci#6); +# non-empty), that an already-published APK is refused before any upload +# (and that --index-only alone never runs that check at all, since it +# cannot upload an APK) -- and, as a negative control, that a +# *non*-colliding APK does NOT trip the same check (fdroid-repo-ci#6); # * the opt-in live round trip against a real bucket, extended to prove the -# two-pass ordering with real uploads (not just command construction) and -# the APK-collision refusal against a real object. +# two-pass ordering **by construction**: `--files-only` then `--index-only` +# as two genuinely separate invocations, with the bucket state asserted in +# between -- no synchronization hook, no `eval` of anything in this public +# repo's production script -- plus the APK-collision refusal against a +# real object, and a deliberate-misuse case proving `--index-only` alone +# really does produce the exact state the ordering guarantee exists to +# prevent (an index naming a file that is not there), which the *default* +# path never reaches. set -euo pipefail here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) @@ -102,6 +110,33 @@ echo "--- --dry-run with no prefix omits the trailing slash entirely ---" out_noprefix=$(bash "$here/publish.sh" --dry-run "$work/repo" mybucket https://s3.example.invalid) assert_contains "no-prefix dry-run targets the bucket root" "$out_noprefix" "s3://mybucket " +echo "--- --files-only and --index-only each print exactly one command, and are mutually exclusive ---" +out_files_only=$(bash "$here/publish.sh" --dry-run --files-only "$work/repo" mybucket https://s3.example.invalid myprefix) +n_files_only=$(echo "$out_files_only" | grep -c 's3 sync' || true) +if [ "$n_files_only" -eq 1 ]; then + pass "--files-only prints exactly one sync command" +else + fail "--files-only prints exactly one sync command (got $n_files_only)" +fi +assert_contains "the --files-only command still excludes index-shaped names" "$out_files_only" "--exclude index" +if echo "$out_files_only" | grep -q -- "--include"; then + fail "--files-only has no --include (it did!)" +else + pass "--files-only has no --include" +fi + +out_index_only=$(bash "$here/publish.sh" --dry-run --index-only "$work/repo" mybucket https://s3.example.invalid myprefix) +n_index_only=$(echo "$out_index_only" | grep -c 's3 sync' || true) +if [ "$n_index_only" -eq 1 ]; then + pass "--index-only prints exactly one sync command" +else + fail "--index-only prints exactly one sync command (got $n_index_only)" +fi +assert_contains "the --index-only command still includes index-shaped names" "$out_index_only" "--include index" + +assert_failure "publish.sh refuses --files-only together with --index-only" \ + bash "$here/publish.sh" --dry-run --files-only --index-only "$work/repo" mybucket https://s3.example.invalid + echo "--- a real invocation fails on the network call, not on arguments -- and fails during credential verification, before any upload ---" # port 1 is a reserved, unassigned TCP port -- nothing listens there, so this # never leaves the loopback interface and needs no real network access. @@ -188,9 +223,29 @@ if [ "$have_python" = true ]; then # once it gets there -- the only thing this proves is that the collision # check itself did not false-positive and let the script proceed past it. assert_contains "with nothing pre-existing, publish.sh proceeds to pass 1" "$out" "pass 1/2" + + echo "--- --index-only alone never runs the APK-collision check at all (it cannot upload one) ---" + # Same colliding key pre-seeded as the refusal test above, but this time + # asking only for the index pass: the check must not even run, since + # index_only_filters can never select a *.apk, so there is nothing it + # could refuse. Proceeding to "pass 2/2" (and failing there, since this + # stub has no sync support) is exactly the proof: a version of publish.sh + # that ran the check regardless of the flags would instead refuse here, + # before ever printing "pass 2/2". + start_stub mybucket "myprefix/de.example.app_1.apk" + out=$(bash "$here/publish.sh" --index-only "$collide_dir" mybucket "http://127.0.0.1:$stub_port" myprefix 2>&1) && rc=0 || rc=$? + kill "$stub_pid" 2>/dev/null || true + wait "$stub_pid" 2>/dev/null || true + assert_contains "--index-only proceeds straight to pass 2/2, skipping the APK check" "$out" "pass 2/2" + if echo "$out" | grep -q "already exists"; then + fail "--index-only must not run the APK-collision check (it refused anyway!)" + else + pass "--index-only does not run the APK-collision check" + fi else skip "credential-check-is-real / APK-collision-refusal (no python3 available, even via with-nixpkgs.sh)" skip "negative control: non-colliding APK does not trip the check" + skip "--index-only never runs the APK-collision check" fi # --- live round trip, opt-in ------------------------------------------------ @@ -223,63 +278,64 @@ if [ -n "${FDROID_CI_LIVE_BUCKET:-}" ] && [ -n "${FDROID_CI_LIVE_ENDPOINT:-}" ] rm -rf "$live_dir" } - echo "--- upload ordering: index must not exist while the file already does (the hard part -- see below) ---" + object_exists() { + # object_exists -- a real head-object against the live bucket. + bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \ + --region "${LIVE_REGION:-us-east-1}" \ + s3api head-object --bucket "$FDROID_CI_LIVE_BUCKET" --key "$1" >/dev/null 2>&1 + } + + echo "--- upload ordering by construction: --files-only, assert, then --index-only, assert (the hard part -- see below) ---" # This is the one property the offline suite structurally cannot see: it # is about the order two *real* uploads land in, which only a real bucket - # can answer. Racing the check in from outside against a real network call - # is exactly the flakiness fdroid-repo-ci#5 warned off ("worth building the - # check into the script's own output rather than trying to catch a race - # from outside"), so instead this uses the synchronization hook publish.sh - # exposes for tests (PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES): a real HEAD - # against the real bucket runs from *inside* publish.sh's own process, - # at the one moment guaranteed to be after pass 1 (files) finished and - # before pass 2 (index) started -- not a poll racing a background upload. - order_result=$(mktemp) - hook_script=$(mktemp) - cat > "$hook_script" <<'HOOK' -#!/usr/bin/env bash -set -uo pipefail -aws=(bash "$FDROID_HOOK_HERE/with-nixpkgs.sh" awscli2 aws \ - --endpoint-url "$FDROID_HOOK_ENDPOINT" --region "${FDROID_HOOK_REGION:-us-east-1}") -file_present=0 -index_absent=0 -if "${aws[@]}" s3api head-object --bucket "$FDROID_HOOK_BUCKET" \ - --key "$FDROID_HOOK_STAMP/probe.txt" >/dev/null 2>&1; then - file_present=1 -fi -if ! "${aws[@]}" s3api head-object --bucket "$FDROID_HOOK_BUCKET" \ - --key "$FDROID_HOOK_STAMP/index-v2.json" >/dev/null 2>&1; then - index_absent=1 -fi -echo "file_present=$file_present index_absent=$index_absent" > "$FDROID_HOOK_RESULT_FILE" -HOOK - - FDROID_HOOK_HERE="$here" FDROID_HOOK_ENDPOINT="$FDROID_CI_LIVE_ENDPOINT" \ - FDROID_HOOK_REGION="${LIVE_REGION:-us-east-1}" FDROID_HOOK_BUCKET="$FDROID_CI_LIVE_BUCKET" \ - FDROID_HOOK_STAMP="$stamp" FDROID_HOOK_RESULT_FILE="$order_result" \ - PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES="bash '$hook_script'" \ - bash "$here/publish.sh" "$live_dir" "$FDROID_CI_LIVE_BUCKET" \ - "$FDROID_CI_LIVE_ENDPOINT" "$stamp" >/tmp/publish_live_out.$$ 2>&1 - publish_rc=$? - if [ "$publish_rc" -eq 0 ]; then - pass "publish.sh uploaded to a real bucket" + # can answer. Racing a check in from outside against one real invocation + # doing both passes is exactly the flakiness fdroid-repo-ci#5 warned off + # ("worth building the check into the script's own output rather than + # trying to catch a race from outside") -- and, per review on #7, doing + # that with an env-var-triggered `eval` synchronization hook inside + # publish.sh was its own problem in a public script that runs with a + # signing key and S3 credentials in its environment. --files-only and + # --index-only remove the need for either: they are two genuinely separate + # invocations, so the bucket can be inspected for real, in between, with + # nothing racing and nothing evaluated. + bash "$here/publish.sh" --files-only "$live_dir" "$FDROID_CI_LIVE_BUCKET" \ + "$FDROID_CI_LIVE_ENDPOINT" "$stamp" >/tmp/publish_live_out.$$ 2>&1 + files_only_rc=$? + if [ "$files_only_rc" -eq 0 ]; then + pass "--files-only uploads to a real bucket" else - fail "publish.sh uploaded to a real bucket (exit $publish_rc)" + fail "--files-only uploads to a real bucket (exit $files_only_rc)" cat /tmp/publish_live_out.$$ >&2 fi rm -f /tmp/publish_live_out.$$ - if [ -s "$order_result" ]; then - order_line=$(cat "$order_result") - if echo "$order_line" | grep -q "file_present=1 index_absent=1"; then - pass "mid-publish (real bucket): the file was already there and the index was not ($order_line)" - else - fail "mid-publish (real bucket): the file was already there and the index was not (got: $order_line)" - fi + if object_exists "$stamp/probe.txt"; then + pass "after --files-only alone: the file object exists" else - fail "the between-passes hook never ran against the real bucket -- ordering was not actually observed" + fail "after --files-only alone: the file object exists" + fi + if object_exists "$stamp/index-v2.json"; then + fail "after --files-only alone: the index object must not exist yet (it does!)" + else + pass "after --files-only alone: the index object does not exist yet" + fi + + bash "$here/publish.sh" --index-only "$live_dir" "$FDROID_CI_LIVE_BUCKET" \ + "$FDROID_CI_LIVE_ENDPOINT" "$stamp" >/tmp/publish_live_out2.$$ 2>&1 + index_only_rc=$? + if [ "$index_only_rc" -eq 0 ]; then + pass "--index-only uploads to a real bucket" + else + fail "--index-only uploads to a real bucket (exit $index_only_rc)" + cat /tmp/publish_live_out2.$$ >&2 + fi + rm -f /tmp/publish_live_out2.$$ + + if object_exists "$stamp/index-v2.json"; then + pass "after --index-only resumes: the index object now exists too" + else + fail "after --index-only resumes: the index object now exists too" fi - rm -f "$order_result" "$hook_script" got=$(bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \ s3 cp "s3://$FDROID_CI_LIVE_BUCKET/$stamp/probe.txt" - 2>/dev/null || true) @@ -296,6 +352,70 @@ HOOK fail "the uploaded index object (pass 2) reads back byte-identical" fi + echo "--- fault injection: --index-only alone (misuse) really does produce the dangerous state; the default never does ---" + # The property that actually matters: --index-only in isolation, against a + # prefix that has never had --files-only run against it, must produce + # exactly the state the whole ordering guarantee exists to prevent -- an + # index naming a file that 404s. If it did not, the two flags would not be + # a faithful decomposition of the default and the "by construction" claim + # above would be empty. + danger_dir=$(mktemp -d) + danger_stamp="live-danger-$(date -u +%s)-$$" + printf 'publish.sh live probe %s\n' "$danger_stamp" > "$danger_dir/probe.txt" + printf '{"probe":"%s"}\n' "$danger_stamp" > "$danger_dir/index-v2.json" + + bash "$here/publish.sh" --index-only "$danger_dir" "$FDROID_CI_LIVE_BUCKET" \ + "$FDROID_CI_LIVE_ENDPOINT" "$danger_stamp" >/tmp/publish_danger.$$ 2>&1 + danger_rc=$? + if [ "$danger_rc" -ne 0 ]; then + fail "misuse setup: --index-only alone uploads to a real bucket (exit $danger_rc)" + cat /tmp/publish_danger.$$ >&2 + fi + rm -f /tmp/publish_danger.$$ + + if object_exists "$danger_stamp/index-v2.json"; then + pass "misuse (--index-only alone) really does publish an index" + else + fail "misuse (--index-only alone) really does publish an index" + fi + if object_exists "$danger_stamp/probe.txt"; then + fail "misuse (--index-only alone) must leave the file it names absent (it does not!)" + else + pass "misuse (--index-only alone) leaves an index naming a file that 404s -- exactly the state the ordering guarantee exists to prevent" + fi + + bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \ + s3 rm --recursive "s3://$FDROID_CI_LIVE_BUCKET/$danger_stamp/" >/dev/null 2>&1 || true + rm -rf "$danger_dir" + + # Contrast: the default (neither flag, one invocation) must never reach + # that state. There is deliberately no mid-flight observation point for a + # single default invocation any more (that was the hook this PR removes), + # so this is not re-proven by racing a check against it -- it holds by + # construction instead: the default's first phase runs the *same command + # array*, built the same way, as --files-only above, so its effect on the + # bucket is identical -- files land, and only then does the index command + # run. There is no code path through publish.sh that reaches "index + # present, file absent" without --index-only having been invoked alone, + # which is opt-in and never the default. What is left to check here is + # only the default's own final state. + safe_dir=$(mktemp -d) + safe_stamp="live-safe-$(date -u +%s)-$$" + printf 'publish.sh live probe %s\n' "$safe_stamp" > "$safe_dir/probe.txt" + printf '{"probe":"%s"}\n' "$safe_stamp" > "$safe_dir/index-v2.json" + bash "$here/publish.sh" "$safe_dir" "$FDROID_CI_LIVE_BUCKET" \ + "$FDROID_CI_LIVE_ENDPOINT" "$safe_stamp" >/tmp/publish_safe.$$ 2>&1 + safe_rc=$? + rm -f /tmp/publish_safe.$$ + if [ "$safe_rc" -eq 0 ] && object_exists "$safe_stamp/probe.txt" && object_exists "$safe_stamp/index-v2.json"; then + pass "the default path (neither flag) never leaves an index without its file: both are present after one call" + else + fail "the default path (neither flag) never leaves an index without its file" + fi + bash "$here/with-nixpkgs.sh" awscli2 aws --endpoint-url "$FDROID_CI_LIVE_ENDPOINT" \ + s3 rm --recursive "s3://$FDROID_CI_LIVE_BUCKET/$safe_stamp/" >/dev/null 2>&1 || true + rm -rf "$safe_dir" + echo "--- APK-collision refusal against a real, already-published object ---" apk_stamp="live-apk-$(date -u +%s)-$$" apk_key="$apk_stamp/de.example.app_1.apk" @@ -332,7 +452,8 @@ HOOK cleanup_live else skip "live round trip (set FDROID_CI_LIVE_BUCKET, FDROID_CI_LIVE_ENDPOINT and AWS credentials)" - skip "live upload ordering (file present, index absent mid-publish)" + skip "live upload ordering by construction (--files-only then --index-only)" + skip "live fault injection: --index-only misuse vs. the default" skip "live APK-collision refusal against a real object" fi