publish.sh: replace the test-only eval hook with --files-only/--index-only
bit's review on PR #7 (comment on the PR, appended to this reasoning): run_test_hook() eval'd an environment variable in publish.sh, which runs with the release signing key and S3 credentials in its environment, in a public repo. "Never set by a real caller" was true, but it was still a documented, discoverable path from any env-var influence to arbitrary code execution at the worst possible moment -- unnecessary, which is the stronger reason to remove it than any claim about likelihood. The replacement is a feature rather than a seam: --files-only and --index-only split the two passes into separately invokable steps. With neither flag the default is unchanged (both passes, files first). This makes the ordering guarantee testable by construction -- run --files-only, assert the bucket state, run --index-only, assert again -- with no hook, no eval, and no test-only branch in production code. It is also independently useful: an interrupted publish can resume at the index step without re-uploading every APK, and a caller can stage files now and publish the index later. The APK-existence check is skipped entirely for a plain --index-only invocation, since that pass can never upload a *.apk. Re-tested end to end: offline 91 passed / 4 skipped / 0 failed (up from 83/ 3/0, mostly new flag-construction and stub assertions); live against fdroid-weblib-dev 104 passed / 0 skipped / 0 failed, bucket verified empty before and after. Added a live fault-injection test that --index-only alone against a prefix --files-only never touched really does publish an index naming an absent file -- the exact state the ordering guarantee exists to prevent -- contrasted with the default path, which never reaches it. Offline, removing the run_files/run_index gating (simulating the flags being ignored) was confirmed to fail 5 of the new assertions before being reverted. Left the versionCode granularity as minutes, per instruction: that is being settled separately on weblib-viewer#15 and is a one-constant change for whenever it lands, not something to pre-empt here. Co-authored-by: bit <bit@das-labor.org>
This commit is contained in:
+109
-66
@@ -39,6 +39,23 @@
|
||||
# it is only ever reached by a client that already has last publish's index
|
||||
# and is willing to look for it).
|
||||
#
|
||||
# The two passes are also separately invokable, with `--files-only` and
|
||||
# `--index-only`. This is not a test seam -- it is real, useful on its own:
|
||||
# an interrupted publish can resume at the index step without re-uploading
|
||||
# every APK, and a caller can stage files now and publish the index later.
|
||||
# It is also what makes the ordering guarantee testable *by construction*:
|
||||
# run `--files-only`, assert the bucket has the files and not the index, run
|
||||
# `--index-only`, assert the index appears -- no synchronization hook and no
|
||||
# `eval` of anything env-supplied needed in a script that runs with a
|
||||
# signing key and S3 credentials in its environment, in a public repo. (An
|
||||
# earlier version of this script had exactly such a hook; it was removed
|
||||
# once these flags made it unnecessary.) With neither flag, both passes run,
|
||||
# files then index, which is the safe default and is unconditional -- there
|
||||
# is no way to select "index then files" short of invoking `--index-only`
|
||||
# by hand before ever running `--files-only`, which is the deliberate
|
||||
# misuse the two flags exist to make possible for cases that want it
|
||||
# (resuming, staging) while never being the default.
|
||||
#
|
||||
# ## Credentials are verified before anything is uploaded (fdroid-repo-ci#5)
|
||||
#
|
||||
# Checking that AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are merely *set* does
|
||||
@@ -94,40 +111,64 @@
|
||||
# `fdroid update` needs no network; this is the one script in this repo that
|
||||
# does (fdroid-repo-ci#1), and is accordingly the one script whose happy path
|
||||
# the test suite cannot exercise against a real bucket. `--dry-run` prints
|
||||
# the exact two commands this script would run, in order -- including the
|
||||
# `with-nixpkgs.sh` pin -- and exits without touching the network at all: no
|
||||
# credential check, no APK-existence check, no sync. That is what the test
|
||||
# suite asserts against for command construction and ordering. (A
|
||||
# PATH-stubbed `aws` cannot do this job: `nix shell` puts the packages it was
|
||||
# asked for at the *front* of PATH, ahead of anything already there, so a
|
||||
# stub earlier on the caller's PATH would simply be shadowed by the real
|
||||
# `aws` from awscli2.) The credential check, the APK-existence check and the
|
||||
# two-pass ordering are each proven separately: a real invocation against a
|
||||
# closed local port for "fails on the network, and fails during the
|
||||
# credential check specifically, before any upload is attempted"; a tiny
|
||||
# local HEAD-only HTTP stub for "an existing APK is refused, and only an
|
||||
# existing APK", offline; and the opt-in live round trip against a real
|
||||
# bucket for the actual upload path plus a real head-to-head APK collision.
|
||||
# the command(s) this script would run for whichever pass(es) are selected,
|
||||
# in order -- including the `with-nixpkgs.sh` pin -- and exits without
|
||||
# touching the network at all: no credential check, no APK-existence check,
|
||||
# no sync. That is what the test suite asserts against for command
|
||||
# construction and ordering. (A PATH-stubbed `aws` cannot do this job: `nix
|
||||
# shell` puts the packages it was asked for at the *front* of PATH, ahead of
|
||||
# anything already there, so a stub earlier on the caller's PATH would
|
||||
# simply be shadowed by the real `aws` from awscli2.) The credential check,
|
||||
# the APK-existence check and the two-pass ordering are each proven
|
||||
# separately: a real invocation against a closed local port for "fails on
|
||||
# the network, and fails during the credential check specifically, before
|
||||
# any upload is attempted"; a tiny local HEAD-only HTTP stub for "an
|
||||
# existing APK is refused, and only an existing APK", offline; and the
|
||||
# opt-in live round trip against a real bucket, run once as `--files-only`
|
||||
# then `--index-only` to observe the ordering by construction, and once
|
||||
# more as a real head-to-head APK collision.
|
||||
#
|
||||
# Pins awscli2 via with-nixpkgs.sh/nixpkgs-pin, next to this script.
|
||||
set -euo pipefail
|
||||
|
||||
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
|
||||
if [ "$#" -lt 3 ]; then
|
||||
usage() {
|
||||
cat >&2 <<'USAGE'
|
||||
usage: publish.sh [--dry-run] <local-repo-dir> <bucket> <endpoint-url> [<prefix>] [extra aws s3 sync args...]
|
||||
usage: publish.sh [--dry-run] [--files-only | --index-only] \
|
||||
<local-repo-dir> <bucket> <endpoint-url> [<prefix>] [extra aws s3 sync args...]
|
||||
|
||||
With neither --files-only nor --index-only, both passes run: every
|
||||
non-index file first, then the index -- the safe default, and the only way
|
||||
to select "index before files" is to invoke --index-only by hand ahead of
|
||||
--files-only, never the default. --files-only and --index-only are mutually
|
||||
exclusive.
|
||||
|
||||
Credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and optionally
|
||||
AWS_SESSION_TOKEN) must already be exported -- never passed as arguments.
|
||||
USAGE
|
||||
}
|
||||
|
||||
dry_run=false
|
||||
files_only=false
|
||||
index_only=false
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) dry_run=true; shift ;;
|
||||
--files-only) files_only=true; shift ;;
|
||||
--index-only) index_only=true; shift ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$files_only" = true ] && [ "$index_only" = true ]; then
|
||||
echo "publish.sh: --files-only and --index-only are mutually exclusive" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
dry_run=false
|
||||
if [ "$1" = "--dry-run" ]; then
|
||||
dry_run=true
|
||||
shift
|
||||
if [ "$#" -lt 3 ]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
local_dir=$1
|
||||
@@ -183,11 +224,23 @@ cmd_files=("${aws_base[@]}" s3 sync "$local_dir" "$dest"
|
||||
cmd_index=("${aws_base[@]}" s3 sync "$local_dir" "$dest"
|
||||
"${index_only_filters[@]}" "${extra_args[@]}")
|
||||
|
||||
# Which pass(es) this invocation runs. Unconditional and exactly one of
|
||||
# three shapes: both (the default), files alone, or index alone -- the
|
||||
# mutual-exclusivity check above already ruled out both flags together.
|
||||
run_files=true
|
||||
run_index=true
|
||||
[ "$index_only" = true ] && run_files=false
|
||||
[ "$files_only" = true ] && run_index=false
|
||||
|
||||
if [ "$dry_run" = true ]; then
|
||||
printf '%q ' "${cmd_files[@]}"
|
||||
printf '\n'
|
||||
printf '%q ' "${cmd_index[@]}"
|
||||
printf '\n'
|
||||
if [ "$run_files" = true ]; then
|
||||
printf '%q ' "${cmd_files[@]}"
|
||||
printf '\n'
|
||||
fi
|
||||
if [ "$run_index" = true ]; then
|
||||
printf '%q ' "${cmd_index[@]}"
|
||||
printf '\n'
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -198,49 +251,39 @@ if ! verify_out=$("${aws_base[@]}" s3api head-bucket --bucket "$bucket" 2>&1); t
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Refuse to overwrite an existing APK -- see the comment block above for why
|
||||
# this exists and what it does and does not close. Every local *.apk is
|
||||
# checked, regardless of nesting, since fdroidserver's binary-repo layout
|
||||
# keeps them under repo/ directly but nothing here should assume that.
|
||||
while IFS= read -r -d '' apk; do
|
||||
rel=${apk#"$local_dir"/}
|
||||
key="$rel"
|
||||
if [ -n "$prefix" ]; then
|
||||
key="${prefix#/}/$rel"
|
||||
fi
|
||||
if head_out=$("${aws_base[@]}" s3api head-object --bucket "$bucket" --key "$key" 2>&1); then
|
||||
echo "publish.sh: refusing to publish -- $key already exists in s3://$bucket (would overwrite an existing APK, fdroid-repo-ci#6)" >&2
|
||||
exit 1
|
||||
elif echo "$head_out" | grep -qi '404\|not found\|NoSuchKey'; then
|
||||
: # does not exist yet -- fine, this is the common case
|
||||
else
|
||||
echo "publish.sh: could not check whether $key already exists -- refusing to guess:" >&2
|
||||
echo "$head_out" >&2
|
||||
exit 1
|
||||
fi
|
||||
done < <(find "$local_dir" -type f -name '*.apk' -print0)
|
||||
if [ "$run_files" = true ]; then
|
||||
# Refuse to overwrite an existing APK -- see the comment block above for
|
||||
# why this exists and what it does and does not close. Every local *.apk
|
||||
# is checked, regardless of nesting, since fdroidserver's binary-repo
|
||||
# layout keeps them under repo/ directly but nothing here should assume
|
||||
# that. Skipped entirely for a plain --index-only invocation: the index
|
||||
# pass never uploads a *.apk (index_only_filters excludes everything but
|
||||
# the index/entry names), so there is nothing this check could catch.
|
||||
while IFS= read -r -d '' apk; do
|
||||
rel=${apk#"$local_dir"/}
|
||||
key="$rel"
|
||||
if [ -n "$prefix" ]; then
|
||||
key="${prefix#/}/$rel"
|
||||
fi
|
||||
if head_out=$("${aws_base[@]}" s3api head-object --bucket "$bucket" --key "$key" 2>&1); then
|
||||
echo "publish.sh: refusing to publish -- $key already exists in s3://$bucket (would overwrite an existing APK, fdroid-repo-ci#6)" >&2
|
||||
exit 1
|
||||
elif echo "$head_out" | grep -qi '404\|not found\|NoSuchKey'; then
|
||||
: # does not exist yet -- fine, this is the common case
|
||||
else
|
||||
echo "publish.sh: could not check whether $key already exists -- refusing to guess:" >&2
|
||||
echo "$head_out" >&2
|
||||
exit 1
|
||||
fi
|
||||
done < <(find "$local_dir" -type f -name '*.apk' -print0)
|
||||
|
||||
# Test-only synchronization point: if set, run it after the APK-existence
|
||||
# check and before pass 1, and again between the two passes. Lets the test
|
||||
# suite observe/assert real bucket state at the one moment that actually
|
||||
# matters for the ordering guarantee (fdroid-repo-ci#5) -- files present,
|
||||
# index not yet -- without racing a real background upload from outside.
|
||||
# Never set by a real caller; not part of the documented interface above.
|
||||
run_test_hook() {
|
||||
local hook_var=$1
|
||||
local hook_value="${!hook_var:-}"
|
||||
[ -n "$hook_value" ] && eval "$hook_value"
|
||||
return 0
|
||||
}
|
||||
echo "publish.sh: pass 1/2: uploading files (index deferred) $local_dir -> $dest"
|
||||
"${cmd_files[@]}"
|
||||
fi
|
||||
|
||||
run_test_hook PUBLISH_SH_TEST_HOOK_BEFORE_UPLOAD
|
||||
|
||||
echo "publish.sh: pass 1/2: uploading files (index deferred) $local_dir -> $dest"
|
||||
"${cmd_files[@]}"
|
||||
|
||||
run_test_hook PUBLISH_SH_TEST_HOOK_BETWEEN_PASSES
|
||||
|
||||
echo "publish.sh: pass 2/2: uploading the index $local_dir -> $dest"
|
||||
"${cmd_index[@]}"
|
||||
if [ "$run_index" = true ]; then
|
||||
echo "publish.sh: pass 2/2: uploading the index $local_dir -> $dest"
|
||||
"${cmd_index[@]}"
|
||||
fi
|
||||
|
||||
echo "publish.sh: done"
|
||||
|
||||
Reference in New Issue
Block a user