claudeandbit
5e4b2fabc6
Measured against the real endpoint, not inferred: every PutObject failed
with
An error occurred (InvalidRequest) when calling the PutObject
operation: Bad request: invalid checksum algorithm
on aws-cli/2.34.24. Recent awscli2 attaches a checksum algorithm by
default and Garage refuses it, so publish.sh could not upload anything --
after passing its entire test suite.
It also reads as a permissions problem, which is the expensive part: reads
succeed, writes fail, and it fails identically against a bucket the same
credentials demonstrably can write. Set here rather than documented,
because a caller should not need to know a client-side default to publish
to the object store this script exists for, and leaving it to the
environment makes it work on one machine and not another.
`when_required` keeps checksums where the protocol needs them, so this is
narrower than switching them off.
Adds the test that would have caught it: an opt-in live round trip that
uploads, reads back and deletes, skipped unless a scratch bucket and
credentials are present so the offline run is unchanged. Everything else in
this suite proves the command is well formed, which is a different claim
from a server accepting it, and this failure lived exactly in that gap.
The live block captures the real credentials at the top of the file,
before the deliberate fakes (AKIAFAKETESTKEY) are exported for the offline
tests -- otherwise it authenticates as the fake and fails every time, and a
live test that always fails gets disabled, which is worse than not having
one.
Verified both directions against the real bucket: with the fix, 11 passed /
0 skipped; with it removed, the two live assertions fail. Offline, 9 passed
/ 1 skipped.
Closes #3.
Co-authored-by: bit <bit@das-labor.org>
114 lines
4.4 KiB
Bash
Executable File
114 lines
4.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Upload a repo/ tree -- signed APKs plus the signed index -- to an
|
|
# S3-compatible bucket.
|
|
#
|
|
# AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... \
|
|
# bash publish.sh <repo-root>/repo <bucket> <endpoint-url> [<prefix>]
|
|
#
|
|
# Bucket, endpoint and prefix are all arguments, never baked in here: this
|
|
# repo does not know, and per fdroid-repo-ci#1 must never come to know, which
|
|
# bucket or host any consumer publishes to. Credentials come from the
|
|
# standard AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN
|
|
# environment variables that `aws` itself reads -- nothing here parses or
|
|
# forwards them as arguments.
|
|
#
|
|
# Region is required by the `aws` CLI even against a non-AWS S3-compatible
|
|
# backend that ignores it; set AWS_DEFAULT_REGION if the endpoint cares,
|
|
# otherwise the placeholder below is never actually consulted.
|
|
#
|
|
# Uses `aws s3 sync`, not `cp --recursive`: re-publishing re-uploads only
|
|
# what changed. Does **not** pass `--delete` by default -- an F-Droid repo
|
|
# normally keeps every APK it has ever served (older index entries can still
|
|
# point at them), so silently pruning the remote on every publish is the
|
|
# wrong default. Pass `--delete` as an extra argument if a caller genuinely
|
|
# wants the remote pruned to match local.
|
|
#
|
|
# `fdroid update` needs no network; this is the one script in this repo that
|
|
# does (fdroid-repo-ci#1), and is accordingly the one script whose happy path
|
|
# the test suite cannot exercise against a real bucket. `--dry-run` prints
|
|
# the exact command this script would run -- including the `with-nixpkgs.sh`
|
|
# pin -- and exits without touching the network, which is what the test
|
|
# suite asserts against. (A PATH-stubbed `aws` cannot do this job: `nix
|
|
# shell` puts the packages it was asked for at the *front* of PATH, ahead of
|
|
# anything already there, so a stub earlier on the caller's PATH would simply
|
|
# be shadowed by the real `aws` from awscli2.) Separately, the test suite
|
|
# also runs this for real against a closed local port, to confirm the
|
|
# constructed command is well-formed and genuinely attempts a network call
|
|
# rather than failing on argument parsing.
|
|
#
|
|
# Pins awscli2 via with-nixpkgs.sh/nixpkgs-pin, next to this script.
|
|
set -euo pipefail
|
|
|
|
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|
|
|
if [ "$#" -lt 3 ]; then
|
|
cat >&2 <<'USAGE'
|
|
usage: publish.sh [--dry-run] <local-repo-dir> <bucket> <endpoint-url> [<prefix>] [extra aws s3 sync args...]
|
|
|
|
Credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and optionally
|
|
AWS_SESSION_TOKEN) must already be exported -- never passed as arguments.
|
|
USAGE
|
|
exit 2
|
|
fi
|
|
|
|
dry_run=false
|
|
if [ "$1" = "--dry-run" ]; then
|
|
dry_run=true
|
|
shift
|
|
fi
|
|
|
|
local_dir=$1
|
|
bucket=$2
|
|
endpoint=$3
|
|
shift 3
|
|
|
|
prefix=""
|
|
if [ "$#" -gt 0 ] && [[ $1 != -* ]]; then
|
|
prefix=$1
|
|
shift
|
|
fi
|
|
|
|
extra_args=("$@")
|
|
|
|
[ -d "$local_dir" ] || { echo "publish.sh: no such directory: $local_dir" >&2; exit 2; }
|
|
: "${AWS_ACCESS_KEY_ID:?publish.sh: AWS_ACCESS_KEY_ID is not set}"
|
|
: "${AWS_SECRET_ACCESS_KEY:?publish.sh: AWS_SECRET_ACCESS_KEY is not set}"
|
|
|
|
dest="s3://$bucket"
|
|
if [ -n "$prefix" ]; then
|
|
dest="$dest/${prefix#/}"
|
|
fi
|
|
|
|
# awscli2 2.23+ attaches a checksum algorithm to every PutObject by default.
|
|
# Garage rejects it outright -- measured against a real bucket:
|
|
#
|
|
# An error occurred (InvalidRequest) when calling the PutObject
|
|
# operation: Bad request: invalid checksum algorithm
|
|
#
|
|
# with aws-cli/2.34.24. Every upload fails, and it reads as a permissions
|
|
# problem until you notice that reads succeed and that writes fail the same
|
|
# way against a bucket the same credentials demonstrably can write.
|
|
#
|
|
# Set here rather than documented, because a caller should not have to know
|
|
# about a client-side default to publish to the object store this script
|
|
# exists for -- and leaving it to the environment means it works on one
|
|
# machine and not another. `when_required` keeps checksums where the protocol
|
|
# actually needs them, so this is narrower than switching them off.
|
|
export AWS_REQUEST_CHECKSUM_CALCULATION="${AWS_REQUEST_CHECKSUM_CALCULATION:-when_required}"
|
|
export AWS_RESPONSE_CHECKSUM_VALIDATION="${AWS_RESPONSE_CHECKSUM_VALIDATION:-when_required}"
|
|
|
|
cmd=(bash "$here/with-nixpkgs.sh" awscli2 aws s3 sync "$local_dir" "$dest"
|
|
--endpoint-url "$endpoint"
|
|
--region "${AWS_DEFAULT_REGION:-us-east-1}"
|
|
"${extra_args[@]}")
|
|
|
|
if [ "$dry_run" = true ]; then
|
|
printf '%q ' "${cmd[@]}"
|
|
printf '\n'
|
|
exit 0
|
|
fi
|
|
|
|
echo "publish.sh: syncing $local_dir -> $dest (endpoint $endpoint)"
|
|
exec "${cmd[@]}"
|