archive-org-collection-harvest

Type: Hermes skill (~/.hermes/hermes-whale/skills/research/archive-org-collection-harvest/SKILL.md) Created: 2026-09-21 Origin: Old iPad iOS 9.3.5 IPA hunt — archive.org was unreachable, this pattern was worked out to get around it.

What it does

Enumerate and bulk-download files from any archive.org collection when archive.org does not resolve on the current network (home router blocks the DNS name). Covers the metadata API, the "filenames are the index" filtering technique, and the double --resolve needed because downloads 302 to a second host.

The two things that make it non-obvious

1. Filenames carry the metadata

Archive.org item filenames often embed app ID + bundle ID + version + min OS:

iOS 4/jp.co.korg.iMS-20/iMS-20-(jp.co.korg.iMS-20)-1.5.2-(iOS_4.2)-b907eac….ipa

So one curl of /metadata/<id> + jq -r '.files[].name' + grep -iE '<terms>' gives you full search, version targeting, and curation. No per-file API calls, no page scraping.

2. Downloads need the redirect host pinned separately

archive.org/download/... returns 302 to https://dn####.ca.archive.org/.... Pinning only archive.org yields http=302 size=0 — looks like a broken file, is actually an unpinned redirect. You must read the Location header, resolve that host (dig @1.1.1.1), and add a second --resolve:

ORIGIN_IP=$(dig +short archive.org @1.1.1.1 | tail -1)
DN=$(curl -sI --resolve archive.org:443:"$ORIGIN_IP" "$url" \
      | grep -i '^location' | tr -d '\r' | sed 's/^[Ll]ocation: //')
HOST=$(printf '%s' "$DN" | sed -E 's#https://([^/]+)/.*#\1#')
IP=$(dig +short "$HOST" @1.1.1.1 | tail -1)
curl -sL --resolve archive.org:443:"$ORIGIN_IP" --resolve "$HOST:443:$IP" -o out.bin "$url"

Other pitfalls recorded

Spaces in filenames are literal → must sed 's/ /%20/g'. + is not accepted.
web_extract on archive.org/details/... returns Blocked: URL targets a private or internal network address — the extractor's own guard. web_search on the same topic can also return empty. Fall back to curl --resolve fast instead of retrying the web tools.
Big metadata items (17k files) truncate in tool output and break json.loads. Always curl -o file.json then jq, never parse the response inline.
HOST/IP change per download host — re-derive, never hardcode dn####.
Cheap probe before a big fetch: curl -r 0-4095 -w 'http=%{http_code} size=%{size_download}' → expect HTTP 206 + non-zero bytes.
Verify with file — an empty file or HTML document means the resolve failed.
Write harvests as a script file run with bash file.sh; inline multi-line curl | jq chains are fragile and each inline call may need approval.

Verification checklist

Control site returns 200 (network up, only archive.org DNS blocked)
/metadata/<id> returns JSON with plausible .files count
Filename list non-empty, filtered hits match expectations
-r 0-4095 probe → HTTP 206, non-zero size
file on probe reports expected archive/binary type

Applied in

personal/how-to/old-ipad-ios935-ipa-sources.md — iOSObscura collection (17,652 IPAs), curated iOS 9.3.5 music/MIDI + utility shortlist
personal/how-to/scripts/fetch-ios935-ipas.sh — working downloader built on this pattern

3.3 KiB Raw Blame History