# archive-org-collection-harvest

**Type:** Hermes skill (`~/.hermes/hermes-whale/skills/research/archive-org-collection-harvest/SKILL.md`)
**Created:** 2026-09-21
**Origin:** Old iPad iOS 9.3.5 IPA hunt — archive.org was unreachable, this pattern was
worked out to get around it.

---

## What it does

Enumerate and bulk-download files from any archive.org collection when
**archive.org does not resolve** on the current network (home router blocks the DNS name).
Covers the metadata API, the "filenames are the index" filtering technique, and the
**double `--resolve`** needed because downloads 302 to a second host.

## The two things that make it non-obvious

### 1. Filenames carry the metadata
Archive.org item filenames often embed app ID + bundle ID + version + min OS:
```
iOS 4/jp.co.korg.iMS-20/iMS-20-(jp.co.korg.iMS-20)-1.5.2-(iOS_4.2)-b907eac….ipa
```
So one `curl` of `/metadata/<id>` + `jq -r '.files[].name'` + `grep -iE '<terms>'` gives you
full search, version targeting, and curation. No per-file API calls, no page scraping.

### 2. Downloads need the redirect host pinned separately
`archive.org/download/...` returns **302** to `https://dn####.ca.archive.org/...`.
Pinning only `archive.org` yields `http=302 size=0` — looks like a broken file, is actually
an unpinned redirect. You must read the `Location` header, resolve that host (`dig @1.1.1.1`),
and add a second `--resolve`:

```bash
ORIGIN_IP=$(dig +short archive.org @1.1.1.1 | tail -1)
DN=$(curl -sI --resolve archive.org:443:"$ORIGIN_IP" "$url" \
      | grep -i '^location' | tr -d '\r' | sed 's/^[Ll]ocation: //')
HOST=$(printf '%s' "$DN" | sed -E 's#https://([^/]+)/.*#\1#')
IP=$(dig +short "$HOST" @1.1.1.1 | tail -1)
curl -sL --resolve archive.org:443:"$ORIGIN_IP" --resolve "$HOST:443:$IP" -o out.bin "$url"
```

## Other pitfalls recorded

- Spaces in filenames are **literal** → must `sed 's/ /%20/g'`. `+` is not accepted.
- `web_extract` on `archive.org/details/...` returns
  `Blocked: URL targets a private or internal network address` — the extractor's own guard.
  `web_search` on the same topic can also return empty. Fall back to `curl --resolve` fast
  instead of retrying the web tools.
- Big metadata items (17k files) truncate in tool output and break `json.loads`.
  **Always** `curl -o file.json` then `jq`, never parse the response inline.
- `HOST`/`IP` change per download host — re-derive, never hardcode `dn####`.
- Cheap probe before a big fetch: `curl -r 0-4095 -w 'http=%{http_code} size=%{size_download}'`
  → expect `HTTP 206` + non-zero bytes.
- Verify with `file` — an empty file or `HTML document` means the resolve failed.
- Write harvests as a **script file** run with `bash file.sh`; inline multi-line
  `curl | jq` chains are fragile and each inline call may need approval.

## Verification checklist

- [ ] Control site returns 200 (network up, only archive.org DNS blocked)
- [ ] `/metadata/<id>` returns JSON with plausible `.files` count
- [ ] Filename list non-empty, filtered hits match expectations
- [ ] `-r 0-4095` probe → HTTP 206, non-zero size
- [ ] `file` on probe reports expected archive/binary type

## Applied in

- `personal/how-to/old-ipad-ios935-ipa-sources.md` — iOSObscura collection (17,652 IPAs),
  curated iOS 9.3.5 music/MIDI + utility shortlist
- `personal/how-to/scripts/fetch-ios935-ipas.sh` — working downloader built on this pattern