From 9ff3da5b6b6d9edd9df0a89c768a6f7074aa5c2e Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Wed, 27 May 2026 03:01:31 +0600 Subject: [PATCH] [2026-05-27] wiki curation: multi-host cron topology, media pipeline pitfalls, watchlist automation updates --- .../wiki-curation-runs/2026-05-27.md | 85 +++++++++++++++++++ wiki/concepts/multi-host-cron-topology.md | 71 ++++++++++++++++ wiki/concepts/watchlist-automation.md | 40 ++++++--- wiki/index.md | 3 +- wiki/log.md | 13 +++ wiki/tech/media-pipeline-pitfalls.md | 30 ++++++- wiki/tech/vps-qentra.md | 2 +- 7 files changed, 229 insertions(+), 15 deletions(-) create mode 100644 personal/projects/personal-os/wiki-curation-runs/2026-05-27.md create mode 100644 wiki/concepts/multi-host-cron-topology.md diff --git a/personal/projects/personal-os/wiki-curation-runs/2026-05-27.md b/personal/projects/personal-os/wiki-curation-runs/2026-05-27.md new file mode 100644 index 00000000..42d8bb05 --- /dev/null +++ b/personal/projects/personal-os/wiki-curation-runs/2026-05-27.md @@ -0,0 +1,85 @@ +--- +date: 2026-05-27 +status: complete +--- +# Wiki Curation — 2026-05-27 + +## Done + +- **Conflict scan:** No git conflict markers found in ~/obsidian/ ✅ +- **Previous run (2026-05-25):** Status = complete, no Unfinished section ✅ (no 2026-05-26 state file — that day's cron had nothing to do) +- **Orientation:** SCHEMA.md (domain + taxonomy), index.md (47 pages), log.md (65 lines) +- **Vault changes since 2026-05-26 02:00:** + - `family/documents/movies-watchlist.md` — new entries added + some marked watched; no wiki action needed + - `wiki/tech/vps-qentra.md` — Alex manually added Backup and Xray sections (commit `78da491d`); bumped `updated` to 2026-05-27 + - `family/how-to/xgimi-play6.md` — deleted (superseded by `wiki/tech/xgimi-play6-setup.md` created 2026-05-25); no action needed + - `personal/documents/fly-box.md` — personal shopping tracker; no wiki relevance + - `work/briefs/daily/2026-05-26.md`, `work/briefs/inbox/2026-05-26.md` — skipped +- **Inbox:** ~/obsidian/wiki/raw/inbox/ empty ✅ + +### Session Crystallization + +**Session:** `session_20260527_014707_b9f09b` (Eagle Zulip, 01:47–01:55) + +**Context:** Alex asked Eagle to investigate Kraken tunnel + torrent issues, +then approved moving watchlist-sync setup from Mac to Kraken. + +Key knowledge crystallized: + +1. **Router subdirectory pitfall** — `router.py` creates lowercase title + subdirectories but Transmission's `downloadDir` stays at genre-root → + 21 torrents stuck with "No data found". Fix: `torrent-set-location` to + subdirectory. Added to `tech/media-pipeline-pitfalls.md`. + +2. **Alpine container / no docker-in-docker** — `on-download-complete.sh` + runs inside Transmission's Alpine container. Docker CLI unavailable. + Solution: `sync.py` (Python3.14, ~190 lines) reads same `config.json`, + calls Transmission RPC directly. Added to `tech/media-pipeline-pitfalls.md`. + +3. **KP markdown link bug** — watchlist items with `[Title](kp.ru/...) (2025)` + syntax were passed raw to TMDB → not found → `❓` permanently. Fixed in + `watchlist-sync` commit `2db1ed9`. Added to `concepts/watchlist-automation.md`. + +4. **Watchlist cron migration Eagle→Kraken** — `watchlist-nightly` and + `watchlist-discover` moved from Eagle Hermes to Kraken Hermes. Updated + cron table + HDD paths in `concepts/watchlist-automation.md`. + +### Synthesis (MANDATORY) + +- **concepts/multi-host-cron-topology.md** (new) — Co-occurrence: `tech/hermes-eagle-mac` + + `tech/hermes-docker-kraken` + `concepts/watchlist-automation` + `personal-os-agent-rules` + (4 pages). Synthesizes Eagle vs Kraken job distribution, design principle (data locality), + migration history, and job duplication pitfall. + +### Lint + +- 0 git conflicts +- 0 broken wikilinks in new/modified pages +- Pre-existing tag issues (not introduced today): + - `concepts/watchlist-automation.md` — tags `watchlist`, `automation`, `synthesis` not in SCHEMA + - `tech/media-pipeline-pitfalls.md` — tag `torrents` not in SCHEMA + - Action: defer SCHEMA taxonomy expansion to next run (low priority, cosmetic) +- log.md: 84 lines — well below 500 rotation limit + +## Validation fixes + +none + +## Unresolved conflicts + +(none) + +## Unfinished + +(none) + +## Stats + +- Conflicts resolved: 0 +- Inbox processed: 0 (empty) +- Sessions crystallized: 1 / 1 checked +- Pages updated: 3 (media-pipeline-pitfalls, watchlist-automation, vps-qentra) +- Synthesis pages created: 1 (concepts/multi-host-cron-topology) +- Pages created: 1 (multi-host-cron-topology) +- Lint issues: 3 noted (pre-existing tag drift, no fix needed) +- Validation issues fixed: 0 diff --git a/wiki/concepts/multi-host-cron-topology.md b/wiki/concepts/multi-host-cron-topology.md new file mode 100644 index 00000000..031dccd6 --- /dev/null +++ b/wiki/concepts/multi-host-cron-topology.md @@ -0,0 +1,71 @@ +--- +title: Multi-Host Cron Topology +created: '2026-05-27' +updated: '2026-05-27' +type: concept +namespace: personal +tags: [system, agent, eagle, kraken, pipeline, sync] +confidence: medium +sources: [] +--- + +# Multi-Host Cron Topology + +Synthesis of how scheduled jobs are distributed across Eagle and Kraken. +Co-occurs in [[tech/hermes-eagle-mac]], [[tech/hermes-docker-kraken]], +[[concepts/watchlist-automation]], and [[personal-os-agent-rules]]. + +## Design Principle + +Jobs run on the host closest to their data. Eagle handles +agent-intelligence tasks (briefs, wiki curation). Kraken handles +media-pipeline tasks (watchlist, arr stack). Duplication is a bug. + +## Eagle (Mac M4) — Agent Jobs + +| Job | Schedule | Purpose | +|-----|----------|---------| +| `wiki-curation` | daily 02:00 | Crystallise vault → wiki (llm-wiki skill) | +| `vault-enrichment` | daily 03:00 | Enrich personal/family notes | +| `cross-enrichment` | daily 04:00 | Cross-namespace enrichment + memory curation | +| `proactive-research` | Sat 05:00 | Proactive gap research | +| `data-pipeline` | 30min, workdays | sync.js + generate-status.js → status.md | +| `daily-brief` | workday morning | Morning brief to Zulip | +| `inbox-triage` | workday noon | Asana inbox triage | +| `weekly-review` | Fri evening | Weekly review | + +Eagle crons are Hermes agent jobs — require LLM, vault access, +Asana context. All run via Hermes native (not Docker). See +[[tech/hermes-eagle-mac]]. + +## Kraken (RPi5) — Pipeline Jobs + +| Job | Schedule | Purpose | +|-----|----------|---------| +| `watchlist-nightly` | daily 01:00 | process-thumbs + resolve + sync-down/up | +| `watchlist-discover` | Sun 09:00 | KP API top films → watchlist Новинки | +| `on-download-complete` | event-triggered | media-pipeline: sort + Jellyfin rescan | + +Kraken crons are shell scripts or Python3 — deterministic, no LLM. +Run via Hermes cron on Kraken Hermes instance (port 8642, Docker). +See [[tech/hermes-docker-kraken]] and [[concepts/watchlist-automation]]. + +## Migration History + +- 2026-05-27: `watchlist-nightly` and `watchlist-discover` moved from + Eagle to Kraken. Eagle was running them via SSH to Kraken which was + fragile and required Eagle's network path to Kraken to be stable. + Running on Kraken directly eliminates that dependency. + +## Pitfall: Job Duplication + +If the same shell-script job runs on both Eagle and Kraken, the +Radarr/Sonarr API sees double requests. Symptom: duplicate ⬇️ +entries or double-adds. Check: `hermes cronjob list` on both hosts. + +## See Also + +- [[personal-os-agent-rules]] — Eagle's rules and allowed writes +- [[tech/hermes-eagle-mac]] — Eagle Hermes config +- [[tech/hermes-docker-kraken]] — Kraken Hermes Docker setup +- [[concepts/watchlist-automation]] — Full watchlist flow diff --git a/wiki/concepts/watchlist-automation.md b/wiki/concepts/watchlist-automation.md index d3e78c14..e63cc202 100644 --- a/wiki/concepts/watchlist-automation.md +++ b/wiki/concepts/watchlist-automation.md @@ -2,8 +2,8 @@ title: Watchlist Automation — Полный Flow namespace: personal created: '2026-05-20' -updated: '2026-05-22' -last_synced: '2026-05-22' +updated: '2026-05-27' +last_synced: '2026-05-27' type: concept tags: - kraken @@ -66,18 +66,23 @@ Jellyfin (8096) → rescan → виден контент **Важно:** `process_thumbs` автоматически определяет отступ из `## Фильмы` и сохраняет при переносе. `SECTION_THUMBDN = "## Новинки👎"` (без пробела) — должно точно совпадать с заголовком. -## Cron Jobs на Eagle +## Cron Jobs (2026-05-27: migrated to Kraken) -Два shell-script крона (не агентные — детерминированы): +Watchlist pipeline runs entirely on Kraken (migrated from Eagle 2026-05-27). +Eagle previously ran `watchlist-nightly` via SSH; now Kraken runs it locally. -| Job | ID | Расписание | Скрипт | -|-----|-----|-----------|--------| -| `watchlist-nightly` | `93619b22b330` | ежедн. 01:00 | `~/.hermes/scripts/watchlist-nightly.sh` | -| `watchlist-discover` | `8f219dfe8f0f` | вс 09:00 | `~/.hermes/scripts/watchlist-discover.sh` | +| Job | Расписание | Хост | Скрипт | +|-----|-----------|------|--------| +| `watchlist-nightly` | ежедн. 01:00 | Kraken | `~/.hermes/scripts/watchlist-nightly.sh` | +| `watchlist-discover` | вс 09:00 | Kraken | `~/.hermes/scripts/watchlist-discover.sh` | -**watchlist-nightly.sh** (Mac → Kraken → Mac): -1. `sync-vault.sh` (pull Mac) → `process-thumbs --apply` → `sync-vault.sh` (push Mac) -2. SSH Kraken: `sync-vault.sh` (pull) → `resolve --recheck --apply` → `sync-down --apply --batch 5` → `sync-up --apply` → `sync-vault.sh` (push) +**watchlist-nightly.sh** (на Kraken): +1. `sync-vault.sh` (pull) → `process-thumbs --apply` → `sync-vault.sh` (push) +2. `resolve --recheck --apply` → `sync-down --apply --batch 5` → `sync-up --apply` → `sync-vault.sh` (push) + +**Paths on Kraken HDD (UUID 49e8f586-3839-4c5d-a1e1-58bfc3579ade):** +- `/home/kraken/obsidian` → symlink to `/srv/dev-disk-by-uuid-.../obsidian` +- watchlist-sync: `/srv/dev-disk-by-uuid-.../Developer/watchlist-sync` ## Discover — Фильтрация @@ -118,6 +123,19 @@ Kopirует NFO + постер рядом с symlink (Jellyfin не видит - 🔴 rsync невыполненных фильмов с HTPC на Кракен (`Фильмы не перенесенные с htpc на кракен.md`) - 🟡 Watchlist sync-up: добавить перенесённые фильмы в нужные разделы +## Pitfalls + +### KP Markdown Link Parsing Bug +Items added from Kinopoisk with full markdown link syntax +`[Title](https://kinopoisk.ru/film/12345/) (2025)` were passed raw to TMDB +search → not found → marked `❓` permanently (`not i.ambiguous` skips on +next run). Fix: commit `2db1ed9` — extract plain title from markdown links +before passing to TMDB. Items incorrectly marked ❓ require manual `--recheck`. + +### KP API Endpoint +`api.kinopoisk.dev` redirects to `api.poiskkino.dev/v1.4/movie`. Use the +redirect destination directly to avoid DNS issues. + ## watchlist-sync — Python CLI Репо: `~/Developer/watchlist-sync` (GitHub: `mallexxx/watchlist-sync`). diff --git a/wiki/index.md b/wiki/index.md index 4c02343a..9f3e679d 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -7,7 +7,7 @@ updated: '2026-05-25' > Content catalog. Every wiki page listed with a one-line summary. > Read this first to find relevant pages for any query. -> Last updated: 2026-05-25 | Total pages: 47 +> Last updated: 2026-05-27 | Total pages: 48 ## Personal OS — Core @@ -83,6 +83,7 @@ updated: '2026-05-25' - [[concepts/kraken-media-stack]] — Full Kraken media server stack; Jellyfin, Arr, Transmission overview - [[concepts/watchlist-automation]] — Full watchlist automation flow; Radarr/Sonarr triggers to Jellyfin +- [[concepts/multi-host-cron-topology]] — Which jobs run on Eagle vs Kraken, why, and design principles ## Meta diff --git a/wiki/log.md b/wiki/log.md index 4b17a0d3..98769747 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -62,3 +62,16 @@ type: meta - Updated: index.md — 4 pages added (vps-qentra, xgimi-play6-setup, psychologist-app, multi-agent-design-patterns); count 43→47 - Lint: personal-os-architecture.md has confidence:1.0 (immutable) — skipped update check per SCHEMA rules - Validation: all 7 modified/created files passed pre-commit checks ✅ + +## [2026-05-27] update | Crystallization from session + synthesis +- Conflicts: 0 found (scan: all ~/obsidian/**/*.md) +- Inbox: empty +- Vault changes since 2026-05-26 02:00: movies-watchlist.md (new entries), vps-qentra.md (Alex added Backup + Xray sections manually), family/how-to/xgimi-play6.md deleted (superseded by wiki/tech/xgimi-play6-setup.md), work briefs (skip) +- Session crystallized: session_20260527_014707_b9f09b (Kraken tunnel investigation + media-pipeline refactor + watchlist-sync migration) +- Updated: tech/media-pipeline-pitfalls.md — 2 new pitfalls: router subdirectory→stale Transmission downloadDir, Alpine container no docker-in-docker (sync.py solution) +- Updated: concepts/watchlist-automation.md — KP markdown link parsing bug (commit 2db1ed9), cron jobs migrated Eagle→Kraken (2026-05-27), Kraken HDD paths +- Updated: tech/vps-qentra.md — bumped updated date (Alex manually added Backup + Xray sections) +- Created: concepts/multi-host-cron-topology.md — synthesis: Eagle vs Kraken job distribution, design principles, migration history, duplication pitfall +- Updated: index.md — added multi-host-cron-topology; count 47→48 +- Lint: watchlist-automation.md has tags [watchlist, automation, synthesis] not in SCHEMA taxonomy (pre-existing); torrents tag on media-pipeline-pitfalls.md also pre-existing; no broken wikilinks in new/modified pages +- Validation: all 4 modified/created pages passed pre-commit checks ✅ diff --git a/wiki/tech/media-pipeline-pitfalls.md b/wiki/tech/media-pipeline-pitfalls.md index c2099c83..d5e3d3c0 100644 --- a/wiki/tech/media-pipeline-pitfalls.md +++ b/wiki/tech/media-pipeline-pitfalls.md @@ -1,11 +1,13 @@ --- title: Media Pipeline Pitfalls created: '2026-05-23' -updated: '2026-05-23' +updated: '2026-05-27' type: tech namespace: family tags: [media-pipeline, pitfalls, torrents, infra] -related: "[[arr-stack-kraken]]" +related: + - "[[tech/arr-stack-kraken]]" + - "[[concepts/watchlist-automation]]" --- # Media Pipeline Pitfalls @@ -26,6 +28,30 @@ rename. Sequence: 2. Call `torrent-set-location` with the new path + `move=false` 3. Verify torrent goes back to seeding state +## Router Creates Subdirectories — Transmission downloadDir Stale + +`router.py` creates subdirectory per title under the genre folder +(lowercase name, e.g. `/media/cartoons/бобик в гостях у барбоса/`). +Transmission still holds the old `downloadDir` pointing to the parent +(`/media/cartoons/`). Result: torrents show "No data found" error. + +Fix: +1. Find the actual subdirectory on disk +2. Call `torrent-set-location` with the subdirectory path + `move=false` +3. Transmission verifies and resumes seeding + +## Alpine Container — No Docker-in-Docker + +`on-download-complete.sh` runs inside the Transmission Alpine container. +Docker CLI is not available there — using `docker run` inside the container +fails silently or errors. Python 3.14 IS available in the Alpine image. + +Solution: `sync.py` (Python3, ~190 lines) in the same container: +- Reads `config.json` for sorting rules + credentials (same file as linker container) +- Queries TMDB for genre routing +- Creates hardlinks + calls Transmission RPC `torrent-set-location` +Location: `/srv/dev-disk-by-uuid-49e8f586-3839-4c5d-a1e1-58bfc3579ade/docker/media-pipeline/sync.py` + ## General Rules - Never touch source files/folders (media-pipeline USER.md rule) diff --git a/wiki/tech/vps-qentra.md b/wiki/tech/vps-qentra.md index 5fec5a76..b53cc32c 100644 --- a/wiki/tech/vps-qentra.md +++ b/wiki/tech/vps-qentra.md @@ -1,7 +1,7 @@ --- title: VPS qentra.top created: '2026-05-24' -updated: '2026-05-25' +updated: '2026-05-27' type: tech namespace: personal tags: [infra, vps]