[2026-05-27] wiki curation: multi-host cron topology, media pipeline pitfalls, watchlist automation updates
This commit is contained in:
Alexey Martemyanov
@@ -0,0 +1,85 @@
|
||||
---
|
||||
date: 2026-05-27
|
||||
status: complete
|
||||
---
|
||||
# Wiki Curation — 2026-05-27
|
||||
|
||||
## Done
|
||||
|
||||
- **Conflict scan:** No git conflict markers found in ~/obsidian/ ✅
|
||||
- **Previous run (2026-05-25):** Status = complete, no Unfinished section ✅ (no 2026-05-26 state file — that day's cron had nothing to do)
|
||||
- **Orientation:** SCHEMA.md (domain + taxonomy), index.md (47 pages), log.md (65 lines)
|
||||
- **Vault changes since 2026-05-26 02:00:**
|
||||
- `family/documents/movies-watchlist.md` — new entries added + some marked watched; no wiki action needed
|
||||
- `wiki/tech/vps-qentra.md` — Alex manually added Backup and Xray sections (commit `78da491d`); bumped `updated` to 2026-05-27
|
||||
- `family/how-to/xgimi-play6.md` — deleted (superseded by `wiki/tech/xgimi-play6-setup.md` created 2026-05-25); no action needed
|
||||
- `personal/documents/fly-box.md` — personal shopping tracker; no wiki relevance
|
||||
- `work/briefs/daily/2026-05-26.md`, `work/briefs/inbox/2026-05-26.md` — skipped
|
||||
- **Inbox:** ~/obsidian/wiki/raw/inbox/ empty ✅
|
||||
|
||||
### Session Crystallization
|
||||
|
||||
**Session:** `session_20260527_014707_b9f09b` (Eagle Zulip, 01:47–01:55)
|
||||
|
||||
**Context:** Alex asked Eagle to investigate Kraken tunnel + torrent issues,
|
||||
then approved moving watchlist-sync setup from Mac to Kraken.
|
||||
|
||||
Key knowledge crystallized:
|
||||
|
||||
1. **Router subdirectory pitfall** — `router.py` creates lowercase title
|
||||
subdirectories but Transmission's `downloadDir` stays at genre-root →
|
||||
21 torrents stuck with "No data found". Fix: `torrent-set-location` to
|
||||
subdirectory. Added to `tech/media-pipeline-pitfalls.md`.
|
||||
|
||||
2. **Alpine container / no docker-in-docker** — `on-download-complete.sh`
|
||||
runs inside Transmission's Alpine container. Docker CLI unavailable.
|
||||
Solution: `sync.py` (Python3.14, ~190 lines) reads same `config.json`,
|
||||
calls Transmission RPC directly. Added to `tech/media-pipeline-pitfalls.md`.
|
||||
|
||||
3. **KP markdown link bug** — watchlist items with `[Title](kp.ru/...) (2025)`
|
||||
syntax were passed raw to TMDB → not found → `❓` permanently. Fixed in
|
||||
`watchlist-sync` commit `2db1ed9`. Added to `concepts/watchlist-automation.md`.
|
||||
|
||||
4. **Watchlist cron migration Eagle→Kraken** — `watchlist-nightly` and
|
||||
`watchlist-discover` moved from Eagle Hermes to Kraken Hermes. Updated
|
||||
cron table + HDD paths in `concepts/watchlist-automation.md`.
|
||||
|
||||
### Synthesis (MANDATORY)
|
||||
|
||||
- **concepts/multi-host-cron-topology.md** (new) — Co-occurrence: `tech/hermes-eagle-mac`
|
||||
+ `tech/hermes-docker-kraken` + `concepts/watchlist-automation` + `personal-os-agent-rules`
|
||||
(4 pages). Synthesizes Eagle vs Kraken job distribution, design principle (data locality),
|
||||
migration history, and job duplication pitfall.
|
||||
|
||||
### Lint
|
||||
|
||||
- 0 git conflicts
|
||||
- 0 broken wikilinks in new/modified pages
|
||||
- Pre-existing tag issues (not introduced today):
|
||||
- `concepts/watchlist-automation.md` — tags `watchlist`, `automation`, `synthesis` not in SCHEMA
|
||||
- `tech/media-pipeline-pitfalls.md` — tag `torrents` not in SCHEMA
|
||||
- Action: defer SCHEMA taxonomy expansion to next run (low priority, cosmetic)
|
||||
- log.md: 84 lines — well below 500 rotation limit
|
||||
|
||||
## Validation fixes
|
||||
|
||||
none
|
||||
|
||||
## Unresolved conflicts
|
||||
|
||||
(none)
|
||||
|
||||
## Unfinished
|
||||
|
||||
(none)
|
||||
|
||||
## Stats
|
||||
|
||||
- Conflicts resolved: 0
|
||||
- Inbox processed: 0 (empty)
|
||||
- Sessions crystallized: 1 / 1 checked
|
||||
- Pages updated: 3 (media-pipeline-pitfalls, watchlist-automation, vps-qentra)
|
||||
- Synthesis pages created: 1 (concepts/multi-host-cron-topology)
|
||||
- Pages created: 1 (multi-host-cron-topology)
|
||||
- Lint issues: 3 noted (pre-existing tag drift, no fix needed)
|
||||
- Validation issues fixed: 0
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: Multi-Host Cron Topology
|
||||
created: '2026-05-27'
|
||||
updated: '2026-05-27'
|
||||
type: concept
|
||||
namespace: personal
|
||||
tags: [system, agent, eagle, kraken, pipeline, sync]
|
||||
confidence: medium
|
||||
sources: []
|
||||
---
|
||||
|
||||
# Multi-Host Cron Topology
|
||||
|
||||
Synthesis of how scheduled jobs are distributed across Eagle and Kraken.
|
||||
Co-occurs in [[tech/hermes-eagle-mac]], [[tech/hermes-docker-kraken]],
|
||||
[[concepts/watchlist-automation]], and [[personal-os-agent-rules]].
|
||||
|
||||
## Design Principle
|
||||
|
||||
Jobs run on the host closest to their data. Eagle handles
|
||||
agent-intelligence tasks (briefs, wiki curation). Kraken handles
|
||||
media-pipeline tasks (watchlist, arr stack). Duplication is a bug.
|
||||
|
||||
## Eagle (Mac M4) — Agent Jobs
|
||||
|
||||
| Job | Schedule | Purpose |
|
||||
|-----|----------|---------|
|
||||
| `wiki-curation` | daily 02:00 | Crystallise vault → wiki (llm-wiki skill) |
|
||||
| `vault-enrichment` | daily 03:00 | Enrich personal/family notes |
|
||||
| `cross-enrichment` | daily 04:00 | Cross-namespace enrichment + memory curation |
|
||||
| `proactive-research` | Sat 05:00 | Proactive gap research |
|
||||
| `data-pipeline` | 30min, workdays | sync.js + generate-status.js → status.md |
|
||||
| `daily-brief` | workday morning | Morning brief to Zulip |
|
||||
| `inbox-triage` | workday noon | Asana inbox triage |
|
||||
| `weekly-review` | Fri evening | Weekly review |
|
||||
|
||||
Eagle crons are Hermes agent jobs — require LLM, vault access,
|
||||
Asana context. All run via Hermes native (not Docker). See
|
||||
[[tech/hermes-eagle-mac]].
|
||||
|
||||
## Kraken (RPi5) — Pipeline Jobs
|
||||
|
||||
| Job | Schedule | Purpose |
|
||||
|-----|----------|---------|
|
||||
| `watchlist-nightly` | daily 01:00 | process-thumbs + resolve + sync-down/up |
|
||||
| `watchlist-discover` | Sun 09:00 | KP API top films → watchlist Новинки |
|
||||
| `on-download-complete` | event-triggered | media-pipeline: sort + Jellyfin rescan |
|
||||
|
||||
Kraken crons are shell scripts or Python3 — deterministic, no LLM.
|
||||
Run via Hermes cron on Kraken Hermes instance (port 8642, Docker).
|
||||
See [[tech/hermes-docker-kraken]] and [[concepts/watchlist-automation]].
|
||||
|
||||
## Migration History
|
||||
|
||||
- 2026-05-27: `watchlist-nightly` and `watchlist-discover` moved from
|
||||
Eagle to Kraken. Eagle was running them via SSH to Kraken which was
|
||||
fragile and required Eagle's network path to Kraken to be stable.
|
||||
Running on Kraken directly eliminates that dependency.
|
||||
|
||||
## Pitfall: Job Duplication
|
||||
|
||||
If the same shell-script job runs on both Eagle and Kraken, the
|
||||
Radarr/Sonarr API sees double requests. Symptom: duplicate ⬇️
|
||||
entries or double-adds. Check: `hermes cronjob list` on both hosts.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[personal-os-agent-rules]] — Eagle's rules and allowed writes
|
||||
- [[tech/hermes-eagle-mac]] — Eagle Hermes config
|
||||
- [[tech/hermes-docker-kraken]] — Kraken Hermes Docker setup
|
||||
- [[concepts/watchlist-automation]] — Full watchlist flow
|
||||
@@ -2,8 +2,8 @@
|
||||
title: Watchlist Automation — Полный Flow
|
||||
namespace: personal
|
||||
created: '2026-05-20'
|
||||
updated: '2026-05-22'
|
||||
last_synced: '2026-05-22'
|
||||
updated: '2026-05-27'
|
||||
last_synced: '2026-05-27'
|
||||
type: concept
|
||||
tags:
|
||||
- kraken
|
||||
@@ -66,18 +66,23 @@ Jellyfin (8096) → rescan → виден контент
|
||||
|
||||
**Важно:** `process_thumbs` автоматически определяет отступ из `## Фильмы` и сохраняет при переносе. `SECTION_THUMBDN = "## Новинки👎"` (без пробела) — должно точно совпадать с заголовком.
|
||||
|
||||
## Cron Jobs на Eagle
|
||||
## Cron Jobs (2026-05-27: migrated to Kraken)
|
||||
|
||||
Два shell-script крона (не агентные — детерминированы):
|
||||
Watchlist pipeline runs entirely on Kraken (migrated from Eagle 2026-05-27).
|
||||
Eagle previously ran `watchlist-nightly` via SSH; now Kraken runs it locally.
|
||||
|
||||
| Job | ID | Расписание | Скрипт |
|
||||
|-----|-----|-----------|--------|
|
||||
| `watchlist-nightly` | `93619b22b330` | ежедн. 01:00 | `~/.hermes/scripts/watchlist-nightly.sh` |
|
||||
| `watchlist-discover` | `8f219dfe8f0f` | вс 09:00 | `~/.hermes/scripts/watchlist-discover.sh` |
|
||||
| Job | Расписание | Хост | Скрипт |
|
||||
|-----|-----------|------|--------|
|
||||
| `watchlist-nightly` | ежедн. 01:00 | Kraken | `~/.hermes/scripts/watchlist-nightly.sh` |
|
||||
| `watchlist-discover` | вс 09:00 | Kraken | `~/.hermes/scripts/watchlist-discover.sh` |
|
||||
|
||||
**watchlist-nightly.sh** (Mac → Kraken → Mac):
|
||||
1. `sync-vault.sh` (pull Mac) → `process-thumbs --apply` → `sync-vault.sh` (push Mac)
|
||||
2. SSH Kraken: `sync-vault.sh` (pull) → `resolve --recheck --apply` → `sync-down --apply --batch 5` → `sync-up --apply` → `sync-vault.sh` (push)
|
||||
**watchlist-nightly.sh** (на Kraken):
|
||||
1. `sync-vault.sh` (pull) → `process-thumbs --apply` → `sync-vault.sh` (push)
|
||||
2. `resolve --recheck --apply` → `sync-down --apply --batch 5` → `sync-up --apply` → `sync-vault.sh` (push)
|
||||
|
||||
**Paths on Kraken HDD (UUID 49e8f586-3839-4c5d-a1e1-58bfc3579ade):**
|
||||
- `/home/kraken/obsidian` → symlink to `/srv/dev-disk-by-uuid-.../obsidian`
|
||||
- watchlist-sync: `/srv/dev-disk-by-uuid-.../Developer/watchlist-sync`
|
||||
|
||||
## Discover — Фильтрация
|
||||
|
||||
@@ -118,6 +123,19 @@ Kopirует NFO + постер рядом с symlink (Jellyfin не видит
|
||||
- 🔴 rsync невыполненных фильмов с HTPC на Кракен (`Фильмы не перенесенные с htpc на кракен.md`)
|
||||
- 🟡 Watchlist sync-up: добавить перенесённые фильмы в нужные разделы
|
||||
|
||||
## Pitfalls
|
||||
|
||||
### KP Markdown Link Parsing Bug
|
||||
Items added from Kinopoisk with full markdown link syntax
|
||||
`[Title](https://kinopoisk.ru/film/12345/) (2025)` were passed raw to TMDB
|
||||
search → not found → marked `❓` permanently (`not i.ambiguous` skips on
|
||||
next run). Fix: commit `2db1ed9` — extract plain title from markdown links
|
||||
before passing to TMDB. Items incorrectly marked ❓ require manual `--recheck`.
|
||||
|
||||
### KP API Endpoint
|
||||
`api.kinopoisk.dev` redirects to `api.poiskkino.dev/v1.4/movie`. Use the
|
||||
redirect destination directly to avoid DNS issues.
|
||||
|
||||
## watchlist-sync — Python CLI
|
||||
|
||||
Репо: `~/Developer/watchlist-sync` (GitHub: `mallexxx/watchlist-sync`).
|
||||
|
||||
+2
-1
@@ -7,7 +7,7 @@ updated: '2026-05-25'
|
||||
|
||||
> Content catalog. Every wiki page listed with a one-line summary.
|
||||
> Read this first to find relevant pages for any query.
|
||||
> Last updated: 2026-05-25 | Total pages: 47
|
||||
> Last updated: 2026-05-27 | Total pages: 48
|
||||
|
||||
## Personal OS — Core
|
||||
|
||||
@@ -83,6 +83,7 @@ updated: '2026-05-25'
|
||||
|
||||
- [[concepts/kraken-media-stack]] — Full Kraken media server stack; Jellyfin, Arr, Transmission overview
|
||||
- [[concepts/watchlist-automation]] — Full watchlist automation flow; Radarr/Sonarr triggers to Jellyfin
|
||||
- [[concepts/multi-host-cron-topology]] — Which jobs run on Eagle vs Kraken, why, and design principles
|
||||
|
||||
## Meta
|
||||
|
||||
|
||||
+13
@@ -62,3 +62,16 @@ type: meta
|
||||
- Updated: index.md — 4 pages added (vps-qentra, xgimi-play6-setup, psychologist-app, multi-agent-design-patterns); count 43→47
|
||||
- Lint: personal-os-architecture.md has confidence:1.0 (immutable) — skipped update check per SCHEMA rules
|
||||
- Validation: all 7 modified/created files passed pre-commit checks ✅
|
||||
|
||||
## [2026-05-27] update | Crystallization from session + synthesis
|
||||
- Conflicts: 0 found (scan: all ~/obsidian/**/*.md)
|
||||
- Inbox: empty
|
||||
- Vault changes since 2026-05-26 02:00: movies-watchlist.md (new entries), vps-qentra.md (Alex added Backup + Xray sections manually), family/how-to/xgimi-play6.md deleted (superseded by wiki/tech/xgimi-play6-setup.md), work briefs (skip)
|
||||
- Session crystallized: session_20260527_014707_b9f09b (Kraken tunnel investigation + media-pipeline refactor + watchlist-sync migration)
|
||||
- Updated: tech/media-pipeline-pitfalls.md — 2 new pitfalls: router subdirectory→stale Transmission downloadDir, Alpine container no docker-in-docker (sync.py solution)
|
||||
- Updated: concepts/watchlist-automation.md — KP markdown link parsing bug (commit 2db1ed9), cron jobs migrated Eagle→Kraken (2026-05-27), Kraken HDD paths
|
||||
- Updated: tech/vps-qentra.md — bumped updated date (Alex manually added Backup + Xray sections)
|
||||
- Created: concepts/multi-host-cron-topology.md — synthesis: Eagle vs Kraken job distribution, design principles, migration history, duplication pitfall
|
||||
- Updated: index.md — added multi-host-cron-topology; count 47→48
|
||||
- Lint: watchlist-automation.md has tags [watchlist, automation, synthesis] not in SCHEMA taxonomy (pre-existing); torrents tag on media-pipeline-pitfalls.md also pre-existing; no broken wikilinks in new/modified pages
|
||||
- Validation: all 4 modified/created pages passed pre-commit checks ✅
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
---
|
||||
title: Media Pipeline Pitfalls
|
||||
created: '2026-05-23'
|
||||
updated: '2026-05-23'
|
||||
updated: '2026-05-27'
|
||||
type: tech
|
||||
namespace: family
|
||||
tags: [media-pipeline, pitfalls, torrents, infra]
|
||||
related: "[[arr-stack-kraken]]"
|
||||
related:
|
||||
- "[[tech/arr-stack-kraken]]"
|
||||
- "[[concepts/watchlist-automation]]"
|
||||
---
|
||||
|
||||
# Media Pipeline Pitfalls
|
||||
@@ -26,6 +28,30 @@ rename. Sequence:
|
||||
2. Call `torrent-set-location` with the new path + `move=false`
|
||||
3. Verify torrent goes back to seeding state
|
||||
|
||||
## Router Creates Subdirectories — Transmission downloadDir Stale
|
||||
|
||||
`router.py` creates subdirectory per title under the genre folder
|
||||
(lowercase name, e.g. `/media/cartoons/бобик в гостях у барбоса/`).
|
||||
Transmission still holds the old `downloadDir` pointing to the parent
|
||||
(`/media/cartoons/`). Result: torrents show "No data found" error.
|
||||
|
||||
Fix:
|
||||
1. Find the actual subdirectory on disk
|
||||
2. Call `torrent-set-location` with the subdirectory path + `move=false`
|
||||
3. Transmission verifies and resumes seeding
|
||||
|
||||
## Alpine Container — No Docker-in-Docker
|
||||
|
||||
`on-download-complete.sh` runs inside the Transmission Alpine container.
|
||||
Docker CLI is not available there — using `docker run` inside the container
|
||||
fails silently or errors. Python 3.14 IS available in the Alpine image.
|
||||
|
||||
Solution: `sync.py` (Python3, ~190 lines) in the same container:
|
||||
- Reads `config.json` for sorting rules + credentials (same file as linker container)
|
||||
- Queries TMDB for genre routing
|
||||
- Creates hardlinks + calls Transmission RPC `torrent-set-location`
|
||||
Location: `/srv/dev-disk-by-uuid-49e8f586-3839-4c5d-a1e1-58bfc3579ade/docker/media-pipeline/sync.py`
|
||||
|
||||
## General Rules
|
||||
|
||||
- Never touch source files/folders (media-pipeline USER.md rule)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: VPS qentra.top
|
||||
created: '2026-05-24'
|
||||
updated: '2026-05-25'
|
||||
updated: '2026-05-27'
|
||||
type: tech
|
||||
namespace: personal
|
||||
tags: [infra, vps]
|
||||
|
||||
Reference in New Issue
Block a user