[2026-05-27] wiki curation: multi-host cron topology, media pipeline pitfalls, watchlist automation updates
This commit is contained in:
Alexey Martemyanov
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: Multi-Host Cron Topology
|
||||
created: '2026-05-27'
|
||||
updated: '2026-05-27'
|
||||
type: concept
|
||||
namespace: personal
|
||||
tags: [system, agent, eagle, kraken, pipeline, sync]
|
||||
confidence: medium
|
||||
sources: []
|
||||
---
|
||||
|
||||
# Multi-Host Cron Topology
|
||||
|
||||
Synthesis of how scheduled jobs are distributed across Eagle and Kraken.
|
||||
Co-occurs in [[tech/hermes-eagle-mac]], [[tech/hermes-docker-kraken]],
|
||||
[[concepts/watchlist-automation]], and [[personal-os-agent-rules]].
|
||||
|
||||
## Design Principle
|
||||
|
||||
Jobs run on the host closest to their data. Eagle handles
|
||||
agent-intelligence tasks (briefs, wiki curation). Kraken handles
|
||||
media-pipeline tasks (watchlist, arr stack). Duplication is a bug.
|
||||
|
||||
## Eagle (Mac M4) — Agent Jobs
|
||||
|
||||
| Job | Schedule | Purpose |
|
||||
|-----|----------|---------|
|
||||
| `wiki-curation` | daily 02:00 | Crystallise vault → wiki (llm-wiki skill) |
|
||||
| `vault-enrichment` | daily 03:00 | Enrich personal/family notes |
|
||||
| `cross-enrichment` | daily 04:00 | Cross-namespace enrichment + memory curation |
|
||||
| `proactive-research` | Sat 05:00 | Proactive gap research |
|
||||
| `data-pipeline` | 30min, workdays | sync.js + generate-status.js → status.md |
|
||||
| `daily-brief` | workday morning | Morning brief to Zulip |
|
||||
| `inbox-triage` | workday noon | Asana inbox triage |
|
||||
| `weekly-review` | Fri evening | Weekly review |
|
||||
|
||||
Eagle crons are Hermes agent jobs — require LLM, vault access,
|
||||
Asana context. All run via Hermes native (not Docker). See
|
||||
[[tech/hermes-eagle-mac]].
|
||||
|
||||
## Kraken (RPi5) — Pipeline Jobs
|
||||
|
||||
| Job | Schedule | Purpose |
|
||||
|-----|----------|---------|
|
||||
| `watchlist-nightly` | daily 01:00 | process-thumbs + resolve + sync-down/up |
|
||||
| `watchlist-discover` | Sun 09:00 | KP API top films → watchlist Новинки |
|
||||
| `on-download-complete` | event-triggered | media-pipeline: sort + Jellyfin rescan |
|
||||
|
||||
Kraken crons are shell scripts or Python3 — deterministic, no LLM.
|
||||
Run via Hermes cron on Kraken Hermes instance (port 8642, Docker).
|
||||
See [[tech/hermes-docker-kraken]] and [[concepts/watchlist-automation]].
|
||||
|
||||
## Migration History
|
||||
|
||||
- 2026-05-27: `watchlist-nightly` and `watchlist-discover` moved from
|
||||
Eagle to Kraken. Eagle was running them via SSH to Kraken which was
|
||||
fragile and required Eagle's network path to Kraken to be stable.
|
||||
Running on Kraken directly eliminates that dependency.
|
||||
|
||||
## Pitfall: Job Duplication
|
||||
|
||||
If the same shell-script job runs on both Eagle and Kraken, the
|
||||
Radarr/Sonarr API sees double requests. Symptom: duplicate ⬇️
|
||||
entries or double-adds. Check: `hermes cronjob list` on both hosts.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[personal-os-agent-rules]] — Eagle's rules and allowed writes
|
||||
- [[tech/hermes-eagle-mac]] — Eagle Hermes config
|
||||
- [[tech/hermes-docker-kraken]] — Kraken Hermes Docker setup
|
||||
- [[concepts/watchlist-automation]] — Full watchlist flow
|
||||
@@ -2,8 +2,8 @@
|
||||
title: Watchlist Automation — Полный Flow
|
||||
namespace: personal
|
||||
created: '2026-05-20'
|
||||
updated: '2026-05-22'
|
||||
last_synced: '2026-05-22'
|
||||
updated: '2026-05-27'
|
||||
last_synced: '2026-05-27'
|
||||
type: concept
|
||||
tags:
|
||||
- kraken
|
||||
@@ -66,18 +66,23 @@ Jellyfin (8096) → rescan → виден контент
|
||||
|
||||
**Важно:** `process_thumbs` автоматически определяет отступ из `## Фильмы` и сохраняет при переносе. `SECTION_THUMBDN = "## Новинки👎"` (без пробела) — должно точно совпадать с заголовком.
|
||||
|
||||
## Cron Jobs на Eagle
|
||||
## Cron Jobs (2026-05-27: migrated to Kraken)
|
||||
|
||||
Два shell-script крона (не агентные — детерминированы):
|
||||
Watchlist pipeline runs entirely on Kraken (migrated from Eagle 2026-05-27).
|
||||
Eagle previously ran `watchlist-nightly` via SSH; now Kraken runs it locally.
|
||||
|
||||
| Job | ID | Расписание | Скрипт |
|
||||
|-----|-----|-----------|--------|
|
||||
| `watchlist-nightly` | `93619b22b330` | ежедн. 01:00 | `~/.hermes/scripts/watchlist-nightly.sh` |
|
||||
| `watchlist-discover` | `8f219dfe8f0f` | вс 09:00 | `~/.hermes/scripts/watchlist-discover.sh` |
|
||||
| Job | Расписание | Хост | Скрипт |
|
||||
|-----|-----------|------|--------|
|
||||
| `watchlist-nightly` | ежедн. 01:00 | Kraken | `~/.hermes/scripts/watchlist-nightly.sh` |
|
||||
| `watchlist-discover` | вс 09:00 | Kraken | `~/.hermes/scripts/watchlist-discover.sh` |
|
||||
|
||||
**watchlist-nightly.sh** (Mac → Kraken → Mac):
|
||||
1. `sync-vault.sh` (pull Mac) → `process-thumbs --apply` → `sync-vault.sh` (push Mac)
|
||||
2. SSH Kraken: `sync-vault.sh` (pull) → `resolve --recheck --apply` → `sync-down --apply --batch 5` → `sync-up --apply` → `sync-vault.sh` (push)
|
||||
**watchlist-nightly.sh** (на Kraken):
|
||||
1. `sync-vault.sh` (pull) → `process-thumbs --apply` → `sync-vault.sh` (push)
|
||||
2. `resolve --recheck --apply` → `sync-down --apply --batch 5` → `sync-up --apply` → `sync-vault.sh` (push)
|
||||
|
||||
**Paths on Kraken HDD (UUID 49e8f586-3839-4c5d-a1e1-58bfc3579ade):**
|
||||
- `/home/kraken/obsidian` → symlink to `/srv/dev-disk-by-uuid-.../obsidian`
|
||||
- watchlist-sync: `/srv/dev-disk-by-uuid-.../Developer/watchlist-sync`
|
||||
|
||||
## Discover — Фильтрация
|
||||
|
||||
@@ -118,6 +123,19 @@ Kopirует NFO + постер рядом с symlink (Jellyfin не видит
|
||||
- 🔴 rsync невыполненных фильмов с HTPC на Кракен (`Фильмы не перенесенные с htpc на кракен.md`)
|
||||
- 🟡 Watchlist sync-up: добавить перенесённые фильмы в нужные разделы
|
||||
|
||||
## Pitfalls
|
||||
|
||||
### KP Markdown Link Parsing Bug
|
||||
Items added from Kinopoisk with full markdown link syntax
|
||||
`[Title](https://kinopoisk.ru/film/12345/) (2025)` were passed raw to TMDB
|
||||
search → not found → marked `❓` permanently (`not i.ambiguous` skips on
|
||||
next run). Fix: commit `2db1ed9` — extract plain title from markdown links
|
||||
before passing to TMDB. Items incorrectly marked ❓ require manual `--recheck`.
|
||||
|
||||
### KP API Endpoint
|
||||
`api.kinopoisk.dev` redirects to `api.poiskkino.dev/v1.4/movie`. Use the
|
||||
redirect destination directly to avoid DNS issues.
|
||||
|
||||
## watchlist-sync — Python CLI
|
||||
|
||||
Репо: `~/Developer/watchlist-sync` (GitHub: `mallexxx/watchlist-sync`).
|
||||
|
||||
+2
-1
@@ -7,7 +7,7 @@ updated: '2026-05-25'
|
||||
|
||||
> Content catalog. Every wiki page listed with a one-line summary.
|
||||
> Read this first to find relevant pages for any query.
|
||||
> Last updated: 2026-05-25 | Total pages: 47
|
||||
> Last updated: 2026-05-27 | Total pages: 48
|
||||
|
||||
## Personal OS — Core
|
||||
|
||||
@@ -83,6 +83,7 @@ updated: '2026-05-25'
|
||||
|
||||
- [[concepts/kraken-media-stack]] — Full Kraken media server stack; Jellyfin, Arr, Transmission overview
|
||||
- [[concepts/watchlist-automation]] — Full watchlist automation flow; Radarr/Sonarr triggers to Jellyfin
|
||||
- [[concepts/multi-host-cron-topology]] — Which jobs run on Eagle vs Kraken, why, and design principles
|
||||
|
||||
## Meta
|
||||
|
||||
|
||||
+13
@@ -62,3 +62,16 @@ type: meta
|
||||
- Updated: index.md — 4 pages added (vps-qentra, xgimi-play6-setup, psychologist-app, multi-agent-design-patterns); count 43→47
|
||||
- Lint: personal-os-architecture.md has confidence:1.0 (immutable) — skipped update check per SCHEMA rules
|
||||
- Validation: all 7 modified/created files passed pre-commit checks ✅
|
||||
|
||||
## [2026-05-27] update | Crystallization from session + synthesis
|
||||
- Conflicts: 0 found (scan: all ~/obsidian/**/*.md)
|
||||
- Inbox: empty
|
||||
- Vault changes since 2026-05-26 02:00: movies-watchlist.md (new entries), vps-qentra.md (Alex added Backup + Xray sections manually), family/how-to/xgimi-play6.md deleted (superseded by wiki/tech/xgimi-play6-setup.md), work briefs (skip)
|
||||
- Session crystallized: session_20260527_014707_b9f09b (Kraken tunnel investigation + media-pipeline refactor + watchlist-sync migration)
|
||||
- Updated: tech/media-pipeline-pitfalls.md — 2 new pitfalls: router subdirectory→stale Transmission downloadDir, Alpine container no docker-in-docker (sync.py solution)
|
||||
- Updated: concepts/watchlist-automation.md — KP markdown link parsing bug (commit 2db1ed9), cron jobs migrated Eagle→Kraken (2026-05-27), Kraken HDD paths
|
||||
- Updated: tech/vps-qentra.md — bumped updated date (Alex manually added Backup + Xray sections)
|
||||
- Created: concepts/multi-host-cron-topology.md — synthesis: Eagle vs Kraken job distribution, design principles, migration history, duplication pitfall
|
||||
- Updated: index.md — added multi-host-cron-topology; count 47→48
|
||||
- Lint: watchlist-automation.md has tags [watchlist, automation, synthesis] not in SCHEMA taxonomy (pre-existing); torrents tag on media-pipeline-pitfalls.md also pre-existing; no broken wikilinks in new/modified pages
|
||||
- Validation: all 4 modified/created pages passed pre-commit checks ✅
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
---
|
||||
title: Media Pipeline Pitfalls
|
||||
created: '2026-05-23'
|
||||
updated: '2026-05-23'
|
||||
updated: '2026-05-27'
|
||||
type: tech
|
||||
namespace: family
|
||||
tags: [media-pipeline, pitfalls, torrents, infra]
|
||||
related: "[[arr-stack-kraken]]"
|
||||
related:
|
||||
- "[[tech/arr-stack-kraken]]"
|
||||
- "[[concepts/watchlist-automation]]"
|
||||
---
|
||||
|
||||
# Media Pipeline Pitfalls
|
||||
@@ -26,6 +28,30 @@ rename. Sequence:
|
||||
2. Call `torrent-set-location` with the new path + `move=false`
|
||||
3. Verify torrent goes back to seeding state
|
||||
|
||||
## Router Creates Subdirectories — Transmission downloadDir Stale
|
||||
|
||||
`router.py` creates subdirectory per title under the genre folder
|
||||
(lowercase name, e.g. `/media/cartoons/бобик в гостях у барбоса/`).
|
||||
Transmission still holds the old `downloadDir` pointing to the parent
|
||||
(`/media/cartoons/`). Result: torrents show "No data found" error.
|
||||
|
||||
Fix:
|
||||
1. Find the actual subdirectory on disk
|
||||
2. Call `torrent-set-location` with the subdirectory path + `move=false`
|
||||
3. Transmission verifies and resumes seeding
|
||||
|
||||
## Alpine Container — No Docker-in-Docker
|
||||
|
||||
`on-download-complete.sh` runs inside the Transmission Alpine container.
|
||||
Docker CLI is not available there — using `docker run` inside the container
|
||||
fails silently or errors. Python 3.14 IS available in the Alpine image.
|
||||
|
||||
Solution: `sync.py` (Python3, ~190 lines) in the same container:
|
||||
- Reads `config.json` for sorting rules + credentials (same file as linker container)
|
||||
- Queries TMDB for genre routing
|
||||
- Creates hardlinks + calls Transmission RPC `torrent-set-location`
|
||||
Location: `/srv/dev-disk-by-uuid-49e8f586-3839-4c5d-a1e1-58bfc3579ade/docker/media-pipeline/sync.py`
|
||||
|
||||
## General Rules
|
||||
|
||||
- Never touch source files/folders (media-pipeline USER.md rule)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: VPS qentra.top
|
||||
created: '2026-05-24'
|
||||
updated: '2026-05-25'
|
||||
updated: '2026-05-27'
|
||||
type: tech
|
||||
namespace: personal
|
||||
tags: [infra, vps]
|
||||
|
||||
Reference in New Issue
Block a user