obsidian-vault/wiki/personal-os-sync-pipeline.md

namespace, tags, last_synced, confidence

namespace	tags	last_synced	confidence
work	system sync pipeline debugging	2026-04-27	1.0

Personal OS — Sync Pipeline

Components

sync.js

Pulls from 4 Asana sources into PostgreSQL. Run: node ~/Developer/personal-os/sync.js

Sources fetched:

1. fetchMyTasks() — My Tasks sections by GID (section-by-section)
2. fetchFollowingTasks() — tasks I follow, 7-day chunks to work around 100-result cap
3. fetchDelegatedTasks() — created by me, assigned to others
4. fetchProjectTasks() — tasks from projects I'm a member of (lookback window on full sync)
5. sweepRecentlyCompleted() — catch completed tasks missed by above

Incremental sync: uses modified_since cursor from sync_state table. Full sync: triggered when full_sync_at is older than configured threshold.

generate-status.js

Reads from PostgreSQL → writes asana_context.md + status.md. Run: node ~/Developer/personal-os/generate-status.js

run-pipeline.sh

Wrapper: runs sync.js + generate-status.js + healthcheck. ~/scripts/run-pipeline.sh

Healthcheck: asana_context.md must be < 40 minutes old. Exits 1 if stale.

Known Bugs (Fixed)

possibly_deleted + overdue tasks appearing

Root cause: Full sync marks tasks absent from Asana response as possibly_deleted=true. These tasks remained completed=false, causing them to appear overdue.

Fix 1 — generate-status.js: Added AND t.possibly_deleted = FALSE to all 5 active-task queries.

Fix 2 — sync.js: Added sweepRecentlyCompleted() called after syncMyTasks(). Uses searchTasksForWorkspace(completed:true, completed_at.after=cursor) to catch tasks that vanished from sections (because completed tasks are excluded from section membership results).

Completion gap

getTasksForSection drops completed tasks. fetchFollowingTasks has completed:false filter. → Completion events never reach DB incrementally. → sweepRecentlyCompleted() fills this gap.

Debugging Checklist

Pipeline seems stuck:

bash ~/scripts/run-pipeline.sh
# Check exit code and output

Check when last sync ran:

SELECT key, last_sync_at, full_sync_at FROM sync_state ORDER BY last_sync_at DESC;

Check possibly_deleted tasks (should be 0 active):

SELECT count(*) FROM tasks WHERE possibly_deleted=true AND completed=false;

Check asana_context.md freshness:

stat -f "%Sm" ~/Developer/personal-os/asana_context.md

Manual full sync (reset cursor):

UPDATE sync_state SET cursor = null, full_sync_at = null WHERE key = 'my_tasks';

Then run node sync.js.

Token Management

ASANA_API_KEY lives in two places — both must be in sync:

• ~/.zprofile — source of truth (export ASANA_API_KEY="...")
• ~/.config/personal-os/env — loaded by run-pipeline.sh via . ~/.config/personal-os/env

If the token is rotated in Asana, update .zprofile first, then sync to env:

NEW_TOKEN=$(grep ASANA_API_KEY ~/.zprofile | sed 's/export ASANA_API_KEY="//;s/"//');
echo "export ASANA_API_KEY=$NEW_TOKEN" > ~/.config/personal-os/env

Verify: curl -s -o /dev/null -w "%{http_code}" "https://app.asana.com/api/1.0/users/me" -H "Authorization: Bearer $NEW_TOKEN" → should return 200.

Symptom of stale token: data-pipeline and inbox-check cron jobs get paused (Hermes auto-pauses on repeated failures), status.md stops updating.

Files

• ~/Developer/personal-os/sync.js — main sync script
• ~/Developer/personal-os/generate-status.js — status generation
• ~/Developer/personal-os/asana.js — thin Asana API client
• ~/Developer/personal-os/config.json — workspace GID, user GID, section GIDs
• ~/.config/personal-os/env — ASANA_API_KEY (sync from .zprofile when rotating)

Related

personal-os-architecture personal-os-agent-rules ddg-asana-workflow