107 lines
3.8 KiB
Markdown
107 lines
3.8 KiB
Markdown
---
|
|
namespace: work
|
|
tags: [system, sync, pipeline, debugging]
|
|
last_synced: 2026-04-27
|
|
confidence: 1.0
|
|
---
|
|
|
|
# Personal OS — Sync Pipeline
|
|
|
|
## Components
|
|
|
|
### sync.js
|
|
Pulls from 4 Asana sources into PostgreSQL. Run: `node ~/Developer/personal-os/sync.js`
|
|
|
|
**Sources fetched:**
|
|
1. `fetchMyTasks()` — My Tasks sections by GID (section-by-section)
|
|
2. `fetchFollowingTasks()` — tasks I follow, 7-day chunks to work around 100-result cap
|
|
3. `fetchDelegatedTasks()` — created by me, assigned to others
|
|
4. `fetchProjectTasks()` — tasks from projects I'm a member of (lookback window on full sync)
|
|
5. `sweepRecentlyCompleted()` — catch completed tasks missed by above
|
|
|
|
**Incremental sync**: uses `modified_since` cursor from `sync_state` table.
|
|
**Full sync**: triggered when `full_sync_at` is older than configured threshold.
|
|
|
|
### generate-status.js
|
|
Reads from PostgreSQL → writes `asana_context.md` + `status.md`.
|
|
Run: `node ~/Developer/personal-os/generate-status.js`
|
|
|
|
### run-pipeline.sh
|
|
Wrapper: runs sync.js + generate-status.js + healthcheck.
|
|
`~/scripts/run-pipeline.sh`
|
|
|
|
Healthcheck: `asana_context.md` must be < 40 minutes old. Exits 1 if stale.
|
|
|
|
## Known Bugs (Fixed)
|
|
|
|
### possibly_deleted + overdue tasks appearing
|
|
**Root cause**: Full sync marks tasks absent from Asana response as `possibly_deleted=true`.
|
|
These tasks remained `completed=false`, causing them to appear overdue.
|
|
|
|
**Fix 1 — generate-status.js**: Added `AND t.possibly_deleted = FALSE` to all 5 active-task queries.
|
|
|
|
**Fix 2 — sync.js**: Added `sweepRecentlyCompleted()` called after `syncMyTasks()`.
|
|
Uses `searchTasksForWorkspace(completed:true, completed_at.after=cursor)` to catch tasks
|
|
that vanished from sections (because completed tasks are excluded from section membership results).
|
|
|
|
### Completion gap
|
|
`getTasksForSection` drops completed tasks. `fetchFollowingTasks` has `completed:false` filter.
|
|
→ Completion events never reach DB incrementally.
|
|
→ `sweepRecentlyCompleted()` fills this gap.
|
|
|
|
## Debugging Checklist
|
|
|
|
**Pipeline seems stuck:**
|
|
```bash
|
|
bash ~/scripts/run-pipeline.sh
|
|
# Check exit code and output
|
|
```
|
|
|
|
**Check when last sync ran:**
|
|
```sql
|
|
SELECT key, last_sync_at, full_sync_at FROM sync_state ORDER BY last_sync_at DESC;
|
|
```
|
|
|
|
**Check possibly_deleted tasks (should be 0 active):**
|
|
```sql
|
|
SELECT count(*) FROM tasks WHERE possibly_deleted=true AND completed=false;
|
|
```
|
|
|
|
**Check asana_context.md freshness:**
|
|
```bash
|
|
stat -f "%Sm" ~/Developer/personal-os/asana_context.md
|
|
```
|
|
|
|
**Manual full sync (reset cursor):**
|
|
```sql
|
|
UPDATE sync_state SET cursor = null, full_sync_at = null WHERE key = 'my_tasks';
|
|
```
|
|
Then run `node sync.js`.
|
|
|
|
## Token Management
|
|
|
|
`ASANA_API_KEY` lives in two places — both must be in sync:
|
|
- `~/.zprofile` — source of truth (`export ASANA_API_KEY="..."`)
|
|
- `~/.config/personal-os/env` — loaded by `run-pipeline.sh` via `. ~/.config/personal-os/env`
|
|
|
|
If the token is rotated in Asana, update `.zprofile` first, then sync to env:
|
|
```bash
|
|
NEW_TOKEN=$(grep ASANA_API_KEY ~/.zprofile | sed 's/export ASANA_API_KEY="//;s/"//');
|
|
echo "export ASANA_API_KEY=$NEW_TOKEN" > ~/.config/personal-os/env
|
|
```
|
|
Verify: `curl -s -o /dev/null -w "%{http_code}" "https://app.asana.com/api/1.0/users/me" -H "Authorization: Bearer $NEW_TOKEN"` → should return `200`.
|
|
|
|
Symptom of stale token: `data-pipeline` and `inbox-check` cron jobs get paused (Hermes auto-pauses on repeated failures), `status.md` stops updating.
|
|
|
|
## Files
|
|
|
|
- `~/Developer/personal-os/sync.js` — main sync script
|
|
- `~/Developer/personal-os/generate-status.js` — status generation
|
|
- `~/Developer/personal-os/asana.js` — thin Asana API client
|
|
- `~/Developer/personal-os/config.json` — workspace GID, user GID, section GIDs
|
|
- `~/.config/personal-os/env` — ASANA_API_KEY (sync from .zprofile when rotating)
|
|
|
|
## Related
|
|
|
|
[[personal-os-architecture]] [[personal-os-agent-rules]] [[ddg-asana-workflow]]
|