---
namespace: work
tags: [system, sync, pipeline, debugging]
last_synced: 2026-04-27
confidence: 1.0
---

# Personal OS — Sync Pipeline

## Components

### sync.js
Pulls from 4 Asana sources into PostgreSQL. Run: `node ~/Developer/personal-os/sync.js`

**Sources fetched:**
1. `fetchMyTasks()` — My Tasks sections by GID (section-by-section)
2. `fetchFollowingTasks()` — tasks I follow, 7-day chunks to work around 100-result cap
3. `fetchDelegatedTasks()` — created by me, assigned to others
4. `fetchProjectTasks()` — tasks from projects I'm a member of (lookback window on full sync)
5. `sweepRecentlyCompleted()` — catch completed tasks missed by above

**Incremental sync**: uses `modified_since` cursor from `sync_state` table.
**Full sync**: triggered when `full_sync_at` is older than configured threshold.

### generate-status.js
Reads from PostgreSQL → writes `asana_context.md` + `status.md`.
Run: `node ~/Developer/personal-os/generate-status.js`

### run-pipeline.sh
Wrapper: runs sync.js + generate-status.js + healthcheck.
`~/scripts/run-pipeline.sh`

Healthcheck: `asana_context.md` must be < 40 minutes old. Exits 1 if stale.

## Known Bugs (Fixed)

### possibly_deleted + overdue tasks appearing
**Root cause**: Full sync marks tasks absent from Asana response as `possibly_deleted=true`.
These tasks remained `completed=false`, causing them to appear overdue.

**Fix 1 — generate-status.js**: Added `AND t.possibly_deleted = FALSE` to all 5 active-task queries.

**Fix 2 — sync.js**: Added `sweepRecentlyCompleted()` called after `syncMyTasks()`.
Uses `searchTasksForWorkspace(completed:true, completed_at.after=cursor)` to catch tasks
that vanished from sections (because completed tasks are excluded from section membership results).

### Completion gap
`getTasksForSection` drops completed tasks. `fetchFollowingTasks` has `completed:false` filter.
→ Completion events never reach DB incrementally.
→ `sweepRecentlyCompleted()` fills this gap.

## Debugging Checklist

**Pipeline seems stuck:**
```bash
bash ~/scripts/run-pipeline.sh
# Check exit code and output
```

**Check when last sync ran:**
```sql
SELECT key, last_sync_at, full_sync_at FROM sync_state ORDER BY last_sync_at DESC;
```

**Check possibly_deleted tasks (should be 0 active):**
```sql
SELECT count(*) FROM tasks WHERE possibly_deleted=true AND completed=false;
```

**Check asana_context.md freshness:**
```bash
stat -f "%Sm" ~/Developer/personal-os/asana_context.md
```

**Manual full sync (reset cursor):**
```sql
UPDATE sync_state SET cursor = null, full_sync_at = null WHERE key = 'my_tasks';
```
Then run `node sync.js`.

## Files

- `~/Developer/personal-os/sync.js` — main sync script
- `~/Developer/personal-os/generate-status.js` — status generation
- `~/Developer/personal-os/asana.js` — thin Asana API client
- `~/Developer/personal-os/config.json` — workspace GID, user GID, section GIDs
- `~/.config/personal-os/env` — ASANA_API_KEY, POSTGRES_URL

## Related

[[personal-os-architecture]] [[personal-os-agent-rules]] [[ddg-asana-workflow]]