obsidian-vault/work/projects/cpm-project-summary-2026-11-03.md

tags, updated, status

tags	updated	status
work cpm summary	2026-11-03	draft

CPM stuck-messaging project: where we are, Nov/3

Internal metrics, keep out of public PRs. Task: CPM: root-cause the stuck embedded extension messaging. Details: cpm-post-release-analysis-2026-10-27, cpm-status-2026-10-15.

What the project has achieved

Success criteria: permanent stuck reports below 1% of DAU (0.2-0.5% counts as success). Permanent = no response for 5 minutes, confirmed by a repeat or a reload.

Progress against it (now = 1.214 / 7.244, Nov/2):

• iOS: 0.44% of DAU (0.43% on 7.243). Inside the target.
• macOS: 1.89% of DAU. Target not reached yet. Nov/9 fixes should bring it to about 1.6% (estimate, upper bound).

Breakage fixes shipped, with what each did:

• Stable extension origin (P3: stable baseURL fix, Disable stable baseURL fix for OS<26.6, OS 26.6+): in a lab reproduction an origin change delays extension startup by 1.5 to 32 s depending on cache size, a stable origin by 0.1 s. Not measured in production. iOS 26.6+ startup-missing and no-view reports are down over 90% vs 7.243 (cohort comparison, P3 state not visible in the pixels).
• macOS pinned tab lost on window close (P5: pinned tab after windowDidClose) and tab merging (Fix Tab Merging causing CPM failure): reproduced and fixed. P5 tests 9 pass with fix → 5 expected failures with it off → 27 pass.
• Wake and startup recovery (P1&P2[&P4]: Fix init loss on background unload): applied to 2,026 iOS / 89 macOS episodes on Oct/26.
• Network process restart (Fix CPM breakage on network process crash): applied 476 iOS / 277 macOS times on Oct/26.
• Public Nov/9: Restart extension context when error 6 leaves the background view alive (up to 0.08% DAU iOS) and Upgrade Web Extensions before loading (macOS 0.26%, iOS 0.05% DAU).

Causes proved by reproduction:

• iOS error-6 loop: stale failed background (WebExtensions breakage harness). Fixed by the Nov/9 error 6 restart.
• Origin change delaying extension startup (lab reproduction with real cache sizes; production impact not measured).
• macOS P6: a tab without a controller never answers. 0.42% of DAU, no fix yet.

Hypotheses tested and rejected:

• Deallocation race fix, A/B, flag activation: flag never activated, data showed no relation.
• Hung or crashed background process: under 1% of failures show it.
• Lost message after wake: 0 lost in ~1,500 wakes in WebKit tests (Reproduce the issue).
• Reload/recover on the Stuck signal: 46% of macOS episodes still got no response after it. Late page reload: 0 fires in production.
• Memory pressure, feature flag toggling, "failed load blocks forever": not reproduced.
• Graveyard mitigation: not supported by the data.

Fixes going public next week (Nov/9)

• Upgrade Web Extensions before loading: load the new extension version first on launch, instead of an outdated one.
  • Expected: removes part of the "extension updated" group (macOS 0.26% DAU, iOS 0.05%). Launch upgrades only, so partial.
• Restart extension context when error 6 leaves the background view alive: restart the stuck background when the failed view is still retained.
  • Expected: up to 0.08% DAU on iOS, negligible on macOS. Partial, most error 6 reports only have it in history.
• Together: macOS 1.89% to about 1.6%, iOS 0.44% to about 0.3-0.35%. Upper bounds, not measured.

Data now (% of DAU, 1.214 / 7.244, Nov/2)

One day, early adopters, macOS has about 9k DAU. Treat as a rough picture.

• Permanent stuck = no response for 5 minutes, confirmed by a repeat or a reload. Browser-wide = 2+ tabs, one-tab = single tab.
• macOS: any failure 15.1%, permanent 1.89%.
  • Browser-wide 1.22%, one-tab 0.68%.
  • Idle wake 0.68%, tab not connected (P6) 0.42%, other 0.30%, extension updated 0.26%, recent load 0.19%.
• iOS: any failure 8.0%, permanent 0.44% (0.39% on 7.243 days 1-2).
  • Browser-wide 0.26%, one-tab 0.18%.
  • Recent load 0.18%, error 6 0.08%, idle wake 0.06%, extension updated 0.05%, other 0.04%.
• iOS is already under the 1% target. macOS stays above it after the Nov/9 fixes.
• Unaddressed on macOS: idle wake (35% of permanent) and P6 (22%). Nothing in flight for either.
• Extension install and load errors are rare (thousandths of a percent), not worth the time.

Plan

Cleanup, this week:

• Feature flags cleanup (task is empty in Asana, content below). Registry tasks per flag:
  • Remove the fix and the flag, they did not work or never ran:
    • cpmBackgroundGraveyardTreatment: graveyard mitigation, not supported by data.
    • cpmLateBackgroundDocumentRecovery: late page reload, default off, 0 fires.
    • cpmMessagingHangRecovery: reload on the stuck signal, 46% still no response; decide keep or remove.
    • cpmBackgroundLoadRecovery: P1/P2 wake and retry, no measurable effect.
  • Remove the diagnostics flags together with the diagnostics layer:
    • cpmDiagnosticsRecorder: recorder and unresponsive token.
    • cpmBackgroundDelegateProxy: delegate proxy for diagnostics.
  • Keep the fix, remove the failsafe flag once its effect is measured:
    • cpmNetworkProcessRecovery: network restart.
    • cpmStableBaseURL: stable origin; flag state is not visible in pixels, check the remote config first.
    • webExtensionSharedPinnedTabWindowClose: P5.
    • webExtensionStateRestorationGate: session restoration race.
    • cpmBackgroundLoadRestart: error 6 restart, after Nov/9 data.
• Pixel data cleanup (task is empty in Asana, content below):
  • Remove the treatment pixels that only say "applied": background_wake, background_load_retry, install/upgrade readiness, late_document_reload, network_process_recovery_applied, startup_response_missing.
  • Remove the temporary episode parameters that only served the investigation (bg_events and the other debug fields), after the new metrics are live.

Metrics, next two weeks:

• Permanent small set: one episode report with a fixed format, rates against DAU, a sample of healthy navigations as baseline. Old and new formats published together for one release to keep the trend. Added to the health indicators dashboard. Task to create.
• Extension lifecycle metrics to diagnose the wake problem: where the init message stops (page, background, native checks, dashboard update, answer back to the page), plus a background state check when a failure is reported. Needs a privacy review and extension and app releases, data in 1-2 release cycles. Task to create.
• Data check of the Nov/9 fixes: Analyze attribution data after stage 3.

Decide later:

• Mitigations that do not need the cause: retry the init from the page when no answer arrives, keep-warm A/B for the background.
• P6 has no fix. The wake group will not be solved without the new metrics. If they show nothing actionable, close the project and reset the target.