CPM stuck-messaging project: where we are, Nov/3

What the project has achieved

Goal: permanent stuck reports below 1% of DAU (0.2-0.5% = success). Permanent = no answer for 5 minutes, repeated or needed a reload.

Now (macOS 1.214, iOS 7.244, Nov/2):

Fixes shipped:

P3, OS<26.6 limit – stable extension identity, avoids slow startup (up to 32 s in lab tests). iOS 26.6+ "background missing" failures 8.2% → 0.3% of DAU, but also fell on older iOS, so P3 is not proven as the cause.
macOS: pinned tab lost on window close, tab merging – real bugs fixed, UI tests added.
P1/P2/P4 – recovery after wake and startup failures. Triggered 2,026 times on iOS, 89 on macOS (Oct/26). No visible drop in stuck reports.
Network process crash – restart after the networking process crashes. Triggered 476 times on iOS, 277 on macOS (Oct/26). No visible drop in stuck reports.
State restoration race – extension failing to start while tabs restore. Effect not measured.

Causes proved by reproduction:

iOS: after "error 6" the background stays broken and every wake fails again (harness). Fix goes public Nov/9.
macOS: a tab the extension is not connected to never gets an answer. 0.42% of DAU, no fix yet.

Hypotheses rejected:

Deallocation race (A/B, flag) – fix shipped behind a flag never turned on, no relation in the data.
Reload on the stuck signal – 46% of macOS cases still got no answer. Late page reload never triggered.
Lost message after wake – 0 of ~1,500 wakes in tests (Reproduce the issue).
Hung background process – under 1% of failures.
Memory pressure, flag toggling, "failed load blocks forever" – not reproduced.

Upgrade before loading – load the new extension version first on launch. Removes part of the failures right after an update (macOS 0.26% of DAU, iOS 0.05%).
Error 6 restart – restart the extension when a load error leaves it broken. Up to 0.08% on iOS, negligible on macOS.
Together: macOS 1.89% → about 1.6%, iOS 0.44% → about 0.3-0.35%. Estimates, upper bounds.

One day, early adopters, macOS about 9k DAU: a rough picture.

macOS: permanent 1.89% (browser-wide 1.22%, one tab 0.68%).
- Failures right after a wake 0.68%, tab not connected 0.42%, other 0.30%, right after an update 0.26%, right after a load 0.19%.
iOS: permanent 0.44% (browser-wide 0.26%, one tab 0.18%).
- Right after a load 0.18%, error 6 0.08%, right after a wake 0.06%, right after an update 0.05%, other 0.04%.
No fix in flight for macOS wake failures (35% of permanent) and tab not connected (22%).

Cleanup, this week:

Feature flags cleanup (task is empty in Asana, content below). Registry tasks per flag:
- Remove the fix and the flag, they did not work or never ran:
  - cpmBackgroundGraveyardTreatment: deallocation race fix, never activated.
  - cpmLateBackgroundDocumentRecovery: late page reload, default off, 0 fires.
  - cpmMessagingHangRecovery: reload on the stuck signal, 46% still no response; decide keep or remove.
  - cpmBackgroundLoadRecovery: wake and retry recovery, no measurable effect.
- Remove the diagnostics flags together with the diagnostics layer:
  - cpmDiagnosticsRecorder: recorder and unresponsive token.
  - cpmBackgroundDelegateProxy: delegate proxy for diagnostics.
- Keep the fix, remove the failsafe flag once its effect is measured:
  - cpmNetworkProcessRecovery: network restart.
  - cpmStableBaseURL: stable origin; flag state is not visible in pixels, check the remote config first.
  - webExtensionSharedPinnedTabWindowClose: P5.
  - webExtensionStateRestorationGate: session restoration race.
  - cpmBackgroundLoadRestart: error 6 restart, after Nov/9 data.
Pixel data cleanup (task is empty in Asana, content below):
- Remove the treatment pixels that only say "applied": background_wake, background_load_retry, install/upgrade readiness, late_document_reload, network_process_recovery_applied, startup_response_missing.
- Remove the temporary episode parameters that only served the investigation (bg_events and the other debug fields), after the new metrics are live.

Metrics, next two weeks:

Permanent small set: one episode report with a fixed format, rates against DAU, a sample of healthy navigations as baseline. Old and new formats published together for one release to keep the trend. Added to the health indicators dashboard. Task to create.
Extension lifecycle metrics to diagnose the wake problem: where the startup message between the page and the browser stops, plus a background state check when a failure is reported. Needs a privacy review and extension and app releases, data in 1-2 release cycles. Task to create.
Data check of the Nov/9 fixes: Analyze attribution data after stage 3.

Decide later:

Mitigations that do not need the cause: retry the init from the page when no answer arrives, keep-warm A/B for the background.
"Tab not connected" has no fix. Failures right after a wake will not be solved without the new metrics. If they show nothing actionable, close the project and reset the target.