92 lines
7.6 KiB
Markdown
92 lines
7.6 KiB
Markdown
---
|
||
tags:
|
||
- work
|
||
- cpm
|
||
- summary
|
||
updated: '2026-11-03'
|
||
status: draft
|
||
---
|
||
# CPM stuck-messaging project: where we are, Nov/3
|
||
|
||
Internal metrics, keep out of public PRs. Task: [CPM: root-cause the stuck embedded extension messaging](https://app.asana.com/1/137249556945/project/72649045549333/task/1216761517055116?focus=true). Details: [[cpm-post-release-analysis-2026-10-27]], [[cpm-status-2026-10-15]].
|
||
|
||
## What the project has achieved
|
||
|
||
Goal: permanent stuck reports below 1% of DAU (0.2-0.5% = success). Permanent = no answer for 5 minutes, repeated or needed a reload.
|
||
|
||
Now (macOS 1.214, iOS 7.244, Nov/2):
|
||
|
||
- iOS: 0.44% of DAU (0.43% before). Inside the target.
|
||
- macOS: 1.89%. Above the target.
|
||
|
||
Fixes shipped:
|
||
|
||
- [P3](https://app.asana.com/1/137249556945/task/1219643764290826?focus=true), [OS<26.6 limit](https://app.asana.com/1/137249556945/task/1220003778559448?focus=true) – stable extension identity, avoids slow startup (up to 32 s in lab tests). iOS 26.6+ "background missing" failures 8.2% → 0.3% of DAU, but also fell on older iOS, so P3 is not proven as the cause.
|
||
- macOS: [pinned tab lost on window close](https://app.asana.com/1/137249556945/task/1219643712648486?focus=true), [tab merging](https://app.asana.com/1/137249556945/task/1218356681060823?focus=true) – real bugs fixed, UI tests added.
|
||
- [P1/P2/P4](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true) – recovery after wake and startup failures. Triggered 2,026 times on iOS, 89 on macOS (Oct/26). No visible drop in stuck reports.
|
||
- [Network process crash](https://app.asana.com/1/137249556945/task/1219531810004073?focus=true) – restart after the networking process crashes. Triggered 476 times on iOS, 277 on macOS (Oct/26). No visible drop in stuck reports.
|
||
- [State restoration race](https://app.asana.com/1/137249556945/task/1218744813699966?focus=true) – extension failing to start while tabs restore. Effect not measured.
|
||
|
||
Causes proved by reproduction:
|
||
|
||
- iOS: after "error 6" the background stays broken and every wake fails again ([harness](https://app.asana.com/1/137249556945/task/1219531810004071?focus=true)). Fix goes public Nov/9.
|
||
- macOS: a tab the extension is not connected to never gets an answer. 0.42% of DAU, no fix yet.
|
||
|
||
Hypotheses rejected:
|
||
|
||
- [Deallocation race](https://app.asana.com/1/137249556945/task/1217907325352233?focus=true) ([A/B](https://app.asana.com/1/137249556945/task/1218356681060819?focus=true), [flag](https://app.asana.com/1/137249556945/task/1219036128743441?focus=true)) – fix shipped behind a flag never turned on, no relation in the data.
|
||
- [Reload on the stuck signal](https://app.asana.com/1/137249556945/task/1218744813699968?focus=true) – 46% of macOS cases still got no answer. Late page reload never triggered.
|
||
- Lost message after wake – 0 of ~1,500 wakes in tests ([Reproduce the issue](https://app.asana.com/1/137249556945/task/1217907325352231?focus=true)).
|
||
- Hung background process – under 1% of failures.
|
||
- Memory pressure, flag toggling, "failed load blocks forever" – not reproduced.
|
||
|
||
## Fixes going public next week (Nov/9)
|
||
|
||
- [Upgrade before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true) – load the new extension version first on launch. Removes part of the failures right after an update (macOS 0.26% of DAU, iOS 0.05%).
|
||
- [Error 6 restart](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true) – restart the extension when a load error leaves it broken. Up to 0.08% on iOS, negligible on macOS.
|
||
- Together: macOS 1.89% → about 1.6%, iOS 0.44% → about 0.3-0.35%. Estimates, upper bounds.
|
||
|
||
## Data now (% of DAU, Nov/2)
|
||
|
||
One day, early adopters, macOS about 9k DAU: a rough picture.
|
||
|
||
- macOS: permanent 1.89% (browser-wide 1.22%, one tab 0.68%).
|
||
- Failures right after a wake 0.68%, tab not connected 0.42%, other 0.30%, right after an update 0.26%, right after a load 0.19%.
|
||
- iOS: permanent 0.44% (browser-wide 0.26%, one tab 0.18%).
|
||
- Right after a load 0.18%, error 6 0.08%, right after a wake 0.06%, right after an update 0.05%, other 0.04%.
|
||
- No fix in flight for macOS wake failures (35% of permanent) and tab not connected (22%).
|
||
|
||
## Plan
|
||
|
||
Cleanup, this week:
|
||
|
||
- [Feature flags cleanup](https://app.asana.com/1/137249556945/task/1220159402040629?focus=true) (task is empty in Asana, content below). Registry tasks per flag:
|
||
- Remove the fix and the flag, they did not work or never ran:
|
||
- [cpmBackgroundGraveyardTreatment](https://app.asana.com/1/137249556945/task/1218860893478440?focus=true): deallocation race fix, never activated.
|
||
- [cpmLateBackgroundDocumentRecovery](https://app.asana.com/1/137249556945/task/1219588497257482?focus=true): late page reload, default off, 0 fires.
|
||
- [cpmMessagingHangRecovery](https://app.asana.com/1/137249556945/task/1218855001659655?focus=true): reload on the stuck signal, 46% still no response; decide keep or remove.
|
||
- [cpmBackgroundLoadRecovery](https://app.asana.com/1/137249556945/task/1219701603183698?focus=true): wake and retry recovery, no measurable effect.
|
||
- Remove the diagnostics flags together with the diagnostics layer:
|
||
- [cpmDiagnosticsRecorder](https://app.asana.com/1/137249556945/task/1218785444896174?focus=true): recorder and unresponsive token.
|
||
- [cpmBackgroundDelegateProxy](https://app.asana.com/1/137249556945/task/1218782530802146?focus=true): delegate proxy for diagnostics.
|
||
- Keep the fix, remove the failsafe flag once its effect is measured:
|
||
- [cpmNetworkProcessRecovery](https://app.asana.com/1/137249556945/task/1219701603183695?focus=true): network restart.
|
||
- [cpmStableBaseURL](https://app.asana.com/1/137249556945/task/1219728649844711?focus=true): stable origin; flag state is not visible in pixels, check the remote config first.
|
||
- [webExtensionSharedPinnedTabWindowClose](https://app.asana.com/1/137249556945/task/1219660990760563?focus=true): P5.
|
||
- [webExtensionStateRestorationGate](https://app.asana.com/1/137249556945/task/1218931841045393?focus=true): session restoration race.
|
||
- [cpmBackgroundLoadRestart](https://app.asana.com/1/137249556945/task/1219970564251181?focus=true): error 6 restart, after Nov/9 data.
|
||
- [Pixel data cleanup](https://app.asana.com/1/137249556945/task/1220159402040630?focus=true) (task is empty in Asana, content below):
|
||
- Remove the treatment pixels that only say "applied": background_wake, background_load_retry, install/upgrade readiness, late_document_reload, network_process_recovery_applied, startup_response_missing.
|
||
- Remove the temporary episode parameters that only served the investigation (bg_events and the other debug fields), after the new metrics are live.
|
||
|
||
Metrics, next two weeks:
|
||
|
||
- Permanent small set: one episode report with a fixed format, rates against DAU, a sample of healthy navigations as baseline. Old and new formats published together for one release to keep the trend. Added to the health indicators dashboard. Task to create.
|
||
- Extension lifecycle metrics to diagnose the wake problem: where the startup message between the page and the browser stops, plus a background state check when a failure is reported. Needs a privacy review and extension and app releases, data in 1-2 release cycles. Task to create.
|
||
- Data check of the Nov/9 fixes: [Analyze attribution data after stage 3](https://app.asana.com/1/137249556945/task/1220159402040627?focus=true).
|
||
|
||
Decide later:
|
||
|
||
- Mitigations that do not need the cause: retry the init from the page when no answer arrives, keep-warm A/B for the background.
|
||
- "Tab not connected" has no fix. Failures right after a wake will not be solved without the new metrics. If they show nothing actionable, close the project and reset the target.
|