81 lines
7.1 KiB
Markdown
81 lines
7.1 KiB
Markdown
---
|
||
tags:
|
||
- work
|
||
- cpm
|
||
- summary
|
||
updated: '2026-11-03'
|
||
status: draft
|
||
---
|
||
# CPM stuck-messaging project: where we are, Nov/3
|
||
|
||
Internal metrics, keep out of public PRs. Task: [CPM: root-cause the stuck embedded extension messaging](https://app.asana.com/1/137249556945/project/72649045549333/task/1216761517055116?focus=true). Details: [[cpm-post-release-analysis-2026-10-27]], [[cpm-status-2026-10-15]].
|
||
|
||
## What the project has achieved
|
||
|
||
Goal: permanent stuck reports below 1% of DAU (0.2-0.5% = success). Permanent = no answer for 5 minutes, repeated or needed a reload.
|
||
|
||
### Now (macOS 1.214, iOS 7.244, Nov/2)
|
||
|
||
- iOS: 0.44% of DAU (0.39% on 7.243, same day). Inside the target.
|
||
- macOS: 1.89%. Above the target.
|
||
|
||
### Fixes shipped
|
||
|
||
- [P3](https://app.asana.com/1/137249556945/task/1219643764290826?focus=true), [OS<26.6 limit](https://app.asana.com/1/137249556945/task/1220003778559448?focus=true) – improves extension startup after a restart from up to 32 s → 0.1 s on iOS/macOS 26.6+ (lab tests). iOS 26.6+: "background missing" failures 8.2% → 0.3% of DAU (users with any failure 14.6% → 8.3%). Effect on permanent not measurable yet (one day, 90k users).
|
||
- macOS: [pinned tab lost on window close](https://app.asana.com/1/137249556945/task/1219643712648486?focus=true), [tab merging](https://app.asana.com/1/137249556945/task/1218356681060823?focus=true) – real bugs fixed, UI tests added. They caused under 1% of stuck reports, about 3x fewer after the fix (macOS 1.211 → 1.213).
|
||
- [P1](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true) – recovery after a slow wake of the extension. Triggers for 19% of iOS and 3% of macOS daily users (Nov/2).
|
||
- [P2](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true) – retry after load error 6. Triggers for 1.7% of iOS and 0.2% of macOS daily users (Nov/2).
|
||
- [Network process crash](https://app.asana.com/1/137249556945/task/1219531810004073?focus=true) – restart after the networking process crashes. Triggers for 6.6% of iOS and 4.2% of macOS daily users (Nov/2).
|
||
- [State restoration race](https://app.asana.com/1/137249556945/task/1218744813699966?focus=true) – restored tabs wait up to 5 s for the extension to start before loading. Stuck restored tabs: iOS 1.32% → 0.52% → 0.06% of DAU, macOS 0.19% → 0.12% → 0.02% (before fix → with fix → now).
|
||
|
||
### Hypotheses rejected
|
||
|
||
- [Deallocation race](https://app.asana.com/1/137249556945/task/1217907325352233?focus=true) ([A/B](https://app.asana.com/1/137249556945/task/1218356681060819?focus=true), [flag](https://app.asana.com/1/137249556945/task/1219036128743441?focus=true)) – fix shipped behind a flag never turned on, no relation in the data.
|
||
- [Reload on the stuck signal](https://app.asana.com/1/137249556945/task/1218744813699968?focus=true) – 46% of macOS cases still got no answer.
|
||
- [P4](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true) – reload the page when the extension starts late. Off by default, never triggered in production.
|
||
|
||
## Fixes going public next week (Nov/9)
|
||
|
||
- [Upgrade before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true) – load the new extension version first on launch. Removes part of the permanent failures right after an update (macOS 0.26% of DAU, iOS 0.05%).
|
||
- [Error 6 restart](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true) – restart the extension when a load error leaves it broken. Up to 0.08% of DAU permanent on iOS, negligible on macOS.
|
||
- Together: macOS 1.89% → about 1.6%, iOS 0.44% → about 0.3-0.35%. Estimates, upper bounds.
|
||
|
||
## Data now (% of DAU)
|
||
|
||
Permanent stuck, daily trend (macOS 1.213 / iOS 7.243: Oct/26 → Nov/2; 1.214 / 7.244: Nov/2):
|
||
|
||
- macOS: 2.07% → 3.07% (Oct/27 peak) → 1.90%; 1.214: 1.89%.
|
||
- iOS: 0.30% → 0.42% (Oct/27 peak) → 0.39%; 7.244: 0.44%.
|
||
|
||
By group: permanent stuck now (1.214 / 7.244) and trend of all failures, permanent and recovered (Oct/26 → Nov/2 on 1.213 / 7.243 → Nov/2 on 1.214 / 7.244):
|
||
|
||
- Failure right after the system wakes the extension: macOS permanent 0.68%, failures 8.80% → 8.50% → 5.89%. iOS permanent 0.06%, failures 1.56% → 2.34% → 1.74%.
|
||
- Tab not connected: macOS permanent 0.42%, failures 1.18% → 0.90% → 1.26%. Not seen on iOS.
|
||
- Right after an extension update: macOS permanent 0.26%, failures 5.40% → 0.92% → 2.94%. iOS permanent 0.05%, failures 0.37% → 0.13% → 0.34%.
|
||
- Right after a load: macOS permanent 0.19%, failures 2.73% → 3.41% → 1.73%. iOS permanent 0.18%, failures 1.86% → 2.21% → 2.54%.
|
||
- Error 6: iOS permanent 0.08%, failures 0.50% → 1.32% → 0.67%. macOS about 0.
|
||
- Background missing: macOS permanent 0.03%, failures 1.06% → 1.13% → **0.20%**. iOS permanent 0.01%, failures 0.63% → 6.90% → **0.27%**.
|
||
- Other: macOS permanent 0.30%, failures 4.75% → 4.17% → 2.92%. iOS permanent 0.04%, failures 1.95% → 1.10% → 2.45%.
|
||
|
||
1.214 / 7.244 is one day of early adopters (macOS about 9k DAU, iOS 110k), a rough picture.
|
||
|
||
## Plan
|
||
|
||
This week, cleanup:
|
||
|
||
- [Feature flags cleanup](https://app.asana.com/1/137249556945/task/1220159402040629?focus=true):
|
||
- Remove fix and flag (did not work or never ran): [cpmBackgroundGraveyardTreatment](https://app.asana.com/1/137249556945/task/1218860893478440?focus=true), [cpmLateBackgroundDocumentRecovery](https://app.asana.com/1/137249556945/task/1219588497257482?focus=true), [cpmMessagingHangRecovery](https://app.asana.com/1/137249556945/task/1218855001659655?focus=true), [cpmBackgroundLoadRecovery](https://app.asana.com/1/137249556945/task/1219701603183698?focus=true).
|
||
- Remove with the diagnostics: [cpmDiagnosticsRecorder](https://app.asana.com/1/137249556945/task/1218785444896174?focus=true), [cpmBackgroundDelegateProxy](https://app.asana.com/1/137249556945/task/1218782530802146?focus=true).
|
||
- Keep the fix, remove the failsafe flag: [cpmNetworkProcessRecovery](https://app.asana.com/1/137249556945/task/1219701603183695?focus=true), [cpmStableBaseURL](https://app.asana.com/1/137249556945/task/1219728649844711?focus=true), [webExtensionSharedPinnedTabWindowClose](https://app.asana.com/1/137249556945/task/1219660990760563?focus=true), [webExtensionStateRestorationGate](https://app.asana.com/1/137249556945/task/1218931841045393?focus=true), [cpmBackgroundLoadRestart](https://app.asana.com/1/137249556945/task/1219970564251181?focus=true) (after Nov/9 data).
|
||
- [Pixel data cleanup](https://app.asana.com/1/137249556945/task/1220159402040630?focus=true): remove the temporary pixels and episode parameters that only served the investigation.
|
||
|
||
Next, metrics:
|
||
|
||
- Permanent breakage metrics: a small fixed set, rates against DAU, added to the health indicators dashboard. Task to create.
|
||
- Extension lifecycle metrics: find where the startup message stops, to diagnose the wake failures. Needs privacy review and releases, data in 1-2 release cycles. Task to create.
|
||
- Data check of the Nov/9 fixes: [Analyze attribution data after stage 3](https://app.asana.com/1/137249556945/task/1220159402040627?focus=true).
|
||
|
||
Open:
|
||
|
||
- No fix for macOS wake failures and tab not connected. If the new metrics show nothing actionable, close the project and reset the target.
|