8.0 KiB
8.0 KiB
tags, updated, status
| tags | updated | status | |||
|---|---|---|---|---|---|
|
2026-11-03 | draft |
CPM stuck-messaging project: where we are, Nov/3
Internal metrics, keep out of public PRs. Task: CPM: root-cause the stuck embedded extension messaging. Details: cpm-post-release-analysis-2026-10-27, cpm-status-2026-10-15.
What the project has achieved
Goal: permanent failures below 1% of DAU (0.2-0.5% = success). Permanent = no answer for 5 minutes, repeated or needed a reload.
Now (macOS 1.214, iOS 7.244, Nov/2)
- iOS: 0.44% of DAU (0.39% on 7.243, same day). Inside the target.
- macOS: 1.89%. Above the target.
Fixes shipped
- P3, OS<26.6 limit – improves extension startup after a restart from up to 32 s → 0.1 s on iOS/macOS 26.6+ (lab tests). iOS 26.6+: "background missing" failures 8.2% → 0.3% of DAU (users with any failure 14.6% → 8.3%). Effect on permanent not measurable yet (one day, 90k users).
- macOS: pinned tab lost on window close, tab merging – real bugs fixed, UI tests added. They caused under 1% of permanent failures, about 3x fewer after the fix (macOS 1.211 → 1.213).
- P1 – recovery after a slow wake of the extension. Triggers for 19% of iOS and 3% of macOS daily users (Nov/2).
- P2 – retry after load error 6. Triggers for 1.7% of iOS and 0.2% of macOS daily users (Nov/2).
- Network process crash – restart after the networking process crashes. Triggers for 6.6% of iOS and 4.2% of macOS daily users (Nov/2).
- State restoration race – restored tabs wait up to 5 s for the extension to start. Init failures on restore: iOS ~20% → ~8.5% of DAU, macOS no clear change.
Hypotheses rejected
- Deallocation race (A/B, flag) – fix shipped behind a flag never turned on, no relation in the data.
- Reload on the stuck signal – 46% of macOS cases still got no answer.
- P4 – reload the page when the extension starts late. Off by default, never triggered in production.
Fixes going public next week (Nov/9)
- Upgrade before loading – load the new extension version first on launch. Removes part of the permanent failures right after an update (macOS 0.26% of DAU, iOS 0.05%).
- Error 6 restart – restart the extension when a load error leaves it broken. Up to 0.08% of DAU permanent on iOS, negligible on macOS.
- Together: macOS 1.89% → about 1.6%, iOS 0.44% → about 0.3-0.35%. Estimates, upper bounds.
Data now (% of DAU)
Permanent failures, daily trend (macOS 1.213 / iOS 7.243: Oct/26 → Nov/2; 1.214 / 7.244: Nov/2):
- macOS: 2.07% → 3.07% (Oct/27 peak) → 1.90%; 1.214: 1.89%.
- iOS: 0.30% → 0.42% (Oct/27 peak) → 0.39%; 7.244: 0.44%.
By group on the new version (1.214 / 7.244, Nov/2), % of DAU. Trend is failures from the first day of the previous version (1.213 / 7.243, Oct/26) to the new version.
| Group | Platform | Failures, % of DAU | % of all failures | Permanent, % of DAU | % of all permanent | Failures trend, % of DAU |
|---|---|---|---|---|---|---|
| Init on wake | macOS | 5.89% | 39% | 0.68% | 36% | 8.80% → 5.89% |
| Init on wake | iOS | 1.74% | 22% | 0.06% | 15% | 1.56% → 1.74% |
| Tab not connected | macOS | 1.26% | 8% | 0.42% | 22% | 1.18% → 1.26% |
| Init after extension update | macOS | 2.94% | 20% | 0.26% | 14% | 5.40% → 2.94% |
| Init after extension update | iOS | 0.34% | 4% | 0.05% | 12% | 0.37% → 0.34% |
| Init after load | macOS | 1.73% | 11% | 0.19% | 10% | 2.73% → 1.73% |
| Init after load | iOS | 2.54% | 32% | 0.18% | 42% | 1.86% → 2.54% |
| Error 6 | macOS | 0.11% | 1% | 0.01% | 1% | 0.14% → 0.11% |
| Error 6 | iOS | 0.67% | 8% | 0.08% | 18% | 0.50% → 0.67% |
| Background missing | macOS | 0.20% | 1% | 0.03% | 2% | 1.06% → 0.20% |
| Background missing | iOS | 0.27% | 3% | 0.01% | 3% | 0.63% → 0.27% |
| Other | macOS | 2.92% | 19% | 0.30% | 16% | 4.75% → 2.92% |
| Other | iOS | 2.45% | 31% | 0.04% | 10% | 1.95% → 2.45% |
1.214 / 7.244 is one day of early adopters (macOS about 9k DAU, iOS 110k), a rough picture.
Plan
This week, cleanup:
- Feature flags cleanup:
- Remove fix and flag (did not work or never ran): cpmBackgroundGraveyardTreatment, cpmLateBackgroundDocumentRecovery, cpmMessagingHangRecovery, cpmBackgroundLoadRecovery.
- Remove with the diagnostics: cpmDiagnosticsRecorder, cpmBackgroundDelegateProxy.
- Keep the fix, remove the failsafe flag: cpmNetworkProcessRecovery, cpmStableBaseURL, webExtensionSharedPinnedTabWindowClose, webExtensionStateRestorationGate, cpmBackgroundLoadRestart (after Nov/9 data).
- Pixel data cleanup: remove the temporary pixels and episode parameters that only served the investigation.
Next, metrics:
- Permanent breakage metrics: a small fixed set, rates against DAU, added to the health indicators dashboard. Task to create.
- Extension lifecycle metrics: find where the startup message stops, to diagnose the wake failures. Needs privacy review and releases, data in 1-2 release cycles. Task to create.
- Data check of the Nov/9 fixes: Analyze attribution data after stage 3.
Open:
- No fix for macOS wake failures and tab not connected. If the new metrics show nothing actionable, close the project and reset the target.