From 0e74956ee61e294f6525d4424c65cb571ac14937 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Tue, 10 Nov 2026 10:54:57 +0600 Subject: [PATCH] [2026-11-10] eagle: work/projects/cpm-project-summary-2026-11-03.md --- .../cpm-project-summary-2026-11-03.md | 76 ++++++++----------- 1 file changed, 33 insertions(+), 43 deletions(-) diff --git a/work/projects/cpm-project-summary-2026-11-03.md b/work/projects/cpm-project-summary-2026-11-03.md index cf2803b5..3006d92f 100644 --- a/work/projects/cpm-project-summary-2026-11-03.md +++ b/work/projects/cpm-project-summary-2026-11-03.md @@ -10,16 +10,16 @@ status: draft Internal metrics, keep out of public PRs. Task: [CPM: root-cause the stuck embedded extension messaging](https://app.asana.com/1/137249556945/project/72649045549333/task/1216761517055116?focus=true). Details: [[cpm-post-release-analysis-2026-10-27]], [[cpm-status-2026-10-15]]. -**What the project has achieved so far** +## What the project has achieved Goal: permanent failures below 1% of DAU (0.2-0.5% = success). Permanent = no answer for 5 minutes, repeated or needed a reload. -**Now (macOS 1.214, iOS 7.244, Nov/2)** +### Now (macOS 1.214, iOS 7.244, Nov/2) -- iOS: 0.44% of DAU (0.39% on 7.243, same day). Inside the target <<1%. -- macOS: 1.89%. Above the target <<1%. +- iOS: 0.44% of DAU (0.39% on 7.243, same day). Inside the target. +- macOS: 1.89%. Above the target. -**Fixes shipped** +### Fixes shipped - [P3](https://app.asana.com/1/137249556945/task/1219643764290826?focus=true), [OS<26.6 limit](https://app.asana.com/1/137249556945/task/1220003778559448?focus=true) – improves extension startup after a restart from up to 32 s → 0.1 s on iOS/macOS 26.6+ (lab tests). iOS 26.6+: "background missing" failures 8.2% → 0.3% of DAU (users with any failure 14.6% → 8.3%). Effect on permanent not measurable yet (one day, 90k users). - macOS: [pinned tab lost on window close](https://app.asana.com/1/137249556945/task/1219643712648486?focus=true), [tab merging](https://app.asana.com/1/137249556945/task/1218356681060823?focus=true) – real bugs fixed, UI tests added. They caused under 1% of permanent failures, about 3x fewer after the fix (macOS 1.211 → 1.213). @@ -28,70 +28,60 @@ Goal: permanent failures below 1% of DAU (0.2-0.5% = success). Permanent = no an - [Network process crash](https://app.asana.com/1/137249556945/task/1219531810004073?focus=true) – restart after the networking process crashes. Triggers for 6.6% of iOS and 4.2% of macOS daily users (Nov/2). - [State restoration race](https://app.asana.com/1/137249556945/task/1218744813699966?focus=true) – restored tabs wait up to 5 s for the extension to start. Init failures on restore: iOS ~20% → ~8.5% of DAU, macOS no clear change. -**Fixes not confirmed by data (to be rolled back)** +### Hypotheses rejected -- [Deallocation race](https://app.asana.com/1/137249556945/task/1217907325352233?focus=true) ([A/B](https://app.asana.com/1/137249556945/task/1218356681060819?focus=true), [flag](https://app.asana.com/1/137249556945/task/1219036128743441?focus=true)) – flag never turned on, no effect seen in the data. -- [Reload on the stuck signal](https://app.asana.com/1/137249556945/task/1218744813699968?focus=true) – 46% of macOS cases still got no answer after the reload. -- [P4](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true) (late page reload) – off by default, never triggered. +- [Deallocation race](https://app.asana.com/1/137249556945/task/1217907325352233?focus=true) ([A/B](https://app.asana.com/1/137249556945/task/1218356681060819?focus=true), [flag](https://app.asana.com/1/137249556945/task/1219036128743441?focus=true)) – fix shipped behind a flag never turned on, no relation in the data. +- [Reload on the stuck signal](https://app.asana.com/1/137249556945/task/1218744813699968?focus=true) – 46% of macOS cases still got no answer. +- [P4](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true) – reload the page when the extension starts late. Off by default, never triggered in production. -**Fixes going public next week (Nov/9)** +## Fixes going public next week (Nov/9) - [Upgrade before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true) – load the new extension version first on launch. Removes part of the permanent failures right after an update (macOS 0.26% of DAU, iOS 0.05%). - [Error 6 restart](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true) – restart the extension when a load error leaves it broken. Up to 0.08% of DAU permanent on iOS, negligible on macOS. - Together: macOS 1.89% → about 1.6%, iOS 0.44% → about 0.3-0.35%. Estimates, upper bounds. -**Data now (% of DAU)** +## Data now (% of DAU) Permanent failures, daily trend (macOS 1.213 / iOS 7.243: Oct/26 → Nov/2; 1.214 / 7.244: Nov/2): - macOS: 2.07% → 3.07% (Oct/27 peak) → 1.90%; 1.214: 1.89%. - iOS: 0.30% → 0.42% (Oct/27 peak) → 0.39%; 7.244: 0.44%. -By group on the new version (1.214 / 7.244, Nov/2), % of DAU. Trend is failures from the first day of the previous version (1.213 / 7.243, Oct/26) to the new version. +By group, % of DAU. Trend is failures: previous version (1.213 / 7.243) on Oct/26 → previous version on Nov/2 → new version (1.214 / 7.244) on Nov/2. Shares and permanent are for the new version. -| Group | Platform | Failures, % of DAU | % of all failures | Permanent, % of DAU | % of all permanent | Failures trend, % of DAU | -| --------------------------- | -------- | ------------------ | ----------------- | ------------------- | ------------------ | ------------------------ | -| Init on wake | macOS | 5.89% | 39% | 0.68% | 36% | 8.80% → 5.89% | -| Init on wake | iOS | 1.74% | 22% | 0.06% | 15% | 1.56% → 1.74% | -| Tab not connected | macOS | 1.26% | 8% | 0.42% | 22% | 1.18% → 1.26% | -| Init after extension update | macOS | 2.94% | 20% | 0.26% | 14% | 5.40% → 2.94% | -| Init after extension update | iOS | 0.34% | 4% | 0.05% | 12% | 0.37% → 0.34% | -| Init after load | macOS | 1.73% | 11% | 0.19% | 10% | 2.73% → 1.73% | -| Init after load | iOS | 2.54% | 32% | 0.18% | 42% | 1.86% → 2.54% | -| Error 6 | macOS | 0.11% | 1% | 0.01% | 1% | 0.14% → 0.11% | -| Error 6 | iOS | 0.67% | 8% | 0.08% | 18% | 0.50% → 0.67% | -| Background missing | macOS | **0.20%** | 1% | 0.03% | 2% | 1.06% → **0.20%** | -| Background missing | iOS | **0.27%** | 3% | 0.01% | 3% | 0.63% → **0.27%** | -| Other | macOS | 2.92% | 19% | 0.30% | 16% | 4.75% → 2.92% | -| Other | iOS | 2.45% | 31% | 0.04% | 10% | 1.95% → 2.45% | +| Group | Platform | Trend | % of all failures | Permanent | % of all permanent | +|---|---|---|---|---|---| +| Init on wake | macOS | 8.80% → 8.50% → 5.89% | 39% | 0.68% | 36% | +| Init on wake | iOS | 1.56% → 2.34% → 1.74% | 22% | 0.06% | 15% | +| Tab not connected | macOS | 1.18% → 0.90% → 1.26% | 8% | 0.42% | 22% | +| Init after extension update | macOS | 5.40% → 0.92% → 2.94% | 20% | 0.26% | 14% | +| Init after extension update | iOS | 0.37% → 0.13% → 0.34% | 4% | 0.05% | 12% | +| Init after load | macOS | 2.73% → 3.41% → 1.73% | 11% | 0.19% | 10% | +| Init after load | iOS | 1.86% → 2.21% → 2.54% | 32% | 0.18% | 42% | +| Error 6 | macOS | 0.14% → 0.17% → 0.11% | 1% | 0.01% | 1% | +| Error 6 | iOS | 0.50% → 1.32% → 0.67% | 8% | 0.08% | 18% | +| Background missing | macOS | 1.06% → 1.13% → **0.20%** | 1% | 0.03% | 2% | +| Background missing | iOS | 0.63% → 6.90% → **0.27%** | 3% | 0.01% | 3% | +| Other | macOS | 4.75% → 4.17% → 2.92% | 19% | 0.30% | 16% | +| Other | iOS | 1.95% → 1.10% → 2.45% | 31% | 0.04% | 10% | 1.214 / 7.244 is one day of early adopters (macOS about 9k DAU, iOS 110k), a rough picture. -**Plan** +## Plan This week, cleanup: - [Feature flags cleanup](https://app.asana.com/1/137249556945/task/1220159402040629?focus=true): - Remove fix and flag (did not work or never ran): [cpmBackgroundGraveyardTreatment](https://app.asana.com/1/137249556945/task/1218860893478440?focus=true), [cpmLateBackgroundDocumentRecovery](https://app.asana.com/1/137249556945/task/1219588497257482?focus=true), [cpmMessagingHangRecovery](https://app.asana.com/1/137249556945/task/1218855001659655?focus=true), [cpmBackgroundLoadRecovery](https://app.asana.com/1/137249556945/task/1219701603183698?focus=true). - Remove with the diagnostics: [cpmDiagnosticsRecorder](https://app.asana.com/1/137249556945/task/1218785444896174?focus=true), [cpmBackgroundDelegateProxy](https://app.asana.com/1/137249556945/task/1218782530802146?focus=true). - - Remove the flag, keep the fix (already in 1.213 and 1.214): [webExtensionSharedPinnedTabWindowClose](https://app.asana.com/1/137249556945/task/1219660990760563?focus=true). - - Keep the fix and the flag as a kill-switch for now (remove after 2-3 releases without issues): [cpmNetworkProcessRecovery](https://app.asana.com/1/137249556945/task/1219701603183695?focus=true), [cpmStableBaseURL](https://app.asana.com/1/137249556945/task/1219728649844711?focus=true), [webExtensionStateRestorationGate](https://app.asana.com/1/137249556945/task/1218931841045393?focus=true), [cpmBackgroundLoadRestart](https://app.asana.com/1/137249556945/task/1219970564251181?focus=true). + - Keep the fix, remove the failsafe flag: [cpmNetworkProcessRecovery](https://app.asana.com/1/137249556945/task/1219701603183695?focus=true), [cpmStableBaseURL](https://app.asana.com/1/137249556945/task/1219728649844711?focus=true), [webExtensionSharedPinnedTabWindowClose](https://app.asana.com/1/137249556945/task/1219660990760563?focus=true), [webExtensionStateRestorationGate](https://app.asana.com/1/137249556945/task/1218931841045393?focus=true), [cpmBackgroundLoadRestart](https://app.asana.com/1/137249556945/task/1219970564251181?focus=true) (after Nov/9 data). - [Pixel data cleanup](https://app.asana.com/1/137249556945/task/1220159402040630?focus=true): remove the temporary pixels and episode parameters that only served the investigation. -Next, metrics (tasks to create): +Next, metrics: -- Permanent breakage metrics, replace the temporary pixels. Goal: track the permanent failure rate against DAU after the project closes. - - Episode report, trimmed to: outcome, tabs affected, repeat count, reload state, cause group (wake, load, update, error 6, tab not connected, other), platform and OS. Same shape on iOS and macOS. - - Healthy-navigation baseline: a 1% sample of navigations reports whether the extension answered and in which time bucket. Gives failures per navigation, not only per user. - - Health indicators dashboard: permanent failures, any failures, one-tab and browser-wide, per platform and version, against the 1% target. - - Old and new reports published together for one release, so the trend is not lost. Then the old ones go. -- Extension lifecycle metrics, to find where init on wake breaks. Goal: for init on wake failures, say at which step the chain stops. - - Stage ledger: the extension reports the last step reached for each navigation. Steps: page script injected, init sent by the page, init received by the background, config read from storage, app checks done (autoconsent on, feature on), dashboard update sent to the app, answer sent to the page, answer received by the page, response received by the app. - - Added to the failure report: last step reached and time spent in it. - - Background probe at failure time: is the background alive, does it answer a ping, how fast. Needs a spike first: does the probe see the background's state, and does it keep the background awake. - - Background lifecycle counters in the report: why it started (wake or load), age bucket, number of wakes in the session. - - Work: extension JS changes to send the steps, app changes to collect them, privacy review, extension and app releases. Data in 1-2 release cycles. -- Data check of the Nov/9 fixes: [Analyze attribution data after stage 3]. +- Permanent breakage metrics: a small fixed set, rates against DAU, added to the health indicators dashboard. Task to create. +- Extension lifecycle metrics: find where the startup message stops, to diagnose the wake failures. Needs privacy review and releases, data in 1-2 release cycles. Task to create. +- Data check of the Nov/9 fixes: [Analyze attribution data after stage 3](https://app.asana.com/1/137249556945/task/1220159402040627?focus=true). Open: