From a3a84e4729ac7e06fe3c54148f7782589dea89fb Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Tue, 3 Nov 2026 18:27:40 +0600 Subject: [PATCH] [2026-11-03] eagle: work/projects/cpm-project-summary-2026-11-03.md --- .../cpm-project-summary-2026-11-03.md | 73 ++++++++----------- 1 file changed, 32 insertions(+), 41 deletions(-) diff --git a/work/projects/cpm-project-summary-2026-11-03.md b/work/projects/cpm-project-summary-2026-11-03.md index d9ed4f82..9be71ec3 100644 --- a/work/projects/cpm-project-summary-2026-11-03.md +++ b/work/projects/cpm-project-summary-2026-11-03.md @@ -12,58 +12,49 @@ Internal metrics, keep out of public PRs. Task: [CPM: root-cause the stuck embed ## What the project has achieved -Success criteria: permanent stuck reports below 1% of DAU (0.2-0.5% counts as success). Permanent = no response for 5 minutes, confirmed by a repeat or a reload. +Goal: permanent stuck reports below 1% of DAU (0.2-0.5% = success). Permanent = no answer for 5 minutes, repeated or needed a reload. -Progress against it (now = 1.214 / 7.244, Nov/2): +Now (macOS 1.214, iOS 7.244, Nov/2): -- iOS: 0.44% of DAU (0.43% on 7.243). Inside the target. -- macOS: 1.89% of DAU. Target not reached yet. Nov/9 fixes should bring it to about 1.6% (estimate, upper bound). +- iOS: 0.44% of DAU (0.43% before). Inside the target. +- macOS: 1.89%. Above the target. -Breakage fixes shipped, with what each did: +Fixes shipped: -- Stable extension origin ([P3: stable baseURL fix](https://app.asana.com/1/137249556945/task/1219643764290826?focus=true), [Disable stable baseURL fix for OS<26.6](https://app.asana.com/1/137249556945/task/1220003778559448?focus=true), OS 26.6+): in a lab reproduction an origin change delays extension startup by 1.5 to 32 s depending on cache size, a stable origin by 0.1 s. Not measured in production. iOS 26.6+ startup-missing and no-view reports are down over 90% vs 7.243 (cohort comparison, P3 state not visible in the pixels). -- macOS pinned tab lost on window close ([P5: pinned tab after windowDidClose](https://app.asana.com/1/137249556945/task/1219643712648486?focus=true)) and tab merging ([Fix Tab Merging causing CPM failure](https://app.asana.com/1/137249556945/task/1218356681060823?focus=true)): reproduced and fixed. P5 tests 9 pass with fix → 5 expected failures with it off → 27 pass. -- Wake and startup recovery ([P1&P2[&P4]: Fix init loss on background unload](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true)): applied to 2,026 iOS / 89 macOS episodes on Oct/26. -- Network process restart ([Fix CPM breakage on network process crash](https://app.asana.com/1/137249556945/task/1219531810004073?focus=true)): applied 476 iOS / 277 macOS times on Oct/26. -- Public Nov/9: [Restart extension context when error 6 leaves the background view alive](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true) (up to 0.08% DAU iOS) and [Upgrade Web Extensions before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true) (macOS 0.26%, iOS 0.05% DAU). +- [P3](https://app.asana.com/1/137249556945/task/1219643764290826?focus=true), [OS<26.6 limit](https://app.asana.com/1/137249556945/task/1220003778559448?focus=true) – stable extension identity, avoids slow startup (up to 32 s in lab tests). iOS 26.6+ "background missing" failures 8.2% → 0.3% of DAU, but also fell on older iOS, so P3 is not proven as the cause. +- macOS: [pinned tab lost on window close](https://app.asana.com/1/137249556945/task/1219643712648486?focus=true), [tab merging](https://app.asana.com/1/137249556945/task/1218356681060823?focus=true) – real bugs fixed, UI tests added. +- [P1/P2/P4](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true) – recovery after wake and startup failures. Triggered 2,026 times on iOS, 89 on macOS (Oct/26). No visible drop in stuck reports. +- [Network process crash](https://app.asana.com/1/137249556945/task/1219531810004073?focus=true) – restart after the networking process crashes. Triggered 476 times on iOS, 277 on macOS (Oct/26). No visible drop in stuck reports. +- [State restoration race](https://app.asana.com/1/137249556945/task/1218744813699966?focus=true) – extension failing to start while tabs restore. Effect not measured. Causes proved by reproduction: -- iOS error-6 loop: stale failed background ([WebExtensions breakage harness](https://app.asana.com/1/137249556945/task/1219531810004071?focus=true)). Fixed by the Nov/9 error 6 restart. -- Origin change delaying extension startup (lab reproduction with real cache sizes; production impact not measured). -- macOS P6: a tab without a controller never answers. 0.42% of DAU, no fix yet. +- iOS: after "error 6" the background stays broken and every wake fails again ([harness](https://app.asana.com/1/137249556945/task/1219531810004071?focus=true)). Fix goes public Nov/9. +- macOS: a tab the extension is not connected to never gets an answer. 0.42% of DAU, no fix yet. -Hypotheses tested and rejected: +Hypotheses rejected: -- [Deallocation race fix](https://app.asana.com/1/137249556945/task/1217907325352233?focus=true), [A/B](https://app.asana.com/1/137249556945/task/1218356681060819?focus=true), [flag activation](https://app.asana.com/1/137249556945/task/1219036128743441?focus=true): flag never activated, data showed no relation. -- Hung or crashed background process: under 1% of failures show it. -- Lost message after wake: 0 lost in ~1,500 wakes in WebKit tests ([Reproduce the issue](https://app.asana.com/1/137249556945/task/1217907325352231?focus=true)). -- [Reload/recover on the Stuck signal](https://app.asana.com/1/137249556945/task/1218744813699968?focus=true): 46% of macOS episodes still got no response after it. Late page reload: 0 fires in production. -- Memory pressure, feature flag toggling, "failed load blocks forever": not reproduced. -- Graveyard mitigation: not supported by the data. +- [Deallocation race](https://app.asana.com/1/137249556945/task/1217907325352233?focus=true) ([A/B](https://app.asana.com/1/137249556945/task/1218356681060819?focus=true), [flag](https://app.asana.com/1/137249556945/task/1219036128743441?focus=true)) – fix shipped behind a flag never turned on, no relation in the data. +- [Reload on the stuck signal](https://app.asana.com/1/137249556945/task/1218744813699968?focus=true) – 46% of macOS cases still got no answer. Late page reload never triggered. +- Lost message after wake – 0 of ~1,500 wakes in tests ([Reproduce the issue](https://app.asana.com/1/137249556945/task/1217907325352231?focus=true)). +- Hung background process – under 1% of failures. +- Memory pressure, flag toggling, "failed load blocks forever" – not reproduced. ## Fixes going public next week (Nov/9) -- [Upgrade Web Extensions before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true): load the new extension version first on launch, instead of an outdated one. - - Expected: removes part of the "extension updated" group (macOS 0.26% DAU, iOS 0.05%). Launch upgrades only, so partial. -- [Restart extension context when error 6 leaves the background view alive](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true): restart the stuck background when the failed view is still retained. - - Expected: up to 0.08% DAU on iOS, negligible on macOS. Partial, most error 6 reports only have it in history. -- Together: macOS 1.89% to about 1.6%, iOS 0.44% to about 0.3-0.35%. Upper bounds, not measured. +- [Upgrade before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true) – load the new extension version first on launch. Removes part of the failures right after an update (macOS 0.26% of DAU, iOS 0.05%). +- [Error 6 restart](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true) – restart the extension when a load error leaves it broken. Up to 0.08% on iOS, negligible on macOS. +- Together: macOS 1.89% → about 1.6%, iOS 0.44% → about 0.3-0.35%. Estimates, upper bounds. -## Data now (% of DAU, 1.214 / 7.244, Nov/2) +## Data now (% of DAU, Nov/2) -One day, early adopters, macOS has about 9k DAU. Treat as a rough picture. +One day, early adopters, macOS about 9k DAU: a rough picture. -- Permanent stuck = no response for 5 minutes, confirmed by a repeat or a reload. Browser-wide = 2+ tabs, one-tab = single tab. -- macOS: any failure 15.1%, permanent 1.89%. - - Browser-wide 1.22%, one-tab 0.68%. - - Idle wake 0.68%, tab not connected (P6) 0.42%, other 0.30%, extension updated 0.26%, recent load 0.19%. -- iOS: any failure 8.0%, permanent 0.44% (0.39% on 7.243 days 1-2). - - Browser-wide 0.26%, one-tab 0.18%. - - Recent load 0.18%, error 6 0.08%, idle wake 0.06%, extension updated 0.05%, other 0.04%. -- iOS is already under the 1% target. macOS stays above it after the Nov/9 fixes. -- Unaddressed on macOS: idle wake (35% of permanent) and P6 (22%). Nothing in flight for either. -- Extension install and load errors are rare (thousandths of a percent), not worth the time. +- macOS: permanent 1.89% (browser-wide 1.22%, one tab 0.68%). + - Failures right after a wake 0.68%, tab not connected 0.42%, other 0.30%, right after an update 0.26%, right after a load 0.19%. +- iOS: permanent 0.44% (browser-wide 0.26%, one tab 0.18%). + - Right after a load 0.18%, error 6 0.08%, right after a wake 0.06%, right after an update 0.05%, other 0.04%. +- No fix in flight for macOS wake failures (35% of permanent) and tab not connected (22%). ## Plan @@ -71,10 +62,10 @@ Cleanup, this week: - [Feature flags cleanup](https://app.asana.com/1/137249556945/task/1220159402040629?focus=true) (task is empty in Asana, content below). Registry tasks per flag: - Remove the fix and the flag, they did not work or never ran: - - [cpmBackgroundGraveyardTreatment](https://app.asana.com/1/137249556945/task/1218860893478440?focus=true): graveyard mitigation, not supported by data. + - [cpmBackgroundGraveyardTreatment](https://app.asana.com/1/137249556945/task/1218860893478440?focus=true): deallocation race fix, never activated. - [cpmLateBackgroundDocumentRecovery](https://app.asana.com/1/137249556945/task/1219588497257482?focus=true): late page reload, default off, 0 fires. - [cpmMessagingHangRecovery](https://app.asana.com/1/137249556945/task/1218855001659655?focus=true): reload on the stuck signal, 46% still no response; decide keep or remove. - - [cpmBackgroundLoadRecovery](https://app.asana.com/1/137249556945/task/1219701603183698?focus=true): P1/P2 wake and retry, no measurable effect. + - [cpmBackgroundLoadRecovery](https://app.asana.com/1/137249556945/task/1219701603183698?focus=true): wake and retry recovery, no measurable effect. - Remove the diagnostics flags together with the diagnostics layer: - [cpmDiagnosticsRecorder](https://app.asana.com/1/137249556945/task/1218785444896174?focus=true): recorder and unresponsive token. - [cpmBackgroundDelegateProxy](https://app.asana.com/1/137249556945/task/1218782530802146?focus=true): delegate proxy for diagnostics. @@ -91,10 +82,10 @@ Cleanup, this week: Metrics, next two weeks: - Permanent small set: one episode report with a fixed format, rates against DAU, a sample of healthy navigations as baseline. Old and new formats published together for one release to keep the trend. Added to the health indicators dashboard. Task to create. -- Extension lifecycle metrics to diagnose the wake problem: where the init message stops (page, background, native checks, dashboard update, answer back to the page), plus a background state check when a failure is reported. Needs a privacy review and extension and app releases, data in 1-2 release cycles. Task to create. +- Extension lifecycle metrics to diagnose the wake problem: where the startup message between the page and the browser stops, plus a background state check when a failure is reported. Needs a privacy review and extension and app releases, data in 1-2 release cycles. Task to create. - Data check of the Nov/9 fixes: [Analyze attribution data after stage 3](https://app.asana.com/1/137249556945/task/1220159402040627?focus=true). Decide later: - Mitigations that do not need the cause: retry the init from the page when no answer arrives, keep-warm A/B for the background. -- P6 has no fix. The wake group will not be solved without the new metrics. If they show nothing actionable, close the project and reset the target. +- "Tab not connected" has no fix. Failures right after a wake will not be solved without the new metrics. If they show nothing actionable, close the project and reset the target.