diff --git a/work/projects/cpm-project-summary-2026-11-03.md b/work/projects/cpm-project-summary-2026-11-03.md index 84621111..eef0e8aa 100644 --- a/work/projects/cpm-project-summary-2026-11-03.md +++ b/work/projects/cpm-project-summary-2026-11-03.md @@ -14,28 +14,46 @@ Goal: permanent stuck reports below 1% of DAU (0.2-0.5% counts as success). ## What the project has achieved -- Telemetry that attributes failures (public since Oct/5). We can now see what state the extension was in when a page got no response. -- Fixed the deterministic per-tab bugs: tab merging, Close Other Tabs, pinned tabs and window close (macOS). -- Reproduced the main failure patterns on real WebKit, with the event sequences matched to production. -- Oct/26 release (macOS 1.213 / iOS 7.243): new episode telemetry, recovery for slow wake, error 6 retry and slow startup, restart after a networking process crash. -- Nov/2 release (1.214 / 7.244): stable extension origin (P3, only on OS 26.6+), and the fix for browser error pages being counted as CPM failures. -- Result so far: the noise is gone, permanent failures are not. On iOS 26.6+, startup-missing and no-view reports are down by over 90% against 7.243 on the same day. Permanent share of DAU is about the same as before. +Success criteria: permanent stuck reports below 1% of DAU (0.2-0.5% counts as success). Permanent = no response for 5 minutes, confirmed by a repeat or a reload. -## Hypotheses and fixes that did not work out +Progress against it (1.213 / 7.243 days 1-2 → 1.214 / 7.244 day 1): -- Deallocation race fix and its A/B test: cancelled because the metrics were too blurry, flag never activated. Data showed no relation to the released fixes. -- Slow wake fix (P1): shipped, no measurable effect on the wake group. The background is ready in under 1s, so the loss happens after that. -- Hung or crashed background process: not the cause, under 1% of failures show it. -- Memory pressure, feature flag toggling, "failed load blocks forever": not reproduced or ruled out as a global cause. -- Lost message after wake: could not be reproduced in WebKit tests (0 lost in ~1,500 wakes, also with real pages and the ad blocker running). -- Late page reload fix: default off and never fired. -- Graveyard mitigation: not supported by the data, not pursued. +- iOS: 0.43% → 0.44% of DAU. Inside the target. +- macOS: 2.87% → 1.89% of DAU (any failure 28.5% → 15.1%). Target not reached yet. Nov/9 fixes bring it to about 1.6%. + +Breakage measurement, built and corrected: + +- Production telemetry for the stuck extension: one episode report with failure groups, tab scope, repeat count, and rates against DAU. Before it we could not separate stuck from recovered failures or attribute a cause. [Improve telemetry](https://app.asana.com/1/137249556945/task/1219531810004069?focus=true), [TD: failure pixels attribution](https://app.asana.com/1/137249556945/task/1218619301158188?focus=true). +- Error pages were reported as breakages on macOS. Fixed, so the macOS figure is now accurate: any failure 28.5% → 15.1%, permanent 2.87% → 1.89% of DAU. [Fix false-positive breakage reports for error pages](https://app.asana.com/1/137249556945/task/1220003778559452?focus=true). + +Breakage fixes shipped, with what each did: + +- Stable extension origin ([P3: stable baseURL fix](https://app.asana.com/1/137249556945/task/1219643764290826?focus=true), [Disable stable baseURL fix for OS<26.6](https://app.asana.com/1/137249556945/task/1220003778559448?focus=true), OS 26.6+): removes a 1.5 to 32 s startup blackout after an origin change. iOS 26.6+ startup-missing and no-view reports down over 90%. +- macOS pinned tab lost on window close ([P5: pinned tab after windowDidClose](https://app.asana.com/1/137249556945/task/1219643712648486?focus=true)) and tab merging ([Fix Tab Merging causing CPM failure](https://app.asana.com/1/137249556945/task/1218356681060823?focus=true)): reproduced and fixed. P5 tests 9 pass with fix → 5 expected failures with it off → 27 pass. +- Wake and startup recovery ([P1&P2[&P4]: Fix init loss on background unload](https://app.asana.com/1/137249556945/task/1219531810004072?focus=true)): applied to 2,026 iOS / 89 macOS episodes on Oct/26. +- Network process restart ([Fix CPM breakage on network process crash](https://app.asana.com/1/137249556945/task/1219531810004073?focus=true)): applied 476 iOS / 277 macOS times on Oct/26. +- Public Nov/9: [Restart extension context when error 6 leaves the background view alive](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true) (up to 0.08% DAU iOS) and [Upgrade Web Extensions before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true) (macOS 0.26%, iOS 0.05% DAU). + +Causes proved by reproduction: + +- iOS error-6 loop: stale failed background ([WebExtensions breakage harness](https://app.asana.com/1/137249556945/task/1219531810004071?focus=true)). Fixed by the Nov/9 error 6 restart. +- Origin change cleaning the cache before startup (fixed by P3). +- macOS P6: a tab without a controller never answers. 0.42% of DAU, no fix yet. + +Hypotheses tested and rejected: + +- [Deallocation race fix](https://app.asana.com/1/137249556945/task/1217907325352233?focus=true), [A/B](https://app.asana.com/1/137249556945/task/1218356681060819?focus=true), [flag activation](https://app.asana.com/1/137249556945/task/1219036128743441?focus=true): flag never activated, data showed no relation. +- Hung or crashed background process: under 1% of failures show it. +- Lost message after wake: 0 lost in ~1,500 wakes in WebKit tests ([Reproduce the issue](https://app.asana.com/1/137249556945/task/1217907325352231?focus=true)). +- [Reload/recover on the Stuck signal](https://app.asana.com/1/137249556945/task/1218744813699968?focus=true): 46% of macOS episodes still got no response after it. Late page reload: 0 fires in production. +- Memory pressure, feature flag toggling, "failed load blocks forever": not reproduced. +- Graveyard mitigation: not supported by the data. ## Fixes going public next week (Nov/9) -- Upgrade-before-load ([#7767](https://github.com/duckduckgo/apple-browsers/pull/7767)): load the new extension version first on launch, instead of an outdated one. +- [Upgrade Web Extensions before loading](https://app.asana.com/1/137249556945/task/1220004977087625?focus=true): load the new extension version first on launch, instead of an outdated one. - Expected: removes part of the "extension updated" group (macOS 0.26% DAU, iOS 0.05%). Launch upgrades only, so partial. -- Error 6 restart ([#7730](https://github.com/duckduckgo/apple-browsers/pull/7730)): restart the stuck background when the failed view is still retained. +- [Restart extension context when error 6 leaves the background view alive](https://app.asana.com/1/137249556945/task/1219939142983214?focus=true): restart the stuck background when the failed view is still retained. - Expected: up to 0.08% DAU on iOS, negligible on macOS. Partial, most error 6 reports only have it in history. - Together: macOS 1.89% to about 1.6%, iOS 0.44% to about 0.3-0.35%. Upper bounds, not measured. @@ -44,7 +62,7 @@ Goal: permanent stuck reports below 1% of DAU (0.2-0.5% counts as success). One day, early adopters, macOS has about 9k DAU. Treat as a rough picture. - Permanent stuck = no response for 5 minutes, confirmed by a repeat or a reload. Browser-wide = 2+ tabs, one-tab = single tab. -- macOS: any failure 15.1% (28.5% on 1.213 days 1-2), permanent 1.89% (2.87% on 1.213 days 1-2, but the mature 1.213 population is also at 1.9%). +- macOS: any failure 15.1%, permanent 1.89%. The earlier 28.5% / 2.87% on 1.213 included error pages, so the drop is the metric fix. Mature 1.213 is also at ~1.9%. - Browser-wide 1.22%, one-tab 0.68%. - Idle wake 0.68%, tab not connected (P6) 0.42%, other 0.30%, extension updated 0.26%, recent load 0.19%. - iOS: any failure 8.0%, permanent 0.44% (0.39% on 7.243 days 1-2). @@ -56,8 +74,34 @@ One day, early adopters, macOS has about 9k DAU. Treat as a rough picture. ## Plan -- Clean up what did not work: remove the never-activated flags and experiment code (deallocation race, graveyard, late reload), the unused readiness pixels and the old debug pixels. Keep the kill-switch flags on fixes that change behavior until their effect is measured. -- Rebuild the metrics as a permanent small set: one episode report with a fixed format, rates against DAU, a sample of healthy navigations as baseline. Drop the tokens and flags that did not help. Publish old and new formats for one release to keep the trend. -- Add extension lifecycle metrics to diagnose the wake problem: where the init message stops (page, background, native checks, dashboard update, answer back to the page). Plus a check of the background state when a failure is reported. Needs a privacy review and extension and app releases, so data in 1-2 release cycles. -- Mitigations to decide on, not planned yet: retry the init from the page when no answer arrives, and a keep-warm A/B for the background. Both work without knowing the cause. -- Open: P6 has no fix, and the wake group will not be solved without the new metrics. If they show nothing actionable, close the project and reset the target. +Cleanup, this week: + +- [Feature flags cleanup](https://app.asana.com/1/137249556945/task/1220159402040629?focus=true) (task is empty in Asana, content below). Registry tasks per flag: + - Remove the fix and the flag, they did not work or never ran: + - [cpmBackgroundGraveyardTreatment](https://app.asana.com/1/137249556945/task/1218860893478440?focus=true): graveyard mitigation, not supported by data. + - [cpmLateBackgroundDocumentRecovery](https://app.asana.com/1/137249556945/task/1219588497257482?focus=true): late page reload, default off, 0 fires. + - [cpmMessagingHangRecovery](https://app.asana.com/1/137249556945/task/1218855001659655?focus=true): reload on the stuck signal, 46% still no response; decide keep or remove. + - [cpmBackgroundLoadRecovery](https://app.asana.com/1/137249556945/task/1219701603183698?focus=true): P1/P2 wake and retry, no measurable effect. + - Remove the diagnostics flags together with the diagnostics layer: + - [cpmDiagnosticsRecorder](https://app.asana.com/1/137249556945/task/1218785444896174?focus=true): recorder and unresponsive token. + - [cpmBackgroundDelegateProxy](https://app.asana.com/1/137249556945/task/1218782530802146?focus=true): delegate proxy for diagnostics. + - Keep the fix, remove the failsafe flag once its effect is measured: + - [cpmNetworkProcessRecovery](https://app.asana.com/1/137249556945/task/1219701603183695?focus=true): network restart. + - [cpmStableBaseURL](https://app.asana.com/1/137249556945/task/1219728649844711?focus=true): stable origin; flag state is not visible in pixels, check the remote config first. + - [webExtensionSharedPinnedTabWindowClose](https://app.asana.com/1/137249556945/task/1219660990760563?focus=true): P5. + - [webExtensionStateRestorationGate](https://app.asana.com/1/137249556945/task/1218931841045393?focus=true): session restoration race. + - [cpmBackgroundLoadRestart](https://app.asana.com/1/137249556945/task/1219970564251181?focus=true): error 6 restart, after Nov/9 data. +- [Pixel data cleanup](https://app.asana.com/1/137249556945/task/1220159402040630?focus=true) (task is empty in Asana, content below): + - Remove the treatment pixels that only say "applied": background_wake, background_load_retry, install/upgrade readiness, late_document_reload, network_process_recovery_applied, startup_response_missing. + - Remove the temporary episode parameters that only served the investigation (bg_events and the other debug fields), after the new metrics are live. + +Metrics, next two weeks: + +- Permanent small set: one episode report with a fixed format, rates against DAU, a sample of healthy navigations as baseline. Old and new formats published together for one release to keep the trend. Added to the health indicators dashboard. Task to create. +- Extension lifecycle metrics to diagnose the wake problem: where the init message stops (page, background, native checks, dashboard update, answer back to the page), plus a background state check when a failure is reported. Needs a privacy review and extension and app releases, data in 1-2 release cycles. Task to create. +- Data check of the Nov/9 fixes: [Analyze attribution data after stage 3](https://app.asana.com/1/137249556945/task/1220159402040627?focus=true). + +Decide later: + +- Mitigations that do not need the cause: retry the init from the page when no answer arrives, keep-warm A/B for the background. +- P6 has no fix. The wake group will not be solved without the new metrics. If they show nothing actionable, close the project and reset the target.