diff --git a/work/projects/cpm-post-release-analysis-2026-10-27.md b/work/projects/cpm-post-release-analysis-2026-10-27.md index 62480d87..0ef4d373 100644 --- a/work/projects/cpm-post-release-analysis-2026-10-27.md +++ b/work/projects/cpm-post-release-analysis-2026-10-27.md @@ -10,20 +10,25 @@ tags: Internal metrics; keep these figures out of public PRs. Previous analysis: [[cpm-metrics-causes-and-mitigations-2026-10-16]], [[cpm-status-2026-10-15]] (§3–5, 12–14 October cut). -## Priorities — macOS permanent failures first +## Priorities — macOS browser-wide breakage first -macOS fails far more (28.5% of DAU with a failure, 2.87% permanent, vs iOS 9.1% / 0.43%), so fixes are ordered by their share of **macOS permanent failures** (26–27 Oct, 598 users). Sources: idle wake 38%, P6 22%, extension updated 15%, other 14%, recent context load 7%, no background view 3%, error 6 0.5%. +macOS fails far more than iOS, so priorities follow **cross-tab permanent failures on macOS**: CPM not answering in two or more tabs for 5 minutes, confirmed by a repeat or reload. These are browser-wide breakage, not one stuck tab. 26–27 Oct, macOS 1.213: **386 users, 1.85% of DAU**. -1. **Origin change → release P3 (#7493).** Cuts across groups: the macOS idle-wake patterns with the highest repeat rates follow `base_url_changed` (58–66% of their episodes permanent, 13–27 episodes per user, small groups). Every M4 reload and launch on released builds changes the origin, with a reproduced 1.5–32 s blackout. Already implemented; verify the effect on internal 1.214 data. -2. **P6, tab not connected (22% of macOS permanent; 39% of P6 users end up permanent).** Trigger unknown; the flag is not toggled in production. ~37% of P6 users show `extension_updated` before the failure, so investigate the mid-session upgrade path first, then fix. -3. **Extension updated (15%).** Load the new bundled version before the old one at launch, and keep the origin across upgrades. Repeats after every app update and likely feeds P6. -4. **Idle wake (38%, cause unknown).** +Share of macOS cross-tab permanent: idle wake 174 (45%), other 78 (20%), extension updated 77 (20%), recent context load 35 (9%), no background view 11 (3%), P6 9 (2%), error 6 2 (0.5%). + +1. **Idle wake (45%, cause unknown).** - Check Network-recovery false positives: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and `died_client` precedes 16% of macOS idle-wake failures. - - Add telemetry: events after the first failure, a healthy-navigation baseline, retry/recovery applied in the episode. + - Add telemetry: events after the first failure (two-phase `bg_events`), a healthy-navigation baseline, retry/recovery applied in the episode. - Keep-warm experiment behind a flag (also a causal test). -5. **Other (14%):** needs the telemetry above. +2. **Extension updated (20%).** Load the new bundled version before the old one at launch, and keep the origin across upgrades. Repeats after every app update. +3. **Other (20%).** No distinguishing signal yet; needs the telemetry from step 1. -Then iOS: error-6 loop (20% of iOS permanent; restart the stale background and re-send `init`), recent context load (40%; origin change, covered by P3), extension updated (13%; same fix as macOS step 3). +Not priorities: +- **P3 / origin change:** the baseURL fix is released. Verify its effect on recent context load, post-reload silence and the origin-change patterns (`base_url_changed`) once the build reaches users; no further work planned. +- **P6:** 96% of P6 users have a single failing tab; 122 of its 131 permanent users are single-tab, only 9 cross-tab. One stuck tab, not browser-wide breakage. Low priority; find the trigger when convenient (~37% show `extension_updated` before the failure). +- **Error 6 on macOS:** 2 cross-tab users. + +iOS (cross-tab permanent 861 users, 0.26% of DAU): recent context load 38% (origin change; covered by the released P3), error 6 20% (error-6 loop: restart the stale background and re-send `init`), extension updated 14% (same fix as macOS step 2), idle wake 14%, other 8%, no background view 5%. ## Summary for the team — 26–27 Oct @@ -38,12 +43,12 @@ Definitions: failure = no CPM dashboard response within 4 s after a page finishe **Where permanent failures come from** (share of permanent users) - iOS (0.43% of DAU): recent context load 40%, error 6 20%, idle wake 16%, extension updated 13%, other 7%. -- macOS (2.87% of DAU): idle wake 38%, P6 (tab not connected) 22%, extension updated 15%, other 14%. +- macOS (2.87% of DAU): idle wake 38%, P6 (tab not connected) 22%, extension updated 15%, other 14%. Browser-wide (cross-tab) only, 1.85% of DAU: idle wake 45%, other 20%, extension updated 20%, recent context load 9%, P6 2%. **Persistent mechanisms** (repeat episodes from the same users confirm a stuck state) - **iOS error-6 loop:** error 6 hits 0.86% of iOS DAU, and 10% of those users stay stuck. Stuck users average 7.4 episodes, recovered ones 1.2. In 38% of error-6 episodes the background keeps failing to load on every wake. Matches the stale-failed-background state reproduced in the harness. *Fix: restart the stale background on error 6 and re-send `init` to the page.* -- **Failures after an origin change (`base_url_changed`):** present in 1,733 iOS users (14% permanent, 3× baseline) and 193 macOS users (16%). On macOS these installs fail repeatedly (13–27 episodes per user in the idle-wake patterns, from small groups). The origin change itself causes a 1.5–32 s blackout depending on cache size (reproduced); P3 removes it. *Fix: release P3; verify on internal 1.214 / 7.244 data.* -- **macOS P6:** 1.6% of DAU, 39% of them permanent, about 2 episodes per user. A stuck tab, not a loop. ~37% of P6 users show `extension_updated` before the failure, so the mid-session upgrade is a likely trigger. *Fix: find the trigger (upgrade path first), then fix.* +- **Failures after an origin change (`base_url_changed`):** present in 1,733 iOS users (14% permanent, 3× baseline) and 193 macOS users (16%). On macOS these installs fail repeatedly (13–27 episodes per user in the idle-wake patterns, from small groups). The origin change itself causes a 1.5–32 s blackout depending on cache size (reproduced); P3 removes it. *P3 is released; verify its effect.* +- **macOS P6:** 1.6% of DAU, 39% of them permanent, about 2 episodes per user. 96% are a single stuck tab (122 of 131 permanent are single-tab), not browser-wide breakage. ~37% of P6 users show `extension_updated` before the failure, so the mid-session upgrade is a likely trigger. *Fix: find the trigger (upgrade path first), then fix.* **Transient, not a stuck state** - Extension updated: one episode per user, at upgrade time. Still worth fixing (load the new version first, keep the origin), since it repeats after every app update. @@ -263,7 +268,7 @@ Remaining untested production ingredients: the real macOS `Tab` stack (DDG user ### Next steps -Order superseded by **Priorities — macOS permanent failures first** at the top of this note (28 Oct, two-day data); the items below remain the detailed list. +Order superseded by **Priorities — macOS browser-wide breakage first** at the top of this note (28 Oct, two-day data); the items below remain the detailed list. 1. **Ship what's understood:** - P3 release.