From 8de391c9bf95bd32961e65943c3aec878752fe9c Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Wed, 28 Oct 2026 13:52:27 +0600 Subject: [PATCH] [2026-10-28] eagle: work/projects/cpm-post-release-analysis-2026-10-27.md --- .../cpm-post-release-analysis-2026-10-27.md | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/work/projects/cpm-post-release-analysis-2026-10-27.md b/work/projects/cpm-post-release-analysis-2026-10-27.md index 66a2456f..62480d87 100644 --- a/work/projects/cpm-post-release-analysis-2026-10-27.md +++ b/work/projects/cpm-post-release-analysis-2026-10-27.md @@ -10,6 +10,47 @@ tags: Internal metrics; keep these figures out of public PRs. Previous analysis: [[cpm-metrics-causes-and-mitigations-2026-10-16]], [[cpm-status-2026-10-15]] (§3–5, 12–14 October cut). +## Priorities — macOS permanent failures first + +macOS fails far more (28.5% of DAU with a failure, 2.87% permanent, vs iOS 9.1% / 0.43%), so fixes are ordered by their share of **macOS permanent failures** (26–27 Oct, 598 users). Sources: idle wake 38%, P6 22%, extension updated 15%, other 14%, recent context load 7%, no background view 3%, error 6 0.5%. + +1. **Origin change → release P3 (#7493).** Cuts across groups: the macOS idle-wake patterns with the highest repeat rates follow `base_url_changed` (58–66% of their episodes permanent, 13–27 episodes per user, small groups). Every M4 reload and launch on released builds changes the origin, with a reproduced 1.5–32 s blackout. Already implemented; verify the effect on internal 1.214 data. +2. **P6, tab not connected (22% of macOS permanent; 39% of P6 users end up permanent).** Trigger unknown; the flag is not toggled in production. ~37% of P6 users show `extension_updated` before the failure, so investigate the mid-session upgrade path first, then fix. +3. **Extension updated (15%).** Load the new bundled version before the old one at launch, and keep the origin across upgrades. Repeats after every app update and likely feeds P6. +4. **Idle wake (38%, cause unknown).** + - Check Network-recovery false positives: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and `died_client` precedes 16% of macOS idle-wake failures. + - Add telemetry: events after the first failure, a healthy-navigation baseline, retry/recovery applied in the episode. + - Keep-warm experiment behind a flag (also a causal test). +5. **Other (14%):** needs the telemetry above. + +Then iOS: error-6 loop (20% of iOS permanent; restart the stale background and re-send `init`), recent context load (40%; origin change, covered by P3), extension updated (13%; same fix as macOS step 3). + +## Summary for the team — 26–27 Oct + +**CPM post-release analysis: iOS 7.243.0 / macOS 1.213.0, 26–27 Oct (2 full days)** + +Definitions: failure = no CPM dashboard response within 4 s after a page finished. Episode = one failure until resolution or the 5-minute deadline. Permanent = no response for 5 minutes, confirmed by a repeat failure or a reload. Affected users = users who reported a failure that day (daily pixel); all percentages are of DAU. Episodes = every failure report (count pixel). + +**Size** +- iOS: 9.1% of DAU hit a failure, 0.43% permanently. About 3× fewer failures than the previous release. +- macOS: 28.5% of DAU hit a failure, 2.87% permanently. No change vs the previous release. +- Most failures recover by themselves: 89% on iOS, 63% on macOS. + +**Where permanent failures come from** (share of permanent users) +- iOS (0.43% of DAU): recent context load 40%, error 6 20%, idle wake 16%, extension updated 13%, other 7%. +- macOS (2.87% of DAU): idle wake 38%, P6 (tab not connected) 22%, extension updated 15%, other 14%. + +**Persistent mechanisms** (repeat episodes from the same users confirm a stuck state) +- **iOS error-6 loop:** error 6 hits 0.86% of iOS DAU, and 10% of those users stay stuck. Stuck users average 7.4 episodes, recovered ones 1.2. In 38% of error-6 episodes the background keeps failing to load on every wake. Matches the stale-failed-background state reproduced in the harness. *Fix: restart the stale background on error 6 and re-send `init` to the page.* +- **Failures after an origin change (`base_url_changed`):** present in 1,733 iOS users (14% permanent, 3× baseline) and 193 macOS users (16%). On macOS these installs fail repeatedly (13–27 episodes per user in the idle-wake patterns, from small groups). The origin change itself causes a 1.5–32 s blackout depending on cache size (reproduced); P3 removes it. *Fix: release P3; verify on internal 1.214 / 7.244 data.* +- **macOS P6:** 1.6% of DAU, 39% of them permanent, about 2 episodes per user. A stuck tab, not a loop. ~37% of P6 users show `extension_updated` before the failure, so the mid-session upgrade is a likely trigger. *Fix: find the trigger (upgrade path first), then fix.* + +**Transient, not a stuck state** +- Extension updated: one episode per user, at upgrade time. Still worth fixing (load the new version first, keep the origin), since it repeats after every app update. +- Idle wake: mostly recovers (iOS 88%, macOS 58%). Its repeat-heavy part overlaps the origin-change patterns. Cause unknown; it may just be the state most macOS navigations are in. + +**Caveats:** two days after release, mostly early updaters (macOS ~17k DAU). Loop figures come from small groups. Not measured yet: retry attempts, events after the first failure, a healthy-navigation baseline. + ## What shipped - iOS **7.243.0** (tag 26 Oct 19:23 UTC), macOS **1.213.0** (tag 26 Oct 16:23 UTC). @@ -221,6 +262,9 @@ Remaining untested production ingredients: the real macOS `Tab` stack (DDG user - **Extension updated: yes, the outdated version is loaded.** `WebExtensionLifecycleCoordinator.loadAndSync` runs `loadInstalledExtensions()` (the previous, now outdated version) before `syncEmbeddedExtensions`, which installs the bundled version under a new identifier, loads it alongside, and unloads the old one. Early pages after an app update run against the old version, then the context is swapped with a new origin and the migration blackout. ### Next steps + +Order superseded by **Priorities — macOS permanent failures first** at the top of this note (28 Oct, two-day data); the items below remain the detailed list. + 1. **Ship what's understood:** - P3 release. - Extension updated: upgrade before load at launch; carry the origin across upgrades.