git_admin/obsidian-vault
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[2026-10-29] eagle: work/projects/cpm-post-release-analysis-2026-10-27.md

Browse Source
This commit is contained in:
Alexey Martemyanov committed 2026-10-29 13:11:17 +06:00
1 parent 0855594c8b
commit 67ab7c2fbe
1 file changed
+30 -11
Download Patch File Download Diff File Expand all files Collapse all files
work/projects/cpm-post-release-analysis-2026-10-27.md
+30 -11
View File
@@ -20,6 +20,7 @@ Share of macOS cross-tab permanent: idle wake 174 (45%), other 78 (20%), extensi
- Check Network-recovery false positives: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and `died_client` precedes 16% of macOS idle-wake failures.
- Add telemetry: events after the first failure (two-phase `bg_events`), a healthy-navigation baseline, retry/recovery applied in the episode.
- Keep-warm experiment behind a flag (also a causal test).
- **Update 28–29 Oct:** error pages were found to be counted as CPM failures with exactly the wake signature (reproduced in the harness and in a macOS integration test); fix in PRs #7752 (macOS 1.214.0) and #7753 (iOS 7.244.0). Details and the other checks in the section "Idle wake investigation — 28 October 2026" below.
2. **Extension updated (20%).** Load the new bundled version before the old one at launch, and keep the origin across upgrades. Repeats after every app update.
3. **Other (20%).** No distinguishing signal yet; needs the telemetry from step 1.
@@ -117,6 +118,7 @@ Per-install rough rates (cohort share × failure rate), old init → new episode
- **A** (`ready=failed`, app_state active): `dealloc → view → proxy_on → interruption → error_6`, often after `died_client`/`died_crash`/`died_crash_limit`. The background load spans an app suspension and fails at resume.
- **B** (`ready=pending`): an older error 6, then a new view whose load is still pending 8–30 s later.
- The P2 retry fires only while foregrounded with a 60 s cooldown (277/day). **Foreground-resume recovery is the missing piece.**
- *Update 28 Oct: checked with a real HOME and suspension: the retry is skipped in the background and nothing happens on foregrounding, but the page was still answered because WebKit re-woke the background itself; see the 28 October section.*
3. **macOS P6 is untreated**: 5.8% of episodes, 70% never answered. iOS now shows `tab_has_ext_scripts=false` in 5.3% of episodes, concentrated on iOS 26.7 (16%) and 27.0 (8%), with 0 on 27.2. Previously iOS was ≈0.
4. **macOS Network replacement is frequent**: the applied pixel is on 9% of 1.213 install-days. With recovery applied it no longer appears as a cohort, but `other` stays at 20%.
5. **After M4 on macOS**: `reload_state=loaded` → 46% no response by deadline; `response_timed_out` → 57%. The reload often does not restore messaging, or no qualifying navigation follows.
@@ -136,6 +138,8 @@ Per-install rough rates (cohort share × failure rate), old init → new episode
| R2 iOS error 6 across suspension | 7.5% eps / 25% stuck, all under 5 s, interruption between view and error | WebKit load timer spans suspension; nothing retries on resume | on foreground resume, retry `loadBackgroundContent` when the last load failed or is pending |
| R3 P6 controller mismatch | macOS 5.8%, 70% unanswered | known main-flag manager replacement | rebind/recreate tab web views to the current controller |
*Update 28 Oct for R2: reproduced with a real HOME and a 45 s suspension. The retry is skipped while the app is in the background, but WebKit created a new view 20–25 ms after the error and the page was answered (0.6 s after resume), so the suspension case self-heals in the simulator. Persistence of this family in production is not explained (172 permanent episodes). See the 28 October section.*
WebKit source pointers (`~/Downloads/WebKit_sparse`): `WebExtensionContext.cpp` `wakeUpBackgroundContentIfNecessaryToFireEvents` (drops events when the listener set lacks the type after the first load), `scheduleBackgroundContentToUnload` (30 s, reset only on dispatch), `WebExtensionContextCocoa.mm` `loadBackgroundWebView` / `performTasksAfterBackgroundContentLoads`, `WebExtensionContextAPIRuntimeCocoa.mm` `runtimeSendMessage` (empty `processes()` → `{}` reply). 324743 (`9bcf5840ef`) holds runtime messages until SW activation; it is not in shipped OS builds.
## Repro log — 27 October 2026
@@ -264,6 +268,7 @@ Remaining untested production ingredients: the real macOS `Tab` stack (DDG user
### Corrections
- **P6 is not the feature flag.** `webExtensions` is on in production, not toggled, and will be removed. The flag-cycle mechanism (also in the `tabControllerMatchesContext` doc comment) was an old unverified hypothesis. Established: a tab's web view gets the controller only at creation and only if `webExtensionManager` exists then (`WKWebViewConfigurationExtensions` → `attachWebExtensions`), and the manager is created once at launch and replaced only by the flag handler. **Trigger unknown.** Candidates: web views created before the launch-time manager; the flag evaluating off at launch for some installs (missing or stale cached config); a web view built outside the standard configuration path. Macos 1.213 on 26 Oct: 51 users with controller mismatch + no scripts, 19 permanent (22% of macOS permanent hangs). The fix depends on the trigger.
- **Error 6: the P2 retry is shipped and helped.** iOS error-6 users fell from 1.3% to 0.5% of DAU, and 455 iOS users/day had a successful `background_load_retry`. Error-6 episodes have outcomes: iOS 86% recovered later, 9% permanent. It is still 25% of iOS stuck because (a) the retry restores the background but not the page whose `init` was lost, which nothing re-sends (late reload off); (b) the retry is skipped while the app is in the background, within the 60 s cooldown, or during M4; (c) when the failed background view is kept (H3, reproduced), the retry cannot complete. Which dominates is unknown: no retry attempted/result field in the episode.
- *Update 28 Oct: (a) was not reproduced for suspension (the page is answered after resume, WebKit creates a new view); the retained-view case (b/H3) is reproduced separately. See the 28 October section.*
- **Extension updated: yes, the outdated version is loaded.** `WebExtensionLifecycleCoordinator.loadAndSync` runs `loadInstalledExtensions()` (the previous, now outdated version) before `syncEmbeddedExtensions`, which installs the bundled version under a new identifier, loads it alongside, and unloads the old one. Early pages after an app update run against the old version, then the context is swapped with a new origin and the migration blackout.
### Next steps
@@ -274,6 +279,7 @@ Order superseded by **Priorities — macOS browser-wide breakage first** at the
- P3 release.
- Extension updated: upgrade before load at launch; carry the origin across upgrades.
- Error 6: restart the stale background on error 6 when the failed view is still alive, and re-init the page that lost `init`.
- *Update 28 Oct: implemented as the restart in the section "Proposed fix" below (PR #7730, closed, branch kept). The "re-init the page that lost `init`" part was not needed in the simulator: the page was answered after the restart.*
- P6: find the trigger first (the three candidates above), then fix.
2. **Verify the macOS Network-recovery false-positive lead:** recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and `died_client` precedes 16% of macOS idle-wake failures. Review `CPMNetworkProcessRecovery` detection; count recovery pixels in the harness screens.
3. **Telemetry** (one privacy review): events after the first failure (two-phase `bg_events`); a healthy-navigation baseline (background state on successful measurements); retry attempted/result; recovery applied in this episode; time since app resume.
@@ -321,6 +327,17 @@ Internal metrics; keep these figures out of public PRs. Data: ClickHouse via `~/
The background page loads with `navigator.serviceWorker.register(...)`; its rejection carries the WebKit error text (`Error: <message>`), obtainable with a `WKUserScript` and `WKScriptMessageHandler` on the background web view delivered by `didCreateBackgroundWebView` (before the page loads). Message only, no file or line. The extension's own `extensionException` messages are not handled by the app (unknown method). Verified once with an injected exception.
### Proposed fix: restart the context when the failed view is retained (PR #7730, closed)
- **Behavior.** On error 6 (current context, recovery flag on, app in the foreground, no M4 in progress, 60 s cooldown) the manager checks `WKWebExtensionContext._backgroundWebView`. If the failed view is still alive it reloads the embedded extension at once (new reload trigger `cpmBackgroundLoadFailure`) and loads the background. If the view was released it keeps the plain P2 retry, which preserves the failed document's init.
- **Why.** WebKit's `loadBackgroundWebViewIfNeeded` returns early while the view exists (`m_backgroundWebView` set), and after a failed `_loadServiceWorker` WebKit records error 6 without an underlying error and does not unload the view. The retry therefore cannot replace it.
- **Flag.** `cpmBackgroundLoadRestart`: remote kill switch, default enabled in iOS and macOS `FeatureFlag`, off by default in the manager API. Separate from `cpmBackgroundLoadRecovery`, so turning it off keeps the plain retry. Feature Flags Registry task: https://app.asana.com/1/137249556945/project/1211834678943996/task/1219970564251181 (created by a parallel session, linked in code).
- **Telemetry.** No new pixel. The only change is a new value `cpm_background_load_failed` of the `trigger` parameter of the existing reload pixels, and those fire only when the reload itself fails. Nobody could count how often the restart ran or whether it helped. Proposal, not done: a recovery pixel like P2, for example `web_extension_cpm_background_load_restart` with `outcome=background_ready`; needs the pixel definition and privacy review.
- **Tests.** Three unit tests in `WebExtensionManagerTests` (restart with a retained view, plain retry with the flag off, plain retry with a released view). They run on mocks and show only the manager's decision; red without the branch, green with it, also in release. They do not reproduce the failure.
- **Real check.** Simulator, real CPM, injected exception at worker start: without the fix the page is unanswered until M4 (about 54 s); with the fix `Reloading extension` comes 8 ms after the error and the page is answered after 0.6 s.
- **Branch.** `alex/cpm-error6-restart`: `49d4bbb309` (test), `4997652d87` (fix, flag), `a417730e60` (registry link). PR #7730 was a draft, closed on 28 Oct because it was opened before the problem was reproduced; the branch is kept.
- **Does not cover.** The 25–35 s loop (view is released each time), the `interruption` family (suspension self-heals in the simulator), and any case where the retained view is not the problem. How often the state occurs in production is unknown (no field).
### Open
- Why 172 permanent episodes of the `interruption` family do not recover, when 2,797 identical-looking ones do.
@@ -360,7 +377,7 @@ Re-cut of the post-release data (macOS 1.213.x, 26–27 Oct, matured since the s
- **Hypothesis, not verified:** pages that fail to load are shown on every tab after wake from sleep or a network drop. Each would count as a CPM failure right after an idle unload (fresh `dealloc>view>proxy_on`, background ready in under 1 s, healthy diagnostics), in several tabs (cross-tab) and with no answer until a normal navigation (permanent-looking, and M4 would not help). A same-tab second timeout also triggers M4 on a healthy extension (see `was_stuck` note above), which then adds the origin-change blackout. The pixel has no URL, so data cannot separate this; the check has to be an app run (debug build, Wi-Fi off, several tabs, wake) with the monitor log, or filtering `.alternateHtmlLoad` / `duck://error` and comparing wake rates.
### Notes
- The "P3" lines above disagree: priorities say the baseURL fix is released; "What shipped" says P3 (#7493) is on main only and not in 1.213. Re-check before relying on either.
- The "P3" lines above disagreed. Checked on 29 Oct: `CPMExtensionBaseURLPolicy` is absent from the 1.213.0 and 7.243.0 release tags and present in `release/macos/1.214.0` and `release/ios/7.244.0`. So P3 is not in 1.213 / 7.243 ("What shipped" is right) and ships with 1.214.0 / 7.244.0, which also makes the 26.6+ gate in #7752 / #7753 relevant to those releases.
- Scratch SQL runner and queries are in the session scratchpad; the harness test is not committed or pushed.
### Addendum 28 Oct — error pages produce the wake signature (replaces the first version of this addendum)
@@ -372,13 +389,15 @@ Re-cut of the post-release data (macOS 1.213.x, 26–27 Oct, matured since the s
- **Weak evidence against, kept for the record:** per-OS correlation of error pages per install with wake episodes per install is r = 0.02 (n = 9); hourly r = 0.88 for both wake and non-wake (rollout trend, uninformative).
- **Unverified:** the running app (the scenario with `log stream --level debug --predicate 'subsystem == "WebExtensions" AND eventMessage CONTAINS "CPM Health Monitor"'`: three tabs idle >30 s, load a dead host in each, expect `timeoutElapsed` without `Dashboard response attributed`); iOS error pages; the share of wake episodes that are error pages (the episode pixel has no URL or error code).
- **Decisive options:** (a) macOS build where the monitor ignores error documents (`.alternateHtmlLoad` / failed provisional navigations) and M4 never fires for them; compare wake share and permanent rate with the current build; (b) a flag parameter on the episode pixel (needs the privacy review you wanted to avoid).
- **Filter implemented (28 Oct, uncommitted, not pushed):** worktree `apple-browsers.git/cpm-error-page-filter`, branch `alex/cpm-error-page-filter` from `8023e4cb5c`. New `CPMNavigationKind.errorPage`; `CPMMessagingHealthMonitor` does not measure a commit/finish of a tab whose current navigation is `.errorPage` (`commitNavigation`, `finishNavigation`), so no timeout, no episode and no M4 for it; macOS `Navigation.cpmMessagingNavigationKind` returns `.errorPage` for `.alternateHtmlLoad` (`PrivacyDashboardTabExtension.swift`). Tests: two monitor unit tests (`testErrorPageIsNotMeasured`, `testNavigationAfterErrorPageIsMeasured`) and the harness `CPMReproErrorPageTests` (3 tests, `CPM_REPRO=1`), all green; the same harness tests failed (`other / failed_tab_response / tabs=1`) before the change, and with it the background still wakes at the navigation but no episode is counted. SwiftLint strict: 0 violations. Not done: iOS error pages (no `loadAlternateHTML` path found on iOS; not checked how they are shown), running the real app, comparing wake share between builds.
- **macOS app build with the filter:** `xcodebuild -scheme "macOS Browser" -configuration Debug CODE_SIGNING_ALLOWED=NO` succeeded (the change compiles in the app target). Product: `/private/tmp/claude-501/-Users-admin-DuckDuckGo-apple-browsers-git/2490a862-c422-4358-b940-1f30df9cbb83/scratchpad/app-dd/Build/Products/Debug/DuckDuckGo.app` (scratchpad of that session; may be cleaned). The app was not launched: no window run on the host Mac. The real-app check is still open: three tabs idle >30 s, a dead host in each, `log stream` for `CPM Health Monitor`; with the filter there should be no `timeoutElapsed` for the error pages.
- **Integration test (28–29 Oct):** `macOS/IntegrationTests/Tab/CPMErrorPageMonitoringTests.swift` (added to both Integration Tests targets in the pbxproj). Real `Tab` + `WKWebView` + `TestSchemeHandler` failing with `NSURLErrorCannotFindHost`; the tab gets a stub `WebExtensionManaging` whose health monitor records events. Asserts the monitor sees `started:other`, `failed`, then `started:error_page`, and the error page finish for the failed URL arrives after it (the real macOS path: `Navigation` alternate load → `.errorPage`). `build-for-testing` of scheme `macOS Integration Tests` succeeded and SwiftLint strict is clean; **not run locally** (integration tests are left to CI, no host windows). Without the macOS mapping the second start would be `started:other` and the test would fail; that was not observed, only reasoned.
- **Integration test run locally (29 Oct), correction of the line above:** with the signed Debug build (dev identity, entitlements present), `macOS Integration Tests` → `CPMErrorPageMonitoringTests` **passes** (1 test, 0 failures). Negative control with the macOS mapping removed from `PrivacyDashboardTabExtension.cpmMessagingNavigationKind`: the same test **fails** in the real app stack, with the monitor receiving `["started:other", "failed", "started:other", "committed:https://testhost.com/", "finished:https://testhost.com/"]`, i.e. the error page is reported as an ordinary navigation of the failed https URL (this is the false positive, observed in the app's own `Tab`/`Navigation` path, not only in the harness). The mapping was restored afterwards. Note for running locally: do not build with `CODE_SIGNING_ALLOWED=NO`; the host app then lacks the keychain entitlement (`-34018`) and macOS asks for keychain access.
- **Corner cases checked (29 Oct):**
- Integration tests (real `Tab`, `macOS Integration Tests`, run locally with the signed Debug build), all pass: ordinary load error (`-1003`), SSL error (`NSURLErrorServerCertificateUntrusted`), malicious-site warning (`MaliciousSiteError`, via the `duck://error` redirect). In each, after the failed load the page starts as `.errorPage`, so the monitor never measures it.
- Harness (`CPMReproCornerCaseTests`, real WebKit, real extension, loopback HTTP), all pass: HTTP Basic auth with accepted credentials (CPM answers) and with a cancelled challenge (401 page, no episode); client redirects at 0.1 s, 2 s, meta refresh and a redirect after the 4 s grace (CPM answers on the target, no episode); PDF, text/plain and JSON documents (CPM answers on all three, content scripts do run on PDF); a Fire-window tab with a non-persistent data store (CPM answers). A 6 s JS timer redirect never fired in a window-less web view (timer throttling), which is a test artefact.
- Not checked: downloads, http→https upgrade redirects, background or minimised windows, mass tab opening, popups from `about:blank`, closing a tab mid-measurement, iOS error pages, the macOS basic-auth dialog UI itself (by code reading nothing is committed while it is open, and cancel leads to no error page).
- **More corner cases checked (29 Oct, harness, pass):** a link that becomes a download (`Content-Disposition: attachment`; WebKit fails the provisional navigation with "Frame load interrupted", no document, no episode); 25 tabs loaded at once with cross-origin frames (all finished in 0.8 s, all answered, no episode). Tab close during a measurement is covered by existing monitor unit tests (`testTabCloseCancelsCurrentMeasurement` and two more). iOS (by code reading): SSL/malicious special pages load via `loadSimulatedRequest` for the failed URL, an ordinary document on which CPM answers (same as every simulated page in the harness); other iOS load errors use a native error view, no document. Policy redirects (`.redirect`, used for malicious sites) are exercised by the malicious integration test. Still not testable here: real minimised/occluded windows and App Nap (harness web views have no window and still answer in milliseconds), http→https upgrade through the app's HTTPS upgrade store.
- **Stable base URL (P3) limited to macOS/iOS 26.6+ (29 Oct, same branch, uncommitted):** `CPMExtensionBaseURLPolicy` has `isSupportedOS` (`guard #available(macOS 26.6, iOS 26.6, *) else { return false }`). On older OS `baseURL(for:)` returns nil (WebKit's random origin) without rotating the stored origin, and `recordBackgroundLoadFailure` does nothing. The 22 existing policy tests pass (this Mac is 26.7.1, CI image is `osx-tahoe-27`); the unsupported branch cannot be exercised on those hosts, so it has no test.
- **Fix, tests and PRs (state on 29 Oct):**
- **PRs, same commit:** [#7752](https://github.com/duckduckgo/apple-browsers/pull/7752) into `release/macos/1.214.0` (branch `alex/cpm-error-page-filter`) and [#7753](https://github.com/duckduckgo/apple-browsers/pull/7753) into `release/ios/7.244.0` (branch `alex/cpm-error-page-filter-ios`); reviewer `miasma13`; CI had not started when they were opened. Worktrees `apple-browsers.git/cpm-error-page-filter` and `…-ios`.
- **Commit content:** `CPMNavigationKind.errorPage` and a monitor that does not measure commit/finish of such a navigation (so no timeout, no episode, no M4); macOS maps `.alternateHtmlLoad` to `.errorPage` in `PrivacyDashboardTabExtension.cpmMessagingNavigationKind`; `CPMExtensionBaseURLPolicy.isSupportedOS` (`#available(macOS 26.6, iOS 26.6, *)`) keeps WebKit's random origin below 26.6 without rotating the stored origin; two monitor unit tests; macOS integration tests `CPMErrorPageMonitoringTests` (both Integration Tests targets in the pbxproj). Harness tests and the reproduction suite are not part of it (the harness is not in the release branches).
- **Run results (macOS release base):** 125 package tests (103 monitor, 22 base URL policy) and the 3 integration tests pass; on the iOS release base the same 125 package tests pass. iOS app and iOS tests were not built (no iOS source changed, no exhaustive `switch` on `CPMNavigationKind` there). The old-OS branch of the base URL gate has no test (hosts run 26.6+).
- **Integration tests** (real `Tab`, `macOS Integration Tests`, run locally with the signed Debug build): ordinary load error (`-1003`), SSL (`NSURLErrorServerCertificateUntrusted`), malicious-site warning (`MaliciousSiteError` via the `duck://error` redirect) all pass. Negative control with the macOS mapping removed: the test fails, the monitor receives `["started:other", "failed", "started:other", "committed:https://testhost.com/", "finished:https://testhost.com/"]`, the error page as an ordinary navigation of the failed URL (the false positive in the app's own `Tab`/`Navigation` path); mapping restored. They use their own `StubWebExtensionManager`; `WebExtensionsTestSupport/MockWebExtensionManaging` exists but that product is not linked to the Integration Tests targets.
- **Running locally:** do not build with `CODE_SIGNING_ALLOWED=NO`: the host app then lacks the keychain entitlement (`-34018`) and macOS asks for keychain access. Use the normal signed Debug build, `xcodebuild … test-without-building -only-testing:"Integration Tests/CPMErrorPageMonitoringTests"`.
- **Corner cases checked (29 Oct), no further false-positive source found:**
- Harness (`CPMReproCornerCaseTests`, real WebKit, real extension, loopback HTTP), all pass: HTTP Basic auth accepted (CPM answers) and cancelled (401 page, no episode); client redirects at 0.1 s, 2 s, meta refresh and a redirect after the 4 s grace (CPM answers on the target, no episode); PDF, text/plain and JSON documents (content scripts run on PDF too); a Fire-window tab with a non-persistent store; a download link (WebKit fails the provisional navigation, no document, no episode); 25 tabs loaded at once (all answered in under a second). A 6 s JS timer redirect never fired in a window-less web view (timer throttling), a test artefact.
- Tab close during a measurement is covered by existing monitor unit tests. Policy redirects are exercised by the malicious-site integration test.
- iOS by code reading only: SSL/malicious special pages load via `loadSimulatedRequest` for the failed URL, an ordinary document on which CPM answers; other load errors use a native error view without a document.
- **Not checked:** real minimised/occluded windows and App Nap (harness web views have no window and still answer in milliseconds); http→https upgrade through the app's HTTPS upgrade store; popups from `about:blank`; iOS app behaviour at runtime.
- **What this does not settle:** the share of wake episodes that are error pages is unknown (the episode pixel has no URL or error code); the install-day ceiling is ~10% (error pages) against 13–15% (wake). Real CPM breakages stay as before: error 6 on iOS, origin-change blackout, P6, extension update, `died_client` (~5% of wake). The check after release: wake episodes per loaded install-day should drop by roughly the error-page share, with `m.mac.errorpageshown.other` as the control.