[2026-10-30] eagle: work/projects/cpm-post-release-analysis-2026-10-27.md work/projects/cpm-status-2026-10-15.md work/projects/cpm-web-extension-breakage-findings.md work/projects/cpm-web-extension-code-guide-2026-10-09.md

This commit is contained in:
Alexey Martemyanov committed 2026-10-30 11:03:45 +06:00
1 parent 558832b4fb
commit 9623dd1e5d
4 files changed
+74 -16

No files matched your search

@@ -1,15 +1,45 @@
---
title: 'CPM — first post-release data (iOS 7.243.0 / macOS 1.213.0)'
title: CPM — first post-release data (iOS 7.243.0 / macOS 1.213.0)
date: '2026-10-27'
tags:
- work
- cpm
- analysis
updated: '2026-10-30'
---
# CPM: first post-release data, 26 October 2026
Internal metrics; keep these figures out of public PRs. Previous analysis: [[cpm-metrics-causes-and-mitigations-2026-10-16]], [[cpm-status-2026-10-15]] (§3–5, 12–14 October cut).
## Release update — 30 October 2026
- **Merged, not yet verified as publicly released:** [iOS #7753](https://github.com/duckduckgo/apple-browsers/pull/7753) into `release/ios/7.244.0` (merge `26a461e30e73608e75e6d84e09bca0ced842993c`) and [macOS #7752](https://github.com/duckduckgo/apple-browsers/pull/7752) into `release/macos/1.214.0` (merge `1e392bac233d7e3e5ccef340ebc82ad7aee72922`). Latest public GitHub releases checked on 30 Oct are still 7.243.0 / 1.213.0. Merge does not establish rollout or production effect.
- **Final stable-origin behavior:** set `baseURL` only on macOS/iOS 26.6+ with the feature flag enabled. Creating a context with an unsupported OS or disabled flag rotates the stored CPM origin and returns `nil` (WebKit chooses the context's origin). Toggling the flag alone does not change a live context; an off→on interval with no intervening context creation does not rotate anything. User reports the remote flag disabled; live remote configuration was not independently audited.
- **Only a storage-key rename:** property `cpmOriginIdentifier` now encodes/decodes JSON key `cpmStableOriginIdentifier`. Old `cpmOriginIdentifier` values are ignored. There is no migration, UUID prefill, OS-version marker or reset on every OS upgrade. Missing new-key value falls back to the already persisted installation `uniqueIdentifier`; this is reuse of that UUID, not creation of a fresh UUID. Subsequent launches reuse it. A later normal rotation saves a replacement under the new key and reuses that replacement.
- **Public-build transition:** 7.243 / 1.213 persist the installation UUID but do not assign it as the extension's baseURL. WebKit chooses a random origin for each new context. The first enabled stable-origin load therefore changes the origin once. An internal build with a replacement saved under the old key falls back to the installation UUID after this key change.
- **Error-page fix:** the health monitor skips error-page commit/finish; macOS maps `.alternateHtmlLoad` to `.errorPage`. These documents no longer start timeout measurements, episodes or M4 reloads. This does not establish what fraction of production wake episodes were error pages.
- **Validation:** the key-only revision passed all 1,025 WebExtensions package tests and strict lint; the final simplified policy suite passed 25 tests, including injected unsupported-OS coverage, with strict lint on both branches. The original release-base error-page change passed three macOS app integration tests (DNS, SSL and malicious-site warning), with a failing negative control when the mapping was removed. Those app tests were not rerun after the key/test-only edits; iOS runtime behavior was not tested. Final branch heads: iOS `fef5f9cdd35977b0cb848ac81f4feaca3e6a9ba6`, macOS `551cb2edd00c02e038b2b46a07a8c32d0b33e666`.
- **Mock follow-up:** the local `StubWebExtensionManager` stays in the release tests by reviewer agreement. Moving them to the shared mock is a [main-branch follow-up](https://app.asana.com/1/137249556945/project/1202406491309510/task/1220003778559454?focus=true).
### Stable-origin evidence and limits
The demonstrated benefit is avoiding WebKit's origin-change data cleanup before injection/background startup: real-cache reproductions measured about 1.5 / 10.9 / 32.2 seconds at 20k / 200k / 600k cache entries with changing origins, versus about 0.1 seconds with a stable origin. See [WebKit bug 320812](https://bugs.webkit.org/show_bug.cgi?id=320812) and the reproduction notes below. Production symptom cohorts are not proof that every matching episode has this cause.
No test demonstrated an extension staying permanently broken after an OS upgrade because its baseURL was saved. A live-session background-process termination did reproduce an error-6 loop that origin rotation recovered. In the real two-process restart baseline, a full app restart restored messaging even while reusing the previous origin. Evidence: `alex/cpm-p3-integration-tests/macOS/IntegrationTests/CPM/README.md`, “two-process” prepare/resume results. Do not turn the live-session failure into a proven persistent-upgrade failure.
The earlier claim that extension workers were *never* persisted was incorrect. [WebKit commit eff9f9231ad90e31e6ae2b128d2269c589550c8d](https://github.com/WebKit/WebKit/commit/eff9f9231ad90e31e6ae2b128d2269c589550c8d) (312316@main, 29 April 2026) stops new extension service-worker registrations being written to disk; that change does not purge existing records. Historical records are therefore possible, but this does not demonstrate the proposed upgrade failure.
Pixels can show no response within five minutes, cross-tab failures, error 6 and reload outcomes. They cannot uniquely identify old-origin reuse after an OS upgrade; `base_url_changed` compares contexts within the current process, not across launches. “Permanent” in this analysis means the five-minute reporting outcome, not proof of an unrecoverable extension.
### OS split of the no-background-view symptom — query on 29 October
26–28 Oct (metrics date in Eastern time), released iOS 7.243.x / macOS 1.213.x without P3. Count-leg episode rows only, not users: `context_loaded=true AND bg_view_alive=false`, excluding a nonempty `bg_load_error_after` or an `error_[0-9]` event in `bg_events`.
- iOS: OS <26.6 — 695 of 14,537 failure episodes (4.78%); OS 26.6+ — 27,670 of 97,982 (28.24%). Thus 97.55% of this symptom cohort is on 26.6+. Confirmed no-response subset: 43 / 347 respectively.
- macOS: OS <26.6 — 274 of 13,206 (2.07%); OS 26.6+ — 961 of 29,770 (3.23%). Thus 77.81% of this symptom cohort is on 26.6+. Confirmed no-response subset: 29 / 98 respectively.
Source: `metrics.pixels`, `pixel_id='web'`, `web.extension.cpm.episode.count%`, agents `ddg_ios` / `ddg_mac_desktop`; OS split by major/minor tuple at (26,6). These are symptom counts, not a causal P3 share or a stable-origin safety comparison. The query was run on 29 Oct; it was not rerun for this documentation update.
## Priorities — macOS browser-wide breakage first
macOS fails far more than iOS, so priorities follow **cross-tab permanent failures on macOS**: CPM not answering in two or more tabs for 5 minutes, confirmed by a repeat or reload. These are browser-wide breakage, not one stuck tab. 26–27 Oct, macOS 1.213: **386 users, 1.85% of DAU**.
@@ -25,11 +55,11 @@ Share of macOS cross-tab permanent: idle wake 174 (45%), other 78 (20%), extensi
3. **Other (20%).** No distinguishing signal yet; needs the telemetry from step 1.
Not priorities:
- **P3 / origin change:** the baseURL fix is released. Verify its effect on recent context load, post-reload silence and the origin-change patterns (`base_url_changed`) once the build reaches users; no further work planned.
- **P3 / origin change:** the baseURL fix is in the upcoming 7.244 / 1.214 release branches. Public rollout and an enabled flag are prerequisites for measuring its effect; recent context load is not by itself proof of origin-change cleanup.
- **P6:** 96% of P6 users have a single failing tab; 122 of its 131 permanent users are single-tab, only 9 cross-tab. One stuck tab, not browser-wide breakage. Low priority; find the trigger when convenient (~37% show `extension_updated` before the failure).
- **Error 6 on macOS:** 2 cross-tab users.
iOS (cross-tab permanent 861 users, 0.26% of DAU): recent context load 38% (origin change; covered by the released P3), error 6 20% (error-6 loop: restart the stale background and re-send `init`), extension updated 14% (same fix as macOS step 2), idle wake 14%, other 8%, no background view 5%.
iOS (cross-tab permanent 861 users, 0.26% of DAU): recent context load 38% (a symptom targeted in part by upcoming P3; P3 is absent from this measured release), error 6 20% (error-6 loop: restart the stale background and re-send `init`), extension updated 14% (same fix as macOS step 2), idle wake 14%, other 8%, no background view 5%.
## Summary for the team — 26–27 Oct
@@ -48,7 +78,7 @@ Definitions: failure = no CPM dashboard response within 4 s after a page finishe
**Persistent mechanisms** (repeat episodes from the same users confirm a stuck state)
- **iOS error-6 loop:** error 6 hits 0.86% of iOS DAU, and 10% of those users stay stuck. Stuck users average 7.4 episodes, recovered ones 1.2. In 38% of error-6 episodes the background keeps failing to load on every wake. Matches the stale-failed-background state reproduced in the harness. *Fix: restart the stale background on error 6 and re-send `init` to the page.*
- **Failures after an origin change (`base_url_changed`):** present in 1,733 iOS users (14% permanent, 3× baseline) and 193 macOS users (16%). On macOS these installs fail repeatedly (13–27 episodes per user in the idle-wake patterns, from small groups). The origin change itself causes a 1.5–32 s blackout depending on cache size (reproduced); P3 removes it. *P3 is released; verify its effect.*
- **Failures after an origin change (`base_url_changed`):** present in 1,733 iOS users (14% permanent, 3× baseline) and 193 macOS users (16%). On macOS these installs fail repeatedly (13–27 episodes per user in the idle-wake patterns, from small groups). The origin change itself causes a 1.5–32 s blackout depending on cache size (reproduced); P3 removes it. *P3 is merged into upcoming release branches; verify its effect after public rollout and flag enablement.*
- **macOS P6:** 1.6% of DAU, 39% of them permanent, about 2 episodes per user. 96% are a single stuck tab (122 of 131 permanent are single-tab), not browser-wide breakage. ~37% of P6 users show `extension_updated` before the failure, so the mid-session upgrade is a likely trigger. *Fix: find the trigger (upgrade path first), then fix.*
**Transient, not a stuck state**
@@ -57,12 +87,12 @@ Definitions: failure = no CPM dashboard response within 4 s after a page finishe
**Caveats:** two days after release, mostly early updaters (macOS ~17k DAU). Loop figures come from small groups. Not measured yet: retry attempts, events after the first failure, a healthy-navigation baseline.
## What shipped
## What shipped in the measured 7.243.0 / 1.213.0 releases
- iOS **7.243.0** (tag 26 Oct 19:23 UTC), macOS **1.213.0** (tag 26 Oct 16:23 UTC).
- Both: episode telemetry #7376/#7484, episode tab scope #7520/#7521, P1/P2/P4 recovery #7379/#7536, Network-process recovery #7459/#7531.
- macOS only: P5 shared-pinned window close #7473, remote/debug disable #7498.
- **Not shipped:** P3 stable origin #7493 (on main only). **No P6 fix exists in either release**: no PR implements the controller rebind.
- **Absent from these measured releases:** P3 stable origin #7493; it is now in the upcoming 7.244 / 1.214 release branches. **No P6 fix exists in either release**: no PR implements the controller rebind.
- Shipped episode contract is the rolled-back variant: `was_stuck` (bool), single-phase `bg_events` (no `;`), plus `tabs_affected`, `repeat_failures`, `interrupted`. No `bg_view_creations`, `context_errors` or `network_restarted`. Old cohort SQL needs adapting (below).
- Monitor timing: 4 s response grace; suspended time no longer counts (8 s after resume, 30 s extension-load wait, 5 min deadline).
@@ -377,7 +407,7 @@ Re-cut of the post-release data (macOS 1.213.x, 26–27 Oct, matured since the s
- **Hypothesis, not verified:** pages that fail to load are shown on every tab after wake from sleep or a network drop. Each would count as a CPM failure right after an idle unload (fresh `dealloc>view>proxy_on`, background ready in under 1 s, healthy diagnostics), in several tabs (cross-tab) and with no answer until a normal navigation (permanent-looking, and M4 would not help). A same-tab second timeout also triggers M4 on a healthy extension (see `was_stuck` note above), which then adds the origin-change blackout. The pixel has no URL, so data cannot separate this; the check has to be an app run (debug build, Wi-Fi off, several tabs, wake) with the monitor log, or filtering `.alternateHtmlLoad` / `duck://error` and comparing wake rates.
### Notes
- The "P3" lines above disagreed. Checked on 29 Oct: `CPMExtensionBaseURLPolicy` is absent from the 1.213.0 and 7.243.0 release tags and present in `release/macos/1.214.0` and `release/ios/7.244.0`. So P3 is not in 1.213 / 7.243 ("What shipped" is right) and ships with 1.214.0 / 7.244.0, which also makes the 26.6+ gate in #7752 / #7753 relevant to those releases.
- **Release status corrected 30 Oct:** P3 is absent from the measured public 1.213.0 / 7.243.0 tags. #7752 / #7753 are merged into the upcoming 1.214.0 / 7.244.0 release branches; public release and remote enablement remain separate checks (see the release update above).
- Scratch SQL runner and queries are in the session scratchpad; the harness test is not committed or pushed.
### Addendum 28 Oct — error pages produce the wake signature (replaces the first version of this addendum)
@@ -389,10 +419,10 @@ Re-cut of the post-release data (macOS 1.213.x, 26–27 Oct, matured since the s
- **Weak evidence against, kept for the record:** per-OS correlation of error pages per install with wake episodes per install is r = 0.02 (n = 9); hourly r = 0.88 for both wake and non-wake (rollout trend, uninformative).
- **Unverified:** the running app (the scenario with `log stream --level debug --predicate 'subsystem == "WebExtensions" AND eventMessage CONTAINS "CPM Health Monitor"'`: three tabs idle >30 s, load a dead host in each, expect `timeoutElapsed` without `Dashboard response attributed`); iOS error pages; the share of wake episodes that are error pages (the episode pixel has no URL or error code).
- **Decisive options:** (a) macOS build where the monitor ignores error documents (`.alternateHtmlLoad` / failed provisional navigations) and M4 never fires for them; compare wake share and permanent rate with the current build; (b) a flag parameter on the episode pixel (needs the privacy review you wanted to avoid).
- **Fix, tests and PRs (state on 29 Oct):**
- **PRs, same commit:** [#7752](https://github.com/duckduckgo/apple-browsers/pull/7752) into `release/macos/1.214.0` (branch `alex/cpm-error-page-filter`) and [#7753](https://github.com/duckduckgo/apple-browsers/pull/7753) into `release/ios/7.244.0` (branch `alex/cpm-error-page-filter-ios`); reviewer `miasma13`; CI had not started when they were opened. Worktrees `apple-browsers.git/cpm-error-page-filter` and `…-ios`.
- **Commit content:** `CPMNavigationKind.errorPage` and a monitor that does not measure commit/finish of such a navigation (so no timeout, no episode, no M4); macOS maps `.alternateHtmlLoad` to `.errorPage` in `PrivacyDashboardTabExtension.cpmMessagingNavigationKind`; `CPMExtensionBaseURLPolicy.isSupportedOS` (`#available(macOS 26.6, iOS 26.6, *)`) keeps WebKit's random origin below 26.6 without rotating the stored origin; two monitor unit tests; macOS integration tests `CPMErrorPageMonitoringTests` (both Integration Tests targets in the pbxproj). Harness tests and the reproduction suite are not part of it (the harness is not in the release branches).
- **Run results (macOS release base):** 125 package tests (103 monitor, 22 base URL policy) and the 3 integration tests pass; on the iOS release base the same 125 package tests pass. iOS app and iOS tests were not built (no iOS source changed, no exhaustive `switch` on `CPMNavigationKind` there). The old-OS branch of the base URL gate has no test (hosts run 26.6+).
- **Fix, tests and PRs (updated 30 Oct):**
- **PRs merged:** #7752 into macOS 1.214.0 and #7753 into iOS 7.244.0; merge commits and final branch heads are recorded in the release update above. The sibling branches contain the same intended WebExtensions/error-page changes on their respective release bases.
- **Commit content:** error-page monitor filtering and macOS alternate-HTML mapping; OS 26.6+ and flag gate; stored-origin rotation on a disabled/unsupported context load; JSON key rename to `cpmStableOriginIdentifier` with installation-UUID fallback. No migration or OS-version tracking. Harness/reproduction suites are not included.
- **Run results:** key-only revision: all 1,025 package tests passed. Final simplified policy suite: 25 passed, including unsupported-OS coverage; strict lint passed on both branches. Three macOS error-page app integration tests passed before the key/test-only edits, with a negative control; no iOS runtime run. See release update for the exact scope.
- **Integration tests** (real `Tab`, `macOS Integration Tests`, run locally with the signed Debug build): ordinary load error (`-1003`), SSL (`NSURLErrorServerCertificateUntrusted`), malicious-site warning (`MaliciousSiteError` via the `duck://error` redirect) all pass. Negative control with the macOS mapping removed: the test fails, the monitor receives `["started:other", "failed", "started:other", "committed:https://testhost.com/", "finished:https://testhost.com/"]`, the error page as an ordinary navigation of the failed URL (the false positive in the app's own `Tab`/`Navigation` path); mapping restored. They use their own `StubWebExtensionManager`; `WebExtensionsTestSupport/MockWebExtensionManaging` exists but that product is not linked to the Integration Tests targets.
- **Running locally:** do not build with `CODE_SIGNING_ALLOWED=NO`: the host app then lacks the keychain entitlement (`-34018`) and macOS asks for keychain access. Use the normal signed Debug build, `xcodebuild … test-without-building -only-testing:"Integration Tests/CPMErrorPageMonitoringTests"`.
- **Corner cases checked (29 Oct), no further false-positive source found:**