obsidian-vault/work/projects/cpm-post-release-analysis-2026-10-27.md

title, date, tags, updated

title	date	tags	updated
CPM — first post-release data (iOS 7.243.0 / macOS 1.213.0)	2026-10-27	work cpm analysis	2026-10-30

CPM: first post-release data, 26 October 2026

iOS-only refresh — 30 October 2026, 17:01 Bishkek data cut

Updated ClickHouse export, independent group checks, background-event footprints and current PR coverage: cpm-ios-uncovered-groups-2026-10-30. This supersedes the older iOS prioritisation below, not its historical measurements. Latest complete metrics day is 29 October (Eastern); 30 October is partial. The next primary investigation/fix is ready-background idle/resume messaging, followed by pending startup. Error-6 history with a replacement view still pending must not be counted as fully covered by #7730. Today's release-branch merges and open PRs have no measured post-rollout effect in this export.

A separate 29 October iOS app-runtime report is also included there: checked DNS/connection errors cancelled measurements; tested warning documents returned CPM responses. The older statements below that iOS runtime was never checked are superseded for those tested paths. macOS is excluded from this refresh.

Error-6 restart — 30 October 2026

PR #7730 is OPEN, non-draft, MERGEABLE, head cf668e2938, with privacy review required. The closed draft described under 28 October is historical. This PR is not merged or shipped; this update does not add a new production data cut.

The implementation restarts only the current embedded context whose original failed background view is still retained at execution. Released views keep the existing P2 retry; WebKit's replacement view is left alone. Deferred errors resume after foreground/wake. The latest pending error survives the 60-second cooldown and is retried after expiry; expiry in background waits for foreground. Equal historical errors are deduplicated, with history reset for a newly created background view so delayed clear/add notifications do not hide a new equal failure.

A dedicated web_extension_cpm_background_load_restart pixel now exists, dailyAndCount on both platforms with no macOS name suffix and bounded outcome. Applying the restart, background readiness and an attributable dashboard response are separate stages. Terminal outcomes distinguish measured no-response at the five-active-minute deadline, unobserved, superseded, reload failure and reload timeout. A new embedded identifier loaded by upgrade supersedes the old restart observation; its response cannot claim that the old restart recovered messaging. Count rows are required for outcome rates; daily deduplication usually keeps restart_started.

Local final validation on cf668e2938: 1,064 production package tests passed, strict lint and both platforms' pixel validators passed. Final review added 8 temporary edge-case probes: 1,072 tests passed, no important findings; the probes were removed. Permanent regressions remain in normal PR CI. Exact-head CI was not re-read for this documentation update.

The separate review harness is synchronized at 5348c7254e and adds only 17 Swift fixture/test files plus its guide. The last actual same-binary macOS OFF/ON was run on d1a2e1de65: OFF functionally failed with real error 6 and a retained failed view; ON restored the original document's native response and real page initResp, with two later idle/wake cycles healthy. This is controlled worker-start fault injection, not a natural production trigger or physical iOS suspend/wake validation. Later queue/cooldown/attribution changes have regression-test evidence, not a new real-WebKit OFF/ON result.

Full behavior, flags, outcomes and evidence: cpm-status-2026-10-15#Error-6 restart — PR 7730. Reviewer commands: CPM Breakage Testing Steps#Error-6 restart — PR 7730 (30 October 2026). Historical production percentages below retain their original dates/denominators; the PR does not establish its production causal share or post-rollout effect.

Internal metrics; keep these figures out of public PRs. Previous analysis: cpm-metrics-causes-and-mitigations-2026-10-16, cpm-status-2026-10-15 (§3–5, 12–14 October cut).

Upgrade-before-load — 30 October 2026

#7767 now implements the extension-updated follow-up. Head 45593bbb4e, latest merged main 3046f67631; OPEN / MERGEABLE, not merged or released. Exact-head CI: 43 passed, 15 skipped, no failed/pending checks, including successful iOS/macOS/BSK/DBP result checks.

The launch pre-sync handles CPM only, without scriptlet downloads, before loading installed contexts. The normal full sync follows; other extension types retain load-before-sync. Failed pre-load upgrades keep the previous installation available and the normal sync can retry after load. Disabled CPM is removed before load. The default-enabled remote failsafe webExtensions.cpmUpgradeBeforeLoad is captured when the manager is created; OFF restores the previous order on the next manager/app launch. Origin reuse additionally depends on stable-origin policy, OS 26.6+, a private CPM manifest and no live old context. Runtime upgrades keep distinct origins.

No new pixel or success tracking was added. Existing install-error pixels mark CPM pre-load errors using .before_load in error domain, preserving code/userInfo/available underlying errors; ordinary sync retains the original domain. Missing bundle/type mismatch use EmbeddedSyncError: DDGError, codes 1/2. The earlier 29 Oct UTC ClickHouse cut found 518 iOS / 16 macOS standard install-error events across versions; 14 macOS events were invalid archive. No installation/session key connects these pixels to failure episodes, and hourly correlation did not establish causation. extension_updated records successful update history. Those queries were not rerun for this update.

Validation is revision-specific: 1,037 package tests passed after the first main merge; 11 real-WebKit harness scenarios passed on fafcb1ffda; 89 manager tests passed after the final telemetry fix. iOS CI initially crashed because the telemetry test read an unavailable package resource bundle. 00ddded2ff replaces that dependency with a generated ZIP fixture. The corrected test passed on iOS 27 Simulator even with the resource bundle removed; the latest PR head subsequently passed CI. Full iOS app upgrade/startup runtime was not locally validated.

Harness medians, 20 launches/scenario with three document replies: old order 180.9 / 173.6 / 208.3 / 205.8 ms; latest CPM-only sample 181.7 / 172.3 / 179.5 / 178.4 ms for first install / unchanged / upgrade / upgrade with 2,000 cookie-bearing sites. All 240 replies in the latest sample arrived. These are separate package-harness samples, not full-app startup measurements or proof of reduced production hangs.

Interpretation of priorities below: the 26–27 Oct macOS wake/other groups predate the error-page fix and may include false positives. The confirmed local mechanism justifies upgrade-before-load as an actionable follow-up; its production effect is unmeasured. iOS should be analysed separately, not assumed to be a proven clean control: error-page behavior at iOS app runtime was not validated in this work. Compare post-rollout versions/OS/flag states before assigning the old wake cohort to real CPM breakage.

The release PRs #7752/#7753 are merged, while GitHub still lists 7.243.0 / 1.213.0 as the latest non-prerelease tags at this check. 7.244.0-8+ios and 1.214.0-862+macos are marked prerelease. This does not establish App Store rollout.

Current implementation, rollback, telemetry, per-head tests and artifact locations: cpm-status-2026-10-15#Текущее состояние — 30 октября 2026, 16:48 Бишкек. Earlier data cuts and reproduction results below remain historical evidence.

Release update — 30 October 2026

• Merged, not yet verified as publicly released: iOS #7753 into release/ios/7.244.0 (merge 26a461e30e73608e75e6d84e09bca0ced842993c) and macOS #7752 into release/macos/1.214.0 (merge 1e392bac233d7e3e5ccef340ebc82ad7aee72922). Latest public GitHub releases checked on 30 Oct are still 7.243.0 / 1.213.0. Merge does not establish rollout or production effect.
• Final stable-origin behavior: set baseURL only on macOS/iOS 26.6+ with the feature flag enabled. Creating a context with an unsupported OS or disabled flag rotates the stored CPM origin and returns nil (WebKit chooses the context's origin). Toggling the flag alone does not change a live context; an off→on interval with no intervening context creation does not rotate anything. User reports the remote flag disabled; live remote configuration was not independently audited.
• Only a storage-key rename: property cpmOriginIdentifier now encodes/decodes JSON key cpmStableOriginIdentifier. Old cpmOriginIdentifier values are ignored. There is no migration, UUID prefill, OS-version marker or reset on every OS upgrade. Missing new-key value falls back to the already persisted installation uniqueIdentifier; this is reuse of that UUID, not creation of a fresh UUID. Subsequent launches reuse it. A later normal rotation saves a replacement under the new key and reuses that replacement.
• Public-build transition: 7.243 / 1.213 persist the installation UUID but do not assign it as the extension's baseURL. WebKit chooses a random origin for each new context. The first enabled stable-origin load therefore changes the origin once. An internal build with a replacement saved under the old key falls back to the installation UUID after this key change.
• Error-page fix: the health monitor skips error-page commit/finish; macOS maps .alternateHtmlLoad to .errorPage. These documents no longer start timeout measurements, episodes or M4 reloads. This does not establish what fraction of production wake episodes were error pages.
• Validation: the key-only revision passed all 1,025 WebExtensions package tests and strict lint; the final simplified policy suite passed 25 tests, including injected unsupported-OS coverage, with strict lint on both branches. The original release-base error-page change passed three macOS app integration tests (DNS, SSL and malicious-site warning), with a failing negative control when the mapping was removed. Those app tests were not rerun after the key/test-only edits; iOS runtime behavior was not tested. Final branch heads: iOS fef5f9cdd35977b0cb848ac81f4feaca3e6a9ba6, macOS 551cb2edd00c02e038b2b46a07a8c32d0b33e666.
• Mock follow-up: the local StubWebExtensionManager stays in the release tests by reviewer agreement. Moving them to the shared mock is a main-branch follow-up.

Stable-origin evidence and limits

The demonstrated benefit is avoiding WebKit's origin-change data cleanup before injection/background startup: real-cache reproductions measured about 1.5 / 10.9 / 32.2 seconds at 20k / 200k / 600k cache entries with changing origins, versus about 0.1 seconds with a stable origin. See WebKit bug 320812 and the reproduction notes below. Production symptom cohorts are not proof that every matching episode has this cause.

No test demonstrated an extension staying permanently broken after an OS upgrade because its baseURL was saved. A live-session background-process termination did reproduce an error-6 loop that origin rotation recovered. In the real two-process restart baseline, a full app restart restored messaging even while reusing the previous origin. Evidence: alex/cpm-p3-integration-tests/macOS/IntegrationTests/CPM/README.md, “two-process” prepare/resume results. Do not turn the live-session failure into a proven persistent-upgrade failure.

The earlier claim that extension workers were never persisted was incorrect. WebKit commit eff9f9231ad90e31e6ae2b128d2269c589550c8d (312316@main, 29 April 2026) stops new extension service-worker registrations being written to disk; that change does not purge existing records. Historical records are therefore possible, but this does not demonstrate the proposed upgrade failure.

Pixels can show no response within five minutes, cross-tab failures, error 6 and reload outcomes. They cannot uniquely identify old-origin reuse after an OS upgrade; base_url_changed compares contexts within the current process, not across launches. “Permanent” in this analysis means the five-minute reporting outcome, not proof of an unrecoverable extension.

OS split of the no-background-view symptom — query on 29 October

26–28 Oct (metrics date in Eastern time), released iOS 7.243.x / macOS 1.213.x without P3. Count-leg episode rows only, not users: context_loaded=true AND bg_view_alive=false, excluding a nonempty bg_load_error_after or an error_[0-9] event in bg_events.

• iOS: OS <26.6 — 695 of 14,537 failure episodes (4.78%); OS 26.6+ — 27,670 of 97,982 (28.24%). Thus 97.55% of this symptom cohort is on 26.6+. Confirmed no-response subset: 43 / 347 respectively.
• macOS: OS <26.6 — 274 of 13,206 (2.07%); OS 26.6+ — 961 of 29,770 (3.23%). Thus 77.81% of this symptom cohort is on 26.6+. Confirmed no-response subset: 29 / 98 respectively.

Source: metrics.pixels, pixel_id='web', web.extension.cpm.episode.count%, agents ddg_ios / ddg_mac_desktop; OS split by major/minor tuple at (26,6). These are symptom counts, not a causal P3 share or a stable-origin safety comparison. The query was run on 29 Oct; it was not rerun for this documentation update.

Priorities — macOS browser-wide breakage first

macOS fails far more than iOS, so priorities follow cross-tab permanent failures on macOS: CPM not answering in two or more tabs for 5 minutes, confirmed by a repeat or reload. These are browser-wide breakage, not one stuck tab. 26–27 Oct, macOS 1.213: 386 users, 1.85% of DAU.

Share of macOS cross-tab permanent: idle wake 174 (45%), other 78 (20%), extension updated 77 (20%), recent context load 35 (9%), no background view 11 (3%), P6 9 (2%), error 6 2 (0.5%).

1. Idle wake (45%, cause unknown).
   • Check Network-recovery false positives: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and died_client precedes 16% of macOS idle-wake failures.
   • Add telemetry: events after the first failure (two-phase bg_events), a healthy-navigation baseline, retry/recovery applied in the episode.
   • Keep-warm experiment behind a flag (also a causal test).
   • Update 28–29 Oct: error pages were found to be counted as CPM failures with exactly the wake signature (reproduced in the harness and in a macOS integration test); fix in PRs #7752 (macOS 1.214.0) and #7753 (iOS 7.244.0). Details and the other checks in the section "Idle wake investigation — 28 October 2026" below.
2. Extension updated (20% in the historical 26–27 Oct cut). Implemented in #7767: prepare CPM before load, carrying a private origin only when allowed and no old context is live. The PR is open with green CI; production effect is unmeasured. See the 30 Oct update above.
3. Other (20%). No distinguishing signal yet; needs the telemetry from step 1.

Not priorities:

• P3 / origin change: the baseURL fix is in the upcoming 7.244 / 1.214 release branches. Public rollout and an enabled flag are prerequisites for measuring its effect; recent context load is not by itself proof of origin-change cleanup.
• P6: 96% of P6 users have a single failing tab; 122 of its 131 permanent users are single-tab, only 9 cross-tab. One stuck tab, not browser-wide breakage. Low priority; find the trigger when convenient (~37% show extension_updated before the failure).
• Error 6 on macOS: 2 cross-tab users.

iOS (cross-tab permanent 861 users, 0.26% of DAU): recent context load 38% (a symptom targeted in part by upcoming P3; P3 is absent from this measured release), error 6 20% (error-6 loop: restart the stale background and re-send init), extension updated 14% (same fix as macOS step 2), idle wake 14%, other 8%, no background view 5%.

Summary for the team — 26–27 Oct

CPM post-release analysis: iOS 7.243.0 / macOS 1.213.0, 26–27 Oct (2 full days)

Definitions: failure = no CPM dashboard response within 4 s after a page finished. Episode = one failure until resolution or the 5-minute deadline. Permanent = no response for 5 minutes, confirmed by a repeat failure or a reload. Affected users = users who reported a failure that day (daily pixel); all percentages are of DAU. Episodes = every failure report (count pixel).

Size

• iOS: 9.1% of DAU hit a failure, 0.43% permanently. About 3× fewer failures than the previous release.
• macOS: 28.5% of DAU hit a failure, 2.87% permanently. No change vs the previous release.
• Most failures recover by themselves: 89% on iOS, 63% on macOS.

Where permanent failures come from (share of permanent users)

• iOS (0.43% of DAU): recent context load 40%, error 6 20%, idle wake 16%, extension updated 13%, other 7%.
• macOS (2.87% of DAU): idle wake 38%, P6 (tab not connected) 22%, extension updated 15%, other 14%. Browser-wide (cross-tab) only, 1.85% of DAU: idle wake 45%, other 20%, extension updated 20%, recent context load 9%, P6 2%.

Persistent mechanisms (repeat episodes from the same users confirm a stuck state)

• iOS error-6 loop: error 6 hits 0.86% of iOS DAU, and 10% of those users stay stuck. Stuck users average 7.4 episodes, recovered ones 1.2. In 38% of error-6 episodes the background keeps failing to load on every wake. Matches the stale-failed-background state reproduced in the harness. Fix: restart the stale background on error 6 and re-send init to the page.
• Failures after an origin change (base_url_changed): present in 1,733 iOS users (14% permanent, 3× baseline) and 193 macOS users (16%). On macOS these installs fail repeatedly (13–27 episodes per user in the idle-wake patterns, from small groups). The origin change itself causes a 1.5–32 s blackout depending on cache size (reproduced); P3 removes it. P3 is merged into upcoming release branches; verify its effect after public rollout and flag enablement.
• macOS P6: 1.6% of DAU, 39% of them permanent, about 2 episodes per user. 96% are a single stuck tab (122 of 131 permanent are single-tab), not browser-wide breakage. ~37% of P6 users show extension_updated before the failure, so the mid-session upgrade is a likely trigger. Fix: find the trigger (upgrade path first), then fix.

Transient, not a stuck state

• Extension updated: one episode per user, at upgrade time. Still worth fixing (load the new version first, keep the origin), since it repeats after every app update.
• Idle wake: mostly recovers (iOS 88%, macOS 58%). Its repeat-heavy part overlaps the origin-change patterns. Cause unknown; it may just be the state most macOS navigations are in.

Caveats: two days after release, mostly early updaters (macOS ~17k DAU). Loop figures come from small groups. Not measured yet: retry attempts, events after the first failure, a healthy-navigation baseline.

What shipped in the measured 7.243.0 / 1.213.0 releases

• iOS 7.243.0 (tag 26 Oct 19:23 UTC), macOS 1.213.0 (tag 26 Oct 16:23 UTC).
• Both: episode telemetry #7376/#7484, episode tab scope #7520/#7521, P1/P2/P4 recovery #7379/#7536, Network-process recovery #7459/#7531.
• macOS only: P5 shared-pinned window close #7473, remote/debug disable #7498.
• Absent from these measured releases: P3 stable origin #7493; it is now in the upcoming 7.244 / 1.214 release branches. No P6 fix exists in either release: no PR implements the controller rebind.
• Shipped episode contract is the rolled-back variant: was_stuck (bool), single-phase bg_events (no ;), plus tabs_affected, repeat_failures, interrupted. No bg_view_creations, context_errors or network_restarted. Old cohort SQL needs adapting (below).
• Monitor timing: 4 s response grace; suspended time no longer counts (8 s after resume, 30 s extension-load wait, 5 min deadline).

How to query

• New pixels have no m_/debug_ prefix: ClickHouse pixel_id = 'web', pixel LIKE 'web.extension.cpm.%' (iOS adds .ios.phone|tablet). Querying pixel_id='m' without an agent filter and a one-day window exceeds the 7B row limit.
• Denominator: pixel_id='m', m.web.extension.loaded.daily% / m.mac.web.extension.loaded.daily, split by appVersion.
• Old-contract attribution was remotely switched off from 21 Oct (old versions keep sending bare debug.web.extension.cpm.* without parameters). The last fully attributed old-version window is 18–20 Oct, iOS 7.241 / macOS 1.211.x.
• Data lag: on 27 Oct the table ended at ~01:10 UTC; date is US-Eastern. 26 Oct is the only post-release day and it is partial for new adopters (exposure starts at update time).
• Runner: ~/Developer/grafana-data-tools reader; ad-hoc SQL and an ARRAY JOIN extractURLParameters(request) parameter dump were used. extractURLParameter needs a constant key.

Rates per extension-loaded install-day (26 Oct)

	iOS 7.242 (old)	iOS 7.243 (new)	macOS 1.212 (old)	macOS 1.213 (new)
loaded.daily	4,257,839	52,979	93,030	2,566
old init_failed.daily (sum of reasons)	21.3%	—	36.8%	—
old stuck.daily	3.3%	—	12.5%	—
episode.daily	—	7.05% (3,735)	—	30.0% (771)
episode with reload_state≠none (hang confirmed, ≈ old stuck), count leg	—	23% of episodes	—	26% of episodes
was_stuck=true (second distinct tab), daily leg	—	8.1%	—	11.2%
network_process_recovery_applied.daily	—	432 (0.8%)	—	232 (9.0%)

Not like-for-like: the episode unit is one report per episode, the old init number sums four reasons, suspended time is now excluded, and day-1 adopters have short exposure. Directionally, iOS fell ~3× (21% → 7%); macOS barely moved (37% → 30%).

was_stuck only becomes true when a different tab fails; M4 reload already runs on a same-tab second timeout. Use reload_state≠none as the old "stuck" proxy.

Cohorts (count leg, adapted classification)

Priority: error 6 (bg_load_error_after present or error_ token) → no view (bg_view_alive=false) → tab not connected (tab_has_ext_scripts=false) → extension_updated ≤2 min → wake (bg_view_age < 30 s and a dealloc token) → recent load/base_url_changed ≤2 min → other. There is no Network-replaced cohort any more.

Cohort	iOS share / of stuck / no-response	macOS share / of stuck / no-response
error 6	7.5% / 24.7% / 39%	0.6% / 1.7% / 44%
no view (P3)	7.7% / 6.8% / 7%	2.9% / 0.9% / 25%
tab not connected (P6)	—	5.8% / 1.3% / 70%
extension_updated ≤2 m	3.9% / 6.2% / 37%	14.6% / 11.7% / 28%
wake, view <30 s	22.8% / 22.0% / 9%	43.9% / 58.4% / 48%
recent context load	22.8% / 17.2% / 12%	11.9% / 6.9% / 25%
other	35.3% / 23.2% / 5%	20.4% / 19.0% / 27%

Baseline with last week's exact SQL, 18–20 Oct (init → stuck shares). iOS 7.241: error6 6.4→22.0, no view 47.5→11.5, network 16.5→38.2, wake 7.7→9.2, recent load 16.7→6.6, other 5.2→12.5. macOS 1.211.x: error6 1.2→2.2, no view 3.6→1.9, tab not connected 5.9→8.7, network 13.2→25.8, wake 33.2→33.3, recent load 19.4→6.3, other 23.6→21.6. This matches the 12–14 Oct cut, so the baseline was stable.

Per-install rough rates (cohort share × failure rate), old init → new episode:

• iOS: no view 10% → 0.5%, recent load 3.5% → 1.6%, error 6 1.3% → 0.5%; the network cohort is absorbed. Wake stays at 1.6% → 1.6%. Other rises 1.1% → 2.5%; it now absorbs former network cases and has no network field.
• macOS: wake ≈12% → ≈13%. Unchanged.

Findings

1. P1 (wake) fix had no measurable effect on either platform. In the wake cohort bg_ready_after=under_1s dominates (macOS 93% overall), so the background is not slow. The background_wake pixel fires only for ≥1 s callbacks (macOS 89/day vs ~670 wake episodes). The remaining failure is message/response loss after a fast idle wake, not readiness. On macOS 48% of wake episodes see no response for 5 min; tabs=3+ → 68%. Typical tail: …dealloc@15m,view@15m,proxy_on@15m,dealloc@5m,view@30s,proxy_on@30s, bg_view_age=4s_to_8s (view created around the navigation). The pattern appears on every macOS version, 15.6–27.2.
2. iOS error 6 changed shape. All attributed error 6 is now bg_load_error_after=under_5s; the 30 s timer bucket is gone.
   • A (ready=failed, app_state active): dealloc → view → proxy_on → interruption → error_6, often after died_client/died_crash/died_crash_limit. The background load spans an app suspension and fails at resume.
   • B (ready=pending): an older error 6, then a new view whose load is still pending 8–30 s later.
   • The P2 retry fires only while foregrounded with a 60 s cooldown (277/day). Foreground-resume recovery is the missing piece.
   • Update 28 Oct: checked with a real HOME and suspension: the retry is skipped in the background and nothing happens on foregrounding, but the page was still answered because WebKit re-woke the background itself; see the 28 October section.
3. macOS P6 is untreated: 5.8% of episodes, 70% never answered. iOS now shows tab_has_ext_scripts=false in 5.3% of episodes, concentrated on iOS 26.7 (16%) and 27.0 (8%), with 0 on 27.2. Previously iOS was ≈0.
4. macOS Network replacement is frequent: the applied pixel is on 9% of 1.213 install-days. With recovery applied it no longer appears as a cohort, but other stays at 20%.
5. After M4 on macOS: reload_state=loaded → 46% no response by deadline; response_timed_out → 57%. The reload often does not restore messaging, or no qualifying navigation follows.
6. Recovery pixels only ever report background_ready. response_received needs late-document reload, which is default OFF.
7. Day-1 effect: macOS extension_updated ≤2m is 14.6% (embedded upgrade on update). Re-check after 2–3 full days.

Next measurements

• Repeat for 27–29 Oct (full days, larger adoption). Track the wake cohort and macOS no-response by OS.
• bg_events per-phase budget and network_replaced_since_view remain missing (see cpm-status-2026-10-15#6. Телеметрия).

Repro targets

Issue	Evidence	Hypothesis to test	Candidate fix
R1 wake message loss	macOS 44% eps / 58% stuck; iOS 23%; ready <1 s	message to a woken non-persistent SW is acknowledged with no reply: (a) processes(RuntimeOnMessage) empty when _loadServiceWorker completion runs before listener IPC; (b) listener set loses RuntimeOnMessage so wakeUpBackgroundContentIfNecessaryToFireEvents resolves {} without waking; (c) background unloaded while a reply is pending	keep the background warm while CPM pages are active, or re-send init on empty reply (native/JS handshake)
R2 iOS error 6 across suspension	7.5% eps / 25% stuck, all under 5 s, interruption between view and error	WebKit load timer spans suspension; nothing retries on resume	on foreground resume, retry loadBackgroundContent when the last load failed or is pending
R3 P6 controller mismatch	macOS 5.8%, 70% unanswered	known main-flag manager replacement	rebind/recreate tab web views to the current controller

Update 28 Oct for R2: reproduced with a real HOME and a 45 s suspension. The retry is skipped while the app is in the background, but WebKit created a new view 20–25 ms after the error and the page was answered (0.6 s after resume), so the suspension case self-heals in the simulator. Persistence of this family in production is not explained (172 permanent episodes). See the 28 October section.

WebKit source pointers (~/Downloads/WebKit_sparse): WebExtensionContext.cpp wakeUpBackgroundContentIfNecessaryToFireEvents (drops events when the listener set lacks the type after the first load), scheduleBackgroundContentToUnload (30 s, reset only on dispatch), WebExtensionContextCocoa.mm loadBackgroundWebView / performTasksAfterBackgroundContentLoads, WebExtensionContextAPIRuntimeCocoa.mm runtimeSendMessage (empty processes() → {} reply). 324743 (9bcf5840ef) holds runtime messages until SW activation; it is not in shipped OS builds.

Repro log — 27 October 2026

Worktree apple-browsers.git/cpm-remaining-failures-repro, branch alex/cpm-remaining-failures-repro (from origin/main 8023e4cb5c, which includes the released P1/P2/P4/Network code plus P3). The package real-WebKit harness was imported from origin/alex/cpm-repro-harness. It compiles unchanged on current main, and its CI-tier tests pass. Local tier: CPM_REPRO=1 swift test --filter <Suite> from SharedPackages/WebExtensions. System WebKit is the host macOS; no app windows are opened.

• R1 idle wake — not reproduced organically. CPMReproIdleWakeTests.testRepeatedIdleWakeCyclesDeliverInit: 3 tabs, 8 idle-unload cycles (WebKit unloads 32.0 s after the last answered navigation), single and double-tab wakes, 0 of 11 lost. Every wake answered in 2–5 ms after didFinish. testNavigationAtIdleUnloadBoundaryDeliversInit: offsets −0.6…+0.2 s around the calibrated unload, 0 of 7 lost. A plain idle wake on an in-memory fixture page is not the production mechanism. The missing ingredient is still unknown: real page loads, a long-lived data store with abandoned registrations, Network replacement, or listener-set loss.
• P3 (#7493) only in internal builds (macOS 1.214.0-860, iOS 7.244.0-x). On 26 Oct: macOS 1.214 had 20 episodes / 9 daily of ~36 loaded installs, with 0 no-view and 0 error 6; iOS 7.244 had none. That is too small to evaluate. Relevance: released builds get a new random extension origin on every context load (launch and M4 reload), and WebKit migrates data before injecting. This may explain the iOS no-view cohort and the macOS "reload loaded → still no response" share (46%). Re-check when 7.244 reaches users.
• R2 suspended wake — macOS analogue does not stay broken. CPMReproSuspendedWakeTests: idle unload, then a navigation wakes the background, and the new background process is SIGSTOPped for 45 s while another tab navigates every 10 s. Two navigations timed out (+9 s, +14 s). Then the background was replaced by a new process (same view slot, new PID) and every later navigation answered in 1–14 ms, including 10 after thaw. No error 6 was recorded on the live context. On macOS the shipped recovery restores messaging within ~14 s of the fault. The harness cannot reproduce iOS suspension, which pauses the UI process's own timers. Likely production sequence: the overdue 30 s unload timer fires at resume, closes the loading view, and records error 6 "under 5 s" (pattern A). Pattern B (new view pending 8–30 s) needs a device or simulator test.
• WebKit hypothesis not hit yet: an SW load failure that keeps the background view (loadBackgroundWebView completion !success without unload). Later wakes would only queue, and every dispatch restarts the 30 s unload timer, so the public loadBackgroundContent/P2 retry cannot escape. A candidate mitigation is to restart the background web content process when error 6 is recorded while a background view still exists. Not implemented, because the state has not been reproduced.

Treatment pixels, P3 ceiling and permanent hangs — 27 October 2026

Treatment pixels received (26 Oct, count leg, iOS 7.243.0 / macOS 1.213.0)

Fix	Pixel (pixel_id='web')	iOS	macOS
P1 wake	web_extension_cpm_background_wake	2,026	89
P2 error-6 retry	web_extension_cpm_background_load_retry	277	12
Install / upgrade readiness	…_install_readiness / …_upgrade_readiness	3 / 24	0 / 1
Network recovery	web_extension_cpm_network_process_recovery_applied	476	277
P4 detection (flag-independent)	web_extension_cpm_startup_response_missing	450 (daily 327)	20 (daily 12)
Late page reload	web_extension_cpm_late_document_reload	0	0

• Every recovery pixel only reports outcome=background_ready, meaning the fix was applied and the background came up. reload_started / response_received exist only for the late page reload, which is default OFF, so it has zero rows on every version including internal builds. No pixel shows whether the page got its answer after a treatment; the effect has to be read from episode rates.
• background_wake fires only when the wake callback took ≥1 s and a navigation waited, so it is not a count of all wakes.
• Detection gap for the late reload. startup_response_missing fires with the flag off but requires startup.isFirstLoad: the first background load of that context in the session (launch, install, upgrade). attemptRecovery (the reload) has no isFirstLoad condition and would also run after later wakes. With the flag off, wake-time cases the reload would treat are not counted anywhere; 327 / 12 is a lower bound of what it could fix. Both detection and reload only arm when the page commits while the startup is still pending. In the wake cohort the background is ready in under 1 s, so many wake failures are outside both. Counting the would-be-reload wake cases needs a pixel change: drop isFirstLoad from detection or add a parameter. Not done.
• Upper bound on the late reload: ~0.6% of iOS / 0.5% of macOS extension-loaded installs; ~9% of iOS / ~2% of macOS installs with a failure episode.

P3 (baseURL) ceiling on the new data

• The earlier "≈half" was the pre-release no-view share: 48% of iOS initial failures, but only 11% of stuck, mostly self-healing.
• On 7.243.0 without P3, no-view is already 7.7% of iOS episodes / 6.8% of stuck (≈10% → 0.5% of loaded installs); macOS 2.9% / 0.9%. The drop comes from the new measurement (suspended time excluded, one report per episode) and the shipped readiness gates.
• Direct P3 ceiling ≈ 8% of iOS episodes, less on macOS. A possible indirect effect: released builds get a new random origin on every M4 reload, and 46% of macOS reload-loaded episodes still got no response. That link is unproven; internal 1.214 / 7.244 data is too thin.

Users affected by permanent hangs (26 Oct)

Denominators were refreshed after more 26 Oct data arrived: extension-loaded installs iOS 69,088 / macOS 3,045. "Permanent" = outcome=no_response_before_deadline (no qualifying response for 5 min). Single = tabs_affected=1, cross = 2/3+. Confirmed = repeat_failures≠0 or reload_state≠none. Daily leg = affected installs.

Installs/day	iOS	macOS
Any failure episode	4,655 (6.7%)	876 (28.8%)
Single-tab, no response	354 (0.51%)	217 (7.1%)
↳ confirmed	76 (0.11%)	25 (0.82%)
Cross-tab, no response	129 (0.19%)	58 (1.9%)
↳ confirmed	123 (0.18%)	49 (1.6%)

• Cross-tab is the reliable permanent-hang figure: ~0.2% of iOS and 1.6–1.9% of macOS users per day; 95% / 84% confirmed.
• Unconfirmed single-tab rows (iOS 278, macOS 192) may be stuck tabs or users who didn't navigate within 5 min.
• Strict permanent (confirmed single + cross): ~0.3% iOS, ~2.4% macOS users/day. Upper bound including unconfirmed single-tab: 0.7% / 9%.
• Lower bounds for users: the daily leg describes only each install's first completed episode of the day, and episodes lost when the process ends before the deadline are not reported. Count leg (episodes per install-day), confirmed cross-tab: iOS 0.41%, macOS 5.2%; confirmed single-tab: iOS 0.26%, macOS 2.3%.
• One partial day; macOS has ~3k installs on 1.213.0. Repeat on 27–29 Oct full days.

Repro statistics note

18 wake navigations cannot rule out a rare loss. At p=8% a clean run happens 22% of the time. The per-wake production rate is likely ≪1% (many wakes per install-day), which needs ~1,000 wakes for 95% power. CPMReproIdleWakeTests now has CPM_REPRO_FAST_UNLOAD=1: it calls +[WKWebView _setApplicationBundleIdentifier:] with com.apple.WebKit.TestWebKitAPI, so WebKit unloads an idle background after 3 s instead of 30 s (~5 s per cycle). It can change other test-runner-only WebKit behavior, so confirm any hit with the 30 s delay. A 1,000-cycle run (~1,500 wake navigations) was started on 27 Oct.

• Fast idle-wake run, 27 Oct: 0 lost of 1,500 wake navigations in 1,000 cycles (CPM_REPRO_FAST_UNLOAD=1, 3 tabs, single and double-tab wakes, 85 min). With zero failures the 95% upper bound on the per-wake loss rate is ≈0.2% (3/1500). A bare idle-unload → wake race on a fixture page is not the main source of the wake cohort; an extra ingredient is needed (hypotheses H1–H5 below).

Hypotheses for the remaining wake hangs (27 Oct)

The embedded extension (background-embedded.js, 2026.10.23) registers runtime.onMessage synchronously (MessageRouter). CPM init = await remoteConfigJson → native isAutoconsentSettingEnabled → native isFeatureEnabled → refreshDashboardState. Every native request has a 20 s timeout, with config_unavailable / setting_disabled fallbacks that still refresh. A 5-minute silence therefore means init never reached the worker, or the refresh was not attributed.

1. H1 — worker without extension bindings after Network-process recovery (WebKit 324754). macOS recovery is applied on 9% of installs/day. A worker without browser.runtime throws at top level: alive, responsive, "ready", never answers, all tabs, M4 may repeat it. Test: Network crash → recovery restart → probe browser.runtime in the worker → N wake cycles. Fix: verify with a ping after restart; restart again or reload if it fails.
2. H2 — listener metadata loss. wakeUpBackgroundContentIfNecessaryToFireEvents returns an empty reply without waking when the saved listener set lacks RuntimeOnMessage. The set is saved after every load, so one load where the worker threw before registering poisons it. Test: a prelude knob that throws or skips onMessage on one load only. Fix: treat an empty sendMessage reply as a failure → restart, plus re-send init.
3. H3 — stale failed background view (iOS error 6, pattern B). _loadServiceWorker failure keeps the view, so wakes only queue; activity keeps restarting the 30 s unload timer; P2 retry uses the same path. Test: fail the worker load without closing the view, with navigations every 10 s. Fix: on error 6 with a live background view, restart its WebContent process, then retry.
4. H4 — refresh not attributed. The monitor needs an exact scheme/host/port/path/query match with the committed URL. A cold wake answers later, giving replaceState / client-side redirects time to change sender.tab.url. other_tab_response is 37–53% of outcomes. Test: a fixture with replaceState 50–300 ms after load, cold vs warm. Fix: attribute by extension tab ID and treat same-document URL changes as the same document.
5. H5 — slow cold-wake init. Three native round-trips plus session storage; explains late answers (response_after ≥30 s in 20–35% of answered episodes), not 5-minute hangs. Test: prelude delay on sendNativeMessage 1–3 after a wake. Fix: answer init from cached config/settings first.

H1/H2 tests — results (27 Oct)

New CPMReproBrokenWorkerTests and harness changes (uncommitted):

• Options.isNetworkRecoveryEnabled, on by default as in the app; off in CPMReproWebKitTests.unprotected.
• Prelude startFaults breaks one worker start inside a wall-clock window: .skipOnMessage (script runs, no runtime.onMessage listener) or .throw (top-level throw before the extension script).
• CPM_REPRO_FAST_UNLOAD moved to CPMReproTestCase.enableFastUnloadIfRequested().

Scenario	Fast unload (3 s)	Production unload (30 s)
H2 skipOnMessage: one woken worker without onMessage	only the broken navigation lost; all later ones answered	not re-run
H2 throw: one woken worker throws at top level	broken navigation + 5 busy + 3 idle-wake losses, tabs 1–2 broken for the rest of the test	broken navigation + 1 busy loss, then everything answered
H1 Network crash, recovery on, 20 / 10 wakes	2 lost after crash, then tabs 1–2 never got initResp, tab-3 always fine; all restarted workers ran	2 lost after crash, then all 10 wakes answered; all restarted workers ran

• H1 not confirmed: every worker restarted after Network recovery ran its script (bindings present). With the real delay, recovery restores messaging after 2 lost navigations.
• H2 (listener metadata loss) not confirmed: one start without a listener only loses that navigation; P1's wake plus re-registration heals it.
• The fast-mode per-tab breakage is an artifact: after a confirmed hang and recovery, tabs with an unanswered init kept receiving the dashboard refresh but never initResp. That is WebKit's silent tabs.sendMessage path (processes.isEmpty() → {}), and the monitor sent one was_stuck=true / reload_state=loaded / other_tab_response episode and nothing more. It did not reproduce with the 30 s delay. The TestWebKitAPI identity or the 3 s unload changes WebKit behavior; fast-mode hits must always be confirmed with the production delay. The blind spot it exposed is still real in principle: the monitor cannot see a tab that gets the dashboard refresh but no initResp.
• H3 (stale failed background) reproduced with production timing: after the throwing start, error 6 is recorded and the failed background view stays present. The next navigation 9 s later got nothing, and only the M4 reload (new context) restored messaging. No background_load_retry pixel fired, so the P2 retry never completed, as predicted by WebKit's loadBackgroundWebViewIfNeeded early return. Cost: at least one extra lost navigation per error 6, plus a full M4 reload. On iOS, error 6 is 7.5% of episodes and 25% of stuck.

Production-gap ideas and coexistence screen (27 Oct)

The harness differs from macOS production in four ways:

1. Other extensions: 83% of macOS 1.213 installs run the ad-blocking extension (m.mac.web.extension.daily.ad.blocking.state.daily is_enabled=true 3,486 of 4,182). It is a module service worker with DNR and four document_start all-frames content scripts. iOS has none; Dark Reader runs on ~140 iOS installs.
2. Real pages: network loads, iframes, redirects, replaceState.
3. Real app tabs and windows: visibility, App Nap, multiple windows and pinned tabs; app-hosted harness only, in a VM.
4. Long-lived data store with abandoned registrations; weak, since ~30 harness runs already shared one persistent store.

• Coexistence screen: CPMReproCoexistenceTests (new Options.embeddedTypes), CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,500 wake navigations (85 min); at 30 s, 0 of 15. The ad-blocking background started once and unloaded at 30 s, but never woke again on fixture pages, so concurrent wakes of both backgrounds were not exercised. Its content scripts ran on every page. A second extension's content scripts alone do not trigger the wake hang.
• App Store vs DMG (macOS 1.213, 26–27 Oct, per loaded install): episodes 20.8% vs 29.7%, wake cohort 9.8% vs 10.6%, cross-tab no-response 1.2% vs 2.4%. Sandboxing does not drive the wake cohort; the DMG excess is likely early-updater bias.
• Real-page screen (CPMReproRealPageTests): earlier repro tests used loadSimulatedRequest, so loads never reached WebKit's Network process. CPMReproFixtureServer serves loopback pages through it: cross-origin iframes (localhost vs 127.0.0.1), 302 redirects, replaceState, slow images, main-thread blocks. CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,333 visits across 7 profiles (~190 each), and no monitor episodes. Real loads, frames, redirects and URL rewrites do not trigger the wake hang.
• Next hypothesis — origin-changing M4 reload stalls on large data: most confirmed macOS cross-tab hangs (1.213, 26–27 Oct, tabs_affected≠1 and no response) have tail proxy_on>dealloc>view>proxy_on, view 4–8 s, ready <1 s, scripts present, reload_state=loaded (97+22+22) or response_timed_out (34+17+13). The reload loaded, yet nothing answered for the rest of the episode. Released builds load the reloaded context under a new random origin, and P3 saw WebKit's origin migration slow with a large cache. Test: CPMReproReloadOriginTests fills the persistent HTTP cache (/fill, default 20,000 × 2 KB) and runs the M4 reload with released vs stable origin, timing post-reload silence. If confirmed, P3 also fixes hang persistence.
• Reproduced: origin-changing M4 reload blacks out CPM proportionally to the HTTP cache (CPMReproReloadOriginTests, production timing, CPM only). Fill the persistent cache through real fetches, run the hang reload (reloadExtension(…, trigger: .cpmMessagingHang)), navigate every 2 s:

Cache entries (2 KB)	Released origin (new random origin per load)	P3 stable origin
20,000	first initResp 1.5 s after reload, 0 lost	0.1 s
200,000	10.9 s, 1 lost	0.1 s

reloadExtension itself returns in ~20 ms; the delay is WebKit migrating extension data to the new origin before content scripts run. Roughly linear, ~50 µs per cache entry. Cookie seeding on 5,000 .test domains did not create website data records (still 2), so per-site record cost is untested. The same migration runs on every app launch in released builds, which links this to the P3 startup cohort. P3 (#7493) removes it in both cases. It explains reload_state=response_timed_out and extended post-reload silence; whether real caches are large enough for a full 5-minute silence is being checked at 600,000 entries.

• 600,000 entries, released origin: first initResp 32.2 s after the reload, 5 navigations lost; still linear (~53 µs/entry). Page loads during the migration were also slower (one took 9 s), so the migration may delay browsing generally, not only CPM. A full 5-minute silence would need ~5–6M entries by extrapolation, so this explains post-reload silences of seconds to about a minute, not the whole deadline. The same blackout hits every app launch on released builds. With P3, 0.1 s at every size.

What survives the hang reload — code search and tests (27 Oct)

Telemetry: every confirmed permanent hang (iOS 0.30%, macOS 2.07% of DAU) had the M4 reload run and stay silent, so the broken state survives a full context replacement or the trigger repeats. Checked:

• Native messaging (WebExtensionMessageRouter, AutoconsentWebExtensionMessageHandler): handlers keyed by extension identifier and re-registered on reload; each request is stateless. Not persistent.
• Monitor reload bookkeeping: responses from measurements begun on the new generation count; harness reload scenarios closed their episodes. No bug found.
• WebKit process reuse: WebProcessCache requires usesWebProcessCache, default false, and the app sets no process-pool options. Ruled out.
• storage.session stall: CPM's refreshRemoteConfig awaits getFromSessionStorage("config") with no timeout, so a stall would hang every init. No stall in any screen log; only tabs.sendMessage on the busy-page profile. Not reproduced.
• Network replaced while the background is unloaded (testNetworkProcessReplacedWhileBackgroundIsUnloadedKeepsWakesWorking, 30 s timing): 0 lost in 6 busy navigations and 4 idle wakes.
• macOS tab URL: Tab.url(for:) returns content.urlForWebView, which follows webView.url one main-queue hop later; WebKit reads it live for sender.tab.url. Harness mode CPM_REPRO_MACOS_TAB_URL=1 reproduces the lag: 133 real-page visits, 0 lost, 0 monitor episodes. init arrives after the hop.
• webExtensionPageWebView: only set for webkit-extension: pages and cleared on content change. Not relevant.

Reinterpretation: "idle wake" is a state at failure, not proven to be a cause. On desktop the background unloads after every 30 s idle, so a large share of all navigations, healthy ones included, run right after a wake, and failures from any cause land in this group at that base rate. That fits P1 changing nothing. Confirming it needs a healthy-navigation baseline (background state on successful measurements); none exists since #7384 was closed.

Remaining untested production ingredients: the real macOS Tab stack (DDG user scripts / content-blocking rule lists in tabs, multiple windows, pinned and hidden tabs, App Nap); needs the app-hosted harness in a VM.

Plan and corrections (27 Oct)

Corrections

• P6 is not the feature flag. webExtensions is on in production, not toggled, and will be removed. The flag-cycle mechanism (also in the tabControllerMatchesContext doc comment) was an old unverified hypothesis. Established: a tab's web view gets the controller only at creation and only if webExtensionManager exists then (WKWebViewConfigurationExtensions → attachWebExtensions), and the manager is created once at launch and replaced only by the flag handler. Trigger unknown. Candidates: web views created before the launch-time manager; the flag evaluating off at launch for some installs (missing or stale cached config); a web view built outside the standard configuration path. Macos 1.213 on 26 Oct: 51 users with controller mismatch + no scripts, 19 permanent (22% of macOS permanent hangs). The fix depends on the trigger.
• Error 6: the P2 retry is shipped and helped. iOS error-6 users fell from 1.3% to 0.5% of DAU, and 455 iOS users/day had a successful background_load_retry. Error-6 episodes have outcomes: iOS 86% recovered later, 9% permanent. It is still 25% of iOS stuck because (a) the retry restores the background but not the page whose init was lost, which nothing re-sends (late reload off); (b) the retry is skipped while the app is in the background, within the 60 s cooldown, or during M4; (c) when the failed background view is kept (H3, reproduced), the retry cannot complete. Which dominates is unknown: no retry attempted/result field in the episode.
  • Update 28 Oct: (a) was not reproduced for suspension (the page is answered after resume, WebKit creates a new view); the retained-view case (b/H3) is reproduced separately. See the 28 October section.
• Extension updated: yes, the outdated version is loaded. WebExtensionLifecycleCoordinator.loadAndSync runs loadInstalledExtensions() (the previous, now outdated version) before syncEmbeddedExtensions, which installs the bundled version under a new identifier, loads it alongside, and unloads the old one. Early pages after an app update run against the old version, then the context is swapped with a new origin and the migration blackout.

Next steps

Order superseded by Priorities — macOS browser-wide breakage first at the top of this note (28 Oct, two-day data); the items below remain the detailed list.

1. Ship what's understood:
   • P3 release.
   • Extension updated: upgrade before load at launch; carry the origin across upgrades.
   • Error 6: restart the stale background on error 6 when the failed view is still alive, and re-init the page that lost init.
   • Update 28 Oct: implemented as the restart in the section "Proposed fix" below (PR #7730, closed, branch kept). The "re-init the page that lost init" part was not needed in the simulator: the page was answered after the restart.
   • P6: find the trigger first (the three candidates above), then fix.
2. Verify the macOS Network-recovery false-positive lead: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and died_client precedes 16% of macOS idle-wake failures. Review CPMNetworkProcessRecovery detection; count recovery pixels in the harness screens.
3. Telemetry (one privacy review): events after the first failure (two-phase bg_events); a healthy-navigation baseline (background state on successful measurements); retry attempted/result; recovery applied in this episode; time since app resume.
4. Targeted harness work only: Network-recovery restart racing a wake; iOS resume-time wake on the simulator (fix the Bundle.module skip); the app-hosted macOS harness in a VM if steps 2–3 leave the wake group unexplained.
5. Mitigation experiment: keep the background warm while browsing (periodic loadBackgroundContent < 30 s), behind a flag; it is also a causal test for the wake group.
6. Re-measure on full days of 7.243 / 1.213 and the first P3 builds.

Data export — 26–27 Oct (built 28 Oct)

One CSV: cpm-failure-groups-2026-10-26_27.csv (iOS 7.243.0 / macOS 1.213.0, both full days combined). Per platform: an ALL row, then each symptom group's total and its top 5 event patterns plus the remainder. Columns: users (failing install-days), % of platform failures, % of DAU, recovered, no response unconfirmed, permanent, % permanent, % of all permanent on the platform. Repeated idle cycles are collapsed as (dealloc>view>proxy_on)xN; [capped] marks timelines that hit the 12-event limit. Totals cross-checked: iOS 29,758 failing install-days, macOS 5,946.

DAU proxies: macOS m.mac.daily.active.user.d; iOS m.app.return.ios.{phone,tablet}.daily. Users = .daily leg (install-days when combined); episodes = .count leg. Group priority: error 6 → no background view → tab not connected (no scripts or controller mismatch) → extension updated ≤2 min → idle wake (view <30 s after a dealloc) → recent context load ≤2 min → other. End state: recovered = any response; unconfirmed = no response, no repeat, no reload; permanent = no response with a repeat or reload.

Two-day headline (% of DAU-days): iOS failing 9.1%, permanent 0.43%; macOS failing 28.5%, permanent 2.87%. Largest permanent source: iOS recent context load (40% of permanent), macOS idle wake (38%), then macOS P6 (22%, 39% of P6 users permanent).

Repro attempts and footprint comparison — 28 October 2026

Internal metrics; keep these figures out of public PRs. Data: ClickHouse via ~/Developer/grafana-data-tools reader, iOS 7.243.0, 26–27 Oct, web.extension.cpm.episode*.count, episodes with bg_load_error_after present.

What production shows

• 3,114 error 6 episodes, 229 permanent (outcome=no_response_before_deadline, 7.4%).
• Permanent, by error timing and whether the background view is alive at report time:

Group	Episodes	Share of permanent
error under_5s, view alive	172	75%
error 25s_to_35s, view not alive (repeated view,proxy_on,dealloc,error_6 cycles, about one per minute)	51	22%
error under_5s, view not alive	5	2%
error over_35s, view not alive	1	0.4%

• The 172 look like the recovered ones. Same group, permanent vs recovered (2,797): interruption in the footprint 98% vs 96%, died_client 22% vs 22%, extension_updated 10% vs 6%, none of the three 3 vs 92. Permanent rate inside the group about 5.8%. These tokens do not explain why they stay broken.
• No unresponsive token in any permanent group.
• bg_view_alive cannot tell "WebKit kept the failed view" from "WebKit created a new view after the error". No field separates the two.

What I reproduced (iOS Simulator, iPhone 17, iOS 26.5, real CPM, real page)

1. Suspension during a background load (SIGSTOP of the app and WebKit helpers for 45 s, with a real HOME; also with the start slowed to 8 s): the overdue 30 s unload timer fires at resume, error 6, view released, WebKit creates a new view, the page is answered. Footprint dealloc,view,proxy_on,interruption,error_6 matches the dominant production footprint, but it self-heals. Persistence of this family in production is NOT reproduced.
2. Exception thrown at service worker start (injected, keyed on the extension origin): error 6 within 0.4 s, no underlying error, the failed view stays alive (WebKit _loadServiceWorker with success NO records the error and returns without unloading), the retry never completes, the page is unanswered, the second wake fails the same way, M4 heals after about 54 s. The footprint is not the dominant production one. PR #7730 (restart the context when the failed view is retained) fixes it in the simulator (restart 8 ms after the error, page answered after 0.6 s). The PR was closed before this was proven; branch alex/cpm-error6-restart kept. How often this state happens in production is unknown.
3. Hung start (busy wait of 90 s in the worker): the 30 s timer unloads the view, error 6 about 35 s after creation, view released, the retry hangs again, M4 heals after about 54 s. Footprint matches the 25–35 s loop group. M4 healed only because it re-extracts the clean extension; real persistence across M4 is NOT reproduced. The pixel the app logged for this run does not equal production (bg_events=load@2m, bg_ready_after=pending, bg_view_age=8s_to_30s), because the snapshot is taken at the first failed navigation, before the first loop ends.
4. Extension update during a background load (2026.10.23 over 2026.10.99): error 6 on the superseded context only, the new context answers.
5. Separate finding: after any app relaunch the embedded worker stays silent in the simulator (stale registration: importRegistrations failed to retrieve main script, then Found directly reusable registration), no error 6. Whether a device does this is unknown.

Reason for a failed start can be read

The background page loads with navigator.serviceWorker.register(...); its rejection carries the WebKit error text (Error: <message>), obtainable with a WKUserScript and WKScriptMessageHandler on the background web view delivered by didCreateBackgroundWebView (before the page loads). Message only, no file or line. The extension's own extensionException messages are not handled by the app (unknown method). Verified once with an injected exception.

Historical draft — 28 October 2026: retained-view restart

Superseded by cpm-post-release-analysis-2026-10-27#Error-6 restart — 30 October 2026. PR #7730 is now OPEN at cf668e2938; the following closed/no-new-pixel/three-test statements describe the 28 October draft, not the current implementation.

• Behavior. On error 6 (current context, recovery flag on, app in the foreground, no M4 in progress, 60 s cooldown) the manager checks WKWebExtensionContext._backgroundWebView. If the failed view is still alive it reloads the embedded extension at once (new reload trigger cpmBackgroundLoadFailure) and loads the background. If the view was released it keeps the plain P2 retry, which preserves the failed document's init.
• Why. WebKit's loadBackgroundWebViewIfNeeded returns early while the view exists (m_backgroundWebView set), and after a failed _loadServiceWorker WebKit records error 6 without an underlying error and does not unload the view. The retry therefore cannot replace it.
• Flag. cpmBackgroundLoadRestart: remote kill switch, default enabled in iOS and macOS FeatureFlag, off by default in the manager API. Separate from cpmBackgroundLoadRecovery, so turning it off keeps the plain retry. Feature Flags Registry task: https://app.asana.com/1/137249556945/project/1211834678943996/task/1219970564251181 (created by a parallel session, linked in code).
• Telemetry. No new pixel. The only change is a new value cpm_background_load_failed of the trigger parameter of the existing reload pixels, and those fire only when the reload itself fails. Nobody could count how often the restart ran or whether it helped. Proposal, not done: a recovery pixel like P2, for example web_extension_cpm_background_load_restart with outcome=background_ready; needs the pixel definition and privacy review.
• Tests. Three unit tests in WebExtensionManagerTests (restart with a retained view, plain retry with the flag off, plain retry with a released view). They run on mocks and show only the manager's decision; red without the branch, green with it, also in release. They do not reproduce the failure.
• Real check. Simulator, real CPM, injected exception at worker start: without the fix the page is unanswered until M4 (about 54 s); with the fix Reloading extension comes 8 ms after the error and the page is answered after 0.6 s.
• Branch. alex/cpm-error6-restart: 49d4bbb309 (test), 4997652d87 (fix, flag), a417730e60 (registry link). PR #7730 was a draft, closed on 28 Oct because it was opened before the problem was reproduced; the branch is kept.
• Does not cover. The 25–35 s loop (view is released each time), the interruption family (suspension self-heals in the simulator), and any case where the retained view is not the problem. How often the state occurs in production is unknown (no field).

Open

• Why 172 permanent episodes of the interruption family do not recover, when 2,797 identical-looking ones do.
• What makes a start hang longer than 30 s (candidates from earlier notes: origin-change data migration, host main thread stall, process throttling), and why that survives M4.
• How often the failed view is retained.
• Historical 28 Oct gap, now implemented in PR #7730: web_extension_cpm_background_load_restart with bounded outcomes. Privacy review and release/rollout are separate gates; see the 30 October update above.

Idle wake investigation — 28 October 2026

Re-cut of the post-release data (macOS 1.213.x, 26–27 Oct, matured since the sections above, so shares differ a little; per-episode web.extension.cpm.episode.count) and of the old contract (1.211.x / 1.210, 18–20 Oct, debug.web.extension.cpm.messaging.stuck.<reason>.count, one report per tab, confirmed by a second timeout). Not like-for-like: compare shapes, not percentages. OS comes from the os_version column of metrics.pixels, not a pixel parameter. Wake = the token rule view@30s plus an earlier dealloc (excluding no-view, no-scripts and error 6).

Was it there before Network recovery and P1? Yes

• 1.211.1 shipped neither fix and already had wake at ≈33% of stuck; 1.213 is the same.
• died_client in bg_events means the app killed the background process. The only shipped code path that does it to a background view is CPMNetworkProcessRecovery (_killWebContentProcessAndResetState). It is 0 of 65,374 reports in 1.211 and 3.6% of 1.213 episodes (5.1% of wake ones). It is more common among wake episodes that got an answer (9.2%) than among those that did not (2.3%). So recovery is a small new source of delays, not the cause. The "16%" in the priorities section did not reproduce with this cut (probably another denominator).
• 1.211 network_restarted is depleted in wake (12% vs 23% outside wake; 1.210: 14% vs 21%), without any recovery shipped.

What the wake group looks like

• Everything the pixel can see is healthy (7,787 episodes): context_loaded, bg_view_alive, bg_process_responsive, handler_registered true in 100%; bg_process_alive 99.95%; bg_ready_after=under_1s 98.8%; scripts present and controller matching 93% (3.8% P6, 2.9% unknown); error 6 0.3%; interrupted 0.03%. Outside wake the same fields do find things (no view 5.8%, no scripts 7.9%).
• bg_view_age 4s–8s in 66% (view created at the navigation), tabs_affected=1 in 78%, no response 57.6%, reload_state none 70% / loaded 19% / response_timed_out 11%, reason=other 90.5%.
• The timeline tail of permanent cross-tab wake (1,134) is the same as for answered ones (dealloc>view>proxy_on twice, 71% vs 67%). It does not separate them.
• No OS signal. Wake share 43–57% on every macOS 15.x / 26.x / 27.x; 27.2 is inside the range (wake 47.7%, no response 42.7%, unmarked 53.7%). Episodes per loaded install-day ≈0.7–0.9, 27.2 at 1.07.
• Unmarked core. Wake no-response episodes with no base_url_changed, extension_updated, load, died_*, error_*, interruption or unresponsive in the 12-event window: 54% in 1.213 (2,319 of 4,279), 52% in 1.211 (15,089 of 28,849; also no network restart, app update or memory pressure). It is uniform by hour, OS, dmg/App Store and tab count. In 1.211 it is 94% app_launch_age >15 min and 86% bg_view_creations >5: long sessions with many 30 s cycles, not startup. Nothing in the available parameters separates it.
• extension_updated is depleted in wake in both versions (2.8% vs 9.4%, 15.1% vs 30.7%); recent updates are not a trigger. base_url_changed was enriched in 1.213 permanent cross-tab wake (25% vs 13% of answered; tail 14.8% vs 5.8%) but is depleted in 1.211 wake (13% vs 28%): the lead does not replicate. Both timeline tokens fall out of the 255-character window after a few idle cycles, so older updates are invisible.
• Context error 7 is not a lead. The old context_errors showed WKWebExtensionContextErrorDomain:7 (= ScriptExecutionError, no underlying pair; WebKit also raises it for console.error) in 27% of 1.211 wake reports vs 17% outside, and 22% vs 16% on 1.210. The rate rises with context age (0 / 6 / 20 / 22 / 26% for bg_view_creations 0 / 1 / 2 / 3–5 / >5) and wake differs by ≤5 points inside a bucket. The shipped contract keeps other context errors locally and sends only error 6.
• Extension load-error pixels are negligible: load.error, embedded.not.loaded, reload.failed are ≈0.1% of loaded installs or less in both versions (1.213: 9 / 8 / 5 events on 18,299 install-days; 1.211 one day: 17 / 101 / 72 on 102,348).
• Session restoration is 4.5% of macOS episodes (16% no response) and 21% of iOS episodes (9.6% no response); only 0.6% of the wake group. It mostly recovers; reason labels only a first failing navigation that is itself restored.

History of reproductions vs this signature

• 8 Oct: a 5 s busy-wait in the worker at wake reproduced the coarse wake signature of that time (responsive=true, no errors, dealloc/view/proxy_on@30s) but recovered about 1 s later; 12 s plus a next navigation gave stuck and M4. Production bg_ready_after is under 1 s, so a slow worker start does not explain the permanent group.
• Reproduced but different: kill -9 of the background (all tabs, self-heals ~1 min), Network crash (fixed by recovery), stale failed view H3 (iOS pattern B), origin-changing M4 blackout (seconds to ~32 s), P6 (single tab).
• Not reproduced: plain idle wake (0 lost of 1,500 and 1,333 wake navigations, with ad-blocking and real pages). Nobody has reproduced the permanent, cross-tab, healthy-looking, ready <1 s case that survives M4.

What the monitor measures: http(s) only, but error pages slip through

• isEligibleForCPMMessagingHealthMeasurement accepts only http/https (CPMMessagingHealthMonitor.swift:1094). about:blank, duck://settings, duck://bookmarks and duck://player are not measured (harness test testNonHTTPPagesAreNotMeasured: no episodes). Events come only from tabs (PrivacyDashboardTabExtension, TabViewController); PIR web views were not checked.
• The embedded extension manifest (duckduckgo-privacy-extension/browsers/embedded/manifest.json, 2026.10.14) matches *://*/* in all frames with no exclude_matches; the app grants every requested pattern (WebExtensionLoader.swift:271). There are no per-site exclusions. The content script is plain autoconsent; a main-frame init always ends in a dashboard refresh.
• Error pages are measured as CPM documents and reported as failures (reproduced in the harness). macOS shows a failed load with loadAlternateHTML(html, baseURL: duck://error, forUnreachableURL: failedURL) (Tab.swift:1620). Then webView.url is the failed http(s) URL, so the filter passes; the navigation action URL for alternate loads is also webView.url (NavigationAction.swift:183); PrivacyDashboardTabExtension does not skip .alternateHtmlLoad (PageSignalsTabExtension does). In CPMReproErrorPageTests.testErrorPageIsNotReportedAsCPMFailure (worktree cpm-remaining-failures-repro, uncommitted; CPM_REPRO=1 swift test --filter CPMReproErrorPageTests): navigation failed, then the alternate load finished with the failed http URL; no init reached the background and no content-script mark appeared (the script does not run on the error document); 4 s later the monitor counted a failure, and the next healthy navigation closed an other / failed_tab_response / tabs=1 episode. The harness feeds the monitor webView.url as iOS does; the macOS Navigation.url path was read in code, not run in the app. iOS error pages were not tested.
• Hypothesis, not verified: pages that fail to load are shown on every tab after wake from sleep or a network drop. Each would count as a CPM failure right after an idle unload (fresh dealloc>view>proxy_on, background ready in under 1 s, healthy diagnostics), in several tabs (cross-tab) and with no answer until a normal navigation (permanent-looking, and M4 would not help). A same-tab second timeout also triggers M4 on a healthy extension (see was_stuck note above), which then adds the origin-change blackout. The pixel has no URL, so data cannot separate this; the check has to be an app run (debug build, Wi-Fi off, several tabs, wake) with the monitor log, or filtering .alternateHtmlLoad / duck://error and comparing wake rates.

Notes

• Release status corrected 30 Oct: P3 is absent from the measured public 1.213.0 / 7.243.0 tags. #7752 / #7753 are merged into the upcoming 1.214.0 / 7.244.0 release branches; public release and remote enablement remain separate checks (see the release update above).
• Scratch SQL runner and queries are in the session scratchpad; the harness test is not committed or pushed.

Addendum 28 Oct — error pages produce the wake signature (replaces the first version of this addendum)

• Pixel: m.mac.errorpageshown.other (+ .daily, pixel_id='m', params d/e = error domain/code, pixelSource; also …webkittermination), fired in Tab.swift when the error page is shown. macOS 1.213, 26–27 Oct: 7,511 shows, 2,284 install-days (≈3.3 per affected install-day). Codes: DNS -1003 29%, TLS -1200/-1202… 26%, -1005 10%, offline -1009 only 1.6%. Any error page counts, not only "no network"; the sleep / offline story from the first version is not needed.
• The wake happens at the navigation, the error comes later. The navigation gate asks initialLoadWaiter, which calls manager.prepareEmbeddedBackgroundForNavigation() → context.loadBackgroundContent at policy time of every main-frame http(s) navigation (WebExtensionLifecycleCoordinator.swift ~80–105, WebExtensionManager.swift ~993; macOS caller PrivacyDashboardTabExtension.decidePolicy). So a navigation to a dead host after an idle unload recreates the background first and fails afterwards.
• Reproduced in the harness (CPMReproErrorPageTests.testErrorPageAfterIdleUnloadDoesNotWakeOrFail, production 30 s unload, real navigation to an unresolvable host, then loadAlternateHTML(… forUnreachableURL:) as the app does; worktree cpm-remaining-failures-repro, uncommitted): the background view was created ~2 ms after the policy decision, the request failed 0.15 s later with the view present, the error document sent no init, the monitor counted the 4 s timeout and closed other / failed_tab_response / tabs=1 on the next healthy navigation. That is the wake signature: dealloc>view>proxy_on with a view younger than 30 s, bg_ready_after under 1 s, healthy diagnostics, reason=other, one tab. The snapshot is taken ≥4 s after the error page finishes, so bg_view_age lands in 4s_to_8s (66% of wake) or 8s_to_30s (33%), as in the data. (A first test without the real navigation did not wake the background.) The harness feeds the monitor webView.url; in the app, PageSignalsTabExtension skips .alternateHtmlLoad in navigationDidFinish, which shows that handler also runs for error pages, and the alternate-load action URL is webView.url (NavigationAction.swift:183).
• Magnitude, install-days (separate pixels, not joined by install): error-page .daily on CPM-capable macOS (15.4+): 1.213 10.3% of loaded.daily (1,891 / 18,299); 1.211.1 10.2% (10,507 / 102,806, 19 Oct). Wake-signature install-days: 1.213 13.3% (2,443 episode.daily rows), 1.211.1 14.6% (15,058 first-failure .daily rows; 5.5% for confirmed stuck). Error pages per affected install-day ≈3.3, wake episodes per wake install-day ≈3.2. This is a ceiling, not a proof: only an error page after an idle unload carries the wake tag (a warm background gives other), and some error pages may not take the alternate-load path.
• Why it looks permanent, cross-tab and immune to M4: a second timeout in a tab runs M4, which reloads a healthy extension (new origin, 1.5–32 s blackout). Error pages in two or more tabs with no healthy navigation within 5 minutes end as no_response_before_deadline, was_stuck=true, reload_state=loaded/response_timed_out, with healthy diagnostics. That matches the cross-tab permanent wake group without any CPM breakage. Not checked: how many permanent cross-tab episodes are such loops.
• Weak evidence against, kept for the record: per-OS correlation of error pages per install with wake episodes per install is r = 0.02 (n = 9); hourly r = 0.88 for both wake and non-wake (rollout trend, uninformative).
• Unverified: the running app (the scenario with log stream --level debug --predicate 'subsystem == "WebExtensions" AND eventMessage CONTAINS "CPM Health Monitor"': three tabs idle >30 s, load a dead host in each, expect timeoutElapsed without Dashboard response attributed); iOS error pages; the share of wake episodes that are error pages (the episode pixel has no URL or error code).
• Decisive options: (a) macOS build where the monitor ignores error documents (.alternateHtmlLoad / failed provisional navigations) and M4 never fires for them; compare wake share and permanent rate with the current build; (b) a flag parameter on the episode pixel (needs the privacy review you wanted to avoid).
• Fix, tests and PRs (updated 30 Oct):
  • PRs merged: #7752 into macOS 1.214.0 and #7753 into iOS 7.244.0; merge commits and final branch heads are recorded in the release update above. The sibling branches contain the same intended WebExtensions/error-page changes on their respective release bases.
  • Commit content: error-page monitor filtering and macOS alternate-HTML mapping; OS 26.6+ and flag gate; stored-origin rotation on a disabled/unsupported context load; JSON key rename to cpmStableOriginIdentifier with installation-UUID fallback. No migration or OS-version tracking. Harness/reproduction suites are not included.
  • Run results: key-only revision: all 1,025 package tests passed. Final simplified policy suite: 25 passed, including unsupported-OS coverage; strict lint passed on both branches. Three macOS error-page app integration tests passed before the key/test-only edits, with a negative control; no iOS runtime run. See release update for the exact scope.
  • Integration tests (real Tab, macOS Integration Tests, run locally with the signed Debug build): ordinary load error (-1003), SSL (NSURLErrorServerCertificateUntrusted), malicious-site warning (MaliciousSiteError via the duck://error redirect) all pass. Negative control with the macOS mapping removed: the test fails, the monitor receives ["started:other", "failed", "started:other", "committed:https://testhost.com/", "finished:https://testhost.com/"], the error page as an ordinary navigation of the failed URL (the false positive in the app's own Tab/Navigation path); mapping restored. They use their own StubWebExtensionManager; WebExtensionsTestSupport/MockWebExtensionManaging exists but that product is not linked to the Integration Tests targets.
  • Running locally: do not build with CODE_SIGNING_ALLOWED=NO: the host app then lacks the keychain entitlement (-34018) and macOS asks for keychain access. Use the normal signed Debug build, xcodebuild … test-without-building -only-testing:"Integration Tests/CPMErrorPageMonitoringTests".
• Corner cases checked (29 Oct), no further false-positive source found:
  • Harness (CPMReproCornerCaseTests, real WebKit, real extension, loopback HTTP), all pass: HTTP Basic auth accepted (CPM answers) and cancelled (401 page, no episode); client redirects at 0.1 s, 2 s, meta refresh and a redirect after the 4 s grace (CPM answers on the target, no episode); PDF, text/plain and JSON documents (content scripts run on PDF too); a Fire-window tab with a non-persistent store; a download link (WebKit fails the provisional navigation, no document, no episode); 25 tabs loaded at once (all answered in under a second). A 6 s JS timer redirect never fired in a window-less web view (timer throttling), a test artefact.
  • Tab close during a measurement is covered by existing monitor unit tests. Policy redirects are exercised by the malicious-site integration test.
  • iOS by code reading only: SSL/malicious special pages load via loadSimulatedRequest for the failed URL, an ordinary document on which CPM answers; other load errors use a native error view without a document.
  • Not checked: real minimised/occluded windows and App Nap (harness web views have no window and still answer in milliseconds); http→https upgrade through the app's HTTPS upgrade store; popups from about:blank; iOS app behaviour at runtime.
• What this does not settle: the share of wake episodes that are error pages is unknown (the episode pixel has no URL or error code); the install-day ceiling is ~10% (error pages) against 13–15% (wake). Real CPM breakages stay as before: error 6 on iOS, origin-change blackout, P6, extension update, died_client (~5% of wake). The check after release: wake episodes per loaded install-day should drop by roughly the error-page share, with m.mac.errorpageshown.other as the control.