Files
obsidian-vault/work/projects/cpm-post-release-analysis-2026-10-27.md
T

60 KiB
Raw Blame History

title, date, tags
title date tags
CPM — first post-release data (iOS 7.243.0 / macOS 1.213.0) 2026-10-27
work
cpm
analysis

CPM: first post-release data, 26 October 2026

Internal metrics; keep these figures out of public PRs. Previous analysis: cpm-metrics-causes-and-mitigations-2026-10-16, cpm-status-2026-10-15 (§3–5, 12–14 October cut).

Priorities — macOS browser-wide breakage first

macOS fails far more than iOS, so priorities follow cross-tab permanent failures on macOS: CPM not answering in two or more tabs for 5 minutes, confirmed by a repeat or reload. These are browser-wide breakage, not one stuck tab. 26–27 Oct, macOS 1.213: 386 users, 1.85% of DAU.

Share of macOS cross-tab permanent: idle wake 174 (45%), other 78 (20%), extension updated 77 (20%), recent context load 35 (9%), no background view 11 (3%), P6 9 (2%), error 6 2 (0.5%).

  1. Idle wake (45%, cause unknown).
    • Check Network-recovery false positives: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and died_client precedes 16% of macOS idle-wake failures.
    • Add telemetry: events after the first failure (two-phase bg_events), a healthy-navigation baseline, retry/recovery applied in the episode.
    • Keep-warm experiment behind a flag (also a causal test).
  2. Extension updated (20%). Load the new bundled version before the old one at launch, and keep the origin across upgrades. Repeats after every app update.
  3. Other (20%). No distinguishing signal yet; needs the telemetry from step 1.

Not priorities:

  • P3 / origin change: the baseURL fix is released. Verify its effect on recent context load, post-reload silence and the origin-change patterns (base_url_changed) once the build reaches users; no further work planned.
  • P6: 96% of P6 users have a single failing tab; 122 of its 131 permanent users are single-tab, only 9 cross-tab. One stuck tab, not browser-wide breakage. Low priority; find the trigger when convenient (~37% show extension_updated before the failure).
  • Error 6 on macOS: 2 cross-tab users.

iOS (cross-tab permanent 861 users, 0.26% of DAU): recent context load 38% (origin change; covered by the released P3), error 6 20% (error-6 loop: restart the stale background and re-send init), extension updated 14% (same fix as macOS step 2), idle wake 14%, other 8%, no background view 5%.

Summary for the team — 26–27 Oct

CPM post-release analysis: iOS 7.243.0 / macOS 1.213.0, 26–27 Oct (2 full days)

Definitions: failure = no CPM dashboard response within 4 s after a page finished. Episode = one failure until resolution or the 5-minute deadline. Permanent = no response for 5 minutes, confirmed by a repeat failure or a reload. Affected users = users who reported a failure that day (daily pixel); all percentages are of DAU. Episodes = every failure report (count pixel).

Size

  • iOS: 9.1% of DAU hit a failure, 0.43% permanently. About 3× fewer failures than the previous release.
  • macOS: 28.5% of DAU hit a failure, 2.87% permanently. No change vs the previous release.
  • Most failures recover by themselves: 89% on iOS, 63% on macOS.

Where permanent failures come from (share of permanent users)

  • iOS (0.43% of DAU): recent context load 40%, error 6 20%, idle wake 16%, extension updated 13%, other 7%.
  • macOS (2.87% of DAU): idle wake 38%, P6 (tab not connected) 22%, extension updated 15%, other 14%. Browser-wide (cross-tab) only, 1.85% of DAU: idle wake 45%, other 20%, extension updated 20%, recent context load 9%, P6 2%.

Persistent mechanisms (repeat episodes from the same users confirm a stuck state)

  • iOS error-6 loop: error 6 hits 0.86% of iOS DAU, and 10% of those users stay stuck. Stuck users average 7.4 episodes, recovered ones 1.2. In 38% of error-6 episodes the background keeps failing to load on every wake. Matches the stale-failed-background state reproduced in the harness. Fix: restart the stale background on error 6 and re-send init to the page.
  • Failures after an origin change (base_url_changed): present in 1,733 iOS users (14% permanent, 3× baseline) and 193 macOS users (16%). On macOS these installs fail repeatedly (13–27 episodes per user in the idle-wake patterns, from small groups). The origin change itself causes a 1.5–32 s blackout depending on cache size (reproduced); P3 removes it. P3 is released; verify its effect.
  • macOS P6: 1.6% of DAU, 39% of them permanent, about 2 episodes per user. 96% are a single stuck tab (122 of 131 permanent are single-tab), not browser-wide breakage. ~37% of P6 users show extension_updated before the failure, so the mid-session upgrade is a likely trigger. Fix: find the trigger (upgrade path first), then fix.

Transient, not a stuck state

  • Extension updated: one episode per user, at upgrade time. Still worth fixing (load the new version first, keep the origin), since it repeats after every app update.
  • Idle wake: mostly recovers (iOS 88%, macOS 58%). Its repeat-heavy part overlaps the origin-change patterns. Cause unknown; it may just be the state most macOS navigations are in.

Caveats: two days after release, mostly early updaters (macOS ~17k DAU). Loop figures come from small groups. Not measured yet: retry attempts, events after the first failure, a healthy-navigation baseline.

What shipped

  • iOS 7.243.0 (tag 26 Oct 19:23 UTC), macOS 1.213.0 (tag 26 Oct 16:23 UTC).
  • Both: episode telemetry #7376/#7484, episode tab scope #7520/#7521, P1/P2/P4 recovery #7379/#7536, Network-process recovery #7459/#7531.
  • macOS only: P5 shared-pinned window close #7473, remote/debug disable #7498.
  • Not shipped: P3 stable origin #7493 (on main only). No P6 fix exists in either release: no PR implements the controller rebind.
  • Shipped episode contract is the rolled-back variant: was_stuck (bool), single-phase bg_events (no ;), plus tabs_affected, repeat_failures, interrupted. No bg_view_creations, context_errors or network_restarted. Old cohort SQL needs adapting (below).
  • Monitor timing: 4 s response grace; suspended time no longer counts (8 s after resume, 30 s extension-load wait, 5 min deadline).

How to query

  • New pixels have no m_/debug_ prefix: ClickHouse pixel_id = 'web', pixel LIKE 'web.extension.cpm.%' (iOS adds .ios.phone|tablet). Querying pixel_id='m' without an agent filter and a one-day window exceeds the 7B row limit.
  • Denominator: pixel_id='m', m.web.extension.loaded.daily% / m.mac.web.extension.loaded.daily, split by appVersion.
  • Old-contract attribution was remotely switched off from 21 Oct (old versions keep sending bare debug.web.extension.cpm.* without parameters). The last fully attributed old-version window is 18–20 Oct, iOS 7.241 / macOS 1.211.x.
  • Data lag: on 27 Oct the table ended at ~01:10 UTC; date is US-Eastern. 26 Oct is the only post-release day and it is partial for new adopters (exposure starts at update time).
  • Runner: ~/Developer/grafana-data-tools reader; ad-hoc SQL and an ARRAY JOIN extractURLParameters(request) parameter dump were used. extractURLParameter needs a constant key.

Rates per extension-loaded install-day (26 Oct)

iOS 7.242 (old) iOS 7.243 (new) macOS 1.212 (old) macOS 1.213 (new)
loaded.daily 4,257,839 52,979 93,030 2,566
old init_failed.daily (sum of reasons) 21.3% — 36.8% —
old stuck.daily 3.3% — 12.5% —
episode.daily — 7.05% (3,735) — 30.0% (771)
episode with reload_state≠none (hang confirmed, ≈ old stuck), count leg — 23% of episodes — 26% of episodes
was_stuck=true (second distinct tab), daily leg — 8.1% — 11.2%
network_process_recovery_applied.daily — 432 (0.8%) — 232 (9.0%)

Not like-for-like: the episode unit is one report per episode, the old init number sums four reasons, suspended time is now excluded, and day-1 adopters have short exposure. Directionally, iOS fell ~3× (21% → 7%); macOS barely moved (37% → 30%).

was_stuck only becomes true when a different tab fails; M4 reload already runs on a same-tab second timeout. Use reload_state≠none as the old "stuck" proxy.

Cohorts (count leg, adapted classification)

Priority: error 6 (bg_load_error_after present or error_ token) → no view (bg_view_alive=false) → tab not connected (tab_has_ext_scripts=false) → extension_updated ≤2 min → wake (bg_view_age < 30 s and a dealloc token) → recent load/base_url_changed ≤2 min → other. There is no Network-replaced cohort any more.

Cohort iOS share / of stuck / no-response macOS share / of stuck / no-response
error 6 7.5% / 24.7% / 39% 0.6% / 1.7% / 44%
no view (P3) 7.7% / 6.8% / 7% 2.9% / 0.9% / 25%
tab not connected (P6) — 5.8% / 1.3% / 70%
extension_updated ≤2 m 3.9% / 6.2% / 37% 14.6% / 11.7% / 28%
wake, view <30 s 22.8% / 22.0% / 9% 43.9% / 58.4% / 48%
recent context load 22.8% / 17.2% / 12% 11.9% / 6.9% / 25%
other 35.3% / 23.2% / 5% 20.4% / 19.0% / 27%

Baseline with last week's exact SQL, 18–20 Oct (init → stuck shares). iOS 7.241: error6 6.4→22.0, no view 47.5→11.5, network 16.5→38.2, wake 7.7→9.2, recent load 16.7→6.6, other 5.2→12.5. macOS 1.211.x: error6 1.2→2.2, no view 3.6→1.9, tab not connected 5.9→8.7, network 13.2→25.8, wake 33.2→33.3, recent load 19.4→6.3, other 23.6→21.6. This matches the 12–14 Oct cut, so the baseline was stable.

Per-install rough rates (cohort share × failure rate), old init → new episode:

  • iOS: no view 10% → 0.5%, recent load 3.5% → 1.6%, error 6 1.3% → 0.5%; the network cohort is absorbed. Wake stays at 1.6% → 1.6%. Other rises 1.1% → 2.5%; it now absorbs former network cases and has no network field.
  • macOS: wake ≈12% → ≈13%. Unchanged.

Findings

  1. P1 (wake) fix had no measurable effect on either platform. In the wake cohort bg_ready_after=under_1s dominates (macOS 93% overall), so the background is not slow. The background_wake pixel fires only for ≥1 s callbacks (macOS 89/day vs ~670 wake episodes). The remaining failure is message/response loss after a fast idle wake, not readiness. On macOS 48% of wake episodes see no response for 5 min; tabs=3+ → 68%. Typical tail: …dealloc@15m,view@15m,proxy_on@15m,dealloc@5m,view@30s,proxy_on@30s, bg_view_age=4s_to_8s (view created around the navigation). The pattern appears on every macOS version, 15.6–27.2.
  2. iOS error 6 changed shape. All attributed error 6 is now bg_load_error_after=under_5s; the 30 s timer bucket is gone.
    • A (ready=failed, app_state active): dealloc → view → proxy_on → interruption → error_6, often after died_client/died_crash/died_crash_limit. The background load spans an app suspension and fails at resume.
    • B (ready=pending): an older error 6, then a new view whose load is still pending 8–30 s later.
    • The P2 retry fires only while foregrounded with a 60 s cooldown (277/day). Foreground-resume recovery is the missing piece.
  3. macOS P6 is untreated: 5.8% of episodes, 70% never answered. iOS now shows tab_has_ext_scripts=false in 5.3% of episodes, concentrated on iOS 26.7 (16%) and 27.0 (8%), with 0 on 27.2. Previously iOS was ≈0.
  4. macOS Network replacement is frequent: the applied pixel is on 9% of 1.213 install-days. With recovery applied it no longer appears as a cohort, but other stays at 20%.
  5. After M4 on macOS: reload_state=loaded → 46% no response by deadline; response_timed_out → 57%. The reload often does not restore messaging, or no qualifying navigation follows.
  6. Recovery pixels only ever report background_ready. response_received needs late-document reload, which is default OFF.
  7. Day-1 effect: macOS extension_updated ≤2m is 14.6% (embedded upgrade on update). Re-check after 2–3 full days.

Next measurements

  • Repeat for 27–29 Oct (full days, larger adoption). Track the wake cohort and macOS no-response by OS.
  • bg_events per-phase budget and network_replaced_since_view remain missing (see cpm-status-2026-10-15#6. Телеметрия).

Repro targets

Issue Evidence Hypothesis to test Candidate fix
R1 wake message loss macOS 44% eps / 58% stuck; iOS 23%; ready <1 s message to a woken non-persistent SW is acknowledged with no reply: (a) processes(RuntimeOnMessage) empty when _loadServiceWorker completion runs before listener IPC; (b) listener set loses RuntimeOnMessage so wakeUpBackgroundContentIfNecessaryToFireEvents resolves {} without waking; (c) background unloaded while a reply is pending keep the background warm while CPM pages are active, or re-send init on empty reply (native/JS handshake)
R2 iOS error 6 across suspension 7.5% eps / 25% stuck, all under 5 s, interruption between view and error WebKit load timer spans suspension; nothing retries on resume on foreground resume, retry loadBackgroundContent when the last load failed or is pending
R3 P6 controller mismatch macOS 5.8%, 70% unanswered known main-flag manager replacement rebind/recreate tab web views to the current controller

WebKit source pointers (~/Downloads/WebKit_sparse): WebExtensionContext.cpp wakeUpBackgroundContentIfNecessaryToFireEvents (drops events when the listener set lacks the type after the first load), scheduleBackgroundContentToUnload (30 s, reset only on dispatch), WebExtensionContextCocoa.mm loadBackgroundWebView / performTasksAfterBackgroundContentLoads, WebExtensionContextAPIRuntimeCocoa.mm runtimeSendMessage (empty processes() → {} reply). 324743 (9bcf5840ef) holds runtime messages until SW activation; it is not in shipped OS builds.

Repro log — 27 October 2026

Worktree apple-browsers.git/cpm-remaining-failures-repro, branch alex/cpm-remaining-failures-repro (from origin/main 8023e4cb5c, which includes the released P1/P2/P4/Network code plus P3). The package real-WebKit harness was imported from origin/alex/cpm-repro-harness. It compiles unchanged on current main, and its CI-tier tests pass. Local tier: CPM_REPRO=1 swift test --filter <Suite> from SharedPackages/WebExtensions. System WebKit is the host macOS; no app windows are opened.

  • R1 idle wake — not reproduced organically. CPMReproIdleWakeTests.testRepeatedIdleWakeCyclesDeliverInit: 3 tabs, 8 idle-unload cycles (WebKit unloads 32.0 s after the last answered navigation), single and double-tab wakes, 0 of 11 lost. Every wake answered in 2–5 ms after didFinish. testNavigationAtIdleUnloadBoundaryDeliversInit: offsets −0.6…+0.2 s around the calibrated unload, 0 of 7 lost. A plain idle wake on an in-memory fixture page is not the production mechanism. The missing ingredient is still unknown: real page loads, a long-lived data store with abandoned registrations, Network replacement, or listener-set loss.
  • P3 (#7493) only in internal builds (macOS 1.214.0-860, iOS 7.244.0-x). On 26 Oct: macOS 1.214 had 20 episodes / 9 daily of ~36 loaded installs, with 0 no-view and 0 error 6; iOS 7.244 had none. That is too small to evaluate. Relevance: released builds get a new random extension origin on every context load (launch and M4 reload), and WebKit migrates data before injecting. This may explain the iOS no-view cohort and the macOS "reload loaded → still no response" share (46%). Re-check when 7.244 reaches users.
  • R2 suspended wake — macOS analogue does not stay broken. CPMReproSuspendedWakeTests: idle unload, then a navigation wakes the background, and the new background process is SIGSTOPped for 45 s while another tab navigates every 10 s. Two navigations timed out (+9 s, +14 s). Then the background was replaced by a new process (same view slot, new PID) and every later navigation answered in 1–14 ms, including 10 after thaw. No error 6 was recorded on the live context. On macOS the shipped recovery restores messaging within ~14 s of the fault. The harness cannot reproduce iOS suspension, which pauses the UI process's own timers. Likely production sequence: the overdue 30 s unload timer fires at resume, closes the loading view, and records error 6 "under 5 s" (pattern A). Pattern B (new view pending 8–30 s) needs a device or simulator test.
  • WebKit hypothesis not hit yet: an SW load failure that keeps the background view (loadBackgroundWebView completion !success without unload). Later wakes would only queue, and every dispatch restarts the 30 s unload timer, so the public loadBackgroundContent/P2 retry cannot escape. A candidate mitigation is to restart the background web content process when error 6 is recorded while a background view still exists. Not implemented, because the state has not been reproduced.

Treatment pixels, P3 ceiling and permanent hangs — 27 October 2026

Treatment pixels received (26 Oct, count leg, iOS 7.243.0 / macOS 1.213.0)

Fix Pixel (pixel_id='web') iOS macOS
P1 wake web_extension_cpm_background_wake 2,026 89
P2 error-6 retry web_extension_cpm_background_load_retry 277 12
Install / upgrade readiness …_install_readiness / …_upgrade_readiness 3 / 24 0 / 1
Network recovery web_extension_cpm_network_process_recovery_applied 476 277
P4 detection (flag-independent) web_extension_cpm_startup_response_missing 450 (daily 327) 20 (daily 12)
Late page reload web_extension_cpm_late_document_reload 0 0
  • Every recovery pixel only reports outcome=background_ready, meaning the fix was applied and the background came up. reload_started / response_received exist only for the late page reload, which is default OFF, so it has zero rows on every version including internal builds. No pixel shows whether the page got its answer after a treatment; the effect has to be read from episode rates.
  • background_wake fires only when the wake callback took ≥1 s and a navigation waited, so it is not a count of all wakes.
  • Detection gap for the late reload. startup_response_missing fires with the flag off but requires startup.isFirstLoad: the first background load of that context in the session (launch, install, upgrade). attemptRecovery (the reload) has no isFirstLoad condition and would also run after later wakes. With the flag off, wake-time cases the reload would treat are not counted anywhere; 327 / 12 is a lower bound of what it could fix. Both detection and reload only arm when the page commits while the startup is still pending. In the wake cohort the background is ready in under 1 s, so many wake failures are outside both. Counting the would-be-reload wake cases needs a pixel change: drop isFirstLoad from detection or add a parameter. Not done.
  • Upper bound on the late reload: ~0.6% of iOS / 0.5% of macOS extension-loaded installs; ~9% of iOS / ~2% of macOS installs with a failure episode.

P3 (baseURL) ceiling on the new data

  • The earlier "≈half" was the pre-release no-view share: 48% of iOS initial failures, but only 11% of stuck, mostly self-healing.
  • On 7.243.0 without P3, no-view is already 7.7% of iOS episodes / 6.8% of stuck (≈10% → 0.5% of loaded installs); macOS 2.9% / 0.9%. The drop comes from the new measurement (suspended time excluded, one report per episode) and the shipped readiness gates.
  • Direct P3 ceiling ≈ 8% of iOS episodes, less on macOS. A possible indirect effect: released builds get a new random origin on every M4 reload, and 46% of macOS reload-loaded episodes still got no response. That link is unproven; internal 1.214 / 7.244 data is too thin.

Users affected by permanent hangs (26 Oct)

Denominators were refreshed after more 26 Oct data arrived: extension-loaded installs iOS 69,088 / macOS 3,045. "Permanent" = outcome=no_response_before_deadline (no qualifying response for 5 min). Single = tabs_affected=1, cross = 2/3+. Confirmed = repeat_failures≠0 or reload_state≠none. Daily leg = affected installs.

Installs/day iOS macOS
Any failure episode 4,655 (6.7%) 876 (28.8%)
Single-tab, no response 354 (0.51%) 217 (7.1%)
↳ confirmed 76 (0.11%) 25 (0.82%)
Cross-tab, no response 129 (0.19%) 58 (1.9%)
↳ confirmed 123 (0.18%) 49 (1.6%)
  • Cross-tab is the reliable permanent-hang figure: ~0.2% of iOS and 1.6–1.9% of macOS users per day; 95% / 84% confirmed.
  • Unconfirmed single-tab rows (iOS 278, macOS 192) may be stuck tabs or users who didn't navigate within 5 min.
  • Strict permanent (confirmed single + cross): ~0.3% iOS, ~2.4% macOS users/day. Upper bound including unconfirmed single-tab: 0.7% / 9%.
  • Lower bounds for users: the daily leg describes only each install's first completed episode of the day, and episodes lost when the process ends before the deadline are not reported. Count leg (episodes per install-day), confirmed cross-tab: iOS 0.41%, macOS 5.2%; confirmed single-tab: iOS 0.26%, macOS 2.3%.
  • One partial day; macOS has ~3k installs on 1.213.0. Repeat on 27–29 Oct full days.

Repro statistics note

18 wake navigations cannot rule out a rare loss. At p=8% a clean run happens 22% of the time. The per-wake production rate is likely ≪1% (many wakes per install-day), which needs ~1,000 wakes for 95% power. CPMReproIdleWakeTests now has CPM_REPRO_FAST_UNLOAD=1: it calls +[WKWebView _setApplicationBundleIdentifier:] with com.apple.WebKit.TestWebKitAPI, so WebKit unloads an idle background after 3 s instead of 30 s (~5 s per cycle). It can change other test-runner-only WebKit behavior, so confirm any hit with the 30 s delay. A 1,000-cycle run (~1,500 wake navigations) was started on 27 Oct.

  • Fast idle-wake run, 27 Oct: 0 lost of 1,500 wake navigations in 1,000 cycles (CPM_REPRO_FAST_UNLOAD=1, 3 tabs, single and double-tab wakes, 85 min). With zero failures the 95% upper bound on the per-wake loss rate is ≈0.2% (3/1500). A bare idle-unload → wake race on a fixture page is not the main source of the wake cohort; an extra ingredient is needed (hypotheses H1–H5 below).

Hypotheses for the remaining wake hangs (27 Oct)

The embedded extension (background-embedded.js, 2026.10.23) registers runtime.onMessage synchronously (MessageRouter). CPM init = await remoteConfigJson → native isAutoconsentSettingEnabled → native isFeatureEnabled → refreshDashboardState. Every native request has a 20 s timeout, with config_unavailable / setting_disabled fallbacks that still refresh. A 5-minute silence therefore means init never reached the worker, or the refresh was not attributed.

  1. H1 — worker without extension bindings after Network-process recovery (WebKit 324754). macOS recovery is applied on 9% of installs/day. A worker without browser.runtime throws at top level: alive, responsive, "ready", never answers, all tabs, M4 may repeat it. Test: Network crash → recovery restart → probe browser.runtime in the worker → N wake cycles. Fix: verify with a ping after restart; restart again or reload if it fails.
  2. H2 — listener metadata loss. wakeUpBackgroundContentIfNecessaryToFireEvents returns an empty reply without waking when the saved listener set lacks RuntimeOnMessage. The set is saved after every load, so one load where the worker threw before registering poisons it. Test: a prelude knob that throws or skips onMessage on one load only. Fix: treat an empty sendMessage reply as a failure → restart, plus re-send init.
  3. H3 — stale failed background view (iOS error 6, pattern B). _loadServiceWorker failure keeps the view, so wakes only queue; activity keeps restarting the 30 s unload timer; P2 retry uses the same path. Test: fail the worker load without closing the view, with navigations every 10 s. Fix: on error 6 with a live background view, restart its WebContent process, then retry.
  4. H4 — refresh not attributed. The monitor needs an exact scheme/host/port/path/query match with the committed URL. A cold wake answers later, giving replaceState / client-side redirects time to change sender.tab.url. other_tab_response is 37–53% of outcomes. Test: a fixture with replaceState 50–300 ms after load, cold vs warm. Fix: attribute by extension tab ID and treat same-document URL changes as the same document.
  5. H5 — slow cold-wake init. Three native round-trips plus session storage; explains late answers (response_after ≥30 s in 20–35% of answered episodes), not 5-minute hangs. Test: prelude delay on sendNativeMessage 1–3 after a wake. Fix: answer init from cached config/settings first.

H1/H2 tests — results (27 Oct)

New CPMReproBrokenWorkerTests and harness changes (uncommitted):

  • Options.isNetworkRecoveryEnabled, on by default as in the app; off in CPMReproWebKitTests.unprotected.
  • Prelude startFaults breaks one worker start inside a wall-clock window: .skipOnMessage (script runs, no runtime.onMessage listener) or .throw (top-level throw before the extension script).
  • CPM_REPRO_FAST_UNLOAD moved to CPMReproTestCase.enableFastUnloadIfRequested().
Scenario Fast unload (3 s) Production unload (30 s)
H2 skipOnMessage: one woken worker without onMessage only the broken navigation lost; all later ones answered not re-run
H2 throw: one woken worker throws at top level broken navigation + 5 busy + 3 idle-wake losses, tabs 1–2 broken for the rest of the test broken navigation + 1 busy loss, then everything answered
H1 Network crash, recovery on, 20 / 10 wakes 2 lost after crash, then tabs 1–2 never got initResp, tab-3 always fine; all restarted workers ran 2 lost after crash, then all 10 wakes answered; all restarted workers ran
  • H1 not confirmed: every worker restarted after Network recovery ran its script (bindings present). With the real delay, recovery restores messaging after 2 lost navigations.
  • H2 (listener metadata loss) not confirmed: one start without a listener only loses that navigation; P1's wake plus re-registration heals it.
  • The fast-mode per-tab breakage is an artifact: after a confirmed hang and recovery, tabs with an unanswered init kept receiving the dashboard refresh but never initResp. That is WebKit's silent tabs.sendMessage path (processes.isEmpty() → {}), and the monitor sent one was_stuck=true / reload_state=loaded / other_tab_response episode and nothing more. It did not reproduce with the 30 s delay. The TestWebKitAPI identity or the 3 s unload changes WebKit behavior; fast-mode hits must always be confirmed with the production delay. The blind spot it exposed is still real in principle: the monitor cannot see a tab that gets the dashboard refresh but no initResp.
  • H3 (stale failed background) reproduced with production timing: after the throwing start, error 6 is recorded and the failed background view stays present. The next navigation 9 s later got nothing, and only the M4 reload (new context) restored messaging. No background_load_retry pixel fired, so the P2 retry never completed, as predicted by WebKit's loadBackgroundWebViewIfNeeded early return. Cost: at least one extra lost navigation per error 6, plus a full M4 reload. On iOS, error 6 is 7.5% of episodes and 25% of stuck.

Production-gap ideas and coexistence screen (27 Oct)

The harness differs from macOS production in four ways:

  1. Other extensions: 83% of macOS 1.213 installs run the ad-blocking extension (m.mac.web.extension.daily.ad.blocking.state.daily is_enabled=true 3,486 of 4,182). It is a module service worker with DNR and four document_start all-frames content scripts. iOS has none; Dark Reader runs on ~140 iOS installs.
  2. Real pages: network loads, iframes, redirects, replaceState.
  3. Real app tabs and windows: visibility, App Nap, multiple windows and pinned tabs; app-hosted harness only, in a VM.
  4. Long-lived data store with abandoned registrations; weak, since ~30 harness runs already shared one persistent store.
  • Coexistence screen: CPMReproCoexistenceTests (new Options.embeddedTypes), CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,500 wake navigations (85 min); at 30 s, 0 of 15. The ad-blocking background started once and unloaded at 30 s, but never woke again on fixture pages, so concurrent wakes of both backgrounds were not exercised. Its content scripts ran on every page. A second extension's content scripts alone do not trigger the wake hang.
  • App Store vs DMG (macOS 1.213, 26–27 Oct, per loaded install): episodes 20.8% vs 29.7%, wake cohort 9.8% vs 10.6%, cross-tab no-response 1.2% vs 2.4%. Sandboxing does not drive the wake cohort; the DMG excess is likely early-updater bias.
  • Real-page screen (CPMReproRealPageTests): earlier repro tests used loadSimulatedRequest, so loads never reached WebKit's Network process. CPMReproFixtureServer serves loopback pages through it: cross-origin iframes (localhost vs 127.0.0.1), 302 redirects, replaceState, slow images, main-thread blocks. CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,333 visits across 7 profiles (~190 each), and no monitor episodes. Real loads, frames, redirects and URL rewrites do not trigger the wake hang.
  • Next hypothesis — origin-changing M4 reload stalls on large data: most confirmed macOS cross-tab hangs (1.213, 26–27 Oct, tabs_affected≠1 and no response) have tail proxy_on>dealloc>view>proxy_on, view 4–8 s, ready <1 s, scripts present, reload_state=loaded (97+22+22) or response_timed_out (34+17+13). The reload loaded, yet nothing answered for the rest of the episode. Released builds load the reloaded context under a new random origin, and P3 saw WebKit's origin migration slow with a large cache. Test: CPMReproReloadOriginTests fills the persistent HTTP cache (/fill, default 20,000 × 2 KB) and runs the M4 reload with released vs stable origin, timing post-reload silence. If confirmed, P3 also fixes hang persistence.
  • Reproduced: origin-changing M4 reload blacks out CPM proportionally to the HTTP cache (CPMReproReloadOriginTests, production timing, CPM only). Fill the persistent cache through real fetches, run the hang reload (reloadExtension(…, trigger: .cpmMessagingHang)), navigate every 2 s:
Cache entries (2 KB) Released origin (new random origin per load) P3 stable origin
20,000 first initResp 1.5 s after reload, 0 lost 0.1 s
200,000 10.9 s, 1 lost 0.1 s

reloadExtension itself returns in ~20 ms; the delay is WebKit migrating extension data to the new origin before content scripts run. Roughly linear, ~50 µs per cache entry. Cookie seeding on 5,000 .test domains did not create website data records (still 2), so per-site record cost is untested. The same migration runs on every app launch in released builds, which links this to the P3 startup cohort. P3 (#7493) removes it in both cases. It explains reload_state=response_timed_out and extended post-reload silence; whether real caches are large enough for a full 5-minute silence is being checked at 600,000 entries.

  • 600,000 entries, released origin: first initResp 32.2 s after the reload, 5 navigations lost; still linear (~53 µs/entry). Page loads during the migration were also slower (one took 9 s), so the migration may delay browsing generally, not only CPM. A full 5-minute silence would need ~5–6M entries by extrapolation, so this explains post-reload silences of seconds to about a minute, not the whole deadline. The same blackout hits every app launch on released builds. With P3, 0.1 s at every size.

What survives the hang reload — code search and tests (27 Oct)

Telemetry: every confirmed permanent hang (iOS 0.30%, macOS 2.07% of DAU) had the M4 reload run and stay silent, so the broken state survives a full context replacement or the trigger repeats. Checked:

  • Native messaging (WebExtensionMessageRouter, AutoconsentWebExtensionMessageHandler): handlers keyed by extension identifier and re-registered on reload; each request is stateless. Not persistent.
  • Monitor reload bookkeeping: responses from measurements begun on the new generation count; harness reload scenarios closed their episodes. No bug found.
  • WebKit process reuse: WebProcessCache requires usesWebProcessCache, default false, and the app sets no process-pool options. Ruled out.
  • storage.session stall: CPM's refreshRemoteConfig awaits getFromSessionStorage("config") with no timeout, so a stall would hang every init. No stall in any screen log; only tabs.sendMessage on the busy-page profile. Not reproduced.
  • Network replaced while the background is unloaded (testNetworkProcessReplacedWhileBackgroundIsUnloadedKeepsWakesWorking, 30 s timing): 0 lost in 6 busy navigations and 4 idle wakes.
  • macOS tab URL: Tab.url(for:) returns content.urlForWebView, which follows webView.url one main-queue hop later; WebKit reads it live for sender.tab.url. Harness mode CPM_REPRO_MACOS_TAB_URL=1 reproduces the lag: 133 real-page visits, 0 lost, 0 monitor episodes. init arrives after the hop.
  • webExtensionPageWebView: only set for webkit-extension: pages and cleared on content change. Not relevant.

Reinterpretation: "idle wake" is a state at failure, not proven to be a cause. On desktop the background unloads after every 30 s idle, so a large share of all navigations, healthy ones included, run right after a wake, and failures from any cause land in this group at that base rate. That fits P1 changing nothing. Confirming it needs a healthy-navigation baseline (background state on successful measurements); none exists since #7384 was closed.

Remaining untested production ingredients: the real macOS Tab stack (DDG user scripts / content-blocking rule lists in tabs, multiple windows, pinned and hidden tabs, App Nap); needs the app-hosted harness in a VM.

Plan and corrections (27 Oct)

Corrections

  • P6 is not the feature flag. webExtensions is on in production, not toggled, and will be removed. The flag-cycle mechanism (also in the tabControllerMatchesContext doc comment) was an old unverified hypothesis. Established: a tab's web view gets the controller only at creation and only if webExtensionManager exists then (WKWebViewConfigurationExtensions → attachWebExtensions), and the manager is created once at launch and replaced only by the flag handler. Trigger unknown. Candidates: web views created before the launch-time manager; the flag evaluating off at launch for some installs (missing or stale cached config); a web view built outside the standard configuration path. Macos 1.213 on 26 Oct: 51 users with controller mismatch + no scripts, 19 permanent (22% of macOS permanent hangs). The fix depends on the trigger.
  • Error 6: the P2 retry is shipped and helped. iOS error-6 users fell from 1.3% to 0.5% of DAU, and 455 iOS users/day had a successful background_load_retry. Error-6 episodes have outcomes: iOS 86% recovered later, 9% permanent. It is still 25% of iOS stuck because (a) the retry restores the background but not the page whose init was lost, which nothing re-sends (late reload off); (b) the retry is skipped while the app is in the background, within the 60 s cooldown, or during M4; (c) when the failed background view is kept (H3, reproduced), the retry cannot complete. Which dominates is unknown: no retry attempted/result field in the episode.
  • Extension updated: yes, the outdated version is loaded. WebExtensionLifecycleCoordinator.loadAndSync runs loadInstalledExtensions() (the previous, now outdated version) before syncEmbeddedExtensions, which installs the bundled version under a new identifier, loads it alongside, and unloads the old one. Early pages after an app update run against the old version, then the context is swapped with a new origin and the migration blackout.

Next steps

Order superseded by Priorities — macOS browser-wide breakage first at the top of this note (28 Oct, two-day data); the items below remain the detailed list.

  1. Ship what's understood:
    • P3 release.
    • Extension updated: upgrade before load at launch; carry the origin across upgrades.
    • Error 6: restart the stale background on error 6 when the failed view is still alive, and re-init the page that lost init.
    • P6: find the trigger first (the three candidates above), then fix.
  2. Verify the macOS Network-recovery false-positive lead: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and died_client precedes 16% of macOS idle-wake failures. Review CPMNetworkProcessRecovery detection; count recovery pixels in the harness screens.
  3. Telemetry (one privacy review): events after the first failure (two-phase bg_events); a healthy-navigation baseline (background state on successful measurements); retry attempted/result; recovery applied in this episode; time since app resume.
  4. Targeted harness work only: Network-recovery restart racing a wake; iOS resume-time wake on the simulator (fix the Bundle.module skip); the app-hosted macOS harness in a VM if steps 2–3 leave the wake group unexplained.
  5. Mitigation experiment: keep the background warm while browsing (periodic loadBackgroundContent < 30 s), behind a flag; it is also a causal test for the wake group.
  6. Re-measure on full days of 7.243 / 1.213 and the first P3 builds.

Data export — 26–27 Oct (built 28 Oct)

One CSV: cpm-failure-groups-2026-10-26_27.csv (iOS 7.243.0 / macOS 1.213.0, both full days combined). Per platform: an ALL row, then each symptom group's total and its top 5 event patterns plus the remainder. Columns: users (failing install-days), % of platform failures, % of DAU, recovered, no response unconfirmed, permanent, % permanent, % of all permanent on the platform. Repeated idle cycles are collapsed as (dealloc>view>proxy_on)xN; [capped] marks timelines that hit the 12-event limit. Totals cross-checked: iOS 29,758 failing install-days, macOS 5,946.

DAU proxies: macOS m.mac.daily.active.user.d; iOS m.app.return.ios.{phone,tablet}.daily. Users = .daily leg (install-days when combined); episodes = .count leg. Group priority: error 6 → no background view → tab not connected (no scripts or controller mismatch) → extension updated ≤2 min → idle wake (view <30 s after a dealloc) → recent context load ≤2 min → other. End state: recovered = any response; unconfirmed = no response, no repeat, no reload; permanent = no response with a repeat or reload.

Two-day headline (% of DAU-days): iOS failing 9.1%, permanent 0.43%; macOS failing 28.5%, permanent 2.87%. Largest permanent source: iOS recent context load (40% of permanent), macOS idle wake (38%), then macOS P6 (22%, 39% of P6 users permanent).

Repro attempts and footprint comparison — 28 October 2026

Internal metrics; keep these figures out of public PRs. Data: ClickHouse via ~/Developer/grafana-data-tools reader, iOS 7.243.0, 26–27 Oct, web.extension.cpm.episode*.count, episodes with bg_load_error_after present.

What production shows

  • 3,114 error 6 episodes, 229 permanent (outcome=no_response_before_deadline, 7.4%).
  • Permanent, by error timing and whether the background view is alive at report time:
Group Episodes Share of permanent
error under_5s, view alive 172 75%
error 25s_to_35s, view not alive (repeated view,proxy_on,dealloc,error_6 cycles, about one per minute) 51 22%
error under_5s, view not alive 5 2%
error over_35s, view not alive 1 0.4%
  • The 172 look like the recovered ones. Same group, permanent vs recovered (2,797): interruption in the footprint 98% vs 96%, died_client 22% vs 22%, extension_updated 10% vs 6%, none of the three 3 vs 92. Permanent rate inside the group about 5.8%. These tokens do not explain why they stay broken.
  • No unresponsive token in any permanent group.
  • bg_view_alive cannot tell "WebKit kept the failed view" from "WebKit created a new view after the error". No field separates the two.

What I reproduced (iOS Simulator, iPhone 17, iOS 26.5, real CPM, real page)

  1. Suspension during a background load (SIGSTOP of the app and WebKit helpers for 45 s, with a real HOME; also with the start slowed to 8 s): the overdue 30 s unload timer fires at resume, error 6, view released, WebKit creates a new view, the page is answered. Footprint dealloc,view,proxy_on,interruption,error_6 matches the dominant production footprint, but it self-heals. Persistence of this family in production is NOT reproduced.
  2. Exception thrown at service worker start (injected, keyed on the extension origin): error 6 within 0.4 s, no underlying error, the failed view stays alive (WebKit _loadServiceWorker with success NO records the error and returns without unloading), the retry never completes, the page is unanswered, the second wake fails the same way, M4 heals after about 54 s. The footprint is not the dominant production one. PR #7730 (restart the context when the failed view is retained) fixes it in the simulator (restart 8 ms after the error, page answered after 0.6 s). The PR was closed before this was proven; branch alex/cpm-error6-restart kept. How often this state happens in production is unknown.
  3. Hung start (busy wait of 90 s in the worker): the 30 s timer unloads the view, error 6 about 35 s after creation, view released, the retry hangs again, M4 heals after about 54 s. Footprint matches the 25–35 s loop group. M4 healed only because it re-extracts the clean extension; real persistence across M4 is NOT reproduced. The pixel the app logged for this run does not equal production (bg_events=load@2m, bg_ready_after=pending, bg_view_age=8s_to_30s), because the snapshot is taken at the first failed navigation, before the first loop ends.
  4. Extension update during a background load (2026.10.23 over 2026.10.99): error 6 on the superseded context only, the new context answers.
  5. Separate finding: after any app relaunch the embedded worker stays silent in the simulator (stale registration: importRegistrations failed to retrieve main script, then Found directly reusable registration), no error 6. Whether a device does this is unknown.

Reason for a failed start can be read

The background page loads with navigator.serviceWorker.register(...); its rejection carries the WebKit error text (Error: <message>), obtainable with a WKUserScript and WKScriptMessageHandler on the background web view delivered by didCreateBackgroundWebView (before the page loads). Message only, no file or line. The extension's own extensionException messages are not handled by the app (unknown method). Verified once with an injected exception.

Open

  • Why 172 permanent episodes of the interruption family do not recover, when 2,797 identical-looking ones do.
  • What makes a start hang longer than 30 s (candidates from earlier notes: origin-change data migration, host main thread stall, process throttling), and why that survives M4.
  • How often the failed view is retained.
  • Pixel for the restart (none exists yet).

Idle wake investigation — 28 October 2026

Re-cut of the post-release data (macOS 1.213.x, 26–27 Oct, matured since the sections above, so shares differ a little; per-episode web.extension.cpm.episode.count) and of the old contract (1.211.x / 1.210, 18–20 Oct, debug.web.extension.cpm.messaging.stuck.<reason>.count, one report per tab, confirmed by a second timeout). Not like-for-like: compare shapes, not percentages. OS comes from the os_version column of metrics.pixels, not a pixel parameter. Wake = the token rule view@30s plus an earlier dealloc (excluding no-view, no-scripts and error 6).

Was it there before Network recovery and P1? Yes

  • 1.211.1 shipped neither fix and already had wake at ≈33% of stuck; 1.213 is the same.
  • died_client in bg_events means the app killed the background process. The only shipped code path that does it to a background view is CPMNetworkProcessRecovery (_killWebContentProcessAndResetState). It is 0 of 65,374 reports in 1.211 and 3.6% of 1.213 episodes (5.1% of wake ones). It is more common among wake episodes that got an answer (9.2%) than among those that did not (2.3%). So recovery is a small new source of delays, not the cause. The "16%" in the priorities section did not reproduce with this cut (probably another denominator).
  • 1.211 network_restarted is depleted in wake (12% vs 23% outside wake; 1.210: 14% vs 21%), without any recovery shipped.

What the wake group looks like

  • Everything the pixel can see is healthy (7,787 episodes): context_loaded, bg_view_alive, bg_process_responsive, handler_registered true in 100%; bg_process_alive 99.95%; bg_ready_after=under_1s 98.8%; scripts present and controller matching 93% (3.8% P6, 2.9% unknown); error 6 0.3%; interrupted 0.03%. Outside wake the same fields do find things (no view 5.8%, no scripts 7.9%).
  • bg_view_age 4s–8s in 66% (view created at the navigation), tabs_affected=1 in 78%, no response 57.6%, reload_state none 70% / loaded 19% / response_timed_out 11%, reason=other 90.5%.
  • The timeline tail of permanent cross-tab wake (1,134) is the same as for answered ones (dealloc>view>proxy_on twice, 71% vs 67%). It does not separate them.
  • No OS signal. Wake share 43–57% on every macOS 15.x / 26.x / 27.x; 27.2 is inside the range (wake 47.7%, no response 42.7%, unmarked 53.7%). Episodes per loaded install-day ≈0.7–0.9, 27.2 at 1.07.
  • Unmarked core. Wake no-response episodes with no base_url_changed, extension_updated, load, died_*, error_*, interruption or unresponsive in the 12-event window: 54% in 1.213 (2,319 of 4,279), 52% in 1.211 (15,089 of 28,849; also no network restart, app update or memory pressure). It is uniform by hour, OS, dmg/App Store and tab count. In 1.211 it is 94% app_launch_age >15 min and 86% bg_view_creations >5: long sessions with many 30 s cycles, not startup. Nothing in the available parameters separates it.
  • extension_updated is depleted in wake in both versions (2.8% vs 9.4%, 15.1% vs 30.7%); recent updates are not a trigger. base_url_changed was enriched in 1.213 permanent cross-tab wake (25% vs 13% of answered; tail 14.8% vs 5.8%) but is depleted in 1.211 wake (13% vs 28%): the lead does not replicate. Both timeline tokens fall out of the 255-character window after a few idle cycles, so older updates are invisible.
  • Context error 7 is not a lead. The old context_errors showed WKWebExtensionContextErrorDomain:7 (= ScriptExecutionError, no underlying pair; WebKit also raises it for console.error) in 27% of 1.211 wake reports vs 17% outside, and 22% vs 16% on 1.210. The rate rises with context age (0 / 6 / 20 / 22 / 26% for bg_view_creations 0 / 1 / 2 / 3–5 / >5) and wake differs by ≤5 points inside a bucket. The shipped contract keeps other context errors locally and sends only error 6.
  • Extension load-error pixels are negligible: load.error, embedded.not.loaded, reload.failed are ≈0.1% of loaded installs or less in both versions (1.213: 9 / 8 / 5 events on 18,299 install-days; 1.211 one day: 17 / 101 / 72 on 102,348).
  • Session restoration is 4.5% of macOS episodes (16% no response) and 21% of iOS episodes (9.6% no response); only 0.6% of the wake group. It mostly recovers; reason labels only a first failing navigation that is itself restored.

History of reproductions vs this signature

  • 8 Oct: a 5 s busy-wait in the worker at wake reproduced the coarse wake signature of that time (responsive=true, no errors, dealloc/view/proxy_on@30s) but recovered about 1 s later; 12 s plus a next navigation gave stuck and M4. Production bg_ready_after is under 1 s, so a slow worker start does not explain the permanent group.
  • Reproduced but different: kill -9 of the background (all tabs, self-heals ~1 min), Network crash (fixed by recovery), stale failed view H3 (iOS pattern B), origin-changing M4 blackout (seconds to ~32 s), P6 (single tab).
  • Not reproduced: plain idle wake (0 lost of 1,500 and 1,333 wake navigations, with ad-blocking and real pages). Nobody has reproduced the permanent, cross-tab, healthy-looking, ready <1 s case that survives M4.

What the monitor measures: http(s) only, but error pages slip through

  • isEligibleForCPMMessagingHealthMeasurement accepts only http/https (CPMMessagingHealthMonitor.swift:1094). about:blank, duck://settings, duck://bookmarks and duck://player are not measured (harness test testNonHTTPPagesAreNotMeasured: no episodes). Events come only from tabs (PrivacyDashboardTabExtension, TabViewController); PIR web views were not checked.
  • The embedded extension manifest (duckduckgo-privacy-extension/browsers/embedded/manifest.json, 2026.10.14) matches *://*/* in all frames with no exclude_matches; the app grants every requested pattern (WebExtensionLoader.swift:271). There are no per-site exclusions. The content script is plain autoconsent; a main-frame init always ends in a dashboard refresh.
  • Error pages are measured as CPM documents and reported as failures (reproduced in the harness). macOS shows a failed load with loadAlternateHTML(html, baseURL: duck://error, forUnreachableURL: failedURL) (Tab.swift:1620). Then webView.url is the failed http(s) URL, so the filter passes; the navigation action URL for alternate loads is also webView.url (NavigationAction.swift:183); PrivacyDashboardTabExtension does not skip .alternateHtmlLoad (PageSignalsTabExtension does). In CPMReproErrorPageTests.testErrorPageIsNotReportedAsCPMFailure (worktree cpm-remaining-failures-repro, uncommitted; CPM_REPRO=1 swift test --filter CPMReproErrorPageTests): navigation failed, then the alternate load finished with the failed http URL; no init reached the background and no content-script mark appeared (the script does not run on the error document); 4 s later the monitor counted a failure, and the next healthy navigation closed an other / failed_tab_response / tabs=1 episode. The harness feeds the monitor webView.url as iOS does; the macOS Navigation.url path was read in code, not run in the app. iOS error pages were not tested.
  • Hypothesis, not verified: pages that fail to load are shown on every tab after wake from sleep or a network drop. Each would count as a CPM failure right after an idle unload (fresh dealloc>view>proxy_on, background ready in under 1 s, healthy diagnostics), in several tabs (cross-tab) and with no answer until a normal navigation (permanent-looking, and M4 would not help). A same-tab second timeout also triggers M4 on a healthy extension (see was_stuck note above), which then adds the origin-change blackout. The pixel has no URL, so data cannot separate this; the check has to be an app run (debug build, Wi-Fi off, several tabs, wake) with the monitor log, or filtering .alternateHtmlLoad / duck://error and comparing wake rates.

Notes

  • The "P3" lines above disagree: priorities say the baseURL fix is released; "What shipped" says P3 (#7493) is on main only and not in 1.213. Re-check before relying on either.
  • Scratch SQL runner and queries are in the session scratchpad; the harness test is not committed or pushed.

Addendum 28 Oct — error pages produce the wake signature (replaces the first version of this addendum)

  • Pixel: m.mac.errorpageshown.other (+ .daily, pixel_id='m', params d/e = error domain/code, pixelSource; also …webkittermination), fired in Tab.swift when the error page is shown. macOS 1.213, 26–27 Oct: 7,511 shows, 2,284 install-days (≈3.3 per affected install-day). Codes: DNS -1003 29%, TLS -1200/-1202… 26%, -1005 10%, offline -1009 only 1.6%. Any error page counts, not only "no network"; the sleep / offline story from the first version is not needed.
  • The wake happens at the navigation, the error comes later. The navigation gate asks initialLoadWaiter, which calls manager.prepareEmbeddedBackgroundForNavigation() → context.loadBackgroundContent at policy time of every main-frame http(s) navigation (WebExtensionLifecycleCoordinator.swift ~80–105, WebExtensionManager.swift ~993; macOS caller PrivacyDashboardTabExtension.decidePolicy). So a navigation to a dead host after an idle unload recreates the background first and fails afterwards.
  • Reproduced in the harness (CPMReproErrorPageTests.testErrorPageAfterIdleUnloadDoesNotWakeOrFail, production 30 s unload, real navigation to an unresolvable host, then loadAlternateHTML(… forUnreachableURL:) as the app does; worktree cpm-remaining-failures-repro, uncommitted): the background view was created ~2 ms after the policy decision, the request failed 0.15 s later with the view present, the error document sent no init, the monitor counted the 4 s timeout and closed other / failed_tab_response / tabs=1 on the next healthy navigation. That is the wake signature: dealloc>view>proxy_on with a view younger than 30 s, bg_ready_after under 1 s, healthy diagnostics, reason=other, one tab. The snapshot is taken ≥4 s after the error page finishes, so bg_view_age lands in 4s_to_8s (66% of wake) or 8s_to_30s (33%), as in the data. (A first test without the real navigation did not wake the background.) The harness feeds the monitor webView.url; in the app, PageSignalsTabExtension skips .alternateHtmlLoad in navigationDidFinish, which shows that handler also runs for error pages, and the alternate-load action URL is webView.url (NavigationAction.swift:183).
  • Magnitude, install-days (separate pixels, not joined by install): error-page .daily on CPM-capable macOS (15.4+): 1.213 10.3% of loaded.daily (1,891 / 18,299); 1.211.1 10.2% (10,507 / 102,806, 19 Oct). Wake-signature install-days: 1.213 13.3% (2,443 episode.daily rows), 1.211.1 14.6% (15,058 first-failure .daily rows; 5.5% for confirmed stuck). Error pages per affected install-day ≈3.3, wake episodes per wake install-day ≈3.2. This is a ceiling, not a proof: only an error page after an idle unload carries the wake tag (a warm background gives other), and some error pages may not take the alternate-load path.
  • Why it looks permanent, cross-tab and immune to M4: a second timeout in a tab runs M4, which reloads a healthy extension (new origin, 1.5–32 s blackout). Error pages in two or more tabs with no healthy navigation within 5 minutes end as no_response_before_deadline, was_stuck=true, reload_state=loaded/response_timed_out, with healthy diagnostics. That matches the cross-tab permanent wake group without any CPM breakage. Not checked: how many permanent cross-tab episodes are such loops.
  • Weak evidence against, kept for the record: per-OS correlation of error pages per install with wake episodes per install is r = 0.02 (n = 9); hourly r = 0.88 for both wake and non-wake (rollout trend, uninformative).
  • Unverified: the running app (the scenario with log stream --level debug --predicate 'subsystem == "WebExtensions" AND eventMessage CONTAINS "CPM Health Monitor"': three tabs idle >30 s, load a dead host in each, expect timeoutElapsed without Dashboard response attributed); iOS error pages; the share of wake episodes that are error pages (the episode pixel has no URL or error code).
  • Decisive options: (a) macOS build where the monitor ignores error documents (.alternateHtmlLoad / failed provisional navigations) and M4 never fires for them; compare wake share and permanent rate with the current build; (b) a flag parameter on the episode pixel (needs the privacy review you wanted to avoid).
  • Filter implemented (28 Oct, uncommitted, not pushed): worktree apple-browsers.git/cpm-error-page-filter, branch alex/cpm-error-page-filter from 8023e4cb5c. New CPMNavigationKind.errorPage; CPMMessagingHealthMonitor does not measure a commit/finish of a tab whose current navigation is .errorPage (commitNavigation, finishNavigation), so no timeout, no episode and no M4 for it; macOS Navigation.cpmMessagingNavigationKind returns .errorPage for .alternateHtmlLoad (PrivacyDashboardTabExtension.swift). Tests: two monitor unit tests (testErrorPageIsNotMeasured, testNavigationAfterErrorPageIsMeasured) and the harness CPMReproErrorPageTests (3 tests, CPM_REPRO=1), all green; the same harness tests failed (other / failed_tab_response / tabs=1) before the change, and with it the background still wakes at the navigation but no episode is counted. SwiftLint strict: 0 violations. Not done: iOS error pages (no loadAlternateHTML path found on iOS; not checked how they are shown), running the real app, comparing wake share between builds.
  • macOS app build with the filter: xcodebuild -scheme "macOS Browser" -configuration Debug CODE_SIGNING_ALLOWED=NO succeeded (the change compiles in the app target). Product: /private/tmp/claude-501/-Users-admin-DuckDuckGo-apple-browsers-git/2490a862-c422-4358-b940-1f30df9cbb83/scratchpad/app-dd/Build/Products/Debug/DuckDuckGo.app (scratchpad of that session; may be cleaned). The app was not launched: no window run on the host Mac. The real-app check is still open: three tabs idle >30 s, a dead host in each, log stream for CPM Health Monitor; with the filter there should be no timeoutElapsed for the error pages.
  • Integration test (28–29 Oct): macOS/IntegrationTests/Tab/CPMErrorPageMonitoringTests.swift (added to both Integration Tests targets in the pbxproj). Real Tab + WKWebView + TestSchemeHandler failing with NSURLErrorCannotFindHost; the tab gets a stub WebExtensionManaging whose health monitor records events. Asserts the monitor sees started:other, failed, then started:error_page, and the error page finish for the failed URL arrives after it (the real macOS path: Navigation alternate load → .errorPage). build-for-testing of scheme macOS Integration Tests succeeded and SwiftLint strict is clean; not run locally (integration tests are left to CI, no host windows). Without the macOS mapping the second start would be started:other and the test would fail; that was not observed, only reasoned.
  • Integration test run locally (29 Oct), correction of the line above: with the signed Debug build (dev identity, entitlements present), macOS Integration Tests → CPMErrorPageMonitoringTests passes (1 test, 0 failures). Negative control with the macOS mapping removed from PrivacyDashboardTabExtension.cpmMessagingNavigationKind: the same test fails in the real app stack, with the monitor receiving ["started:other", "failed", "started:other", "committed:https://testhost.com/", "finished:https://testhost.com/"], i.e. the error page is reported as an ordinary navigation of the failed https URL (this is the false positive, observed in the app's own Tab/Navigation path, not only in the harness). The mapping was restored afterwards. Note for running locally: do not build with CODE_SIGNING_ALLOWED=NO; the host app then lacks the keychain entitlement (-34018) and macOS asks for keychain access.
  • Corner cases checked (29 Oct):
    • Integration tests (real Tab, macOS Integration Tests, run locally with the signed Debug build), all pass: ordinary load error (-1003), SSL error (NSURLErrorServerCertificateUntrusted), malicious-site warning (MaliciousSiteError, via the duck://error redirect). In each, after the failed load the page starts as .errorPage, so the monitor never measures it.
    • Harness (CPMReproCornerCaseTests, real WebKit, real extension, loopback HTTP), all pass: HTTP Basic auth with accepted credentials (CPM answers) and with a cancelled challenge (401 page, no episode); client redirects at 0.1 s, 2 s, meta refresh and a redirect after the 4 s grace (CPM answers on the target, no episode); PDF, text/plain and JSON documents (CPM answers on all three, content scripts do run on PDF); a Fire-window tab with a non-persistent data store (CPM answers). A 6 s JS timer redirect never fired in a window-less web view (timer throttling), which is a test artefact.
    • Not checked: downloads, http→https upgrade redirects, background or minimised windows, mass tab opening, popups from about:blank, closing a tab mid-measurement, iOS error pages, the macOS basic-auth dialog UI itself (by code reading nothing is committed while it is open, and cancel leads to no error page).