Files
obsidian-vault/work/projects/cpm-post-release-analysis-2026-10-27.md
T

32 KiB
Raw Blame History

title, date, tags
title date tags
CPM — first post-release data (iOS 7.243.0 / macOS 1.213.0) 2026-10-27
work
cpm
analysis

CPM: first post-release data, 26 October 2026

Internal metrics; keep these figures out of public PRs. Previous analysis: cpm-metrics-causes-and-mitigations-2026-10-16, cpm-status-2026-10-15 (§3–5, 12–14 October cut).

What shipped

  • iOS 7.243.0 (tag 26 Oct 19:23 UTC), macOS 1.213.0 (tag 26 Oct 16:23 UTC).
  • Both: episode telemetry #7376/#7484, episode tab scope #7520/#7521, P1/P2/P4 recovery #7379/#7536, Network-process recovery #7459/#7531.
  • macOS only: P5 shared-pinned window close #7473, remote/debug disable #7498.
  • Not shipped: P3 stable origin #7493 (on main only). No P6 fix exists in either release: no PR implements the controller rebind.
  • Shipped episode contract is the rolled-back variant: was_stuck (bool), single-phase bg_events (no ;), plus tabs_affected, repeat_failures, interrupted. No bg_view_creations, context_errors or network_restarted. Old cohort SQL needs adapting (below).
  • Monitor timing: 4 s response grace; suspended time no longer counts (8 s after resume, 30 s extension-load wait, 5 min deadline).

How to query

  • New pixels have no m_/debug_ prefix: ClickHouse pixel_id = 'web', pixel LIKE 'web.extension.cpm.%' (iOS adds .ios.phone|tablet). Querying pixel_id='m' without an agent filter and a one-day window exceeds the 7B row limit.
  • Denominator: pixel_id='m', m.web.extension.loaded.daily% / m.mac.web.extension.loaded.daily, split by appVersion.
  • Old-contract attribution was remotely switched off from 21 Oct (old versions keep sending bare debug.web.extension.cpm.* without parameters). The last fully attributed old-version window is 18–20 Oct, iOS 7.241 / macOS 1.211.x.
  • Data lag: on 27 Oct the table ended at ~01:10 UTC; date is US-Eastern. 26 Oct is the only post-release day and it is partial for new adopters (exposure starts at update time).
  • Runner: ~/Developer/grafana-data-tools reader; ad-hoc SQL and an ARRAY JOIN extractURLParameters(request) parameter dump were used. extractURLParameter needs a constant key.

Rates per extension-loaded install-day (26 Oct)

iOS 7.242 (old) iOS 7.243 (new) macOS 1.212 (old) macOS 1.213 (new)
loaded.daily 4,257,839 52,979 93,030 2,566
old init_failed.daily (sum of reasons) 21.3% — 36.8% —
old stuck.daily 3.3% — 12.5% —
episode.daily — 7.05% (3,735) — 30.0% (771)
episode with reload_state≠none (hang confirmed, ≈ old stuck), count leg — 23% of episodes — 26% of episodes
was_stuck=true (second distinct tab), daily leg — 8.1% — 11.2%
network_process_recovery_applied.daily — 432 (0.8%) — 232 (9.0%)

Not like-for-like: the episode unit is one report per episode, the old init number sums four reasons, suspended time is now excluded, and day-1 adopters have short exposure. Directionally, iOS fell ~3× (21% → 7%); macOS barely moved (37% → 30%).

was_stuck only becomes true when a different tab fails; M4 reload already runs on a same-tab second timeout. Use reload_state≠none as the old "stuck" proxy.

Cohorts (count leg, adapted classification)

Priority: error 6 (bg_load_error_after present or error_ token) → no view (bg_view_alive=false) → tab not connected (tab_has_ext_scripts=false) → extension_updated ≤2 min → wake (bg_view_age < 30 s and a dealloc token) → recent load/base_url_changed ≤2 min → other. There is no Network-replaced cohort any more.

Cohort iOS share / of stuck / no-response macOS share / of stuck / no-response
error 6 7.5% / 24.7% / 39% 0.6% / 1.7% / 44%
no view (P3) 7.7% / 6.8% / 7% 2.9% / 0.9% / 25%
tab not connected (P6) — 5.8% / 1.3% / 70%
extension_updated ≤2 m 3.9% / 6.2% / 37% 14.6% / 11.7% / 28%
wake, view <30 s 22.8% / 22.0% / 9% 43.9% / 58.4% / 48%
recent context load 22.8% / 17.2% / 12% 11.9% / 6.9% / 25%
other 35.3% / 23.2% / 5% 20.4% / 19.0% / 27%

Baseline with last week's exact SQL, 18–20 Oct (init → stuck shares). iOS 7.241: error6 6.4→22.0, no view 47.5→11.5, network 16.5→38.2, wake 7.7→9.2, recent load 16.7→6.6, other 5.2→12.5. macOS 1.211.x: error6 1.2→2.2, no view 3.6→1.9, tab not connected 5.9→8.7, network 13.2→25.8, wake 33.2→33.3, recent load 19.4→6.3, other 23.6→21.6. This matches the 12–14 Oct cut, so the baseline was stable.

Per-install rough rates (cohort share × failure rate), old init → new episode:

  • iOS: no view 10% → 0.5%, recent load 3.5% → 1.6%, error 6 1.3% → 0.5%; the network cohort is absorbed. Wake stays at 1.6% → 1.6%. Other rises 1.1% → 2.5%; it now absorbs former network cases and has no network field.
  • macOS: wake ≈12% → ≈13%. Unchanged.

Findings

  1. P1 (wake) fix had no measurable effect on either platform. In the wake cohort bg_ready_after=under_1s dominates (macOS 93% overall), so the background is not slow. The background_wake pixel fires only for ≥1 s callbacks (macOS 89/day vs ~670 wake episodes). The remaining failure is message/response loss after a fast idle wake, not readiness. On macOS 48% of wake episodes see no response for 5 min; tabs=3+ → 68%. Typical tail: …dealloc@15m,view@15m,proxy_on@15m,dealloc@5m,view@30s,proxy_on@30s, bg_view_age=4s_to_8s (view created around the navigation). The pattern appears on every macOS version, 15.6–27.2.
  2. iOS error 6 changed shape. All attributed error 6 is now bg_load_error_after=under_5s; the 30 s timer bucket is gone.
    • A (ready=failed, app_state active): dealloc → view → proxy_on → interruption → error_6, often after died_client/died_crash/died_crash_limit. The background load spans an app suspension and fails at resume.
    • B (ready=pending): an older error 6, then a new view whose load is still pending 8–30 s later.
    • The P2 retry fires only while foregrounded with a 60 s cooldown (277/day). Foreground-resume recovery is the missing piece.
  3. macOS P6 is untreated: 5.8% of episodes, 70% never answered. iOS now shows tab_has_ext_scripts=false in 5.3% of episodes, concentrated on iOS 26.7 (16%) and 27.0 (8%), with 0 on 27.2. Previously iOS was ≈0.
  4. macOS Network replacement is frequent: the applied pixel is on 9% of 1.213 install-days. With recovery applied it no longer appears as a cohort, but other stays at 20%.
  5. After M4 on macOS: reload_state=loaded → 46% no response by deadline; response_timed_out → 57%. The reload often does not restore messaging, or no qualifying navigation follows.
  6. Recovery pixels only ever report background_ready. response_received needs late-document reload, which is default OFF.
  7. Day-1 effect: macOS extension_updated ≤2m is 14.6% (embedded upgrade on update). Re-check after 2–3 full days.

Next measurements

  • Repeat for 27–29 Oct (full days, larger adoption). Track the wake cohort and macOS no-response by OS.
  • bg_events per-phase budget and network_replaced_since_view remain missing (see cpm-status-2026-10-15#6. Телеметрия).

Repro targets

Issue Evidence Hypothesis to test Candidate fix
R1 wake message loss macOS 44% eps / 58% stuck; iOS 23%; ready <1 s message to a woken non-persistent SW is acknowledged with no reply: (a) processes(RuntimeOnMessage) empty when _loadServiceWorker completion runs before listener IPC; (b) listener set loses RuntimeOnMessage so wakeUpBackgroundContentIfNecessaryToFireEvents resolves {} without waking; (c) background unloaded while a reply is pending keep the background warm while CPM pages are active, or re-send init on empty reply (native/JS handshake)
R2 iOS error 6 across suspension 7.5% eps / 25% stuck, all under 5 s, interruption between view and error WebKit load timer spans suspension; nothing retries on resume on foreground resume, retry loadBackgroundContent when the last load failed or is pending
R3 P6 controller mismatch macOS 5.8%, 70% unanswered known main-flag manager replacement rebind/recreate tab web views to the current controller

WebKit source pointers (~/Downloads/WebKit_sparse): WebExtensionContext.cpp wakeUpBackgroundContentIfNecessaryToFireEvents (drops events when the listener set lacks the type after the first load), scheduleBackgroundContentToUnload (30 s, reset only on dispatch), WebExtensionContextCocoa.mm loadBackgroundWebView / performTasksAfterBackgroundContentLoads, WebExtensionContextAPIRuntimeCocoa.mm runtimeSendMessage (empty processes() → {} reply). 324743 (9bcf5840ef) holds runtime messages until SW activation; it is not in shipped OS builds.

Repro log — 27 October 2026

Worktree apple-browsers.git/cpm-remaining-failures-repro, branch alex/cpm-remaining-failures-repro (from origin/main 8023e4cb5c, which includes the released P1/P2/P4/Network code plus P3). The package real-WebKit harness was imported from origin/alex/cpm-repro-harness. It compiles unchanged on current main, and its CI-tier tests pass. Local tier: CPM_REPRO=1 swift test --filter <Suite> from SharedPackages/WebExtensions. System WebKit is the host macOS; no app windows are opened.

  • R1 idle wake — not reproduced organically. CPMReproIdleWakeTests.testRepeatedIdleWakeCyclesDeliverInit: 3 tabs, 8 idle-unload cycles (WebKit unloads 32.0 s after the last answered navigation), single and double-tab wakes, 0 of 11 lost. Every wake answered in 2–5 ms after didFinish. testNavigationAtIdleUnloadBoundaryDeliversInit: offsets −0.6…+0.2 s around the calibrated unload, 0 of 7 lost. A plain idle wake on an in-memory fixture page is not the production mechanism. The missing ingredient is still unknown: real page loads, a long-lived data store with abandoned registrations, Network replacement, or listener-set loss.
  • P3 (#7493) only in internal builds (macOS 1.214.0-860, iOS 7.244.0-x). On 26 Oct: macOS 1.214 had 20 episodes / 9 daily of ~36 loaded installs, with 0 no-view and 0 error 6; iOS 7.244 had none. That is too small to evaluate. Relevance: released builds get a new random extension origin on every context load (launch and M4 reload), and WebKit migrates data before injecting. This may explain the iOS no-view cohort and the macOS "reload loaded → still no response" share (46%). Re-check when 7.244 reaches users.
  • R2 suspended wake — macOS analogue does not stay broken. CPMReproSuspendedWakeTests: idle unload, then a navigation wakes the background, and the new background process is SIGSTOPped for 45 s while another tab navigates every 10 s. Two navigations timed out (+9 s, +14 s). Then the background was replaced by a new process (same view slot, new PID) and every later navigation answered in 1–14 ms, including 10 after thaw. No error 6 was recorded on the live context. On macOS the shipped recovery restores messaging within ~14 s of the fault. The harness cannot reproduce iOS suspension, which pauses the UI process's own timers. Likely production sequence: the overdue 30 s unload timer fires at resume, closes the loading view, and records error 6 "under 5 s" (pattern A). Pattern B (new view pending 8–30 s) needs a device or simulator test.
  • WebKit hypothesis not hit yet: an SW load failure that keeps the background view (loadBackgroundWebView completion !success without unload). Later wakes would only queue, and every dispatch restarts the 30 s unload timer, so the public loadBackgroundContent/P2 retry cannot escape. A candidate mitigation is to restart the background web content process when error 6 is recorded while a background view still exists. Not implemented, because the state has not been reproduced.

Treatment pixels, P3 ceiling and permanent hangs — 27 October 2026

Treatment pixels received (26 Oct, count leg, iOS 7.243.0 / macOS 1.213.0)

Fix Pixel (pixel_id='web') iOS macOS
P1 wake web_extension_cpm_background_wake 2,026 89
P2 error-6 retry web_extension_cpm_background_load_retry 277 12
Install / upgrade readiness …_install_readiness / …_upgrade_readiness 3 / 24 0 / 1
Network recovery web_extension_cpm_network_process_recovery_applied 476 277
P4 detection (flag-independent) web_extension_cpm_startup_response_missing 450 (daily 327) 20 (daily 12)
Late page reload web_extension_cpm_late_document_reload 0 0
  • Every recovery pixel only reports outcome=background_ready, meaning the fix was applied and the background came up. reload_started / response_received exist only for the late page reload, which is default OFF, so it has zero rows on every version including internal builds. No pixel shows whether the page got its answer after a treatment; the effect has to be read from episode rates.
  • background_wake fires only when the wake callback took ≥1 s and a navigation waited, so it is not a count of all wakes.
  • Detection gap for the late reload. startup_response_missing fires with the flag off but requires startup.isFirstLoad: the first background load of that context in the session (launch, install, upgrade). attemptRecovery (the reload) has no isFirstLoad condition and would also run after later wakes. With the flag off, wake-time cases the reload would treat are not counted anywhere; 327 / 12 is a lower bound of what it could fix. Both detection and reload only arm when the page commits while the startup is still pending. In the wake cohort the background is ready in under 1 s, so many wake failures are outside both. Counting the would-be-reload wake cases needs a pixel change: drop isFirstLoad from detection or add a parameter. Not done.
  • Upper bound on the late reload: ~0.6% of iOS / 0.5% of macOS extension-loaded installs; ~9% of iOS / ~2% of macOS installs with a failure episode.

P3 (baseURL) ceiling on the new data

  • The earlier "≈half" was the pre-release no-view share: 48% of iOS initial failures, but only 11% of stuck, mostly self-healing.
  • On 7.243.0 without P3, no-view is already 7.7% of iOS episodes / 6.8% of stuck (≈10% → 0.5% of loaded installs); macOS 2.9% / 0.9%. The drop comes from the new measurement (suspended time excluded, one report per episode) and the shipped readiness gates.
  • Direct P3 ceiling ≈ 8% of iOS episodes, less on macOS. A possible indirect effect: released builds get a new random origin on every M4 reload, and 46% of macOS reload-loaded episodes still got no response. That link is unproven; internal 1.214 / 7.244 data is too thin.

Users affected by permanent hangs (26 Oct)

Denominators were refreshed after more 26 Oct data arrived: extension-loaded installs iOS 69,088 / macOS 3,045. "Permanent" = outcome=no_response_before_deadline (no qualifying response for 5 min). Single = tabs_affected=1, cross = 2/3+. Confirmed = repeat_failures≠0 or reload_state≠none. Daily leg = affected installs.

Installs/day iOS macOS
Any failure episode 4,655 (6.7%) 876 (28.8%)
Single-tab, no response 354 (0.51%) 217 (7.1%)
↳ confirmed 76 (0.11%) 25 (0.82%)
Cross-tab, no response 129 (0.19%) 58 (1.9%)
↳ confirmed 123 (0.18%) 49 (1.6%)
  • Cross-tab is the reliable permanent-hang figure: ~0.2% of iOS and 1.6–1.9% of macOS users per day; 95% / 84% confirmed.
  • Unconfirmed single-tab rows (iOS 278, macOS 192) may be stuck tabs or users who didn't navigate within 5 min.
  • Strict permanent (confirmed single + cross): ~0.3% iOS, ~2.4% macOS users/day. Upper bound including unconfirmed single-tab: 0.7% / 9%.
  • Lower bounds for users: the daily leg describes only each install's first completed episode of the day, and episodes lost when the process ends before the deadline are not reported. Count leg (episodes per install-day), confirmed cross-tab: iOS 0.41%, macOS 5.2%; confirmed single-tab: iOS 0.26%, macOS 2.3%.
  • One partial day; macOS has ~3k installs on 1.213.0. Repeat on 27–29 Oct full days.

Repro statistics note

18 wake navigations cannot rule out a rare loss. At p=8% a clean run happens 22% of the time. The per-wake production rate is likely ≪1% (many wakes per install-day), which needs ~1,000 wakes for 95% power. CPMReproIdleWakeTests now has CPM_REPRO_FAST_UNLOAD=1: it calls +[WKWebView _setApplicationBundleIdentifier:] with com.apple.WebKit.TestWebKitAPI, so WebKit unloads an idle background after 3 s instead of 30 s (~5 s per cycle). It can change other test-runner-only WebKit behavior, so confirm any hit with the 30 s delay. A 1,000-cycle run (~1,500 wake navigations) was started on 27 Oct.

  • Fast idle-wake run, 27 Oct: 0 lost of 1,500 wake navigations in 1,000 cycles (CPM_REPRO_FAST_UNLOAD=1, 3 tabs, single and double-tab wakes, 85 min). With zero failures the 95% upper bound on the per-wake loss rate is ≈0.2% (3/1500). A bare idle-unload → wake race on a fixture page is not the main source of the wake cohort; an extra ingredient is needed (hypotheses H1–H5 below).

Hypotheses for the remaining wake hangs (27 Oct)

The embedded extension (background-embedded.js, 2026.10.23) registers runtime.onMessage synchronously (MessageRouter). CPM init = await remoteConfigJson → native isAutoconsentSettingEnabled → native isFeatureEnabled → refreshDashboardState. Every native request has a 20 s timeout, with config_unavailable / setting_disabled fallbacks that still refresh. A 5-minute silence therefore means init never reached the worker, or the refresh was not attributed.

  1. H1 — worker without extension bindings after Network-process recovery (WebKit 324754). macOS recovery is applied on 9% of installs/day. A worker without browser.runtime throws at top level: alive, responsive, "ready", never answers, all tabs, M4 may repeat it. Test: Network crash → recovery restart → probe browser.runtime in the worker → N wake cycles. Fix: verify with a ping after restart; restart again or reload if it fails.
  2. H2 — listener metadata loss. wakeUpBackgroundContentIfNecessaryToFireEvents returns an empty reply without waking when the saved listener set lacks RuntimeOnMessage. The set is saved after every load, so one load where the worker threw before registering poisons it. Test: a prelude knob that throws or skips onMessage on one load only. Fix: treat an empty sendMessage reply as a failure → restart, plus re-send init.
  3. H3 — stale failed background view (iOS error 6, pattern B). _loadServiceWorker failure keeps the view, so wakes only queue; activity keeps restarting the 30 s unload timer; P2 retry uses the same path. Test: fail the worker load without closing the view, with navigations every 10 s. Fix: on error 6 with a live background view, restart its WebContent process, then retry.
  4. H4 — refresh not attributed. The monitor needs an exact scheme/host/port/path/query match with the committed URL. A cold wake answers later, giving replaceState / client-side redirects time to change sender.tab.url. other_tab_response is 37–53% of outcomes. Test: a fixture with replaceState 50–300 ms after load, cold vs warm. Fix: attribute by extension tab ID and treat same-document URL changes as the same document.
  5. H5 — slow cold-wake init. Three native round-trips plus session storage; explains late answers (response_after ≥30 s in 20–35% of answered episodes), not 5-minute hangs. Test: prelude delay on sendNativeMessage 1–3 after a wake. Fix: answer init from cached config/settings first.

H1/H2 tests — results (27 Oct)

New CPMReproBrokenWorkerTests and harness changes (uncommitted):

  • Options.isNetworkRecoveryEnabled, on by default as in the app; off in CPMReproWebKitTests.unprotected.
  • Prelude startFaults breaks one worker start inside a wall-clock window: .skipOnMessage (script runs, no runtime.onMessage listener) or .throw (top-level throw before the extension script).
  • CPM_REPRO_FAST_UNLOAD moved to CPMReproTestCase.enableFastUnloadIfRequested().
Scenario Fast unload (3 s) Production unload (30 s)
H2 skipOnMessage: one woken worker without onMessage only the broken navigation lost; all later ones answered not re-run
H2 throw: one woken worker throws at top level broken navigation + 5 busy + 3 idle-wake losses, tabs 1–2 broken for the rest of the test broken navigation + 1 busy loss, then everything answered
H1 Network crash, recovery on, 20 / 10 wakes 2 lost after crash, then tabs 1–2 never got initResp, tab-3 always fine; all restarted workers ran 2 lost after crash, then all 10 wakes answered; all restarted workers ran
  • H1 not confirmed: every worker restarted after Network recovery ran its script (bindings present). With the real delay, recovery restores messaging after 2 lost navigations.
  • H2 (listener metadata loss) not confirmed: one start without a listener only loses that navigation; P1's wake plus re-registration heals it.
  • The fast-mode per-tab breakage is an artifact: after a confirmed hang and recovery, tabs with an unanswered init kept receiving the dashboard refresh but never initResp. That is WebKit's silent tabs.sendMessage path (processes.isEmpty() → {}), and the monitor sent one was_stuck=true / reload_state=loaded / other_tab_response episode and nothing more. It did not reproduce with the 30 s delay. The TestWebKitAPI identity or the 3 s unload changes WebKit behavior; fast-mode hits must always be confirmed with the production delay. The blind spot it exposed is still real in principle: the monitor cannot see a tab that gets the dashboard refresh but no initResp.
  • H3 (stale failed background) reproduced with production timing: after the throwing start, error 6 is recorded and the failed background view stays present. The next navigation 9 s later got nothing, and only the M4 reload (new context) restored messaging. No background_load_retry pixel fired, so the P2 retry never completed, as predicted by WebKit's loadBackgroundWebViewIfNeeded early return. Cost: at least one extra lost navigation per error 6, plus a full M4 reload. On iOS, error 6 is 7.5% of episodes and 25% of stuck.

Production-gap ideas and coexistence screen (27 Oct)

The harness differs from macOS production in four ways:

  1. Other extensions: 83% of macOS 1.213 installs run the ad-blocking extension (m.mac.web.extension.daily.ad.blocking.state.daily is_enabled=true 3,486 of 4,182). It is a module service worker with DNR and four document_start all-frames content scripts. iOS has none; Dark Reader runs on ~140 iOS installs.
  2. Real pages: network loads, iframes, redirects, replaceState.
  3. Real app tabs and windows: visibility, App Nap, multiple windows and pinned tabs; app-hosted harness only, in a VM.
  4. Long-lived data store with abandoned registrations; weak, since ~30 harness runs already shared one persistent store.
  • Coexistence screen: CPMReproCoexistenceTests (new Options.embeddedTypes), CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,500 wake navigations (85 min); at 30 s, 0 of 15. The ad-blocking background started once and unloaded at 30 s, but never woke again on fixture pages, so concurrent wakes of both backgrounds were not exercised. Its content scripts ran on every page. A second extension's content scripts alone do not trigger the wake hang.
  • App Store vs DMG (macOS 1.213, 26–27 Oct, per loaded install): episodes 20.8% vs 29.7%, wake cohort 9.8% vs 10.6%, cross-tab no-response 1.2% vs 2.4%. Sandboxing does not drive the wake cohort; the DMG excess is likely early-updater bias.
  • Real-page screen (CPMReproRealPageTests): earlier repro tests used loadSimulatedRequest, so loads never reached WebKit's Network process. CPMReproFixtureServer serves loopback pages through it: cross-origin iframes (localhost vs 127.0.0.1), 302 redirects, replaceState, slow images, main-thread blocks. CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,333 visits across 7 profiles (~190 each), and no monitor episodes. Real loads, frames, redirects and URL rewrites do not trigger the wake hang.
  • Next hypothesis — origin-changing M4 reload stalls on large data: most confirmed macOS cross-tab hangs (1.213, 26–27 Oct, tabs_affected≠1 and no response) have tail proxy_on>dealloc>view>proxy_on, view 4–8 s, ready <1 s, scripts present, reload_state=loaded (97+22+22) or response_timed_out (34+17+13). The reload loaded, yet nothing answered for the rest of the episode. Released builds load the reloaded context under a new random origin, and P3 saw WebKit's origin migration slow with a large cache. Test: CPMReproReloadOriginTests fills the persistent HTTP cache (/fill, default 20,000 × 2 KB) and runs the M4 reload with released vs stable origin, timing post-reload silence. If confirmed, P3 also fixes hang persistence.
  • Reproduced: origin-changing M4 reload blacks out CPM proportionally to the HTTP cache (CPMReproReloadOriginTests, production timing, CPM only). Fill the persistent cache through real fetches, run the hang reload (reloadExtension(…, trigger: .cpmMessagingHang)), navigate every 2 s:
Cache entries (2 KB) Released origin (new random origin per load) P3 stable origin
20,000 first initResp 1.5 s after reload, 0 lost 0.1 s
200,000 10.9 s, 1 lost 0.1 s

reloadExtension itself returns in ~20 ms; the delay is WebKit migrating extension data to the new origin before content scripts run. Roughly linear, ~50 µs per cache entry. Cookie seeding on 5,000 .test domains did not create website data records (still 2), so per-site record cost is untested. The same migration runs on every app launch in released builds, which links this to the P3 startup cohort. P3 (#7493) removes it in both cases. It explains reload_state=response_timed_out and extended post-reload silence; whether real caches are large enough for a full 5-minute silence is being checked at 600,000 entries.

  • 600,000 entries, released origin: first initResp 32.2 s after the reload, 5 navigations lost; still linear (~53 µs/entry). Page loads during the migration were also slower (one took 9 s), so the migration may delay browsing generally, not only CPM. A full 5-minute silence would need ~5–6M entries by extrapolation, so this explains post-reload silences of seconds to about a minute, not the whole deadline. The same blackout hits every app launch on released builds. With P3, 0.1 s at every size.

What survives the hang reload — code search and tests (27 Oct)

Telemetry: every confirmed permanent hang (iOS 0.30%, macOS 2.07% of DAU) had the M4 reload run and stay silent, so the broken state survives a full context replacement or the trigger repeats. Checked:

  • Native messaging (WebExtensionMessageRouter, AutoconsentWebExtensionMessageHandler): handlers keyed by extension identifier and re-registered on reload; each request is stateless. Not persistent.
  • Monitor reload bookkeeping: responses from measurements begun on the new generation count; harness reload scenarios closed their episodes. No bug found.
  • WebKit process reuse: WebProcessCache requires usesWebProcessCache, default false, and the app sets no process-pool options. Ruled out.
  • storage.session stall: CPM's refreshRemoteConfig awaits getFromSessionStorage("config") with no timeout, so a stall would hang every init. No stall in any screen log; only tabs.sendMessage on the busy-page profile. Not reproduced.
  • Network replaced while the background is unloaded (testNetworkProcessReplacedWhileBackgroundIsUnloadedKeepsWakesWorking, 30 s timing): 0 lost in 6 busy navigations and 4 idle wakes.
  • macOS tab URL: Tab.url(for:) returns content.urlForWebView, which follows webView.url one main-queue hop later; WebKit reads it live for sender.tab.url. Harness mode CPM_REPRO_MACOS_TAB_URL=1 reproduces the lag: 133 real-page visits, 0 lost, 0 monitor episodes. init arrives after the hop.
  • webExtensionPageWebView: only set for webkit-extension: pages and cleared on content change. Not relevant.

Reinterpretation: "idle wake" is a state at failure, not proven to be a cause. On desktop the background unloads after every 30 s idle, so a large share of all navigations, healthy ones included, run right after a wake, and failures from any cause land in this group at that base rate. That fits P1 changing nothing. Confirming it needs a healthy-navigation baseline (background state on successful measurements); none exists since #7384 was closed.

Remaining untested production ingredients: the real macOS Tab stack (DDG user scripts / content-blocking rule lists in tabs, multiple windows, pinned and hidden tabs, App Nap); needs the app-hosted harness in a VM.

Plan and corrections (27 Oct)

Corrections

  • P6 is not the feature flag. webExtensions is on in production, not toggled, and will be removed. The flag-cycle mechanism (also in the tabControllerMatchesContext doc comment) was an old unverified hypothesis. Established: a tab's web view gets the controller only at creation and only if webExtensionManager exists then (WKWebViewConfigurationExtensions → attachWebExtensions), and the manager is created once at launch and replaced only by the flag handler. Trigger unknown. Candidates: web views created before the launch-time manager; the flag evaluating off at launch for some installs (missing or stale cached config); a web view built outside the standard configuration path. Macos 1.213 on 26 Oct: 51 users with controller mismatch + no scripts, 19 permanent (22% of macOS permanent hangs). The fix depends on the trigger.
  • Error 6: the P2 retry is shipped and helped. iOS error-6 users fell from 1.3% to 0.5% of DAU, and 455 iOS users/day had a successful background_load_retry. Error-6 episodes have outcomes: iOS 86% recovered later, 9% permanent. It is still 25% of iOS stuck because (a) the retry restores the background but not the page whose init was lost, which nothing re-sends (late reload off); (b) the retry is skipped while the app is in the background, within the 60 s cooldown, or during M4; (c) when the failed background view is kept (H3, reproduced), the retry cannot complete. Which dominates is unknown: no retry attempted/result field in the episode.
  • Extension updated: yes, the outdated version is loaded. WebExtensionLifecycleCoordinator.loadAndSync runs loadInstalledExtensions() (the previous, now outdated version) before syncEmbeddedExtensions, which installs the bundled version under a new identifier, loads it alongside, and unloads the old one. Early pages after an app update run against the old version, then the context is swapped with a new origin and the migration blackout.

Next steps

  1. Ship what's understood:
    • P3 release.
    • Extension updated: upgrade before load at launch; carry the origin across upgrades.
    • Error 6: restart the stale background on error 6 when the failed view is still alive, and re-init the page that lost init.
    • P6: find the trigger first (the three candidates above), then fix.
  2. Verify the macOS Network-recovery false-positive lead: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and died_client precedes 16% of macOS idle-wake failures. Review CPMNetworkProcessRecovery detection; count recovery pixels in the harness screens.
  3. Telemetry (one privacy review): events after the first failure (two-phase bg_events); a healthy-navigation baseline (background state on successful measurements); retry attempted/result; recovery applied in this episode; time since app resume.
  4. Targeted harness work only: Network-recovery restart racing a wake; iOS resume-time wake on the simulator (fix the Bundle.module skip); the app-hosted macOS harness in a VM if steps 2–3 leave the wake group unexplained.
  5. Mitigation experiment: keep the background warm while browsing (periodic loadBackgroundContent < 30 s), behind a flag; it is also a causal test for the wake group.
  6. Re-measure on full days of 7.243 / 1.213 and the first P3 builds.