32 KiB
title, date, tags
| title | date | tags | |||
|---|---|---|---|---|---|
| CPM — first post-release data (iOS 7.243.0 / macOS 1.213.0) | 2026-10-27 |
|
CPM: first post-release data, 26 October 2026
Internal metrics; keep these figures out of public PRs. Previous analysis: cpm-metrics-causes-and-mitigations-2026-10-16, cpm-status-2026-10-15 (§3–5, 12–14 October cut).
What shipped
- iOS 7.243.0 (tag 26 Oct 19:23 UTC), macOS 1.213.0 (tag 26 Oct 16:23 UTC).
- Both: episode telemetry #7376/#7484, episode tab scope #7520/#7521, P1/P2/P4 recovery #7379/#7536, Network-process recovery #7459/#7531.
- macOS only: P5 shared-pinned window close #7473, remote/debug disable #7498.
- Not shipped: P3 stable origin #7493 (on main only). No P6 fix exists in either release: no PR implements the controller rebind.
- Shipped episode contract is the rolled-back variant:
was_stuck(bool), single-phasebg_events(no;), plustabs_affected,repeat_failures,interrupted. Nobg_view_creations,context_errorsornetwork_restarted. Old cohort SQL needs adapting (below). - Monitor timing: 4 s response grace; suspended time no longer counts (8 s after resume, 30 s extension-load wait, 5 min deadline).
How to query
- New pixels have no
m_/debug_prefix: ClickHousepixel_id = 'web',pixel LIKE 'web.extension.cpm.%'(iOS adds.ios.phone|tablet). Queryingpixel_id='m'without an agent filter and a one-day window exceeds the 7B row limit. - Denominator:
pixel_id='m',m.web.extension.loaded.daily%/m.mac.web.extension.loaded.daily, split byappVersion. - Old-contract attribution was remotely switched off from 21 Oct (old versions keep sending bare
debug.web.extension.cpm.*without parameters). The last fully attributed old-version window is 18–20 Oct, iOS 7.241 / macOS 1.211.x. - Data lag: on 27 Oct the table ended at ~01:10 UTC;
dateis US-Eastern. 26 Oct is the only post-release day and it is partial for new adopters (exposure starts at update time). - Runner:
~/Developer/grafana-data-toolsreader; ad-hoc SQL and anARRAY JOIN extractURLParameters(request)parameter dump were used.extractURLParameterneeds a constant key.
Rates per extension-loaded install-day (26 Oct)
| iOS 7.242 (old) | iOS 7.243 (new) | macOS 1.212 (old) | macOS 1.213 (new) | |
|---|---|---|---|---|
| loaded.daily | 4,257,839 | 52,979 | 93,030 | 2,566 |
| old init_failed.daily (sum of reasons) | 21.3% | — | 36.8% | — |
| old stuck.daily | 3.3% | — | 12.5% | — |
| episode.daily | — | 7.05% (3,735) | — | 30.0% (771) |
episode with reload_state≠none (hang confirmed, ≈ old stuck), count leg |
— | 23% of episodes | — | 26% of episodes |
was_stuck=true (second distinct tab), daily leg |
— | 8.1% | — | 11.2% |
| network_process_recovery_applied.daily | — | 432 (0.8%) | — | 232 (9.0%) |
Not like-for-like: the episode unit is one report per episode, the old init number sums four reasons, suspended time is now excluded, and day-1 adopters have short exposure. Directionally, iOS fell ~3× (21% → 7%); macOS barely moved (37% → 30%).
was_stuck only becomes true when a different tab fails; M4 reload already runs on a same-tab second timeout. Use reload_state≠none as the old "stuck" proxy.
Cohorts (count leg, adapted classification)
Priority: error 6 (bg_load_error_after present or error_ token) → no view (bg_view_alive=false) → tab not connected (tab_has_ext_scripts=false) → extension_updated ≤2 min → wake (bg_view_age < 30 s and a dealloc token) → recent load/base_url_changed ≤2 min → other. There is no Network-replaced cohort any more.
| Cohort | iOS share / of stuck / no-response | macOS share / of stuck / no-response |
|---|---|---|
| error 6 | 7.5% / 24.7% / 39% | 0.6% / 1.7% / 44% |
| no view (P3) | 7.7% / 6.8% / 7% | 2.9% / 0.9% / 25% |
| tab not connected (P6) | — | 5.8% / 1.3% / 70% |
| extension_updated ≤2 m | 3.9% / 6.2% / 37% | 14.6% / 11.7% / 28% |
| wake, view <30 s | 22.8% / 22.0% / 9% | 43.9% / 58.4% / 48% |
| recent context load | 22.8% / 17.2% / 12% | 11.9% / 6.9% / 25% |
| other | 35.3% / 23.2% / 5% | 20.4% / 19.0% / 27% |
Baseline with last week's exact SQL, 18–20 Oct (init → stuck shares). iOS 7.241: error6 6.4→22.0, no view 47.5→11.5, network 16.5→38.2, wake 7.7→9.2, recent load 16.7→6.6, other 5.2→12.5. macOS 1.211.x: error6 1.2→2.2, no view 3.6→1.9, tab not connected 5.9→8.7, network 13.2→25.8, wake 33.2→33.3, recent load 19.4→6.3, other 23.6→21.6. This matches the 12–14 Oct cut, so the baseline was stable.
Per-install rough rates (cohort share × failure rate), old init → new episode:
- iOS: no view 10% → 0.5%, recent load 3.5% → 1.6%, error 6 1.3% → 0.5%; the network cohort is absorbed. Wake stays at 1.6% → 1.6%. Other rises 1.1% → 2.5%; it now absorbs former network cases and has no network field.
- macOS: wake ≈12% → ≈13%. Unchanged.
Findings
- P1 (wake) fix had no measurable effect on either platform. In the wake cohort
bg_ready_after=under_1sdominates (macOS 93% overall), so the background is not slow. Thebackground_wakepixel fires only for ≥1 s callbacks (macOS 89/day vs ~670 wake episodes). The remaining failure is message/response loss after a fast idle wake, not readiness. On macOS 48% of wake episodes see no response for 5 min; tabs=3+ → 68%. Typical tail:…dealloc@15m,view@15m,proxy_on@15m,dealloc@5m,view@30s,proxy_on@30s,bg_view_age=4s_to_8s(view created around the navigation). The pattern appears on every macOS version, 15.6–27.2. - iOS error 6 changed shape. All attributed error 6 is now
bg_load_error_after=under_5s; the 30 s timer bucket is gone.- A (
ready=failed, app_state active):dealloc → view → proxy_on → interruption → error_6, often afterdied_client/died_crash/died_crash_limit. The background load spans an app suspension and fails at resume. - B (
ready=pending): an older error 6, then a new view whose load is still pending 8–30 s later. - The P2 retry fires only while foregrounded with a 60 s cooldown (277/day). Foreground-resume recovery is the missing piece.
- A (
- macOS P6 is untreated: 5.8% of episodes, 70% never answered. iOS now shows
tab_has_ext_scripts=falsein 5.3% of episodes, concentrated on iOS 26.7 (16%) and 27.0 (8%), with 0 on 27.2. Previously iOS was ≈0. - macOS Network replacement is frequent: the applied pixel is on 9% of 1.213 install-days. With recovery applied it no longer appears as a cohort, but
otherstays at 20%. - After M4 on macOS:
reload_state=loaded→ 46% no response by deadline;response_timed_out→ 57%. The reload often does not restore messaging, or no qualifying navigation follows. - Recovery pixels only ever report
background_ready.response_receivedneeds late-document reload, which is default OFF. - Day-1 effect: macOS
extension_updated ≤2mis 14.6% (embedded upgrade on update). Re-check after 2–3 full days.
Next measurements
- Repeat for 27–29 Oct (full days, larger adoption). Track the wake cohort and macOS no-response by OS.
bg_eventsper-phase budget andnetwork_replaced_since_viewremain missing (see cpm-status-2026-10-15#6. Телеметрия).
Repro targets
| Issue | Evidence | Hypothesis to test | Candidate fix |
|---|---|---|---|
| R1 wake message loss | macOS 44% eps / 58% stuck; iOS 23%; ready <1 s | message to a woken non-persistent SW is acknowledged with no reply: (a) processes(RuntimeOnMessage) empty when _loadServiceWorker completion runs before listener IPC; (b) listener set loses RuntimeOnMessage so wakeUpBackgroundContentIfNecessaryToFireEvents resolves {} without waking; (c) background unloaded while a reply is pending |
keep the background warm while CPM pages are active, or re-send init on empty reply (native/JS handshake) |
| R2 iOS error 6 across suspension | 7.5% eps / 25% stuck, all under 5 s, interruption between view and error | WebKit load timer spans suspension; nothing retries on resume | on foreground resume, retry loadBackgroundContent when the last load failed or is pending |
| R3 P6 controller mismatch | macOS 5.8%, 70% unanswered | known main-flag manager replacement | rebind/recreate tab web views to the current controller |
WebKit source pointers (~/Downloads/WebKit_sparse): WebExtensionContext.cpp wakeUpBackgroundContentIfNecessaryToFireEvents (drops events when the listener set lacks the type after the first load), scheduleBackgroundContentToUnload (30 s, reset only on dispatch), WebExtensionContextCocoa.mm loadBackgroundWebView / performTasksAfterBackgroundContentLoads, WebExtensionContextAPIRuntimeCocoa.mm runtimeSendMessage (empty processes() → {} reply). 324743 (9bcf5840ef) holds runtime messages until SW activation; it is not in shipped OS builds.
Repro log — 27 October 2026
Worktree apple-browsers.git/cpm-remaining-failures-repro, branch alex/cpm-remaining-failures-repro (from origin/main 8023e4cb5c, which includes the released P1/P2/P4/Network code plus P3). The package real-WebKit harness was imported from origin/alex/cpm-repro-harness. It compiles unchanged on current main, and its CI-tier tests pass. Local tier: CPM_REPRO=1 swift test --filter <Suite> from SharedPackages/WebExtensions. System WebKit is the host macOS; no app windows are opened.
- R1 idle wake — not reproduced organically.
CPMReproIdleWakeTests.testRepeatedIdleWakeCyclesDeliverInit: 3 tabs, 8 idle-unload cycles (WebKit unloads 32.0 s after the last answered navigation), single and double-tab wakes, 0 of 11 lost. Every wake answered in 2–5 ms after didFinish.testNavigationAtIdleUnloadBoundaryDeliversInit: offsets −0.6…+0.2 s around the calibrated unload, 0 of 7 lost. A plain idle wake on an in-memory fixture page is not the production mechanism. The missing ingredient is still unknown: real page loads, a long-lived data store with abandoned registrations, Network replacement, or listener-set loss. - P3 (#7493) only in internal builds (macOS 1.214.0-860, iOS 7.244.0-x). On 26 Oct: macOS 1.214 had 20 episodes / 9 daily of ~36 loaded installs, with 0 no-view and 0 error 6; iOS 7.244 had none. That is too small to evaluate. Relevance: released builds get a new random extension origin on every context load (launch and M4 reload), and WebKit migrates data before injecting. This may explain the iOS no-view cohort and the macOS "reload loaded → still no response" share (46%). Re-check when 7.244 reaches users.
- R2 suspended wake — macOS analogue does not stay broken.
CPMReproSuspendedWakeTests: idle unload, then a navigation wakes the background, and the new background process is SIGSTOPped for 45 s while another tab navigates every 10 s. Two navigations timed out (+9 s, +14 s). Then the background was replaced by a new process (same view slot, new PID) and every later navigation answered in 1–14 ms, including 10 after thaw. No error 6 was recorded on the live context. On macOS the shipped recovery restores messaging within ~14 s of the fault. The harness cannot reproduce iOS suspension, which pauses the UI process's own timers. Likely production sequence: the overdue 30 s unload timer fires at resume, closes the loading view, and records error 6 "under 5 s" (pattern A). Pattern B (new view pending 8–30 s) needs a device or simulator test. - WebKit hypothesis not hit yet: an SW load failure that keeps the background view (
loadBackgroundWebViewcompletion!successwithout unload). Later wakes would only queue, and every dispatch restarts the 30 s unload timer, so the publicloadBackgroundContent/P2 retry cannot escape. A candidate mitigation is to restart the background web content process when error 6 is recorded while a background view still exists. Not implemented, because the state has not been reproduced.
Treatment pixels, P3 ceiling and permanent hangs — 27 October 2026
Treatment pixels received (26 Oct, count leg, iOS 7.243.0 / macOS 1.213.0)
| Fix | Pixel (pixel_id='web') |
iOS | macOS |
|---|---|---|---|
| P1 wake | web_extension_cpm_background_wake |
2,026 | 89 |
| P2 error-6 retry | web_extension_cpm_background_load_retry |
277 | 12 |
| Install / upgrade readiness | …_install_readiness / …_upgrade_readiness |
3 / 24 | 0 / 1 |
| Network recovery | web_extension_cpm_network_process_recovery_applied |
476 | 277 |
| P4 detection (flag-independent) | web_extension_cpm_startup_response_missing |
450 (daily 327) | 20 (daily 12) |
| Late page reload | web_extension_cpm_late_document_reload |
0 | 0 |
- Every recovery pixel only reports
outcome=background_ready, meaning the fix was applied and the background came up.reload_started/response_receivedexist only for the late page reload, which is default OFF, so it has zero rows on every version including internal builds. No pixel shows whether the page got its answer after a treatment; the effect has to be read from episode rates. background_wakefires only when the wake callback took ≥1 s and a navigation waited, so it is not a count of all wakes.- Detection gap for the late reload.
startup_response_missingfires with the flag off but requiresstartup.isFirstLoad: the first background load of that context in the session (launch, install, upgrade).attemptRecovery(the reload) has noisFirstLoadcondition and would also run after later wakes. With the flag off, wake-time cases the reload would treat are not counted anywhere; 327 / 12 is a lower bound of what it could fix. Both detection and reload only arm when the page commits while the startup is still pending. In the wake cohort the background is ready in under 1 s, so many wake failures are outside both. Counting the would-be-reload wake cases needs a pixel change: dropisFirstLoadfrom detection or add a parameter. Not done. - Upper bound on the late reload: ~0.6% of iOS / 0.5% of macOS extension-loaded installs; ~9% of iOS / ~2% of macOS installs with a failure episode.
P3 (baseURL) ceiling on the new data
- The earlier "≈half" was the pre-release no-view share: 48% of iOS initial failures, but only 11% of stuck, mostly self-healing.
- On 7.243.0 without P3, no-view is already 7.7% of iOS episodes / 6.8% of stuck (≈10% → 0.5% of loaded installs); macOS 2.9% / 0.9%. The drop comes from the new measurement (suspended time excluded, one report per episode) and the shipped readiness gates.
- Direct P3 ceiling ≈ 8% of iOS episodes, less on macOS. A possible indirect effect: released builds get a new random origin on every M4 reload, and 46% of macOS reload-loaded episodes still got no response. That link is unproven; internal 1.214 / 7.244 data is too thin.
Users affected by permanent hangs (26 Oct)
Denominators were refreshed after more 26 Oct data arrived: extension-loaded installs iOS 69,088 / macOS 3,045. "Permanent" = outcome=no_response_before_deadline (no qualifying response for 5 min). Single = tabs_affected=1, cross = 2/3+. Confirmed = repeat_failures≠0 or reload_state≠none. Daily leg = affected installs.
| Installs/day | iOS | macOS |
|---|---|---|
| Any failure episode | 4,655 (6.7%) | 876 (28.8%) |
| Single-tab, no response | 354 (0.51%) | 217 (7.1%) |
| ↳ confirmed | 76 (0.11%) | 25 (0.82%) |
| Cross-tab, no response | 129 (0.19%) | 58 (1.9%) |
| ↳ confirmed | 123 (0.18%) | 49 (1.6%) |
- Cross-tab is the reliable permanent-hang figure: ~0.2% of iOS and 1.6–1.9% of macOS users per day; 95% / 84% confirmed.
- Unconfirmed single-tab rows (iOS 278, macOS 192) may be stuck tabs or users who didn't navigate within 5 min.
- Strict permanent (confirmed single + cross): ~0.3% iOS, ~2.4% macOS users/day. Upper bound including unconfirmed single-tab: 0.7% / 9%.
- Lower bounds for users: the daily leg describes only each install's first completed episode of the day, and episodes lost when the process ends before the deadline are not reported. Count leg (episodes per install-day), confirmed cross-tab: iOS 0.41%, macOS 5.2%; confirmed single-tab: iOS 0.26%, macOS 2.3%.
- One partial day; macOS has ~3k installs on 1.213.0. Repeat on 27–29 Oct full days.
Repro statistics note
18 wake navigations cannot rule out a rare loss. At p=8% a clean run happens 22% of the time. The per-wake production rate is likely ≪1% (many wakes per install-day), which needs ~1,000 wakes for 95% power. CPMReproIdleWakeTests now has CPM_REPRO_FAST_UNLOAD=1: it calls +[WKWebView _setApplicationBundleIdentifier:] with com.apple.WebKit.TestWebKitAPI, so WebKit unloads an idle background after 3 s instead of 30 s (~5 s per cycle). It can change other test-runner-only WebKit behavior, so confirm any hit with the 30 s delay. A 1,000-cycle run (~1,500 wake navigations) was started on 27 Oct.
- Fast idle-wake run, 27 Oct: 0 lost of 1,500 wake navigations in 1,000 cycles (
CPM_REPRO_FAST_UNLOAD=1, 3 tabs, single and double-tab wakes, 85 min). With zero failures the 95% upper bound on the per-wake loss rate is ≈0.2% (3/1500). A bare idle-unload → wake race on a fixture page is not the main source of the wake cohort; an extra ingredient is needed (hypotheses H1–H5 below).
Hypotheses for the remaining wake hangs (27 Oct)
The embedded extension (background-embedded.js, 2026.10.23) registers runtime.onMessage synchronously (MessageRouter). CPM init = await remoteConfigJson → native isAutoconsentSettingEnabled → native isFeatureEnabled → refreshDashboardState. Every native request has a 20 s timeout, with config_unavailable / setting_disabled fallbacks that still refresh. A 5-minute silence therefore means init never reached the worker, or the refresh was not attributed.
- H1 — worker without extension bindings after Network-process recovery (WebKit 324754). macOS recovery is applied on 9% of installs/day. A worker without
browser.runtimethrows at top level: alive, responsive, "ready", never answers, all tabs, M4 may repeat it. Test: Network crash → recovery restart → probebrowser.runtimein the worker → N wake cycles. Fix: verify with a ping after restart; restart again or reload if it fails. - H2 — listener metadata loss.
wakeUpBackgroundContentIfNecessaryToFireEventsreturns an empty reply without waking when the saved listener set lacksRuntimeOnMessage. The set is saved after every load, so one load where the worker threw before registering poisons it. Test: a prelude knob that throws or skipsonMessageon one load only. Fix: treat an empty sendMessage reply as a failure → restart, plus re-send init. - H3 — stale failed background view (iOS error 6, pattern B).
_loadServiceWorkerfailure keeps the view, so wakes only queue; activity keeps restarting the 30 s unload timer; P2 retry uses the same path. Test: fail the worker load without closing the view, with navigations every 10 s. Fix: on error 6 with a live background view, restart its WebContent process, then retry. - H4 — refresh not attributed. The monitor needs an exact scheme/host/port/path/query match with the committed URL. A cold wake answers later, giving
replaceState/ client-side redirects time to changesender.tab.url.other_tab_responseis 37–53% of outcomes. Test: a fixture withreplaceState50–300 ms after load, cold vs warm. Fix: attribute by extension tab ID and treat same-document URL changes as the same document. - H5 — slow cold-wake init. Three native round-trips plus session storage; explains late answers (
response_after≥30 s in 20–35% of answered episodes), not 5-minute hangs. Test: preludedelayonsendNativeMessage1–3 after a wake. Fix: answer init from cached config/settings first.
H1/H2 tests — results (27 Oct)
New CPMReproBrokenWorkerTests and harness changes (uncommitted):
Options.isNetworkRecoveryEnabled, on by default as in the app; off inCPMReproWebKitTests.unprotected.- Prelude
startFaultsbreaks one worker start inside a wall-clock window:.skipOnMessage(script runs, noruntime.onMessagelistener) or.throw(top-level throw before the extension script). CPM_REPRO_FAST_UNLOADmoved toCPMReproTestCase.enableFastUnloadIfRequested().
| Scenario | Fast unload (3 s) | Production unload (30 s) |
|---|---|---|
H2 skipOnMessage: one woken worker without onMessage |
only the broken navigation lost; all later ones answered | not re-run |
| H2 throw: one woken worker throws at top level | broken navigation + 5 busy + 3 idle-wake losses, tabs 1–2 broken for the rest of the test | broken navigation + 1 busy loss, then everything answered |
| H1 Network crash, recovery on, 20 / 10 wakes | 2 lost after crash, then tabs 1–2 never got initResp, tab-3 always fine; all restarted workers ran |
2 lost after crash, then all 10 wakes answered; all restarted workers ran |
- H1 not confirmed: every worker restarted after Network recovery ran its script (bindings present). With the real delay, recovery restores messaging after 2 lost navigations.
- H2 (listener metadata loss) not confirmed: one start without a listener only loses that navigation; P1's wake plus re-registration heals it.
- The fast-mode per-tab breakage is an artifact: after a confirmed hang and recovery, tabs with an unanswered init kept receiving the dashboard refresh but never
initResp. That is WebKit's silenttabs.sendMessagepath (processes.isEmpty()→{}), and the monitor sent onewas_stuck=true / reload_state=loaded / other_tab_responseepisode and nothing more. It did not reproduce with the 30 s delay. The TestWebKitAPI identity or the 3 s unload changes WebKit behavior; fast-mode hits must always be confirmed with the production delay. The blind spot it exposed is still real in principle: the monitor cannot see a tab that gets the dashboard refresh but noinitResp. - H3 (stale failed background) reproduced with production timing: after the throwing start, error 6 is recorded and the failed background view stays present. The next navigation 9 s later got nothing, and only the M4 reload (new context) restored messaging. No
background_load_retrypixel fired, so the P2 retry never completed, as predicted by WebKit'sloadBackgroundWebViewIfNeededearly return. Cost: at least one extra lost navigation per error 6, plus a full M4 reload. On iOS, error 6 is 7.5% of episodes and 25% of stuck.
Production-gap ideas and coexistence screen (27 Oct)
The harness differs from macOS production in four ways:
- Other extensions: 83% of macOS 1.213 installs run the ad-blocking extension (
m.mac.web.extension.daily.ad.blocking.state.dailyis_enabled=true3,486 of 4,182). It is a module service worker with DNR and fourdocument_startall-frames content scripts. iOS has none; Dark Reader runs on ~140 iOS installs. - Real pages: network loads, iframes, redirects,
replaceState. - Real app tabs and windows: visibility, App Nap, multiple windows and pinned tabs; app-hosted harness only, in a VM.
- Long-lived data store with abandoned registrations; weak, since ~30 harness runs already shared one persistent store.
- Coexistence screen:
CPMReproCoexistenceTests(newOptions.embeddedTypes), CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,500 wake navigations (85 min); at 30 s, 0 of 15. The ad-blocking background started once and unloaded at 30 s, but never woke again on fixture pages, so concurrent wakes of both backgrounds were not exercised. Its content scripts ran on every page. A second extension's content scripts alone do not trigger the wake hang. - App Store vs DMG (macOS 1.213, 26–27 Oct, per loaded install): episodes 20.8% vs 29.7%, wake cohort 9.8% vs 10.6%, cross-tab no-response 1.2% vs 2.4%. Sandboxing does not drive the wake cohort; the DMG excess is likely early-updater bias.
- Real-page screen (
CPMReproRealPageTests): earlier repro tests usedloadSimulatedRequest, so loads never reached WebKit's Network process.CPMReproFixtureServerserves loopback pages through it: cross-origin iframes (localhostvs127.0.0.1), 302 redirects,replaceState, slow images, main-thread blocks. CPM + ad-blocking, fast unload, 1,000 cycles: 0 lost of 1,333 visits across 7 profiles (~190 each), and no monitor episodes. Real loads, frames, redirects and URL rewrites do not trigger the wake hang. - Next hypothesis — origin-changing M4 reload stalls on large data: most confirmed macOS cross-tab hangs (1.213, 26–27 Oct,
tabs_affected≠1and no response) have tailproxy_on>dealloc>view>proxy_on, view 4–8 s, ready <1 s, scripts present,reload_state=loaded(97+22+22) orresponse_timed_out(34+17+13). The reload loaded, yet nothing answered for the rest of the episode. Released builds load the reloaded context under a new random origin, and P3 saw WebKit's origin migration slow with a large cache. Test:CPMReproReloadOriginTestsfills the persistent HTTP cache (/fill, default 20,000 × 2 KB) and runs the M4 reload with released vs stable origin, timing post-reload silence. If confirmed, P3 also fixes hang persistence. - Reproduced: origin-changing M4 reload blacks out CPM proportionally to the HTTP cache (
CPMReproReloadOriginTests, production timing, CPM only). Fill the persistent cache through real fetches, run the hang reload (reloadExtension(…, trigger: .cpmMessagingHang)), navigate every 2 s:
| Cache entries (2 KB) | Released origin (new random origin per load) | P3 stable origin |
|---|---|---|
| 20,000 | first initResp 1.5 s after reload, 0 lost |
0.1 s |
| 200,000 | 10.9 s, 1 lost | 0.1 s |
reloadExtension itself returns in ~20 ms; the delay is WebKit migrating extension data to the new origin before content scripts run. Roughly linear, ~50 µs per cache entry. Cookie seeding on 5,000 .test domains did not create website data records (still 2), so per-site record cost is untested. The same migration runs on every app launch in released builds, which links this to the P3 startup cohort. P3 (#7493) removes it in both cases. It explains reload_state=response_timed_out and extended post-reload silence; whether real caches are large enough for a full 5-minute silence is being checked at 600,000 entries.
- 600,000 entries, released origin: first
initResp32.2 s after the reload, 5 navigations lost; still linear (~53 µs/entry). Page loads during the migration were also slower (one took 9 s), so the migration may delay browsing generally, not only CPM. A full 5-minute silence would need ~5–6M entries by extrapolation, so this explains post-reload silences of seconds to about a minute, not the whole deadline. The same blackout hits every app launch on released builds. With P3, 0.1 s at every size.
What survives the hang reload — code search and tests (27 Oct)
Telemetry: every confirmed permanent hang (iOS 0.30%, macOS 2.07% of DAU) had the M4 reload run and stay silent, so the broken state survives a full context replacement or the trigger repeats. Checked:
- Native messaging (
WebExtensionMessageRouter,AutoconsentWebExtensionMessageHandler): handlers keyed by extension identifier and re-registered on reload; each request is stateless. Not persistent. - Monitor reload bookkeeping: responses from measurements begun on the new generation count; harness reload scenarios closed their episodes. No bug found.
- WebKit process reuse:
WebProcessCacherequiresusesWebProcessCache, default false, and the app sets no process-pool options. Ruled out. storage.sessionstall: CPM'srefreshRemoteConfigawaitsgetFromSessionStorage("config")with no timeout, so a stall would hang everyinit. No stall in any screen log; onlytabs.sendMessageon the busy-page profile. Not reproduced.- Network replaced while the background is unloaded (
testNetworkProcessReplacedWhileBackgroundIsUnloadedKeepsWakesWorking, 30 s timing): 0 lost in 6 busy navigations and 4 idle wakes. - macOS tab URL:
Tab.url(for:)returnscontent.urlForWebView, which followswebView.urlone main-queue hop later; WebKit reads it live forsender.tab.url. Harness modeCPM_REPRO_MACOS_TAB_URL=1reproduces the lag: 133 real-page visits, 0 lost, 0 monitor episodes.initarrives after the hop. webExtensionPageWebView: only set forwebkit-extension:pages and cleared on content change. Not relevant.
Reinterpretation: "idle wake" is a state at failure, not proven to be a cause. On desktop the background unloads after every 30 s idle, so a large share of all navigations, healthy ones included, run right after a wake, and failures from any cause land in this group at that base rate. That fits P1 changing nothing. Confirming it needs a healthy-navigation baseline (background state on successful measurements); none exists since #7384 was closed.
Remaining untested production ingredients: the real macOS Tab stack (DDG user scripts / content-blocking rule lists in tabs, multiple windows, pinned and hidden tabs, App Nap); needs the app-hosted harness in a VM.
Plan and corrections (27 Oct)
Corrections
- P6 is not the feature flag.
webExtensionsis on in production, not toggled, and will be removed. The flag-cycle mechanism (also in thetabControllerMatchesContextdoc comment) was an old unverified hypothesis. Established: a tab's web view gets the controller only at creation and only ifwebExtensionManagerexists then (WKWebViewConfigurationExtensions→attachWebExtensions), and the manager is created once at launch and replaced only by the flag handler. Trigger unknown. Candidates: web views created before the launch-time manager; the flag evaluating off at launch for some installs (missing or stale cached config); a web view built outside the standard configuration path. Macos 1.213 on 26 Oct: 51 users with controller mismatch + no scripts, 19 permanent (22% of macOS permanent hangs). The fix depends on the trigger. - Error 6: the P2 retry is shipped and helped. iOS error-6 users fell from 1.3% to 0.5% of DAU, and 455 iOS users/day had a successful
background_load_retry. Error-6 episodes have outcomes: iOS 86% recovered later, 9% permanent. It is still 25% of iOS stuck because (a) the retry restores the background but not the page whoseinitwas lost, which nothing re-sends (late reload off); (b) the retry is skipped while the app is in the background, within the 60 s cooldown, or during M4; (c) when the failed background view is kept (H3, reproduced), the retry cannot complete. Which dominates is unknown: no retry attempted/result field in the episode. - Extension updated: yes, the outdated version is loaded.
WebExtensionLifecycleCoordinator.loadAndSyncrunsloadInstalledExtensions()(the previous, now outdated version) beforesyncEmbeddedExtensions, which installs the bundled version under a new identifier, loads it alongside, and unloads the old one. Early pages after an app update run against the old version, then the context is swapped with a new origin and the migration blackout.
Next steps
- Ship what's understood:
- P3 release.
- Extension updated: upgrade before load at launch; carry the origin across upgrades.
- Error 6: restart the stale background on error 6 when the failed view is still alive, and re-init the page that lost
init. - P6: find the trigger first (the three candidates above), then fix.
- Verify the macOS Network-recovery false-positive lead: recovery applies on 7.5% of macOS DAU vs 0.9% on iOS, and
died_clientprecedes 16% of macOS idle-wake failures. ReviewCPMNetworkProcessRecoverydetection; count recovery pixels in the harness screens. - Telemetry (one privacy review): events after the first failure (two-phase
bg_events); a healthy-navigation baseline (background state on successful measurements); retry attempted/result; recovery applied in this episode; time since app resume. - Targeted harness work only: Network-recovery restart racing a wake; iOS resume-time wake on the simulator (fix the
Bundle.moduleskip); the app-hosted macOS harness in a VM if steps 2–3 leave the wake group unexplained. - Mitigation experiment: keep the background warm while browsing (periodic
loadBackgroundContent< 30 s), behind a flag; it is also a causal test for the wake group. - Re-measure on full days of 7.243 / 1.213 and the first P3 builds.