From 5b68b3c8fc74681684ca50792c67180da4febc5c Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Wed, 9 Sep 2026 10:30:24 +0600 Subject: [PATCH] [2026-09-09] eagle: work/projects/cpm-web-extension-breakage-findings.md --- .../cpm-web-extension-breakage-findings.md | 96 +++++++++++++++++-- 1 file changed, 87 insertions(+), 9 deletions(-) diff --git a/work/projects/cpm-web-extension-breakage-findings.md b/work/projects/cpm-web-extension-breakage-findings.md index 8beb69f6..79031984 100644 --- a/work/projects/cpm-web-extension-breakage-findings.md +++ b/work/projects/cpm-web-extension-breakage-findings.md @@ -11,6 +11,8 @@ tags: # CPM Web Extension Breakage Findings > Investigation summary as of 2026-09-08. This note distinguishes confirmed code paths from hypotheses that still need production evidence. +> +> **Updated 2026-09-09 after source validation** (WebKit trunk `0c7e7ad97b`, see [[cpm-web-extension-breakage-validation]] for file:line evidence). Corrections are marked **[corrected]**; the ranked production sequences are in the new section "Ranked production trigger chains" at the end. Headline: the retained-failed-view state is real but **self-heals via the 30 s idle eviction**; the state that actually matches "stuck across all tabs" is a **terminated service worker under a background view WebKit still considers loaded**, which nothing in WebKit's extension layer detects. ## Executive summary @@ -20,7 +22,8 @@ The same visible symptom, CPM no longer managing consent, can come from three ma |---|---|---|---| | CPM content script misses a document | Current document | A navigation or reload after the extension finishes loading | Confirmed | | WebKit loses routing for an individual tab | Tab or WebContent process | A fresh tab/window, or rebuilding the affected tab | Confirmed historically; the known app lifecycle cause is fixed | -| WebKit retains a background service worker that failed to load | Extension context/controller | No ordinary message can recover it; the context must be unloaded or the failed background view cleaned up | Confirmed WebKit state machine; natural production trigger remains unknown | +| WebKit retains a background service worker that failed to load | Extension context/controller | **[corrected]** Recovers by itself: the next wake-up arms the 30 s idle-eviction timer, the failed view is unloaded, and the following message retries. Persistent only if every retry fails (e.g. SW script missing on disk) | Confirmed WebKit state machine; self-healing unless the failure cause is repeatable | +| **[new]** Background service worker terminated while WebKit still reports the background as loaded | Extension context (all tabs) | Only after ≥30 s with no extension events (idle eviction), or context unload/reload | Confirmed by source: triggered by critical memory pressure (macOS) and by Network-process termination; `WKWebExtensionContext.errors` stays empty | The debug-menu simulation enters the third state deliberately. It proves the health detection and failure shape, but its exact trigger cannot happen naturally in the same way because only the debug delegate closes the newly created background web view in that synchronous interval. @@ -97,7 +100,7 @@ The important retained state is: Every later CPM message appends another pending action, calls `loadBackgroundWebViewIfNeeded`, and immediately returns because `m_backgroundWebView` already exists. The pending actions therefore never run and the worker is never replaced. -That is a stable stuck state, not merely a slow startup. +**[corrected]** This is not stable. `wakeUpBackgroundContentIfNecessary` calls `scheduleBackgroundContentToUnload()` before checking whether the background is loaded (`WebExtensionContext.cpp:1854`). The first message finds no view and arms nothing, but the **second** message (any tab) finds the retained view and arms the 30 s one-shot. `unloadBackgroundContentIfPossible` then calls `unloadBackgroundWebView()` (CPM holds no ports, is not inspected, has no permission prompts), the pending actions are kept, and the next message creates a fresh view and retries. The state persists only if every retry fails for the same reason. ### Normal cleanup paths @@ -107,7 +110,7 @@ These paths do clean up the background web view: - WebContent process termination: `webViewWebContentProcessDidTerminate` calls `unloadBackgroundWebView`. - Normal nonpersistent-worker eviction: `unloadBackgroundContentIfPossible` calls `unloadBackgroundWebView`. -Consequently, ordinary idle eviction, memory pressure, or a conventional process crash alone do not explain the retained failed-view state. +**[corrected]** Ordinary idle eviction *does* clear the retained failed view (see above). Memory pressure and Network-process termination do not produce the retained-failed-view state, but they produce a different and worse one: they terminate the running service worker while the background view, its frame, and its listener registration remain, so WebKit keeps reporting the background as loaded (see "Ranked production trigger chains"). ## Debug simulation @@ -214,14 +217,15 @@ The following ideas do not yet have a traced path into a stable stuck state: - Multiple installed browser versions - Sleep/wake by itself -- Memory pressure by itself -- Normal service-worker idle termination -- A routine WebContent-process crash +- ~~Memory pressure by itself~~ **[corrected]** Critical system memory pressure is a traced trigger on macOS: `WebProcessPool::memoryPressureStatusChangedForProcess` → `NetworkProcess::TerminateIdleServiceWorkers` → `SWServer::terminateIdleServiceWorkers` kills the extension worker, which the SW server always considers idle because extension messages bypass it. +- ~~Normal service-worker idle termination~~ **[corrected]** same mechanism as above; the extension worker *is* idle-terminable under pressure. +- A routine WebContent-process crash (still unsupported: the background process crash path unloads the view and the next message reloads it) +- **[corrected]** A Network-process crash or unresponsive-kill is a traced trigger: `WebProcessPool::terminateServiceWorkers()` and `WebProcess::networkProcessConnectionClosed → SWContextManager::stopAllServiceWorkers()` stop the extension worker without touching the background view. - Compiled content-rule cache corruption They may alter timing or storage pressure, but should not be presented as reproduction scenarios without a corresponding failing handler sequence or production trace. -Safari 26.6 includes a fix for service-worker registration database files accumulating on launch, and WebKit has also fixed cold-launch cleanup that delayed extension content injection. These can widen or shrink startup timing windows, but neither source demonstrates the retained failed-background state. +Safari 26.6 includes a fix for service-worker registration database files accumulating on launch, and WebKit has also fixed cold-launch cleanup that delayed extension content injection. **[corrected]** The registration-database fix is irrelevant here: extension service workers are never persisted (`SWServerWorker::shouldPersistToDisk` returns false for service-worker-page registrations) and the registration is cleared whenever the SW page client goes away. ## Evidence needed next @@ -230,7 +234,7 @@ The next useful instrumentation should classify the failure before attempting mo - Inspect `WKWebExtensionContext.errors` for `WKWebExtensionContextErrorBackgroundContentFailedToLoad`. - Invoke `loadBackgroundContent(completionHandler:)` diagnostically after health detection. - A failure or completion that never arrives points to the background load/pending-action path. -- A successful background load while CPM messaging still fails points to tab/content-process routing. +- A successful background load while CPM messaging still fails points to tab/content-process routing **[corrected] — or to a terminated worker under a loaded view.** Discriminate by sending a probe `runtime.sendMessage` from a tab (or the native `healthCheckRequest`): `undefined` reply + empty `errors` + no `Tab not found` log = dead worker; `tab not found` error = routing. - Capture WebKit logs around “Tab not found for message for content script message.” - Correlate “Loading background content” with a missing “Background content loaded” and any registration error. @@ -461,7 +465,7 @@ This state ends only when something calls `unloadBackgroundWebView` or unloads/r - Explicit extension-context unload/reload - App process restart -The ordinary 30-second nonpersistent-background eviction does not recover the failed-first-load sequence. The wake-up call tries to schedule eviction before the background view is created, so it returns without creating a timer; the failure path never schedules the timer afterward. +**[corrected]** The 30-second eviction *does* recover this sequence. Only the first wake-up (view still nil) skips the timer; every later wake-up sees `m_backgroundWebView != nil` and arms it. Expected timeline: message 1 → load fails; message 2 → timer armed; +30 s → view unloaded; message 3 → fresh load. The debug simulation is one-shot (`WebExtensionManager.swift:648-652`), so it should recover on that schedule — verify empirically; persistence beyond that would mean shipping WebKit differs from trunk. Opening an ordinary new browser window does not itself unload or reload the shared extension context. Therefore: @@ -493,3 +497,77 @@ runtimeSendMessage This freezes CPM across all tabs without growing `m_actionsToPerformAfterBackgroundContentLoads`. It recovers only when another event loads the worker and repairs listener registration, or when the extension context is reloaded. On WebKit versions before `7682d9817b`, the same sequence can occur before the first successful background load because an empty persisted listener set was incorrectly considered authoritative. + + +## Ranked production trigger chains (added 2026-09-09) + +Ordered by estimated probability of producing the observed "CPM dead in every tab, reload doesn't help" state on macOS. Line refs: WebKit trunk `0c7e7ad97b`; details in [[cpm-web-extension-breakage-validation]]. + +### 1. Critical memory pressure terminates the "idle" extension worker — most probable + +Why first: needs no crash, no user action, no app bug; happens routinely on 8/16 GB Macs with many tabs; leaves `WKWebExtensionContext.errors` empty; matches "stuck, then recovered without reload" once the user pauses browsing for 30 s. macOS-only (`ENABLE(WEB_PROCESS_SUSPENSION_DELAY)`), which fits a macOS-only report. + +Chain: +1. CPM context loaded; background SW registered and activated. ≥10 s pass. The SW server marks the worker idle (`SWServerWorker::isIdle`, `SWServerWorker.cpp:341`) — extension `runtime.sendMessage` traffic goes over WebKit IPC and never calls `needsRunning()`, so the worker is *always* idle from the server's view. +2. System memory pressure → Critical. The WebProcess hosting the SW page reports `MemoryPressureStatusChanged` (`WebProcess.cpp:554`). +3. `WebProcessPool::memoryPressureStatusChangedForProcess`: `Critical && isRunningServiceWorkers()` → `NetworkProcess::TerminateIdleServiceWorkers(pid)` (`WebProcessPool.cpp:2937`). +4. `SWServer::terminateIdleServiceWorkers` terminates the CPM worker (`SWServer.cpp:1807`). +5. WebProcess: worker thread stops, `ServiceWorkerGlobalScope` destroyed, `Page::m_serviceWorkerGlobalScope` → null. SW page, frame, and `RuntimeOnMessage/Main` listener entry survive. +6. UIProcess `WebExtensionContext` unchanged: `m_backgroundWebView != nil`, `m_backgroundContentIsLoaded == true`. No error recorded, no delegate callback. +7. Any tab, any frame: `cpm.js` → `runtime.sendMessage(init)` → `runtimeSendMessage` → tab found → listener present → `wakeUpBackgroundContentIfNecessary` re-arms the 30 s eviction timer → `backgroundContentIsLoaded()` true → `DispatchRuntimeMessageEvent` to the SW page process. +8. `enumerateFramesAndNamespaceObjects` → `jsContextForServiceWorkerWorld` → null → frame skipped → null reply → `{}` → content script's `await sendMessage` resolves `undefined`. No `initResp`; `AutoConsent` stays in `waitingForInitResponse`. +9. Every navigation in every tab repeats 7–8 and keeps re-arming the timer (`all_frames: true` makes this continuous while browsing). +10. Exit: ≥30 s with zero extension events → `unloadBackgroundContentIfPossible` → view unloaded → next message reloads a fresh worker (`cpmMessagingRecoveredWithoutExtensionReload`); or Fire/extension reload; or relaunch. + +Repro: +1. Launch DDG, open one tab, wait ≥15 s. +2. `sudo memory_pressure -S -l critical` (simulate) or `memory_pressure -l critical` (real), hold ~5 s. Watch `log stream --predicate 'subsystem == "com.apple.WebKit" AND (eventMessage CONTAINS "terminateIdleServiceWorkers" OR eventMessage CONTAINS "memoryPressureStatusChangedForProcess")'`. +3. In Safari-style Web Inspector target list (Develop → DuckDuckGo) the "DuckDuckGo Embedded Extension — Extension Service Worker" target should disappear while the extension stays loaded. +4. Load any CMP page in any tab / new tab / new window → banner not handled; health monitor fires `cpmInitializationFailed` after 4 s, `cpmMessagingStuck` on the second navigation. `WKWebExtensionContext.errors` is empty; no "Tab not found" log line. +5. Stop navigating for ≥35 s, load a CMP page → works; `cpmMessagingRecoveredWithoutExtensionReload`. + +### 2. Network process crash / unresponsive termination + +Why second: less frequent than memory pressure, but every occurrence lands in the same dead-worker state; also fits an "after the network hiccuped, cookie banners stopped being handled" report. + +Chain: +1. `NetworkProcessProxy::didClose` (crash), `didBecomeUnresponsive` (UIProcess kills it), or `ExceededMemoryLimit`. +2. UIProcess: `WebProcessPool::networkProcessDidTerminate` → `terminateServiceWorkers()` → `WebProcessProxy::disableRemoteWorkers(ServiceWorker)` → `WebSWContextManagerConnection::Close` to the SW page process (`WebProcessPool.cpp:497-509, 1850`; `WebProcessProxy.cpp:3094`). Process survives because it owns the SW page. +3. WebProcess: `networkProcessConnectionClosed` → `SWContextManager::stopAllServiceWorkers()` (`WebProcess.cpp:1467`). +4. Steps 5–10 of chain 1. The in-memory registration is gone; nothing re-registers until the view is recreated. + +Repro: launch, wait 15 s, `kill -9` DuckDuckGo's `com.apple.WebKit.Networking` process (pick the one whose parent is the DDG PID), wait for the network process to relaunch, then step 4–5 of chain 1. + +### 3. Startup / session-restoration miss (transient, per-tab, very common) + +Not the stuck state, but the most frequent CPM miss and the one most likely to be *misread* as stuck. Amplified by an app-side detail: the context's `baseURL` is a random UUID per `WKWebExtensionContext` object and the app never sets it, so every load runs `_renameOrigin` and content injection waits on a Network-process round trip (`WebExtensionContextCocoa.mm:222, 297-311, 516-525`). + +Chain: launch → `setupWebExtensions()` → async `loadAndSync()` → restored tabs commit before `addInjectedContent` → no `document_start` → no CPM until reload. Recovery: any navigation after the extension finishes loading. + +Repro: 10+ restored tabs, quit, relaunch; several restored tabs have no CPM until reloaded. Fix: set `context.baseURL = webkit-extension:///` in `WebExtensionLoader.makeContext`. + +### 4. Tab/window routing: sender page not found + +The only chain that explains "one window broken, new window fine". `getCurrentTab` iterates `windowControllersManager.mainWindowControllers` → `loadedPinnedTabs + loadedTabs` → `Tab.webView`; a web view outside that set fails with `runtime.sendMessage(): tab not found` before any wake-up (`WebExtensionContextAPIRuntimeCocoa.mm:141-147`). Requires a `MainWindowController` missing from `mainWindowControllers` while its window is still open, or a `Tab` not in `loadedTabs`. Trigger not identified in source; needs the "Tab not found for message for content script message" log correlated with window registration. Probability unknown; ranked here because it matches a reported observation, not because a trigger is known. + +### 5. Repeatable registration failure (SW script unreadable) + +The retained-failed-view state becomes persistent only if each retry fails. Only repeatable cause found: `background-embedded.js` is read from disk on every SW load, uncached (`WebExtensionURLSchemeHandlerCocoa.mm:120`, `WebExtension::resourceDataForPath` default `CacheResult::No`). If the installed extension directory is removed/unreadable while loaded, every retry → `NSURLErrorFileDoesNotExist` → `completion(false)`. Content scripts (read at context load) keep running → exactly "content script talks, background silent". Unlike chains 1–2, `errors` contains `BackgroundContentFailedToLoad` on every retry. No in-app path deletes a loaded extension's files; external causes only. Low probability. + +Repro: with the extension loaded, `mv` the `public/js/background-embedded.js` inside `~/Library/Application Support/DuckDuckGo/…//`; wait for the worker to be evicted (30 s idle) or force via memory pressure; load a CMP page. + +### 6. Transient registration failure (retained failed view, self-healing) + +Debug-simulation shape. Any one-off `_loadServiceWorker` failure (Network-process disconnect mid-registration, etc.) retains the view, but the second message arms the eviction timer and the third message (≥30 s later) recovers. Visible as at most ~30 s of failure plus one extra navigation. Low impact. + +### 7. Listener-gating freeze + +`RuntimeOnMessage` absent from `m_backgroundContentEventListeners` after first load would drop every message before wake-up. Counts only decrement via JS `removeListener`, and the bundle never removes `onMessage`. No trigger found; theoretical. + +### Discriminating the three observable classes + +| Probe result after a stuck episode | Class | +|---|---| +| `errors == []`, probe `runtime.sendMessage` resolves `undefined`, no "Tab not found" log, SW inspector target gone | Dead worker under loaded view (chains 1, 2) | +| `errors` contains `BackgroundContentFailedToLoad`, repeating every ~30 s | Repeatable registration failure (chain 5) | +| Content script rejected with `tab not found`, log "Tab not found for message for content script message" | Routing (chain 4) |