CPM Web Extension failures — reproduction, fixes and mitigations

Companion note: cpm-web-extension-data-analysis-summary-2026-10-08 (cohorts and theories). Detailed working notes (Russian): cpm-web-extension-repro-and-fix-2026-10-08.

Status: nothing is committed or merged yet. Code lives in three local branches, listed at the end.

Test harness

XCTest suite against the real system WebKit. Tested on macOS 26.7.1.
It loads the production embedded extension and drives the real WebExtensionManager, the lifecycle coordinator with the navigation gate, the CPM health monitor (4 s grace period), M4 recovery, and the production diagnostics recorder. Failures therefore produce the same pixel parameters as in production.
Extension logic is unchanged. Faults are injected from outside:
- kill or freeze WebKit processes (SIGKILL, SIGSTOP);
- call WebKit SPI, such as terminating service workers;
- prepend delays to the extension scripts, either once at startup or only for one wake;
- fill the network cache;
- remove extension files.
Opt-in, so it does not slow down CI: CPM_REPRO=1 swift test --filter CPMRepro in SharedPackages/WebExtensions. CPM_REPRO_WEBKIT=1 adds tests that confirm the WebKit bugs are still present; they will fail once Apple ships fixes.
The iOS Simulator is not representative: test processes get no RunningBoard assertions. Device checks used an iPhone 12 mini with iOS 26.7.1.

Failure modes reproduced

Lost first message on a new extension identifier (first install or upgrade). WebKit loads the background eagerly, and an init sent before the background script has run is dropped. Without the gate, 14/20 messages were lost on install and 18/20 on upgrade. At launch the existing gate already prevents this; mid-session install or upgrade did not.
Full failure after a Network process crash. The background view and process survive, but the service worker is dead. Every tab loses CPM, including pages that were already open. M4 fixes new navigations only; pages opened before the crash stay broken. Reproduced 20 out of 20 times.
Service worker dead while the Network process is alive. Triggered by terminating service workers or removing their registrations. Same symptoms as above. The app cannot detect it: the registration still says "activated".
Error 6. A woken background that cannot finish loading within 30 s is closed by WebKit's idle-unload timer, which reports error 6. The test reproduces the production signature (bg_load_error_after=25s_to_35s). Not reproduced on the iPhone in four scenarios.
Slow worker start on wake (the group 1 signature).
- Start ≤ 2.5 s: messages are queued and delivered, nothing is lost.
- Start of 5 s: initialization_failed with a healthy snapshot and the production-like timeline tail, then the late answer arrives and the episode recovers without a reload.
- Start of 12 s plus a second navigation: messaging_stuck, then M4. M4 kills the slow but healthy worker, and the pending init messages of already-open pages are lost.
Context-load cleanup delay (WebKit 312231@main). With about 20–24k network-cache entries, content scripts ran 0.75–1.35 s late on relaunch and on reload, against 23 ms with an empty cache. That is under 4 s on a fast Mac; slow devices are still to be measured.

Recovery findings:

Killing only the background WebContent process makes WebKit start a fresh worker. It fixes every tab, including already-open pages.
An extension reload (M4) fixes new navigations only.
Calling loadBackgroundContent does nothing for a dead worker.

Readiness signals measured during a 5 s slow start:

The background view finishes its own page load after 0.1 s, so it says nothing about the worker.
The loadBackgroundContent completion and the worker's first native message both arrive right after the worker's startup code has run. Both are accurate readiness signals.

Fixes (app side, branch `alex/cpm-web-extension-ready-handshake`)

Gate mid-session install and upgrade. After a new embedded extension is installed, navigations wait (up to 5 s) for its first background load, as they already do at launch.
Network process watchdog. Records the Network process PID of the background. If it changes, the watchdog restarts only the background WebContent process. Checked on every navigation and every second.
Restart the background on the first failure, but only if the background process responds to a script call within 1 s. Killing a frozen process left the replacement worker deaf.
Items 2 and 3 share one switch, which is on by default and not yet tied to a remote flag.

Results, without fix → with fix:

Mid-session install: 3/3 lost → 5/5 delivered.
Mid-session upgrade: 5/5 lost → 5/5 delivered.
Network process crash with 4 tabs and open pages: all broken → all working, including already-open pages and with M4 disabled.
Worker death without a Network process crash: two failures, then M4, and old pages stay broken → one lost navigation, then everything works without a reload.

Known gap: fix 3 can hurt a slow wake. If the slow worker does not keep the CPU busy (for example it waits on I/O), the process still answers, gets restarted, and the late answer turns into a lost one. Restarts should be held back while the background is still starting. The phase 2 detector below covers this.

Telemetry (branch `alex/cpm-messaging-readiness-telemetry`, phase 1)

Telemetry only; the detector's behavior is unchanged. Ship it before the fixes, or together with them in an A/B experiment.
On failure pixels: bg_ready_after, bg_view_age, app_state, open_tabs, loading_tabs, and the error 6 fields bg_load_error_app_state and bg_load_error_after.
On recovery pixels: recovery_document (failed navigation answered late, later navigation, or unattributed) and recovery_after.
Readiness source: the app calls loadBackgroundContent once, when WebKit creates a background view. The view already exists, so the call joins the load in progress and wakes nothing. It is behind the same remote flag as the existing background delegate proxy.
Verified against real WebKit with a 5 s slow wake: bg_ready_after=pending, bg_view_age=4s_to_8s, then recovery_document=failed_navigation, recovery_after=under_4s.
677 package tests pass. SwiftLint is clean. Pixel definitions validate on iOS and macOS. Both apps build.

Mitigations still to decide

Detector phase 2.
- Start the 4 s grace period only once the background is ready.
- Report "background did not start within 30 s" as its own pixel.
- Escalate to stuck, M4 or a restart only for failures with a ready background.
- This removes slow-start false positives.
Wake the background early when a navigation starts or the app comes to the foreground, without blocking the navigation. Wake-up time then overlaps with the page load.
Stable base URL for embedded extensions (branch alex/cpm-stable-base-url). Removes the cleanup on every launch and reload.
- Privacy check: no embedded extension declares web_accessible_resources, and pages cannot see the URL.
- Still to verify: M4 and crash recovery with a stable URL, and the upgrade path.
Error 6: once data shows the app state, restart the background when the app returns to the foreground if error 6 was recorded.
Extension side:
- a retry handshake for init, covering lost and late messages whatever the cause;
- a ping or heartbeat channel, which would allow zero-loss detection of a dead worker.
Feature flags and A/B: put the watchdog and the restart behind remote flags and expose the cohort in CPM pixels. Comparing release versions proved inconclusive.
Report to WebKit: the dead worker after a Network process crash, and the 30 s unload timer that starts before the load finishes.

Branches (local, uncommitted)

alex/cpm-web-extension-ready-handshake: the fixes and the main test harness.
alex/cpm-web-extension-remaining-failures: production code without fixes; the error 6 and slow-wake reproductions.
alex/cpm-messaging-readiness-telemetry: phase 1 telemetry, based on current main.
alex/cpm-stable-base-url: the stable base URL experiment.

8.3 KiB Raw Blame History Unescape Escape