99 lines
8.3 KiB
Markdown
99 lines
8.3 KiB
Markdown
---
|
||
title: CPM Web Extension failures — reproduction, fixes and mitigations
|
||
date: '2026-10-08'
|
||
status: draft-for-asana
|
||
tags:
|
||
- work
|
||
- cpm
|
||
- webkit
|
||
---
|
||
|
||
# CPM Web Extension failures — reproduction, fixes and mitigations
|
||
|
||
Companion note: [[cpm-web-extension-data-analysis-summary-2026-10-08]] (cohorts and theories). Detailed working notes (Russian): [[cpm-web-extension-repro-and-fix-2026-10-08]].
|
||
|
||
Status: nothing is committed or merged yet. Code lives in three local branches, listed at the end.
|
||
|
||
## Test harness
|
||
|
||
- XCTest suite against the **real system WebKit**. Tested on macOS 26.7.1.
|
||
- It loads the **production embedded extension** and drives the real `WebExtensionManager`, the lifecycle coordinator with the navigation gate, the CPM health monitor (4 s grace period), M4 recovery, and the production diagnostics recorder. Failures therefore produce the same pixel parameters as in production.
|
||
- Extension logic is unchanged. Faults are injected from outside:
|
||
- kill or freeze WebKit processes (`SIGKILL`, `SIGSTOP`);
|
||
- call WebKit SPI, such as terminating service workers;
|
||
- prepend delays to the extension scripts, either once at startup or only for one wake;
|
||
- fill the network cache;
|
||
- remove extension files.
|
||
- Opt-in, so it does not slow down CI: `CPM_REPRO=1 swift test --filter CPMRepro` in `SharedPackages/WebExtensions`. `CPM_REPRO_WEBKIT=1` adds tests that confirm the WebKit bugs are still present; they will fail once Apple ships fixes.
|
||
- The iOS Simulator is not representative: test processes get no RunningBoard assertions. Device checks used an iPhone 12 mini with iOS 26.7.1.
|
||
|
||
## Failure modes reproduced
|
||
|
||
1. **Lost first message on a new extension identifier** (first install or upgrade). WebKit loads the background eagerly, and an `init` sent before the background script has run is dropped. Without the gate, 14/20 messages were lost on install and 18/20 on upgrade. At launch the existing gate already prevents this; mid-session install or upgrade did not.
|
||
2. **Full failure after a Network process crash.** The background view and process survive, but the service worker is dead. Every tab loses CPM, including pages that were already open. M4 fixes new navigations only; pages opened before the crash stay broken. Reproduced 20 out of 20 times.
|
||
3. **Service worker dead while the Network process is alive.** Triggered by terminating service workers or removing their registrations. Same symptoms as above. The app cannot detect it: the registration still says "activated".
|
||
4. **Error 6.** A woken background that cannot finish loading within 30 s is closed by WebKit's idle-unload timer, which reports error 6. The test reproduces the production signature (`bg_load_error_after=25s_to_35s`). Not reproduced on the iPhone in four scenarios.
|
||
5. **Slow worker start on wake** (the group 1 signature).
|
||
- Start ≤ 2.5 s: messages are queued and delivered, nothing is lost.
|
||
- Start of 5 s: `initialization_failed` with a healthy snapshot and the production-like timeline tail, then the late answer arrives and the episode recovers without a reload.
|
||
- Start of 12 s plus a second navigation: `messaging_stuck`, then M4. M4 kills the slow but healthy worker, and the pending `init` messages of already-open pages are lost.
|
||
6. **Context-load cleanup delay** (WebKit 312231@main). With about 20–24k network-cache entries, content scripts ran 0.75–1.35 s late on relaunch and on reload, against 23 ms with an empty cache. That is under 4 s on a fast Mac; slow devices are still to be measured.
|
||
|
||
**Recovery findings:**
|
||
- Killing only the background WebContent process makes WebKit start a fresh worker. It fixes every tab, including already-open pages.
|
||
- An extension reload (M4) fixes new navigations only.
|
||
- Calling `loadBackgroundContent` does nothing for a dead worker.
|
||
|
||
**Readiness signals measured during a 5 s slow start:**
|
||
- The background view finishes its own page load after 0.1 s, so it says nothing about the worker.
|
||
- The `loadBackgroundContent` completion and the worker's first native message both arrive right after the worker's startup code has run. Both are accurate readiness signals.
|
||
|
||
## Fixes (app side, branch `alex/cpm-web-extension-ready-handshake`)
|
||
|
||
1. **Gate mid-session install and upgrade.** After a new embedded extension is installed, navigations wait (up to 5 s) for its first background load, as they already do at launch.
|
||
2. **Network process watchdog.** Records the Network process PID of the background. If it changes, the watchdog restarts only the background WebContent process. Checked on every navigation and every second.
|
||
3. **Restart the background on the first failure**, but only if the background process responds to a script call within 1 s. Killing a frozen process left the replacement worker deaf.
|
||
4. Items 2 and 3 share one switch, which is on by default and not yet tied to a remote flag.
|
||
|
||
**Results, without fix → with fix:**
|
||
- Mid-session install: 3/3 lost → 5/5 delivered.
|
||
- Mid-session upgrade: 5/5 lost → 5/5 delivered.
|
||
- Network process crash with 4 tabs and open pages: all broken → all working, including already-open pages and with M4 disabled.
|
||
- Worker death without a Network process crash: two failures, then M4, and old pages stay broken → one lost navigation, then everything works without a reload.
|
||
|
||
**Known gap:** fix 3 can hurt a slow wake. If the slow worker does not keep the CPU busy (for example it waits on I/O), the process still answers, gets restarted, and the late answer turns into a lost one. Restarts should be held back while the background is still starting. The phase 2 detector below covers this.
|
||
|
||
## Telemetry (branch `alex/cpm-messaging-readiness-telemetry`, phase 1)
|
||
|
||
- Telemetry only; the detector's behavior is unchanged. Ship it before the fixes, or together with them in an A/B experiment.
|
||
- **On failure pixels:** `bg_ready_after`, `bg_view_age`, `app_state`, `open_tabs`, `loading_tabs`, and the error 6 fields `bg_load_error_app_state` and `bg_load_error_after`.
|
||
- **On recovery pixels:** `recovery_document` (failed navigation answered late, later navigation, or unattributed) and `recovery_after`.
|
||
- **Readiness source:** the app calls `loadBackgroundContent` once, when WebKit creates a background view. The view already exists, so the call joins the load in progress and wakes nothing. It is behind the same remote flag as the existing background delegate proxy.
|
||
- Verified against real WebKit with a 5 s slow wake: `bg_ready_after=pending`, `bg_view_age=4s_to_8s`, then `recovery_document=failed_navigation`, `recovery_after=under_4s`.
|
||
- 677 package tests pass. SwiftLint is clean. Pixel definitions validate on iOS and macOS. Both apps build.
|
||
|
||
## Mitigations still to decide
|
||
|
||
- **Detector phase 2.**
|
||
- Start the 4 s grace period only once the background is ready.
|
||
- Report "background did not start within 30 s" as its own pixel.
|
||
- Escalate to stuck, M4 or a restart only for failures with a ready background.
|
||
- This removes slow-start false positives.
|
||
- **Wake the background early** when a navigation starts or the app comes to the foreground, without blocking the navigation. Wake-up time then overlaps with the page load.
|
||
- **Stable base URL** for embedded extensions (branch `alex/cpm-stable-base-url`). Removes the cleanup on every launch and reload.
|
||
- Privacy check: no embedded extension declares `web_accessible_resources`, and pages cannot see the URL.
|
||
- Still to verify: M4 and crash recovery with a stable URL, and the upgrade path.
|
||
- **Error 6:** once data shows the app state, restart the background when the app returns to the foreground if error 6 was recorded.
|
||
- **Extension side:**
|
||
- a retry handshake for `init`, covering lost and late messages whatever the cause;
|
||
- a ping or heartbeat channel, which would allow zero-loss detection of a dead worker.
|
||
- **Feature flags and A/B:** put the watchdog and the restart behind remote flags and expose the cohort in CPM pixels. Comparing release versions proved inconclusive.
|
||
- **Report to WebKit:** the dead worker after a Network process crash, and the 30 s unload timer that starts before the load finishes.
|
||
|
||
## Branches (local, uncommitted)
|
||
|
||
- `alex/cpm-web-extension-ready-handshake`: the fixes and the main test harness.
|
||
- `alex/cpm-web-extension-remaining-failures`: production code without fixes; the error 6 and slow-wake reproductions.
|
||
- `alex/cpm-messaging-readiness-telemetry`: phase 1 telemetry, based on current `main`.
|
||
- `alex/cpm-stable-base-url`: the stable base URL experiment.
|