[2026-10-09] eagle: work/projects/cpm-web-extension-data-analysis-summary-2026-10-08.md work/projects/cpm-web-extension-repro-and-fixes-summary-2026-10-08.md
This commit is contained in:
Alexey Martemyanov committed
1 parent
7f45813b83
commit
3f95226bc4
2 files changed
+172
-149
No files matched your search
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: CPM Web Extension failures — data analysis summary
|
||||
title: CPM Web Extension failures — what the data shows
|
||||
date: '2026-10-08'
|
||||
status: draft-for-asana
|
||||
tags:
|
||||
@@ -8,106 +8,123 @@ tags:
|
||||
- webkit
|
||||
---
|
||||
|
||||
# CPM Web Extension failures — data analysis summary
|
||||
# CPM Web Extension failures: what the data shows
|
||||
|
||||
Companion note: [[cpm-web-extension-repro-and-fixes-summary-2026-10-08]] (tests and fixes). Detailed working notes (Russian): [[cpm-web-extension-repro-and-fix-2026-10-08]], [[cpm-web-extension-breakage-session-summary-2026-10-07]].
|
||||
## Context
|
||||
|
||||
## TL;DR
|
||||
Cookie Pop-up Management (CPM) runs in the iOS and macOS browsers as a built-in web extension. It has two parts:
|
||||
- a **content script** inside each web page;
|
||||
- a **background service worker**, the extension's shared "brain". WebKit starts it on demand and unloads it after about 30 seconds of inactivity.
|
||||
|
||||
- Most CPM "failures" are not a dead extension. The failure pixels mean that within 4 s after a page finished loading, the app received no `refreshCpmDashboardState` call from the extension. A late answer counts as a failure too.
|
||||
- The largest cohort, a failure while the app-side state looks healthy, mostly happens right after WebKit wakes the extension's background after idle. A test reproduces it exactly when the background service worker takes longer than 4 s to start. Many of these failures are late responses, not lost ones.
|
||||
- The second-largest cohort on iOS is WebKit error 6, where the background fails to load. Its root cause is in WebKit: the 30 s idle-unload timer starts before the background has finished loading.
|
||||
- A Network process crash explains only 20–45% of the "healthy" stuck cases. It is the only mechanism that breaks every tab at once, including pages that were already open.
|
||||
- New telemetry (phase 1) separates "slow start" from "lost message" and "late" from "lost". It is ready and should ship before or together with the fixes.
|
||||
On every page load, the content script asks the background worker for its state. The worker then reports to the app.
|
||||
|
||||
## What the pixels measure
|
||||
The app watches this exchange with a health monitor:
|
||||
- If no report arrives within **4 seconds** after a page finishes loading, the app sends an **"initialization failed"** pixel.
|
||||
- If a later page load in the same episode also fails, the app sends a **"messaging stuck"** pixel and reloads the extension to recover.
|
||||
- When answers resume, the app sends a **"recovered"** pixel.
|
||||
- Failure pixels carry a snapshot of the extension's state: whether it is loaded, whether its background is alive, recent background lifecycle events, and WebKit errors.
|
||||
|
||||
- `initialization_failed_after_<reason>`: the first navigation with no CPM answer within 4 s. Sent once per day per reason.
|
||||
- `messaging_stuck_<reason>`: a second failure in the same episode, from a navigation that started after the first failure. This triggers recovery M4, a reload of the extension.
|
||||
- `recovered_without_extension_reload` and `recovered_after_extension_reload`: the episode closed because an answer finally arrived.
|
||||
- Each failure pixel carries a snapshot of app state: whether the context is loaded, background view and process state, tab binding, context errors, and `bg_events`, a short timeline of background lifecycle events.
|
||||
Important: a failure pixel means "no answer within 4 seconds". It does not mean the extension is broken. A late answer also counts as a failure.
|
||||
|
||||
## Scope
|
||||
## Key findings
|
||||
|
||||
- One UTC day, October 5, 2026. Only events after the release tag of iOS 7.240 and macOS 1.210.
|
||||
- First `initialization_failed`, daily: 6,624 on iOS, 1,377 on macOS.
|
||||
- `messaging_stuck`, daily: 1,440 on iOS, 487 on macOS.
|
||||
- Daily pixels count installs per day, not users or episodes.
|
||||
1. **Most failures happen right after WebKit wakes the background worker** following idle unload, while the app-side state looks healthy. A test reproduces this exact pattern when the worker takes longer than 4 seconds to start. Many of these failures are late answers rather than lost ones.
|
||||
2. **About a third of iOS "stuck" reports carry WebKit error 6, "background failed to load".** The root cause is in WebKit: the 30-second idle-unload timer starts before the background has finished loading. A load slower than 30 seconds is cancelled.
|
||||
3. **A crash of WebKit's networking process explains only a minority (20–45%) of the cases that look healthy.** It is the only cause we found that breaks every tab at once, including pages that were already open.
|
||||
4. **Phase 1 telemetry, ready but not shipped, separates "slow start" from "lost message" and "late answer" from "no answer".** It should ship before or together with any fix, so the effect of fixes can be measured.
|
||||
|
||||
## Cohorts of stuck reports
|
||||
## Data used
|
||||
|
||||
These are snapshot states at the time of the timeout, not root causes. Each report is assigned to one cohort, by priority.
|
||||
- One full day, October 5, 2026 (UTC), on iOS 7.240 and macOS 1.210.
|
||||
- The first daily "initialization failed" pixel: 6,624 iOS installs and 1,377 macOS installs.
|
||||
- The daily "messaging stuck" pixel: 1,440 iOS installs and 487 macOS installs.
|
||||
- Daily pixels fire at most once per install per day, so these are install-days, not users or episodes.
|
||||
|
||||
1. **Healthy snapshot, no visible error.** iOS 1,410, macOS 705 (stuck counts); about 55% of iOS stuck daily.
|
||||
- The Network process had changed since the context loaded in 17–21% of iOS and 30–45% of macOS reports.
|
||||
- The rest is called **"group 1"** below: iOS 748, macOS 228 (daily).
|
||||
2. **WebKit error 6** (`BackgroundContentFailedToLoad`). iOS 579, about a third of iOS stuck reports; macOS 3.
|
||||
3. **No background view.** iOS 158, macOS 35.
|
||||
4. **Tab not attached to the extension.** Wrong controller or missing content scripts. iOS 81, macOS 102. On macOS these appear only after the 1.210 release tag.
|
||||
5. **Background process died** (`died_*`). iOS 14; 86% of these also had a Network process change, which looks like iOS killing both processes at once.
|
||||
## Failure cohorts
|
||||
|
||||
## Group 1: the largest unexplained cohort
|
||||
"Stuck" reports split by the state the app recorded at the time of failure. This describes symptoms, not root causes.
|
||||
|
||||
- **About half follow a background wake.** The timeline tail is "old view deallocated, new view created", with the new view under 30 s old: iOS 373 of 748, macOS 120 of 228.
|
||||
- **About a quarter on iOS and 15% on macOS follow a context load or reload**: `load → view` or `base_url_changed → view`.
|
||||
- On macOS, 63% of reports were sent shortly after an app update, against 52% across all stuck reports.
|
||||
- Memory pressure and context errors are almost always absent.
|
||||
- Both macOS 26.7 and 27.0 are affected. The iOS OS-version split has not been queried yet.
|
||||
- **Looks healthy (no visible error): the largest cohort, about 55% of iOS stuck reports.**
|
||||
- In 17–21% of iOS and 30–45% of macOS reports, the networking process had been replaced since the extension loaded.
|
||||
- The rest has no visible cause, which is 748 iOS and 228 macOS daily reports. This is the main open question, covered below.
|
||||
- **WebKit error 6, background failed to load:** about a third of iOS stuck reports; almost none on macOS.
|
||||
- **No background at all:** 158 iOS and 35 macOS reports.
|
||||
- **Tab not connected to the extension** (the tab has the wrong extension controller or no content scripts): 81 iOS and 102 macOS reports. On macOS this appeared only with version 1.210.
|
||||
- **Background process died:** 14 iOS reports, almost always together with a networking process restart. This looks like iOS killing both processes under memory pressure.
|
||||
|
||||
**Recoveries:**
|
||||
- An `initialization_failed` episode recovers without a reload on 23% of iOS and 25% of macOS device-days. Counting only "other" navigations, the figure is 33% on both platforms. Recoveries repeat on the same devices, about 1.9 per device-day on iOS.
|
||||
- After M4, a stuck episode recovers on 64% of iOS and 74% of macOS device-days.
|
||||
- Recovery pixels do not yet say whether the answer was late or came from a later page. Phase 1 telemetry adds this.
|
||||
## The largest unexplained cohort: healthy-looking failures
|
||||
|
||||
**Init failures compared with stuck reports, iOS healthy snapshots:**
|
||||
- After a context load: 32% of init failures, 18% of stuck reports.
|
||||
- After a wake: 44% of init failures, 61% of stuck reports.
|
||||
- So failures after a context load usually clear on their own, while failures on wake more often escalate to stuck.
|
||||
- **About half come right after the background was woken up again after idle unload.** The background had been created less than 30 seconds earlier: 373 of 748 on iOS, 120 of 228 on macOS.
|
||||
- **A quarter on iOS and 15% on macOS come right after the extension was loaded or reloaded.**
|
||||
- On macOS, 63% of these reports come shortly after an app update, against 52% across all stuck reports.
|
||||
- Memory pressure and WebKit errors are almost never present.
|
||||
- Both macOS 26.7 and 27.0 are affected. The iOS version split has not been checked yet.
|
||||
|
||||
## Theories and status
|
||||
**How often failures recover:**
|
||||
- After a single failure, answers resume without a reload on about a quarter of install-days (23% iOS, 25% macOS). The same devices recover repeatedly, about twice a day on iOS.
|
||||
- After a "stuck" episode and the extension reload, answers resume on 64% of iOS and 74% of macOS install-days.
|
||||
- The recovery pixels cannot yet tell a late answer from a later page that worked. Phase 1 telemetry adds this.
|
||||
|
||||
- **Network process crash leaves a dead service worker.** Confirmed by test. Explains at most 20–45% of healthy stuck reports. Fixed in the app; fix not merged.
|
||||
- **First-load message drop**, WebKit 317981-like. Confirmed by test. At launch the existing gate already covers it; mid-session install or upgrade was a gap, now fixed in the app (not merged).
|
||||
- **Service worker dies while the Network process stays alive.** Confirmed by test, but the app cannot detect it. Mitigation: restart the background after the first failure.
|
||||
- **Error 6.** Root cause found in WebKit source and reproduced on macOS. Not reproduced on an iPhone with iOS 26.7.1. Telemetry added.
|
||||
- **Slow service-worker start on wake.** A start longer than 4 s reproduces the group 1 signature exactly. Why real devices are that slow is unknown; candidates are CPU contention with the loading tab, process launch, the extension process's priority, and a busy Network process. Telemetry added.
|
||||
- **Context-load cleanup when the base URL changes** (WebKit 312231@main, bug 320812). Reproduced on macOS: with about 20–24k network-cache entries, content scripts are delayed by 0.75–1.35 s. Explains part of the "after load or reload" subset. Possible mitigation: give the extension a stable base URL.
|
||||
- **Ruled out:** device lock and data protection, a stale iOS tab URL, a macOS feature-flag flip explaining the tab-binding cohort, the low-memory notification, background freezes of 10–75 s as a cascade trigger, and killing the background WebContent process.
|
||||
- **Not reproduced yet:**
|
||||
- the macOS tab-binding cohort;
|
||||
- the "no background view" cohort;
|
||||
- error code 7;
|
||||
**Single failures and stuck episodes have different patterns:**
|
||||
- Failures right after an extension load make up 32% of single failures but only 18% of stuck reports.
|
||||
- Failures right after a wake-up make up 44% of single failures but 61% of stuck reports.
|
||||
- So failures after a load usually clear on their own, while failures after a wake-up more often turn into "stuck".
|
||||
|
||||
## Theories and their status
|
||||
|
||||
- **Networking process crash leaves the background worker dead.** Reproduced in tests. Explains at most 20–45% of healthy-looking stuck reports. App fix written, not merged.
|
||||
- **The first message is lost when a new extension version loads.** Reproduced in tests. At launch the app already waits for the extension, so this is covered. Installing or updating the extension mid-session was not covered; app fix written, not merged.
|
||||
- **The background worker dies while the networking process stays alive.** Reproduced in tests. The app cannot detect it directly, so it can only react after the first failure.
|
||||
- **Error 6.** The cause was found in WebKit source code and reproduced on macOS. It did not reproduce on an iPhone with iOS 26.7.1 in four scenarios. Telemetry has been added to tell which conditions trigger it.
|
||||
- **The worker starts slowly on wake-up.** A start longer than 4 seconds reproduces the healthy-looking pattern exactly. Why real devices would be that slow is still unknown. Candidates:
|
||||
- CPU contention with the page that is loading;
|
||||
- process startup cost;
|
||||
- the extension process running at low priority;
|
||||
- a busy networking process.
|
||||
|
||||
Telemetry has been added for this.
|
||||
- **WebKit cleans up website data every time the extension loads.** WebKit change 312231@main, shipped in OS 26.6 and later, does this when an extension's internal base URL changes, which for us happens on every launch and every reload. It delays content scripts and the background start. On a Mac with a large browser cache, the delay was 0.75–1.35 seconds. This explains part of the "right after load" failures. A possible mitigation is to give the extension a stable base URL.
|
||||
- **Ruled out:**
|
||||
- device lock and iOS data protection;
|
||||
- an outdated tab URL on iOS;
|
||||
- a macOS feature flag toggling (as the cause of tabs not connected to the extension);
|
||||
- low-memory notifications;
|
||||
- long freezes of the background as a cause of all-tab failures.
|
||||
- **Not explained yet:**
|
||||
- tabs not connected to the extension on macOS;
|
||||
- the "no background" cohort;
|
||||
- WebKit error code 7;
|
||||
- Lockdown Mode;
|
||||
- the first probe after M4 failing (670 on iOS, 232 on macOS).
|
||||
- the first page after an extension reload still failing (670 iOS and 232 macOS daily reports).
|
||||
|
||||
## Relevant WebKit bugs
|
||||
|
||||
- **317981**: a message sent while the background is still loading is dropped. Reproduced on macOS 26.7.1 for the first load of a new extension identifier. Fixed in the `safari-7625` branch.
|
||||
- **324743**: a service worker is treated as loaded before its registration completes. Fixed September 22, not in 26.7 or 27.0. Fits the wake path; no deterministic repro.
|
||||
- **320812, introduced by 312231@main** (26.6+): when the base URL changes, context load deletes all website data of the previous origin, including the whole disk cache, before it injects content or loads the background. The DDG extension gets a new base URL on every launch and every reload. Fixed by 318756@main and cherry-picked to `safari-7625`; which OS build ships it is unverified.
|
||||
- **314194** (open): Lockdown Mode blocks every service worker, so CPM cannot work. The share of affected users is unknown.
|
||||
- **322881** (clients lost after reload) and **324754** (reloaded worker without extension bindings under site isolation): low relevance to CPM, unproven.
|
||||
- **319976** (fixed): `loadBackgroundContent` completion called twice on error. We use this API.
|
||||
- **To file:**
|
||||
1. A dead worker after a Network process crash goes unnoticed by `WebExtensionContext`.
|
||||
2. The 30 s unload timer is started before the background finishes loading, which causes error 6.
|
||||
- [Bug 317981](https://bugs.webkit.org/show_bug.cgi?id=317981): a message sent while the background is still starting is dropped. Fixed upstream, but still present on macOS 26.7.1, where we reproduced it for the first load of a new extension version.
|
||||
- [Bug 324743](https://bugs.webkit.org/show_bug.cgi?id=324743): a background worker is treated as ready before it has finished starting. Fixed upstream on September 22; not in 26.7 or 27.0. It fits the wake-up pattern, but we have no deterministic reproduction.
|
||||
- [Bug 320812](https://bugs.webkit.org/show_bug.cgi?id=320812): "Web Extensions load with significant delay on iOS 26.6 when Safari is cold launched". This is the slow website-data cleanup described above. Fixed upstream; which OS release ships the fix is unverified.
|
||||
- [Bug 314194](https://bugs.webkit.org/show_bug.cgi?id=314194), still open: Lockdown Mode blocks all service workers, so CPM cannot work for those users. We do not know how many users this affects.
|
||||
- [Bug 322881](https://bugs.webkit.org/show_bug.cgi?id=322881) and [bug 324754](https://bugs.webkit.org/show_bug.cgi?id=324754): issues after a background reload. Probably not relevant to CPM, unproven.
|
||||
- [Bug 319976](https://bugs.webkit.org/show_bug.cgi?id=319976), fixed: a WebKit callback could fire twice on error. Relevant because the app uses this API.
|
||||
- **Not yet reported to WebKit:**
|
||||
1. after a networking process crash, WebKit does not notice that the background worker is dead;
|
||||
2. the idle-unload timer starts before the background finishes loading, which causes error 6.
|
||||
|
||||
## Telemetry added (phase 1, not merged)
|
||||
## Telemetry added (phase 1, not shipped yet)
|
||||
|
||||
All values are buckets or enums, with no URLs or identifiers.
|
||||
All values are coarse buckets or fixed labels, with no URLs or identifiers.
|
||||
|
||||
- **On failure pixels:**
|
||||
- `bg_ready_after`: whether the background had finished loading at the time of the failure, and how long the load took;
|
||||
- `bg_view_age`: how long ago the current background view was created;
|
||||
- `app_state`;
|
||||
- `open_tabs` and `loading_tabs`;
|
||||
- `bg_load_error_app_state` and `bg_load_error_after`: app state and timing for error 6.
|
||||
- whether the background had finished starting, and how long it took (`bg_ready_after`);
|
||||
- how long ago the background was created (`bg_view_age`);
|
||||
- whether the app was in the foreground (`app_state`);
|
||||
- how many tabs were open and how many were loading (`open_tabs`, `loading_tabs`);
|
||||
- for error 6, the app state and the time until the error (`bg_load_error_app_state`, `bg_load_error_after`).
|
||||
- **On recovery pixels:**
|
||||
- `recovery_document`: whether the failed navigation answered late, a later navigation answered, or the response matched no single tab;
|
||||
- `recovery_after`: time from the episode's first failure to the recovery.
|
||||
- whether the failed page answered late or a later page answered (`recovery_document`);
|
||||
- how long after the first failure answers resumed (`recovery_after`).
|
||||
|
||||
**How to read the results:**
|
||||
- `bg_ready_after=pending` with a young view means a slow wake.
|
||||
- `bg_ready_after` loaded with no answer means a lost message.
|
||||
- `recovery_document=failed_navigation` means a late answer, not a loss.
|
||||
- For error 6, `bg_load_error_after=25s_to_35s` points at WebKit's unload timer.
|
||||
**How to read it:**
|
||||
- "Background not ready" together with "created a few seconds ago" means a slow wake-up.
|
||||
- "Background ready" with no answer means a lost message.
|
||||
- A recovery on the failed page means the answer was late, not lost.
|
||||
- An error 6 about 30 seconds after the background was created points at WebKit's unload timer.
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: CPM Web Extension failures — reproduction, fixes and mitigations
|
||||
title: CPM Web Extension failures — reproduction and fixes
|
||||
date: '2026-10-08'
|
||||
status: draft-for-asana
|
||||
tags:
|
||||
@@ -8,91 +8,97 @@ tags:
|
||||
- webkit
|
||||
---
|
||||
|
||||
# CPM Web Extension failures — reproduction, fixes and mitigations
|
||||
# CPM Web Extension failures: reproduction and fixes
|
||||
|
||||
Companion note: [[cpm-web-extension-data-analysis-summary-2026-10-08]] (cohorts and theories). Detailed working notes (Russian): [[cpm-web-extension-repro-and-fix-2026-10-08]].
|
||||
## Context
|
||||
|
||||
Status: nothing is committed or merged yet. Code lives in three local branches, listed at the end.
|
||||
Cookie Pop-up Management (CPM) runs in the iOS and macOS browsers as a built-in web extension: a content script in every page plus a shared background service worker that WebKit starts on demand and unloads after about 30 seconds of inactivity.
|
||||
|
||||
## Test harness
|
||||
The app's health monitor reports a failure when a page gets no CPM answer within 4 seconds. A second failure marks the episode as "stuck", and the app then reloads the extension to recover. The companion summary of the production data explains which failure patterns we see and how often.
|
||||
|
||||
- XCTest suite against the **real system WebKit**. Tested on macOS 26.7.1.
|
||||
- It loads the **production embedded extension** and drives the real `WebExtensionManager`, the lifecycle coordinator with the navigation gate, the CPM health monitor (4 s grace period), M4 recovery, and the production diagnostics recorder. Failures therefore produce the same pixel parameters as in production.
|
||||
- Extension logic is unchanged. Faults are injected from outside:
|
||||
- kill or freeze WebKit processes (`SIGKILL`, `SIGSTOP`);
|
||||
- call WebKit SPI, such as terminating service workers;
|
||||
- prepend delays to the extension scripts, either once at startup or only for one wake;
|
||||
- fill the network cache;
|
||||
- remove extension files.
|
||||
- Opt-in, so it does not slow down CI: `CPM_REPRO=1 swift test --filter CPMRepro` in `SharedPackages/WebExtensions`. `CPM_REPRO_WEBKIT=1` adds tests that confirm the WebKit bugs are still present; they will fail once Apple ships fixes.
|
||||
- The iOS Simulator is not representative: test processes get no RunningBoard assertions. Device checks used an iPhone 12 mini with iOS 26.7.1.
|
||||
**Status:** all code described here is still local. None of it is in review or merged yet.
|
||||
|
||||
## Failure modes reproduced
|
||||
## How we reproduced the failures
|
||||
|
||||
1. **Lost first message on a new extension identifier** (first install or upgrade). WebKit loads the background eagerly, and an `init` sent before the background script has run is dropped. Without the gate, 14/20 messages were lost on install and 18/20 on upgrade. At launch the existing gate already prevents this; mid-session install or upgrade did not.
|
||||
2. **Full failure after a Network process crash.** The background view and process survive, but the service worker is dead. Every tab loses CPM, including pages that were already open. M4 fixes new navigations only; pages opened before the crash stay broken. Reproduced 20 out of 20 times.
|
||||
3. **Service worker dead while the Network process is alive.** Triggered by terminating service workers or removing their registrations. Same symptoms as above. The app cannot detect it: the registration still says "activated".
|
||||
4. **Error 6.** A woken background that cannot finish loading within 30 s is closed by WebKit's idle-unload timer, which reports error 6. The test reproduces the production signature (`bg_load_error_after=25s_to_35s`). Not reproduced on the iPhone in four scenarios.
|
||||
5. **Slow worker start on wake** (the group 1 signature).
|
||||
- Start ≤ 2.5 s: messages are queued and delivered, nothing is lost.
|
||||
- Start of 5 s: `initialization_failed` with a healthy snapshot and the production-like timeline tail, then the late answer arrives and the episode recovers without a reload.
|
||||
- Start of 12 s plus a second navigation: `messaging_stuck`, then M4. M4 kills the slow but healthy worker, and the pending `init` messages of already-open pages are lost.
|
||||
6. **Context-load cleanup delay** (WebKit 312231@main). With about 20–24k network-cache entries, content scripts ran 0.75–1.35 s late on relaunch and on reload, against 23 ms with an empty cache. That is under 4 s on a fast Mac; slow devices are still to be measured.
|
||||
- We built an automated test suite that runs the **real WebKit framework on macOS 26.7.1** and the **production CPM extension**. It drives the app's real extension manager, startup logic, health monitor and recovery.
|
||||
- Failures in the tests therefore produce the same diagnostic pixels as in production, which lets us compare them with real reports.
|
||||
- The extension itself is unchanged. Problems are injected from outside:
|
||||
- crashing or freezing WebKit's processes;
|
||||
- calling private WebKit testing API, for example to stop service workers;
|
||||
- adding a startup delay to the extension, once or for a single wake-up;
|
||||
- filling the browser cache;
|
||||
- removing extension files.
|
||||
- The suite is opt-in, so it does not slow down regular CI.
|
||||
- The iOS Simulator turned out to be unsuitable: test processes there do not get the system's background-execution grants. Device checks were done on an iPhone 12 mini with iOS 26.7.1.
|
||||
|
||||
**Recovery findings:**
|
||||
- Killing only the background WebContent process makes WebKit start a fresh worker. It fixes every tab, including already-open pages.
|
||||
- An extension reload (M4) fixes new navigations only.
|
||||
- Calling `loadBackgroundContent` does nothing for a dead worker.
|
||||
## What we reproduced
|
||||
|
||||
**Readiness signals measured during a 5 s slow start:**
|
||||
- The background view finishes its own page load after 0.1 s, so it says nothing about the worker.
|
||||
- The `loadBackgroundContent` completion and the worker's first native message both arrive right after the worker's startup code has run. Both are accurate readiness signals.
|
||||
1. **The first message is lost when a new extension version loads** (first install or update). WebKit starts the background immediately, and a message sent before the background code has run is dropped: 14 of 20 attempts on install, 18 of 20 on update. The app already waits for the extension at launch, but not when the extension is installed or updated mid-session.
|
||||
2. **A crash of WebKit's networking process breaks CPM in every tab.** The background looks alive, but its worker is dead. All tabs lose CPM, including pages that were already open. The current recovery, an extension reload, fixes new pages only; pages opened earlier stay broken. Reproduced 20 out of 20 times.
|
||||
3. **The worker can die while the networking process survives**, for example when service worker data is cleared. The symptoms are the same, and the app cannot detect this state directly.
|
||||
4. **WebKit error 6, background failed to load.** If waking the background takes longer than 30 seconds, WebKit's idle timer closes it mid-load and reports error 6. The test matches the production signature. It did not reproduce on the iPhone in four scenarios.
|
||||
5. **A slow worker start on wake-up reproduces the most common production pattern.**
|
||||
- Up to 2.5 seconds: messages wait and are delivered, nothing is lost.
|
||||
- At 5 seconds: the app reports a failure with a "healthy" snapshot exactly like production, then the late answer arrives and the app reports a recovery.
|
||||
- At 12 seconds with a second page load: the episode becomes "stuck" and the app reloads the extension. That reload kills a worker that was slow but working, and the pending messages of already-open pages are lost.
|
||||
6. **WebKit's website-data cleanup on every extension load delays CPM.** With a large browser cache (about 20,000 entries), content scripts started 0.75–1.35 seconds late, against 23 milliseconds with an empty cache. On a fast Mac this stays under the 4-second limit; slower phones still need to be measured.
|
||||
|
||||
## Fixes (app side, branch `alex/cpm-web-extension-ready-handshake`)
|
||||
**What recovers what:**
|
||||
- Restarting only the extension's background process makes WebKit start a fresh worker. That fixes every tab, including already-open pages.
|
||||
- Reloading the extension, the current recovery, fixes new pages only.
|
||||
|
||||
1. **Gate mid-session install and upgrade.** After a new embedded extension is installed, navigations wait (up to 5 s) for its first background load, as they already do at launch.
|
||||
2. **Network process watchdog.** Records the Network process PID of the background. If it changes, the watchdog restarts only the background WebContent process. Checked on every navigation and every second.
|
||||
3. **Restart the background on the first failure**, but only if the background process responds to a script call within 1 s. Killing a frozen process left the replacement worker deaf.
|
||||
4. Items 2 and 3 share one switch, which is on by default and not yet tied to a remote flag.
|
||||
**Detecting when the worker has actually started:**
|
||||
- The background's own page finishes loading after about 0.1 seconds, long before the worker runs, so it is no use as a signal.
|
||||
- WebKit's "background content loaded" callback and the worker's first message to the app both arrive right after the worker has started. Either is a reliable signal.
|
||||
|
||||
**Results, without fix → with fix:**
|
||||
- Mid-session install: 3/3 lost → 5/5 delivered.
|
||||
- Mid-session upgrade: 5/5 lost → 5/5 delivered.
|
||||
- Network process crash with 4 tabs and open pages: all broken → all working, including already-open pages and with M4 disabled.
|
||||
- Worker death without a Network process crash: two failures, then M4, and old pages stay broken → one lost navigation, then everything works without a reload.
|
||||
## Proposed fixes (app side, not merged)
|
||||
|
||||
**Known gap:** fix 3 can hurt a slow wake. If the slow worker does not keep the CPU busy (for example it waits on I/O), the process still answers, gets restarted, and the late answer turns into a lost one. Restarts should be held back while the background is still starting. The phase 2 detector below covers this.
|
||||
1. **Wait for the extension after a mid-session install or update**, as the app already does at launch. Page loads wait at most 5 seconds.
|
||||
2. **Watch the networking process.** When it is replaced, restart only the extension's background process. This repairs every tab, including open pages.
|
||||
3. **Restart the background after the first failure**, but only if its process still responds. Restarting a frozen process left the new worker unable to receive messages.
|
||||
|
||||
## Telemetry (branch `alex/cpm-messaging-readiness-telemetry`, phase 1)
|
||||
**Results in tests, without fix → with fix:**
|
||||
- Mid-session install: 3 of 3 messages lost → 5 of 5 delivered.
|
||||
- Mid-session update: 5 of 5 lost → 5 of 5 delivered.
|
||||
- Networking process crash with several tabs and open pages: everything broken → everything works, without an extension reload.
|
||||
- Worker dies while the networking process survives: two failures, an extension reload, and old pages stay broken → one failed page, then everything works without a reload.
|
||||
|
||||
- Telemetry only; the detector's behavior is unchanged. Ship it before the fixes, or together with them in an A/B experiment.
|
||||
- **On failure pixels:** `bg_ready_after`, `bg_view_age`, `app_state`, `open_tabs`, `loading_tabs`, and the error 6 fields `bg_load_error_app_state` and `bg_load_error_after`.
|
||||
- **On recovery pixels:** `recovery_document` (failed navigation answered late, later navigation, or unattributed) and `recovery_after`.
|
||||
- **Readiness source:** the app calls `loadBackgroundContent` once, when WebKit creates a background view. The view already exists, so the call joins the load in progress and wakes nothing. It is behind the same remote flag as the existing background delegate proxy.
|
||||
- Verified against real WebKit with a 5 s slow wake: `bg_ready_after=pending`, `bg_view_age=4s_to_8s`, then `recovery_document=failed_navigation`, `recovery_after=under_4s`.
|
||||
- 677 package tests pass. SwiftLint is clean. Pixel definitions validate on iOS and macOS. Both apps build.
|
||||
**Known risk:** fix 3 can make a slow wake-up worse. If the slow worker is not busy on the CPU (for example, it is waiting on disk or network), its process still responds, gets restarted, and a late answer becomes a lost one. Restarts should wait until the background has finished starting; see "Detector changes" below.
|
||||
|
||||
## Mitigations still to decide
|
||||
## Telemetry (phase 1, ready, not merged)
|
||||
|
||||
- **Detector phase 2.**
|
||||
- Start the 4 s grace period only once the background is ready.
|
||||
- Report "background did not start within 30 s" as its own pixel.
|
||||
- Escalate to stuck, M4 or a restart only for failures with a ready background.
|
||||
- This removes slow-start false positives.
|
||||
- **Wake the background early** when a navigation starts or the app comes to the foreground, without blocking the navigation. Wake-up time then overlaps with the page load.
|
||||
- **Stable base URL** for embedded extensions (branch `alex/cpm-stable-base-url`). Removes the cleanup on every launch and reload.
|
||||
- Privacy check: no embedded extension declares `web_accessible_resources`, and pages cannot see the URL.
|
||||
- Still to verify: M4 and crash recovery with a stable URL, and the upgrade path.
|
||||
- **Error 6:** once data shows the app state, restart the background when the app returns to the foreground if error 6 was recorded.
|
||||
- **Extension side:**
|
||||
- a retry handshake for `init`, covering lost and late messages whatever the cause;
|
||||
- a ping or heartbeat channel, which would allow zero-loss detection of a dead worker.
|
||||
- **Feature flags and A/B:** put the watchdog and the restart behind remote flags and expose the cohort in CPM pixels. Comparing release versions proved inconclusive.
|
||||
- **Report to WebKit:** the dead worker after a Network process crash, and the 30 s unload timer that starts before the load finishes.
|
||||
- Adds diagnostic fields only and does not change behaviour. It should ship before the fixes, or together with them in an A/B experiment, so the fixes can be measured.
|
||||
- **Failure pixels gain:**
|
||||
- whether the background had finished starting and how long that took;
|
||||
- the age of the background;
|
||||
- whether the app was in the foreground;
|
||||
- the number of open and loading tabs;
|
||||
- for error 6, the app state and the time to the error.
|
||||
- **Recovery pixels gain:**
|
||||
- whether the failed page answered late or a later page answered;
|
||||
- how long recovery took.
|
||||
- **How readiness is detected:** the app asks WebKit to report when the background it has just created finishes loading. This wakes nothing, and it can be switched off remotely with the existing diagnostics flag.
|
||||
- **Verification:**
|
||||
- In the slow wake-up test, the failure pixel reported "background not ready, created 4–8 seconds ago", and the recovery pixel reported "the failed page answered late, under 4 seconds after the failure".
|
||||
- All 677 automated tests of the extension package pass. Pixel definitions validate on iOS and macOS, and both apps build.
|
||||
|
||||
## Branches (local, uncommitted)
|
||||
## Next steps to decide
|
||||
|
||||
- `alex/cpm-web-extension-ready-handshake`: the fixes and the main test harness.
|
||||
- `alex/cpm-web-extension-remaining-failures`: production code without fixes; the error 6 and slow-wake reproductions.
|
||||
- `alex/cpm-messaging-readiness-telemetry`: phase 1 telemetry, based on current `main`.
|
||||
- `alex/cpm-stable-base-url`: the stable base URL experiment.
|
||||
- **Detector changes (phase 2):**
|
||||
- start the 4-second timer only once the background is ready;
|
||||
- report "background did not start within 30 seconds" as a separate pixel;
|
||||
- recover (restart or reload) only when the background was ready but did not answer.
|
||||
|
||||
This removes false alarms from slow starts and avoids killing workers that are slow but working.
|
||||
- **Wake the background early,** when a page load starts or the app returns to the foreground, without blocking the page. Startup then overlaps with page loading.
|
||||
- **Give the extension a stable internal base URL.** This removes WebKit's data cleanup on every launch and reload.
|
||||
- Privacy check: our embedded extensions expose no files to web pages, so pages cannot observe the URL.
|
||||
- Still to test: extension reload and crash recovery with a stable URL, and the update path.
|
||||
- **Error 6:** if the data confirms the background-app pattern, restart the background when the app returns to the foreground after an error 6.
|
||||
- **Extension changes:**
|
||||
- retry the first message when no answer comes, which covers both lost and late messages;
|
||||
- add a heartbeat between the extension and the app, so a dead worker can be detected without waiting for a failed page.
|
||||
- **Rollout:** put the fixes behind remote feature flags and run them as an A/B experiment that is visible in CPM pixels. Comparing app versions has not given conclusive results.
|
||||
- **Report to WebKit:**
|
||||
1. after a networking process crash, WebKit does not notice that the background worker is dead;
|
||||
2. the idle timer starts before the background finishes loading, which causes error 6.
|
||||
Reference in new issue
Block a user