diff --git a/work/projects/cpm-post-release-analysis-2026-10-27.md b/work/projects/cpm-post-release-analysis-2026-10-27.md index f49663ad..dd090bbf 100644 --- a/work/projects/cpm-post-release-analysis-2026-10-27.md +++ b/work/projects/cpm-post-release-analysis-2026-10-27.md @@ -100,3 +100,49 @@ Worktree `apple-browsers.git/cpm-remaining-failures-repro`, branch `alex/cpm-rem - **P3 (#7493) only in internal builds** (macOS 1.214.0-860, iOS 7.244.0-x). On 26 Oct: macOS 1.214 had 20 episodes / 9 daily of ~36 loaded installs, with 0 no-view and 0 error 6; iOS 7.244 had none. That is too small to evaluate. Relevance: released builds get a new random extension origin on every context load (launch and M4 reload), and WebKit migrates data before injecting. This may explain the iOS no-view cohort and the macOS "reload loaded → still no response" share (46%). Re-check when 7.244 reaches users. - **R2 suspended wake — macOS analogue does not stay broken.** `CPMReproSuspendedWakeTests`: idle unload, then a navigation wakes the background, and the new background process is SIGSTOPped for 45 s while another tab navigates every 10 s. Two navigations timed out (+9 s, +14 s). Then the background was replaced by a new process (same view slot, new PID) and every later navigation answered in 1–14 ms, including 10 after thaw. No error 6 was recorded on the live context. On macOS the shipped recovery restores messaging within ~14 s of the fault. The harness cannot reproduce iOS suspension, which pauses the UI process's own timers. Likely production sequence: the overdue 30 s unload timer fires at resume, closes the loading view, and records error 6 "under 5 s" (pattern A). Pattern B (new view pending 8–30 s) needs a device or simulator test. - WebKit hypothesis not hit yet: an SW load failure that **keeps** the background view (`loadBackgroundWebView` completion `!success` without unload). Later wakes would only queue, and every dispatch restarts the 30 s unload timer, so the public `loadBackgroundContent`/P2 retry cannot escape. A candidate mitigation is to restart the background web content process when error 6 is recorded while a background view still exists. Not implemented, because the state has not been reproduced. + +## Treatment pixels, P3 ceiling and permanent hangs — 27 October 2026 + +### Treatment pixels received (26 Oct, count leg, iOS 7.243.0 / macOS 1.213.0) + +| Fix | Pixel (`pixel_id='web'`) | iOS | macOS | +|---|---|---|---| +| P1 wake | `web_extension_cpm_background_wake` | 2,026 | 89 | +| P2 error-6 retry | `web_extension_cpm_background_load_retry` | 277 | 12 | +| Install / upgrade readiness | `…_install_readiness` / `…_upgrade_readiness` | 3 / 24 | 0 / 1 | +| Network recovery | `web_extension_cpm_network_process_recovery_applied` | 476 | 277 | +| P4 detection (flag-independent) | `web_extension_cpm_startup_response_missing` | 450 (daily 327) | 20 (daily 12) | +| Late page reload | `web_extension_cpm_late_document_reload` | 0 | 0 | + +- Every recovery pixel only reports `outcome=background_ready`, meaning the fix was applied and the background came up. `reload_started` / `response_received` exist only for the late page reload, which is default OFF, so it has zero rows on every version including internal builds. **No pixel shows whether the page got its answer after a treatment; the effect has to be read from episode rates.** +- `background_wake` fires only when the wake callback took ≥1 s and a navigation waited, so it is not a count of all wakes. +- **Detection gap for the late reload.** `startup_response_missing` fires with the flag off but requires `startup.isFirstLoad`: the first background load of that context in the session (launch, install, upgrade). `attemptRecovery` (the reload) has no `isFirstLoad` condition and would also run after later wakes. With the flag off, **wake-time cases the reload would treat are not counted anywhere**; 327 / 12 is a lower bound of what it could fix. Both detection and reload only arm when the page commits while the startup is still pending. In the wake cohort the background is ready in under 1 s, so many wake failures are outside both. Counting the would-be-reload wake cases needs a pixel change: drop `isFirstLoad` from detection or add a parameter. Not done. +- Upper bound on the late reload: ~0.6% of iOS / 0.5% of macOS extension-loaded installs; ~9% of iOS / ~2% of macOS installs with a failure episode. + +### P3 (baseURL) ceiling on the new data + +- The earlier "≈half" was the pre-release no-view share: **48% of iOS initial failures**, but only 11% of stuck, mostly self-healing. +- On 7.243.0 without P3, no-view is already **7.7% of iOS episodes / 6.8% of stuck** (≈10% → 0.5% of loaded installs); macOS 2.9% / 0.9%. The drop comes from the new measurement (suspended time excluded, one report per episode) and the shipped readiness gates. +- Direct P3 ceiling ≈ **8% of iOS episodes**, less on macOS. A possible indirect effect: released builds get a new random origin on every M4 reload, and 46% of macOS reload-loaded episodes still got no response. That link is unproven; internal 1.214 / 7.244 data is too thin. + +### Users affected by permanent hangs (26 Oct) + +Denominators were refreshed after more 26 Oct data arrived: extension-loaded installs **iOS 69,088 / macOS 3,045**. "Permanent" = `outcome=no_response_before_deadline` (no qualifying response for 5 min). Single = `tabs_affected=1`, cross = `2`/`3+`. Confirmed = `repeat_failures≠0` or `reload_state≠none`. Daily leg = affected installs. + +| Installs/day | iOS | macOS | +|---|---|---| +| Any failure episode | 4,655 (6.7%) | 876 (28.8%) | +| Single-tab, no response | 354 (0.51%) | 217 (7.1%) | +| ↳ confirmed | 76 (0.11%) | 25 (0.82%) | +| Cross-tab, no response | 129 (0.19%) | 58 (1.9%) | +| ↳ confirmed | 123 (0.18%) | 49 (1.6%) | + +- **Cross-tab is the reliable permanent-hang figure:** ~0.2% of iOS and 1.6–1.9% of macOS users per day; 95% / 84% confirmed. +- Unconfirmed single-tab rows (iOS 278, macOS 192) may be stuck tabs or users who didn't navigate within 5 min. +- **Strict permanent (confirmed single + cross): ~0.3% iOS, ~2.4% macOS users/day.** Upper bound including unconfirmed single-tab: 0.7% / 9%. +- Lower bounds for users: the daily leg describes only each install's *first* completed episode of the day, and episodes lost when the process ends before the deadline are not reported. Count leg (episodes per install-day), confirmed cross-tab: iOS 0.41%, macOS 5.2%; confirmed single-tab: iOS 0.26%, macOS 2.3%. +- One partial day; macOS has ~3k installs on 1.213.0. Repeat on 27–29 Oct full days. + +### Repro statistics note + +18 wake navigations cannot rule out a rare loss. At p=8% a clean run happens 22% of the time. The per-wake production rate is likely ≪1% (many wakes per install-day), which needs ~1,000 wakes for 95% power. `CPMReproIdleWakeTests` now has `CPM_REPRO_FAST_UNLOAD=1`: it calls `+[WKWebView _setApplicationBundleIdentifier:]` with `com.apple.WebKit.TestWebKitAPI`, so WebKit unloads an idle background after 3 s instead of 30 s (~5 s per cycle). It can change other test-runner-only WebKit behavior, so confirm any hit with the 30 s delay. A 1,000-cycle run (~1,500 wake navigations) was started on 27 Oct.