obsidian-vault/work/projects/cpm-post-release-analysis-2026-10-27.md

title, date, tags

title	date	tags
CPM — first post-release data (iOS 7.243.0 / macOS 1.213.0)	2026-10-27	work cpm analysis

CPM: first post-release data, 26 October 2026

Internal metrics; keep these figures out of public PRs. Previous analysis: cpm-metrics-causes-and-mitigations-2026-10-16, cpm-status-2026-10-15 (§3–5, 12–14 October cut).

What shipped

• iOS 7.243.0 (tag 26 Oct 19:23 UTC), macOS 1.213.0 (tag 26 Oct 16:23 UTC).
• Both: episode telemetry #7376/#7484, episode tab scope #7520/#7521, P1/P2/P4 recovery #7379/#7536, Network-process recovery #7459/#7531.
• macOS only: P5 shared-pinned window close #7473, remote/debug disable #7498.
• Not shipped: P3 stable origin #7493 (on main only). No P6 fix exists in either release: no PR implements the controller rebind.
• Shipped episode contract is the rolled-back variant: was_stuck (bool), single-phase bg_events (no ;), plus tabs_affected, repeat_failures, interrupted. No bg_view_creations, context_errors or network_restarted. Old cohort SQL needs adapting (below).
• Monitor timing: 4 s response grace; suspended time no longer counts (8 s after resume, 30 s extension-load wait, 5 min deadline).

How to query

• New pixels have no m_/debug_ prefix: ClickHouse pixel_id = 'web', pixel LIKE 'web.extension.cpm.%' (iOS adds .ios.phone|tablet). Querying pixel_id='m' without an agent filter and a one-day window exceeds the 7B row limit.
• Denominator: pixel_id='m', m.web.extension.loaded.daily% / m.mac.web.extension.loaded.daily, split by appVersion.
• Old-contract attribution was remotely switched off from 21 Oct (old versions keep sending bare debug.web.extension.cpm.* without parameters). The last fully attributed old-version window is 18–20 Oct, iOS 7.241 / macOS 1.211.x.
• Data lag: on 27 Oct the table ended at ~01:10 UTC; date is US-Eastern. 26 Oct is the only post-release day and it is partial for new adopters (exposure starts at update time).
• Runner: ~/Developer/grafana-data-tools reader; ad-hoc SQL and an ARRAY JOIN extractURLParameters(request) parameter dump were used. extractURLParameter needs a constant key.

Rates per extension-loaded install-day (26 Oct)

	iOS 7.242 (old)	iOS 7.243 (new)	macOS 1.212 (old)	macOS 1.213 (new)
loaded.daily	4,257,839	52,979	93,030	2,566
old init_failed.daily (sum of reasons)	21.3%	—	36.8%	—
old stuck.daily	3.3%	—	12.5%	—
episode.daily	—	7.05% (3,735)	—	30.0% (771)
episode with reload_state≠none (hang confirmed, ≈ old stuck), count leg	—	23% of episodes	—	26% of episodes
was_stuck=true (second distinct tab), daily leg	—	8.1%	—	11.2%
network_process_recovery_applied.daily	—	432 (0.8%)	—	232 (9.0%)

Not like-for-like: the episode unit is one report per episode, the old init number sums four reasons, suspended time is now excluded, and day-1 adopters have short exposure. Directionally, iOS fell ~3× (21% → 7%); macOS barely moved (37% → 30%).

was_stuck only becomes true when a different tab fails; M4 reload already runs on a same-tab second timeout. Use reload_state≠none as the old "stuck" proxy.

Cohorts (count leg, adapted classification)

Priority: error 6 (bg_load_error_after present or error_ token) → no view (bg_view_alive=false) → tab not connected (tab_has_ext_scripts=false) → extension_updated ≤2 min → wake (bg_view_age < 30 s and a dealloc token) → recent load/base_url_changed ≤2 min → other. There is no Network-replaced cohort any more.

Cohort	iOS share / of stuck / no-response	macOS share / of stuck / no-response
error 6	7.5% / 24.7% / 39%	0.6% / 1.7% / 44%
no view (P3)	7.7% / 6.8% / 7%	2.9% / 0.9% / 25%
tab not connected (P6)	—	5.8% / 1.3% / 70%
extension_updated ≤2 m	3.9% / 6.2% / 37%	14.6% / 11.7% / 28%
wake, view <30 s	22.8% / 22.0% / 9%	43.9% / 58.4% / 48%
recent context load	22.8% / 17.2% / 12%	11.9% / 6.9% / 25%
other	35.3% / 23.2% / 5%	20.4% / 19.0% / 27%

Baseline with last week's exact SQL, 18–20 Oct (init → stuck shares). iOS 7.241: error6 6.4→22.0, no view 47.5→11.5, network 16.5→38.2, wake 7.7→9.2, recent load 16.7→6.6, other 5.2→12.5. macOS 1.211.x: error6 1.2→2.2, no view 3.6→1.9, tab not connected 5.9→8.7, network 13.2→25.8, wake 33.2→33.3, recent load 19.4→6.3, other 23.6→21.6. This matches the 12–14 Oct cut, so the baseline was stable.

Per-install rough rates (cohort share × failure rate), old init → new episode:

• iOS: no view 10% → 0.5%, recent load 3.5% → 1.6%, error 6 1.3% → 0.5%; the network cohort is absorbed. Wake stays at 1.6% → 1.6%. Other rises 1.1% → 2.5%; it now absorbs former network cases and has no network field.
• macOS: wake ≈12% → ≈13%. Unchanged.

Findings

1. P1 (wake) fix had no measurable effect on either platform. In the wake cohort bg_ready_after=under_1s dominates (macOS 93% overall), so the background is not slow. The background_wake pixel fires only for ≥1 s callbacks (macOS 89/day vs ~670 wake episodes). The remaining failure is message/response loss after a fast idle wake, not readiness. On macOS 48% of wake episodes see no response for 5 min; tabs=3+ → 68%. Typical tail: …dealloc@15m,view@15m,proxy_on@15m,dealloc@5m,view@30s,proxy_on@30s, bg_view_age=4s_to_8s (view created around the navigation). The pattern appears on every macOS version, 15.6–27.2.
2. iOS error 6 changed shape. All attributed error 6 is now bg_load_error_after=under_5s; the 30 s timer bucket is gone.
   • A (ready=failed, app_state active): dealloc → view → proxy_on → interruption → error_6, often after died_client/died_crash/died_crash_limit. The background load spans an app suspension and fails at resume.
   • B (ready=pending): an older error 6, then a new view whose load is still pending 8–30 s later.
   • The P2 retry fires only while foregrounded with a 60 s cooldown (277/day). Foreground-resume recovery is the missing piece.
3. macOS P6 is untreated: 5.8% of episodes, 70% never answered. iOS now shows tab_has_ext_scripts=false in 5.3% of episodes, concentrated on iOS 26.7 (16%) and 27.0 (8%), with 0 on 27.2. Previously iOS was ≈0.
4. macOS Network replacement is frequent: the applied pixel is on 9% of 1.213 install-days. With recovery applied it no longer appears as a cohort, but other stays at 20%.
5. After M4 on macOS: reload_state=loaded → 46% no response by deadline; response_timed_out → 57%. The reload often does not restore messaging, or no qualifying navigation follows.
6. Recovery pixels only ever report background_ready. response_received needs late-document reload, which is default OFF.
7. Day-1 effect: macOS extension_updated ≤2m is 14.6% (embedded upgrade on update). Re-check after 2–3 full days.

Next measurements

• Repeat for 27–29 Oct (full days, larger adoption). Track the wake cohort and macOS no-response by OS.
• bg_events per-phase budget and network_replaced_since_view remain missing (see cpm-status-2026-10-15#6. Телеметрия).

Repro targets

Issue	Evidence	Hypothesis to test	Candidate fix
R1 wake message loss	macOS 44% eps / 58% stuck; iOS 23%; ready <1 s	message to a woken non-persistent SW is acknowledged with no reply: (a) processes(RuntimeOnMessage) empty when _loadServiceWorker completion runs before listener IPC; (b) listener set loses RuntimeOnMessage so wakeUpBackgroundContentIfNecessaryToFireEvents resolves {} without waking; (c) background unloaded while a reply is pending	keep the background warm while CPM pages are active, or re-send init on empty reply (native/JS handshake)
R2 iOS error 6 across suspension	7.5% eps / 25% stuck, all under 5 s, interruption between view and error	WebKit load timer spans suspension; nothing retries on resume	on foreground resume, retry loadBackgroundContent when the last load failed or is pending
R3 P6 controller mismatch	macOS 5.8%, 70% unanswered	known main-flag manager replacement	rebind/recreate tab web views to the current controller

WebKit source pointers (~/Downloads/WebKit_sparse): WebExtensionContext.cpp wakeUpBackgroundContentIfNecessaryToFireEvents (drops events when the listener set lacks the type after the first load), scheduleBackgroundContentToUnload (30 s, reset only on dispatch), WebExtensionContextCocoa.mm loadBackgroundWebView / performTasksAfterBackgroundContentLoads, WebExtensionContextAPIRuntimeCocoa.mm runtimeSendMessage (empty processes() → {} reply). 324743 (9bcf5840ef) holds runtime messages until SW activation; it is not in shipped OS builds.

Repro log — 27 October 2026

Worktree apple-browsers.git/cpm-remaining-failures-repro, branch alex/cpm-remaining-failures-repro (from origin/main 8023e4cb5c, which includes the released P1/P2/P4/Network code plus P3). The package real-WebKit harness was imported from origin/alex/cpm-repro-harness. It compiles unchanged on current main, and its CI-tier tests pass. Local tier: CPM_REPRO=1 swift test --filter <Suite> from SharedPackages/WebExtensions. System WebKit is the host macOS; no app windows are opened.

• R1 idle wake — not reproduced organically. CPMReproIdleWakeTests.testRepeatedIdleWakeCyclesDeliverInit: 3 tabs, 8 idle-unload cycles (WebKit unloads 32.0 s after the last answered navigation), single and double-tab wakes, 0 of 11 lost. Every wake answered in 2–5 ms after didFinish. testNavigationAtIdleUnloadBoundaryDeliversInit: offsets −0.6…+0.2 s around the calibrated unload, 0 of 7 lost. A plain idle wake on an in-memory fixture page is not the production mechanism. The missing ingredient is still unknown: real page loads, a long-lived data store with abandoned registrations, Network replacement, or listener-set loss.
• P3 (#7493) only in internal builds (macOS 1.214.0-860, iOS 7.244.0-x). On 26 Oct: macOS 1.214 had 20 episodes / 9 daily of ~36 loaded installs, with 0 no-view and 0 error 6; iOS 7.244 had none. That is too small to evaluate. Relevance: released builds get a new random extension origin on every context load (launch and M4 reload), and WebKit migrates data before injecting. This may explain the iOS no-view cohort and the macOS "reload loaded → still no response" share (46%). Re-check when 7.244 reaches users.
• R2 suspended wake — macOS analogue does not stay broken. CPMReproSuspendedWakeTests: idle unload, then a navigation wakes the background, and the new background process is SIGSTOPped for 45 s while another tab navigates every 10 s. Two navigations timed out (+9 s, +14 s). Then the background was replaced by a new process (same view slot, new PID) and every later navigation answered in 1–14 ms, including 10 after thaw. No error 6 was recorded on the live context. On macOS the shipped recovery restores messaging within ~14 s of the fault. The harness cannot reproduce iOS suspension, which pauses the UI process's own timers. Likely production sequence: the overdue 30 s unload timer fires at resume, closes the loading view, and records error 6 "under 5 s" (pattern A). Pattern B (new view pending 8–30 s) needs a device or simulator test.
• WebKit hypothesis not hit yet: an SW load failure that keeps the background view (loadBackgroundWebView completion !success without unload). Later wakes would only queue, and every dispatch restarts the 30 s unload timer, so the public loadBackgroundContent/P2 retry cannot escape. A candidate mitigation is to restart the background web content process when error 6 is recorded while a background view still exists. Not implemented, because the state has not been reproduced.

Treatment pixels, P3 ceiling and permanent hangs — 27 October 2026

Treatment pixels received (26 Oct, count leg, iOS 7.243.0 / macOS 1.213.0)

Fix	Pixel (pixel_id='web')	iOS	macOS
P1 wake	web_extension_cpm_background_wake	2,026	89
P2 error-6 retry	web_extension_cpm_background_load_retry	277	12
Install / upgrade readiness	…_install_readiness / …_upgrade_readiness	3 / 24	0 / 1
Network recovery	web_extension_cpm_network_process_recovery_applied	476	277
P4 detection (flag-independent)	web_extension_cpm_startup_response_missing	450 (daily 327)	20 (daily 12)
Late page reload	web_extension_cpm_late_document_reload	0	0

• Every recovery pixel only reports outcome=background_ready, meaning the fix was applied and the background came up. reload_started / response_received exist only for the late page reload, which is default OFF, so it has zero rows on every version including internal builds. No pixel shows whether the page got its answer after a treatment; the effect has to be read from episode rates.
• background_wake fires only when the wake callback took ≥1 s and a navigation waited, so it is not a count of all wakes.
• Detection gap for the late reload. startup_response_missing fires with the flag off but requires startup.isFirstLoad: the first background load of that context in the session (launch, install, upgrade). attemptRecovery (the reload) has no isFirstLoad condition and would also run after later wakes. With the flag off, wake-time cases the reload would treat are not counted anywhere; 327 / 12 is a lower bound of what it could fix. Both detection and reload only arm when the page commits while the startup is still pending. In the wake cohort the background is ready in under 1 s, so many wake failures are outside both. Counting the would-be-reload wake cases needs a pixel change: drop isFirstLoad from detection or add a parameter. Not done.
• Upper bound on the late reload: ~0.6% of iOS / 0.5% of macOS extension-loaded installs; ~9% of iOS / ~2% of macOS installs with a failure episode.

P3 (baseURL) ceiling on the new data

• The earlier "≈half" was the pre-release no-view share: 48% of iOS initial failures, but only 11% of stuck, mostly self-healing.
• On 7.243.0 without P3, no-view is already 7.7% of iOS episodes / 6.8% of stuck (≈10% → 0.5% of loaded installs); macOS 2.9% / 0.9%. The drop comes from the new measurement (suspended time excluded, one report per episode) and the shipped readiness gates.
• Direct P3 ceiling ≈ 8% of iOS episodes, less on macOS. A possible indirect effect: released builds get a new random origin on every M4 reload, and 46% of macOS reload-loaded episodes still got no response. That link is unproven; internal 1.214 / 7.244 data is too thin.

Users affected by permanent hangs (26 Oct)

Denominators were refreshed after more 26 Oct data arrived: extension-loaded installs iOS 69,088 / macOS 3,045. "Permanent" = outcome=no_response_before_deadline (no qualifying response for 5 min). Single = tabs_affected=1, cross = 2/3+. Confirmed = repeat_failures≠0 or reload_state≠none. Daily leg = affected installs.

Installs/day	iOS	macOS
Any failure episode	4,655 (6.7%)	876 (28.8%)
Single-tab, no response	354 (0.51%)	217 (7.1%)
↳ confirmed	76 (0.11%)	25 (0.82%)
Cross-tab, no response	129 (0.19%)	58 (1.9%)
↳ confirmed	123 (0.18%)	49 (1.6%)

• Cross-tab is the reliable permanent-hang figure: ~0.2% of iOS and 1.6–1.9% of macOS users per day; 95% / 84% confirmed.
• Unconfirmed single-tab rows (iOS 278, macOS 192) may be stuck tabs or users who didn't navigate within 5 min.
• Strict permanent (confirmed single + cross): ~0.3% iOS, ~2.4% macOS users/day. Upper bound including unconfirmed single-tab: 0.7% / 9%.
• Lower bounds for users: the daily leg describes only each install's first completed episode of the day, and episodes lost when the process ends before the deadline are not reported. Count leg (episodes per install-day), confirmed cross-tab: iOS 0.41%, macOS 5.2%; confirmed single-tab: iOS 0.26%, macOS 2.3%.
• One partial day; macOS has ~3k installs on 1.213.0. Repeat on 27–29 Oct full days.

Repro statistics note

18 wake navigations cannot rule out a rare loss. At p=8% a clean run happens 22% of the time. The per-wake production rate is likely ≪1% (many wakes per install-day), which needs ~1,000 wakes for 95% power. CPMReproIdleWakeTests now has CPM_REPRO_FAST_UNLOAD=1: it calls +[WKWebView _setApplicationBundleIdentifier:] with com.apple.WebKit.TestWebKitAPI, so WebKit unloads an idle background after 3 s instead of 30 s (~5 s per cycle). It can change other test-runner-only WebKit behavior, so confirm any hit with the 30 s delay. A 1,000-cycle run (~1,500 wake navigations) was started on 27 Oct.