obsidian-vault/work/projects/cpm-readiness-telemetry-tech-design-2026-10-15.md

Изолированная ветка фиксов — 16 Oct, 13:33 UTC. alex/cpm-fix-p1-p2 (/Users/admin/DuckDuckGo/apple-browsers.git/alex/cpm-fix-p1-p2) создана от текущего alex/cpm-messaging-readiness-telemetry / 01d4a0ec22. Выделены P1 navigation wake wait (cap 5 s) и P2 same-context background retry по реальной error 6, с существующими gate/recovery flags и ограничением частоты. Только 2 app-регрессии + 4 новых lifecycle unit-теста; прежние 22 — весь research-набор. Signed app 2/2 PASS, unit 165 PASS, strict lint 0. На тех же бинарниках оба disabled-flag контроля функционально FAIL; deallocation guard в финальном positive run PASS после test-only autorelease fix. База уже использует cpmMessagingEpisode: P2 outcome failed_measurement_response, reload_state none; доставка page initResp и неизменность context проверяются отдельно. Добавлен только read-only recorder isApplicationInBackground для guard, без изменения telemetry schema/ZIP/dependencies. Более поздние baseline/init-progress PR не перенесены. Нет коммитов, исходный scratch сохранён. Артефакты /private/tmp/cpm-p1-p2-clean-20261016/validation-summary.json; история и границы macOS injected-fault доказательства в cpm-status-2026-10-15.

Final local P1/P2 validation — 16 Oct, 12:30 UTC. Current background-only candidate: 22/22 unique app methods PASS across 2+19+1+1 runs (P2 repeated); both disabled-fix controls fail functionally. P2 restores native dashboard and page initResp on the same failed document without navigation or context replacement, with recovered_without_extension_reload / failed_navigation. The full-context-reload prototype was rejected after repeated page-delivery loss. Package: 806 tests, 14 skips, 0 failures; strict lint: 0 violations. No commits, PR mutations or telemetry schema changes. P1's 5 s bound, macOS injected-fault scope, unresolved natural iOS/P3/P5 limits and nonfatal owned-cache SQLite cleanup warning remain explicit in cpm-status-2026-10-15. Earlier pending/20-method statements are dated history.

Interpretation clarification — 16 Oct. The 20 passing app methods do not mean all production-data cases except P3 were reproduced. P1/P2 use injected delay/SIGSTOP; scoped message/native/dashboard loss tests are diagnostic fault injections absent from the shipping bundle. P5/P6 and Network tests exercise browser/WebKit lifecycle paths, under instrumented app-test conditions; occurrence of those exact triggers in affected production sessions remains a separate question. Even with no active scoped fault, the extracted extension carries runtime/DOM observation wrappers. Natural listener metadata loss, P5/tab=true, actual iOS lifecycle/device chains and P3 remain unresolved. The original longer app P5 did not pass and was narrowed to one failed navigation; its limit remains explicit.

Local app validation complete — 16 Oct 2026, 01:48 Bishkek / 15 Oct 19:48 UTC. Twenty app-hosted methods passed with 0 failures, skips or expected failures in three invocations of the same signed build: 18 together, P5 separately, persistent cache separately. This validates the requested slow-load, one-navigation/recovery, single-tab, several-tabs/one-window and all-windows diagnostics scenarios. Source-linked app P1/P2/P5/P6 and Network paths are covered; P3 was not reproduced with 20,000 cached resources, background SIGKILL self-healed, and app P5 proves one lost navigation plus controls. The earlier red joint run and long P5 attempts remain documented. See cpm-status-2026-10-15#Финальный app-hosted итог — 16 октября 2026 for the full matrix, evidence and limits. No commits, PR changes or telemetry schema changes. PR/CI and production-data statements below retain their original dated provenance.

TD: Pixel Definition: CPM Readiness Telemetry (macOS/iOS)

Internal documentation update — 2026-10-15, 17:26 UTC. Current evidence, unresolved review decisions and PR stack: cpm-status-2026-10-15. This update does not approve or implement the proposed schema changes.

PR #7376 is OPEN at 619869ee84c6, review required, with no failing/pending checks; #7384/#7385 remain open and each has a failing Danger status. Their skipped checks are not completed tests. JS #3727 is open, checks successful, review required.

Source-linked P5/shared-window loss and conditional P6/main-flag replacement are now reproduced; natural metadata loss, P5/tab=true and actual production flag transitions remain unproven. The main status contains the full 14-test matrix, page-receipt controls, cold Network PID caveat, 494/499 denominator distinction and pending telemetry simplification decision. No new pixel or parameter was added by those tests. Internal ClickHouse figures stay in Obsidian, not public PR comments or issues.

Author: Alex M
Reviewer: TBD (previous reviewer: Michal)
Stakeholders: Michal, Konrad, Russell (from the previous Tech Designs; confirm for this proposal)
Project: CPM: root-cause the stuck embedded extension messaging (macOS/iOS)
Task: Improve telemetry
Previous Tech Designs: CPM breakage pixels · CPM Failure Diagnostics (cpm-m3-telemetry-privacy-triage)
Privacy Triage: not requested yet; see Privacy below.
Status (checked 2026-10-15, 17:26 UTC): PR #7376 OPEN at 619869ee84c6; REVIEW_REQUIRED; no failing/pending published checks (39 success, 18 skipped entries). Not merged; this check does not establish shipping.
Analysis this builds on: cpm-web-extension-public-attribution-2026-10-05, cpm-web-extension-breakage-session-summary-2026-10-07, cpm-web-extension-code-guide-2026-10-09 (section 3).

Background & Requirements

CPM Failure Diagnostics added process, extension and tab state to the existing failure pixels. The first release with that data (iOS 7.240.0, macOS 1.210.0) showed what it cannot tell apart:

• Most failure snapshots look healthy: context loaded, background view alive, process responsive. The largest group (about 56% of first failures on iOS, 66% on macOS) has no visible error. A live view does not show whether the background script has run.
• A background that is still starting (a wake that takes seconds) looks the same as one that is gone.
• Background load error 6 (BackgroundContentFailedToLoad) is recorded, but not when it happened relative to the view, nor in which app state.
• Recovery pixels say that CPM answered again, not whether the late answer to the failed navigation arrived, a later navigation answered, or the answer could not be attributed.

Add facts that separate slow starts from lost messages, to the existing failure and recovery pixels.

Problem Statement

Tell apart, per failure and per recovery:

1. A background still loading when the failure was recorded, from one that finished loading or failed to load.
2. A failure during a background wake (the view is seconds old) from one long after it.
3. A background load error cut off by WebKit's 30 s unload timer from one that failed quickly, and the app state it happened in.
4. A late response from a lost one: did the failed navigation answer after its grace period, did a later navigation, or neither.
5. A failure with many loading tabs (a busy session restoration) from a quiet one.

Collect no page content, URLs or persistent identifiers.

This proposal covers diagnostic collection only. The failure detector, recovery behavior and reload logic do not change. No experiment.

Recommended Approach

Keep the existing detection. Extend the existing recorder and pixels with new optional parameters. No new pixel names.

Background readiness. When WebKit creates the background view, an observer records that the load is pending. On the next run loop turn it calls WKWebExtensionContext.loadBackgroundContent. The view already exists, so the call joins the load in progress and does not wake a background. Its completion fires once the background script has run, and records loaded(seconds since view creation) or failed. A response for a replaced view is ignored. readiness(of:) answers only for the current view. The observer is attached by the same switch as the delegate proxy (cpmBackgroundDelegateProxy).

Error 6 timing. When error 6 is recorded, the recorder also records the app state and the seconds since the failing view was created. Values near 30 s point at WebKit's unload timer.

Snapshot additions. When a failure pixel fires, the recorder adds app state, background view age, background readiness and tab counts, read synchronously on the main actor. The app supplies the app state (UIApplication.applicationState on iOS; NSApp.isActive mapped to active/inactive on macOS). Facts that cannot be read are omitted.

Recovery details. The health monitor already tracks an episode. On recovery it now says which navigation answered, and the time since the episode's first failure, bucketed. Detector behavior is unchanged.

Pixels

No new pixels. Parameters are added to existing ones.

Pixel	Change
debug_web_extension_cpm_initialization_failed_after_<reason> (4 reasons × daily)	Failure parameters below.
debug_web_extension_cpm_messaging_stuck_<reason> (4 reasons × daily + count)	Failure parameters below.
debug_web_extension_cpm_messaging_recovered_without_extension_reload	Recovery parameters below.
debug_web_extension_cpm_messaging_recovered_after_extension_reload	Recovery parameters below.

<reason>: session_restoration, tab_crash, extension_reload, other. Eight failure pixels and two recovery pixels per platform. Names, triggers and frequencies are unchanged. iOS defines them in debug_pixels.json5, macOS in web_extensions.json5; both add the parameters to params_dictionary.json5.

Failure / stuck parameters

Unavailable optional facts are omitted. PIDs, URLs and tab identifiers are never sent.

Parameter	Values
app_state	active, inactive, background. macOS reports only active / inactive. Omitted if the app does not report it.
bg_view_age	Time since the current background view was created: under_4s, 4s_to_8s, 8s_to_30s, 30s_to_2m, over_2m. Omitted if there is no background view.
bg_ready_after	none (no background view), pending (still loading), failed, or time from view creation to load completion: under_1s, 1s_to_4s, 4s_to_10s, 10s_to_30s, over_30s. Omitted if readiness is not observed (observer flag off).
open_tabs	Tabs in WKWebExtensionContext.openTabs: 0, 1, 2_to_5, 6_to_10, 11_to_20, 21_to_50, over_50.
loading_tabs	How many of those tabs were loading: 0, 1, 2, 3_to_5, over_5.
bg_load_error_app_state	App state when error 6 was last recorded for this context: none (no error 6), active, inactive, background. Omitted if unknown.
bg_load_error_after	Time from view creation to that error 6: none, under_5s, 5s_to_25s, 25s_to_35s, over_35s. The 25s_to_35s bucket surrounds WebKit's 30 s timer. Omitted if unknown.

bg_view_age and bg_ready_after use buckets around the monitor's 4 s grace period and WebKit's 30 s timer. They are boundaries chosen for the questions above, not measured thresholds.

Recovery parameters

Parameter	Values
recovery_document	failed_navigation (the navigation that failed answered after its grace period: late, not lost), later_navigation, unattributed (a response matched an open document but no single tab).
recovery_after	Time from the episode's first failure to recovery: under_4s, 4s_to_10s, 10s_to_30s, 30s_to_2m, over_2m. A failure is reported 4 s after the navigation finished.

Existing recovery_from is unchanged.

Testing

• Unit tests: bucket boundaries, parameter serialization and omission rules, error 6 timing (including none versus unknown), readiness (loaded, failed, replaced view, observation off), app state, open and loading tabs on a real controller with a navigation that does not finish, recovery details and buckets in the health monitor. 115 tests in the affected suites pass locally.
• Pixel validation: both platforms' schema validators run in CI.
• Real WebKit: during development the harness was temporarily copied into this branch. A background start slowed to 5 s produced bg_ready_after=pending, bg_view_age=4s_to_8s, then recovery_document=failed_navigation with recovery_after=under_4s. This was a manual check; it is not in the PR. The CPM repro harness (PR #7378) can reproduce it in CI once it lands.
• Privacy validation: inspect the real payloads and confirm no URLs, identifiers or message content. Not done yet.

Additional Considerations (if applicable)

Privacy

Adds enumerated, bucketed parameters to existing pixels; no page content, URLs, search queries, message payloads, tab/context identifiers or PIDs. App state, tab counts and timings are bucketed. These are new parameters on existing pixels, so Privacy Triage is required before release. I have not requested it.

Security

No new private WebKit selectors. loadBackgroundContent is a public API.

Site Breakage

No intended page-behavior changes. The one behavioral addition is the loadBackgroundContent call on a background view that already exists. It should join the load in progress and wake nothing, but the earlier delegate proxy changed the reproduction of this bug once, so check the observer with its flag on and off. A background wake the observer causes would also corrupt bg_ready_after.

Experimentation

None.

Operational

• The observer shares the cpmBackgroundDelegateProxy switch; the recorder and its snapshot additions follow cpmDiagnosticsRecorder. No new flag. Whether sharing is acceptable needs a decision from the flag owner, because turning the proxy off also turns readiness off and the parameter disappears.
• Update the CPM Grafana board for the new fields.
• Diagnostic removal is still planned for approximately November 2026; no removal date is enforced in code.
• The limits of the previous Tech Design stand: failure-only snapshots cannot measure failure rates across all process deaths or prove that every tab failed.
• bg_ready_after tells whether the background script ran. It does not tell whether the init message for a given document reached it. That needs per-message timings (content init sent, background init received, initResp sent and received, dashboard refresh sent and received), which is the next telemetry candidate and out of scope here.

Localization / Internationalization

None.

App-hosted validation, 16 October 2026

A separate local suite now runs inside the signed DuckDuckGo.app, using actual AppDelegate lifecycle/flag callbacks, Tab/MainWindowController/WindowsManager, macOS WebExtension providers/native delegate, production monitor/recorder and system WebKit. Worktree: cpm-e2e-init, macOS/IntegrationTests/CPM/; scheme CPM Integration Tests, without profile-cleaning pre-actions. No commits or PR changes.

The ten core methods passed together (signed-core.xcresult, 0 failures): healthy baseline, slow HTTP document, slow native settings, one failed navigation followed by recovery, persistent single-tab failure, several tabs in one window, all windows including a newly opened tab, automatic M4 recovery, dashboard-transport negative control, and a runtime diagnostic-proxy flag cycle. Healthy checks require page-side initResp as well as a forwarded real native dashboard message. Fault scopes are injected into a temporary extracted copy at the extension messaging boundary; navigation/monitor events are not fabricated.

Observed library metadata: message loss gives init_stage=none; delayed settings gives received, then recovery on failed_navigation (4s_to_10s); a missing dashboard confirmation despite page receipt gives response_sent; later-document recovery is later_navigation. These are metadata captured before the platform pixel adapter/daily deduplication, not proof of an actual production HTTP payload or production causal attribution. JSON artifacts contain only owned fixture page URLs and test observations; pixel parameters remain the existing bucketed metadata.

Additional app/WebKit characterization tests cover real shared-pinned-window closure, main/embedded flag transitions, startup race, process termination, slow idle wake and the real error-6 load timeout. Their final outcomes and the preserved red experiments are tracked in cpm-status-2026-10-15#13. Integration tests в настоящем приложении — реализация и проверки. iOS lifecycle and natural listener-metadata loss remain unproven by the macOS suite. The initial unsigned CLI workaround was unnecessary; the original project signing configuration builds successfully and is used for the reported results. The temporary vault bypass has been removed.

Scope and source-linked characterization follow-up

The known-path methods also execute the real AppDelegate main/embedded flag transitions (including a first main enable), actual shared-pin window closure, the owned test host's WebKit process faults and cache reload. P5's final app method passed (signed-pinned-single-probe.xcresult, 73.806 s): after two healthy/idle cycles, real window closure causes one failure with no page initResp, tab_in_context=false, matching controller and background creation bucket 2; regular and new-tab controls remain healthy. Longer app attempts reached the enabled DEBUG MainWindow lifetime guard during the second timeout and remain documented as red experiments. Persistent P5/stuck proof belongs to the separate SPM suite, not this narrower app method.

The persistent-cache characterization now completes cleanup (signed-large-cache-session-end.xcresult, 47.519 s). A fresh UUID website/controller profile actually receives 20,000 cacheable resources and has disk-cache records; real reload changes the extension baseURL. In that run, immediate post-reload messaging was healthy and p3NoBackgroundFailure=false. Passing this test means the experiment and recovery/cleanup controls succeeded; P3 itself was not reproduced. _terminateNetworkProcess is used only in teardown of the owned test host, after evidence capture/data clearing, to permit removal of that UUID store. It adds no selector to production diagnostics and changes no telemetry schema.

P1 slow idle wake and P2 real WebKit load timeout have signed app passes; the latter records actual error 6 near the 30-second deadline after stopping the owned worker. These are macOS mechanisms, not proof of the causal attribution of iOS production groups. Runtime listener-metadata loss, iOS suspend/resume/device errors and actual production flag changes remain unresolved. The final current-source regression outcome will be recorded in the main status; original CI/PR and ClickHouse snapshots above are dated 15 October and have not been refreshed by these local app tests.

Final current-source verification

The final build uses owned UUID extension resources under unswizzled NSTemporaryDirectory(), outside TestRunHelper's shared cleanup root. The same 18-method selection that previously failed now passes completely (signed-isolated-regression.xcresult): actual Network termination fails 4/4 tabs across two windows and explicit reload/readiness restores all four; the following slow-wake test also passes. The prior deleting actor remains unidentified. Final P5 passes separately in 72.644 s; persistent cache passes separately in 43.053 s, records p3NoBackgroundFailure=false, and removes its own UUID store. All three XCTest summaries have zero failures, skips and expected failures. The 20 exported reports, source hashes and trial history are indexed by /private/tmp/cpm-app-integration-20261015/app-test-results.json. Strict SwiftLint on all 11 changed Swift files has zero violations; both Integration Tests targets contain the six new test files. Main app/XCTest use normal Apple Development signing; no unsigned or SecureVault bypass remains. The nested VPN Debug packaging deep-verify limitation is recorded in the main status and is not claimed fixed.

Rerun requested by user — 16 Oct 04:37 UTC

Current versions of the formerly failed selections passed again: 18+1+1 methods, zero XCTest failures/skips/expected failures, same signed binary/source hashes. Artifacts: /private/tmp/cpm-app-integration-20261015/rerun-failed-20261016-042858/rerun-summary.json. P3 remains negative (20k cache entries). Cache teardown logs a nonfatal SQLite vnode unlinked while in use warning for its owned store; this infrastructure warning and the historical longer P5 limitation are not claimed resolved. The current P5 repeats one failed navigation plus controls; the old two-timeout source variant was not rerun as if fixed. Repeatability does not change the controlled-fault versus production-cause boundary documented at the top of this design.

P1/P2 local fix validation — 16 Oct, background-only candidate

The first P2 full-context-reload prototype was rejected: a repeated real error-6 test lost the existing page's initResp despite a successful native dashboard. The current local candidate preserves WKWebExtensionContext and retries the public background-load API after the actual load-error notification. Its app regression requires recovery of the same failed document without navigation or context replacement, with both native dashboard and page initResp; the observed pixel is recovered_without_extension_reload / recovery_document=failed_navigation / recovery_after=30s_to_2m. The existing confirmed-hang M4 remains the full-reset fallback.

P1 prepares background on each main-frame web navigation with a real 5 s bound/cancellation; the imposed 8 s startup now yields a healthy first document. This neither accelerates worker execution nor establishes a universal fix for longer delays. P1/P2 point cases initially passed 2/2; the package passed 806 tests with 14 opt-in skips, 0 failures. Full final app matrix, disabled-fix controls and limits remain in cpm-status-2026-10-15 and /private/tmp/cpm-p1-p2-fixes-20261016/. No telemetry schema change, commit, PR mutation or iOS production attribution is implied.