160 lines
13 KiB
Markdown
Executable File
160 lines
13 KiB
Markdown
Executable File
---
|
||
updated: '2026-10-20'
|
||
---
|
||
# TD: Pixel Definition: CPM Failure Diagnostics (macOS/iOS)
|
||
|
||
> **Status checked 20 October 2026.** This note preserves the original M3 schema merged in #6844, with later production evidence in iOS 7.240+ / macOS 1.210+. It is not the schema of the current #7376 branch.
|
||
>
|
||
> #7376 `5d1cb4abd2` replaces 11 separate CPM events with one terminal `web_extension_cpm_episode`, **daily+count**; it retains eight booleans and bounded `bg_events`, adds the episode/load fields, and drops nine old wire fields including `context_errors` and `network_restarted`. Graveyard treatment attribution is explicitly removed. Exact lists and values: [[cpm-pr0-episode-telemetry-summary-2026-10-16]]. That PR is closed without merge; macOS #7484 is an older contract.
|
||
>
|
||
> New `stuck` and `initial;continuation` history/`...` are later changes than the quoted triage's `was_stuck`. Approval of that latest combination was not established here. Sanitization/buckets and absence of IDs do not eliminate combined distinguishability. Current evidence, actual gates and remaining gaps: [[cpm-status-2026-10-15]].
|
||
|
||
## Historical context — recorded 15 October 2026
|
||
|
||
> **Internal status context — 2026-10-15:** [[cpm-status-2026-10-15]] is the current source for production cohort evidence, PR stack and local reproduction results. The merge/rollout statement below is the historical September 23 record; it must not hide the subsequently collected diagnostic data for iOS 7.240+ and macOS 1.210+.
|
||
>
|
||
> Current P5/shared-window and conditional P6/main-flag tests exercise the existing diagnostics; they add no pixel/schema changes. Reproducing a mechanism does not establish its production share, a flag transition, or page delivery from dashboard-only telemetry. Latest readiness proposal and unresolved privacy/review choices are separate: [[cpm-readiness-telemetry-tech-design-2026-10-15]], main status §6–8.
|
||
|
||
|
||
|
||
**Author:** Alex M
|
||
**Reviewer:** Michal
|
||
**Stakeholders:** Michal, Konrad, Russell (from the first Tech Design; confirm for this proposal)
|
||
**Project:** [CPM: root-cause the stuck embedded extension messaging (macOS/iOS)](https://app.asana.com/1/137249556945/task/1216761517055116)
|
||
**Previous Tech Design:** [CPM breakage pixels](https://app.asana.com/1/137249556945/task/1218005985662983)
|
||
**Tech Design:** [Asana](https://app.asana.com/1/137249556945/task/1218619301158188)
|
||
**Privacy Triage:** [Asana](https://app.asana.com/1/137249556945/task/1218627883499612)
|
||
**Historical status (2026-09-23):** [PR #6844](https://github.com/duckduckgo/apple-browsers/pull/6844) merged 2026-09-23; CI green. Merge commit `2d58ad8cf4fae728a92e2aae0d3ec6073ddf49f2`. Production rollout not yet confirmed.
|
||
|
||
## Background & Requirements
|
||
|
||
The first CPM pixels detect initialization failure, a stuck messaging episode and recovery. They do not show whether the background process stopped, the extension failed to load, or the tab used the wrong extension configuration.
|
||
|
||
Terminating the extension's background process can leave CPM unable to respond, even after page reloads or opening new tabs. Add process, extension and tab state to existing iOS/macOS failure pixels, even when the extension cannot respond.
|
||
|
||
## Problem Statement
|
||
|
||
Distinguish background-process and extension-context failures from problems affecting one tab, and identify failed reloads. Collect no page content, URLs or persistent identifiers.
|
||
|
||
This proposal covers diagnostic collection only; it does not change recovery behavior or introduce an experiment.
|
||
|
||
## Recommended Approach
|
||
|
||
Keep the existing failure/stuck detection. A shared recorder holds lifecycle and critical-memory-pressure events in memory. When a pixel fires, it captures current WebKit/tab state synchronously on the main actor. Earlier context loads cannot add events to this timeline. There are no JavaScript probes or periodic uploads.
|
||
|
||
`cpmDiagnosticsRecorder` controls recorder creation when the extension manager is created; changes apply to the next manager instance, normally after app restart. Disabled: no recorder or proxy, and existing failure/stuck pixels carry no recorder diagnostics.
|
||
|
||
A delegate proxy records process termination and responsiveness, then forwards callbacks to WebKit. WebKit's delegate owns the proxy; the proxy holds it weakly. The separate `cpmBackgroundDelegateProxy` flag applies at runtime: enabling installs the proxy on the current view; disabling restores original delegates on all surviving views, including after unload. Other recorder diagnostics remain available. Both flags default to enabled and support local overrides.
|
||
|
||
| Pixel | Trigger | Frequency |
|
||
|---|---|---|
|
||
| `debug_web_extension_cpm_initialization_failed_after_<reason>` | Existing initialization failure; add diagnostic parameters. | Daily |
|
||
| `debug_web_extension_cpm_messaging_stuck_<reason>` | Existing failure confirmed by a later eligible navigation; add diagnostic parameters. | Daily + count, once per episode |
|
||
| `debug_web_extension_reload_failed` | **New:** extension unload/load fails during a reload operation. Applies to all supported extension types. | Daily + count |
|
||
|
||
`<reason>`: `session_restoration`, `tab_crash`, `extension_reload`, `other`. PixelKit applies standard frequency/platform suffixes. Standard metadata includes `appVersion`; macOS also declares `pixelSource` and `channel`.
|
||
|
||
Both existing recovery pixels (`debug_web_extension_cpm_messaging_recovered_without_extension_reload` and `debug_web_extension_cpm_messaging_recovered_after_extension_reload`) gain `recovery_from`: `initialization_failed` or `messaging_stuck`, based on the episode state before recovery. Names, triggers and daily + count frequency stay unchanged.
|
||
|
||
The new reload-failure pixel reports a failed reload operation, unlike the existing `cpm_messaging_extension_reload_failed`, which reports CPM failure **after a successful reload**.
|
||
|
||
### Failure / stuck parameters
|
||
|
||
Unavailable optional facts are omitted. The first nine parameters are booleans (`true` / `false`); PIDs are never sent.
|
||
|
||
| Parameter | What is collected |
|
||
|---|---|
|
||
| `context_loaded` | Context is loaded. |
|
||
| `bg_view_alive` | Current background WKWebView exists. |
|
||
| `bg_process_alive` | Background process identifier is nonzero. |
|
||
| `bg_process_responsive` | WebKit's process-responsiveness verdict. |
|
||
| `network_restarted` | Network PID differs from the value at context load. |
|
||
| `tab_in_context` | Failing tab is in WebKit's open-tabs set. |
|
||
| `tab_controller_match` | Tab and context use the same extension controller. |
|
||
| `tab_has_ext_scripts` | Registered scripts include the expected extension world; not proof of execution. |
|
||
| `handler_registered` | Autoconsent native-message handler is registered; not proof of message delivery. |
|
||
| `critical_memory_age` | Time since critical memory pressure: `none` (not observed), `1m` (<1 min), `5m` (1–<5 min), `30m` (5–<30 min), `over_30` (≥30 min). |
|
||
| `bg_view_creations` | Creations since context load: `0`, `1`, `2`, `3_to_5`, `over_5`. |
|
||
| `bg_view_leaked_count` | Retained views beyond the expected current view; same buckets. Not proof of a leak. |
|
||
| `context_errors` | `none` or accumulated error descriptors: raw domain/code pairs, including the immediate underlying error. No localized text; no explicit payload cap. |
|
||
| `bg_events` | Recent lifecycle sequence with bucketed ages relative to the snapshot. |
|
||
| `app_version_change` | `updated` or `downgraded`; omitted if unchanged or no previous value exists. Fixed for the app session. Version/build values are stored locally, not sent in this parameter. |
|
||
| `app_launch_age` | Time since app startup using the same age buckets as `bg_events`. Foregrounding and recorder recreation do not reset it. |
|
||
|
||
The recorder keeps 40 events. `bg_events` sends up to the last 12, within 255 characters, dropping older entries first. Format: `event_name@age_bucket`. Buckets: `30s` (<30s), `1m` (30–<60s), `2m` (1–<2 min), `5m` (2–<5 min), `15m` (5–<15 min), `over_15` (≥15 min). Events remain oldest first; order and repeated events within a bucket are preserved. Exact ages are not sent.
|
||
|
||
Events: `load`, `view`, `dealloc`, `unresponsive`, `responsive`, `proxy_on`, `proxy_off`, `died_<reason>`, `error_<descriptor>`, `context_unload_failed_<descriptor>`, `extension_files_remove_failed_<descriptor>`, `extension_updated`, `base_url_changed`.
|
||
|
||
- `extension_updated`: successful replacement of an installed CPM extension, not first installation or ordinary reload. The latest update and its original timestamp survive context reloads.
|
||
- `base_url_changed`: base URL differs from the previous context load with the same `uniqueIdentifier` in this recorder's lifetime. No event on first load or identifier change. Neither URL nor identifier is sent. This observes URL changes; it does not implement the stable-base-URL fix.
|
||
|
||
Future `hold` / `release` events belong to the graveyard mitigation and are not part of this PR.
|
||
|
||
Termination reasons: `memory`, `cpu`, `client`, `crash`, `crash_limit`, `unknown`. The `error_` events record background-load failures. CPM unload failures (reload, data clearing or uninstall) and uninstall file-removal failures are also recorded. Failed unloads keep the context; file-removal errors can still be recorded after unload. These entries reach the server only with a later failure/stuck pixel; existing standalone error pixels are unchanged.
|
||
|
||
Timeline labels are lowercased, character-filtered and truncated to 64 characters, which can cut off error codes. `context_errors` and reload error fields preserve raw domain/code pairs.
|
||
|
||
Example: `view@2m,died_crash@1m` means the view was created 1–<2 minutes before the snapshot and its process crashed 30–<60 seconds before it.
|
||
|
||
### Reload-failure parameters
|
||
|
||
| Parameter | Values |
|
||
|---|---|
|
||
| `extension_type` | `embedded`, `darkMode`, `adBlocking`, `searchToken`, `unknown` |
|
||
| `reload_trigger` | `data_clearing`, `scriptlet_update`, `explicit` |
|
||
| `reload_phase` | `unload`, `load`, `light_load`, `full_load`, `fallback_load` |
|
||
|
||
The three fields above are allowlisted enums. The pixel also sends standard error parameters:
|
||
|
||
| Parameter | Values |
|
||
|---|---|
|
||
| `d` (`errorDomain`) | Original NSError domain. |
|
||
| `e` (`errorCode`) | Original numeric NSError code. |
|
||
| `ud` (`underlyingErrorDomain`) | Immediate underlying NSError domain, when present. |
|
||
| `ue` (`underlyingErrorCode`) | Immediate underlying numeric code, when present. |
|
||
|
||
Localized descriptions, file paths and other userInfo contents are not serialized. Only the immediate underlying error is included.
|
||
|
||
## Testing
|
||
|
||
- Unit tests: parameter serialization, buckets, timeline limits, attachment to failure/stuck pixels, reload-error mapping, delegate forwarding, ownership and runtime flag changes.
|
||
- Integration validation on iOS/macOS: force background-process termination, check the emitted diagnostics, test proxy on/off and extension reload failures. Confirm unknown SPI values are omitted.
|
||
- Privacy validation: inspect actual payloads and run both platform schema validators. Confirm URLs, identifiers and error messages are absent. Verify age-bucket boundaries and event order; review error-domain/code values before release.
|
||
|
||
## Additional Considerations (if applicable)
|
||
|
||
**Privacy**
|
||
|
||
New pixel and additional parameters require Privacy Triage. These payloads do not intentionally collect browsing URLs, search queries, page content, message payloads, tab/context IDs or PIDs. Event ages are bucketed; unrestricted domain/code pairs still require Privacy Triage review. `data_clearing` discloses the operation preceding a failed reload.
|
||
|
||
Local context-error diagnostics log only original and immediate underlying error domains/codes, without localized descriptions or other userInfo contents.
|
||
|
||
**Security**
|
||
|
||
Uses guarded private WebKit selectors for process/script facts. Validate supported OS versions; missing facts must not block pixel delivery.
|
||
|
||
**Site Breakage**
|
||
|
||
No intended page-behavior changes. The proxy changed the bug's reproduction in an earlier test. Test the current proxy on/off to check whether it still affects the failure.
|
||
|
||
**Experimentation**
|
||
|
||
None. Recovery changes and experiments are outside this proposal.
|
||
|
||
**Operational**
|
||
|
||
Update the existing CPM Grafana board for the new fields and reload failures. Diagnostic removal is planned for approximately November 2026; no removal date is enforced in code. Feature flag review is due October 23, 2026. Failure-only snapshots cannot measure the failure rate across all process deaths or prove that all tabs failed. Older events, including the trigger, can drop out of the timeline.
|
||
|
||
Feature flag registry tasks (For review, assigned to Alex M):
|
||
|
||
| Flag | iOS | macOS |
|
||
|---|---|---|
|
||
| `cpmDiagnosticsRecorder` | [Task](https://app.asana.com/1/137249556945/project/1211834678943996/task/1218785444896174) | [Task](https://app.asana.com/1/137249556945/project/1211834678943996/task/1218781680888931) |
|
||
| `cpmBackgroundDelegateProxy` | [Task](https://app.asana.com/1/137249556945/project/1211834678943996/task/1218782530802146) | [Task](https://app.asana.com/1/137249556945/project/1211834678943996/task/1218785444683902) |
|
||
|
||
Task descriptions contain the template values; registry custom fields still require Platform, Is failsafe=True and Is permanent=False unless filled manually since creation.
|
||
|
||
**Localization / Internationalization**
|
||
|
||
None.
|