Files
obsidian-vault/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md
T

131 lines
8.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
tags:
- work
- cpm
- tech-design
- pixel
updated: '2026-11-10'
status: draft
---
# TD: Pixel Definition: CPM Permanent Breakage Metric (macOS/iOS)
> Internal metrics, keep out of public PRs. Companion: [[cpm-wake-init-detection-tech-design-2026-11-10]] (adds `last_stage`). Where we are: [[cpm-project-summary-2026-11-03]].
**Author:** Alex M
**Reviewer:** TBD (previous CPM pixel designs: Michal)
**Stakeholders:** Michal, Konrad, Russell (from the previous Tech Designs; confirm)
**Project:** [CPM: root-cause the stuck embedded extension messaging (macOS/iOS)](https://app.asana.com/1/137249556945/task/1216761517055116)
**Task:** to create ("Permanent breakage metrics")
**Previous Tech Designs:** [[cpm-readiness-telemetry-tech-design-2026-10-15]], [[cpm-m3-telemetry-privacy-triage]]
**Privacy Triage:** to request (new pixel names)
## Background & Requirements
The goal is permanent CPM failures below 1% of DAU. There is no pixel for it; the number is derived from `web_extension_cpm_episode` with a long SQL. The episode is wrong in two ways:
- **A timer decides.** A four-second navigation timeout opens an episode. On iOS 7.244, 6.0% of episodes closed with a late answer on the same page, 42% of those later than 30 s. Slow answers count as failures.
- **Episodes are lost.** An open episode lives in memory and is dropped when the app exits.
Requirements: a metric without a timer, as a rate against DAU; no page content, URLs or persistent identifiers. End state: the only CPM pixels are the two defined here.
## Problem Statement
Report pages that really never got the extension's answer, and tell persistent breakage from a single page.
## Recommended Approach
**A page is healthy when an answer reaches it at any time while its document lives. A page that never got an answer is judged when its document ends. There are no navigation timers.**
| Event | Effect |
|---|---|
| Navigation finished, extension loaded | The document becomes judgeable and its clock starts. If the extension loads later, the clock starts then. |
| Attributed answer (dashboard refresh) | Healthy. Resets the streak. |
| Document ends: new navigation or tab close | If unanswered and alive at least **30 s**: breakage. Shorter silence is not judged (it may be a slow wake). |
| Web content process terminated | Not judged. The tab's next navigation gets `reason=tab_crash`. |
| Extension signals `cpmStage` = `config_unavailable` or `settings_missing` | Breakage at once, no lifetime limit. The signal normally arrives while the page is still loading, so it is remembered and applied when the page finishes. |
| Answer that cannot be tied to one document | That document is not judged. |
| App suspended | Nothing is judged while suspended. |
Judged documents are never re-judged. Measurement state never expires (the old five-minute drop is gone).
**Series:** reported pages that started after the latest answered page, any tab, in memory. Pages are ordered by navigation start, not by the time of the verdict, so a page judged late (tab closed long after) does not join a series that an answered newer page has already ended. At 3 the persistent pixel fires once for the series.
### Pixels
| Pixel | Trigger | Frequency |
|---|---|---|
| `web_extension_cpm_breakage` | A document is judged broken. | Daily + count |
| `web_extension_cpm_breakage_persistent` | The streak reaches 3. Once per streak. | Daily + count |
`_daily` is installs per day; `_count` is events. Metadata follows the existing episode pixel on each platform: `appVersion` on both; macOS also `pixelSource`, `channel`, suffix `daily_count`; iOS keeps its suffix set. The macOS definition must match what is sent.
`web_extension_cpm_breakage` parameters:
| Parameter | Values |
|---|---|
| `reason` | `session_restoration`, `tab_crash`, `extension_reload`, `other` (as in the episode pixel). |
| `background` | `woken` (a background view was created during this navigation), `warm` (views were seen, none created for it), `unknown` (no background view seen yet). From the manager's `didCreateBackgroundWebView` callback. |
`web_extension_cpm_breakage_persistent` has no parameters. `last_stage` is added by the companion design.
Dashboard: persistent installs / DAU from `breakage_persistent_daily` (the success metric); any-breakage installs / DAU from `breakage_daily`; breakages per day from `breakage_count` by `reason` and `background`.
### Removed in the same PR
The episode pixel `web_extension_cpm_episode` with its timers and episode state, `web_extension_cpm_background_load_restart` (and its observation), and `web_extension_cpm_network_process_recovery_applied`: definitions, events and firing code. The recovery behavior (error-6 restart, network recovery) stays; only the pixels go.
### First attempt (`e6419bb910`): defects fixed here
Measurements older than five minutes dropped; no minimum lifetime; tab process crash counted; `background` from load duration and one app-wide signal; macOS definition mismatch; `navigationWaitContexts` leak (the wait-context set is deleted); no `cpmStage` test.
### PR stack
| PR | Content | State |
|---|---|---|
| 1, [#8025](https://github.com/duckduckgo/apple-browsers/pull/8025) | Remove ineffective recovery fixes and flags | pushed, in review |
| 2, `alex/cpm-pixel-cleanup` | Remove diagnostics, trim the episode pixel | local, tests pass, not pushed |
| 3, `alex/cpm-breakage-metric` (`b8a4361c26`) | This design: new pixels and removal of the old ones, one commit | local; package tests 1013 pass; both pixel validators pass; macOS app and tests build; iOS build running; not pushed |
| 4, `alex/cpm-init-marks` (`7bb259e9cf`) | Marks and `last_stage` ([[cpm-wake-init-detection-tech-design-2026-11-10]]) | local; package tests 1023 pass; validators pass; not pushed. Extension: `alex/cpm-init-progress` has the new commit `908f9a62`, not pushed |
### Known limits
- A page that never reaches `navigationFinished`, or a document alive when the app is killed, is not judged.
- If a new navigation fails after `navigationStarted`, the old document was already judged and reset; its late answer is not attributed.
- A mark or answer for a reloaded page with the same URL cannot be told from the old document's.
- The dashboard refresh is sent independently of `initResp`, so a page whose `initResp` was never delivered counts as answered unless the extension reports `response_failed`.
- `page_init_sent` travels through the same background, so `last_stage=none` cannot tell a lost mark from an init that never started.
### Cons
- Documents alive when the app exits or suspends are never judged; breakage on pages left open is missed.
- 30 s hides breakage on short pages.
- The series is new: no overlap with the episode series, so comparison is by app version against the previous derived number.
- Streak resets on relaunch.
### Risks
- A page that works but whose answer is not attributed counts as breakage (`other_tab_response` was 37–53% of outcomes). Documents with an unattributable answer are not judged.
- Pixel volume for a persistently broken install: one count request per page.
- The Debug "Simulate CPM Messaging Breakage" switch must keep hiding answers.
## Testing
- Unit tests per row of the table, streak at 3 once and reset, `reason` values, `background` values, simulation switch, `cpmStage` parsing, extension reload.
- Pixel definitions pass both validators; parameters match what is sent.
- Manual: [[CPM Breakage Testing Steps]] with the Debug switch.
- Real WebKit (CPM repro harness, [PR #7378](https://github.com/duckduckgo/apple-browsers/pull/7378)): not run.
## Additional Considerations (if applicable)
**Privacy:** two new pixel names with enumerated parameters; no URLs, identifiers or page content. Privacy Triage required, not requested yet.
**Security:** no new private WebKit selectors.
**Site Breakage:** none; the detector only observes events and recovery is unchanged.
**Experimentation:** none.
**Operational:** pixel expiry 2027-01-15 as existing definitions. Danger label for the removed pixels. Data checks: `metrics.pixels_aggregation` and `metrics._pixels_new` (`pixel_id = 'web'`).
**Localization / Internationalization:** none.