131 lines
8.1 KiB
Markdown
131 lines
8.1 KiB
Markdown
---
|
||
tags:
|
||
- work
|
||
- cpm
|
||
- tech-design
|
||
- pixel
|
||
updated: '2026-11-10'
|
||
status: draft
|
||
---
|
||
# TD: Pixel Definition: CPM Permanent Breakage Metric (macOS/iOS)
|
||
|
||
> Internal metrics, keep out of public PRs. Companion: [[cpm-wake-init-detection-tech-design-2026-11-10]] (adds `last_stage`). Where we are: [[cpm-project-summary-2026-11-03]].
|
||
|
||
**Author:** Alex M
|
||
**Reviewer:** TBD (previous CPM pixel designs: Michal)
|
||
**Stakeholders:** Michal, Konrad, Russell (from the previous Tech Designs; confirm)
|
||
**Project:** [CPM: root-cause the stuck embedded extension messaging (macOS/iOS)](https://app.asana.com/1/137249556945/task/1216761517055116)
|
||
**Task:** to create ("Permanent breakage metrics")
|
||
**Previous Tech Designs:** [[cpm-readiness-telemetry-tech-design-2026-10-15]], [[cpm-m3-telemetry-privacy-triage]]
|
||
**Privacy Triage:** to request (new pixel names)
|
||
|
||
## Background & Requirements
|
||
|
||
The goal is permanent CPM failures below 1% of DAU. There is no pixel for it; the number is derived from `web_extension_cpm_episode` with a long SQL. The episode is wrong in two ways:
|
||
|
||
- **A timer decides.** A four-second navigation timeout opens an episode. On iOS 7.244, 6.0% of episodes closed with a late answer on the same page, 42% of those later than 30 s. Slow answers count as failures.
|
||
- **Episodes are lost.** An open episode lives in memory and is dropped when the app exits.
|
||
|
||
Requirements: a metric without a timer, as a rate against DAU; no page content, URLs or persistent identifiers. End state: the only CPM pixels are the two defined here.
|
||
|
||
## Problem Statement
|
||
|
||
Report pages that really never got the extension's answer, and tell persistent breakage from a single page.
|
||
|
||
## Recommended Approach
|
||
|
||
**A page is healthy when an answer reaches it at any time while its document lives. A page that never got an answer is judged when its document ends. There are no navigation timers.**
|
||
|
||
| Event | Effect |
|
||
|---|---|
|
||
| Navigation finished, extension loaded | The document becomes judgeable and its clock starts. If the extension loads later, the clock starts then. |
|
||
| Attributed answer (dashboard refresh) | Healthy. Resets the streak. |
|
||
| Document ends: new navigation or tab close | If unanswered and alive at least **30 s**: breakage. Shorter silence is not judged (it may be a slow wake). |
|
||
| Web content process terminated | Not judged. The tab's next navigation gets `reason=tab_crash`. |
|
||
| Extension signals `cpmStage` = `config_unavailable` or `settings_missing` | Breakage at once, no lifetime limit. The signal normally arrives while the page is still loading, so it is remembered and applied when the page finishes. |
|
||
| Answer that cannot be tied to one document | That document is not judged. |
|
||
| App suspended | Nothing is judged while suspended. |
|
||
|
||
Judged documents are never re-judged. Measurement state never expires (the old five-minute drop is gone).
|
||
|
||
**Series:** reported pages that started after the latest answered page, any tab, in memory. Pages are ordered by navigation start, not by the time of the verdict, so a page judged late (tab closed long after) does not join a series that an answered newer page has already ended. At 3 the persistent pixel fires once for the series.
|
||
|
||
### Pixels
|
||
|
||
| Pixel | Trigger | Frequency |
|
||
|---|---|---|
|
||
| `web_extension_cpm_breakage` | A document is judged broken. | Daily + count |
|
||
| `web_extension_cpm_breakage_persistent` | The streak reaches 3. Once per streak. | Daily + count |
|
||
|
||
`_daily` is installs per day; `_count` is events. Metadata follows the existing episode pixel on each platform: `appVersion` on both; macOS also `pixelSource`, `channel`, suffix `daily_count`; iOS keeps its suffix set. The macOS definition must match what is sent.
|
||
|
||
`web_extension_cpm_breakage` parameters:
|
||
|
||
| Parameter | Values |
|
||
|---|---|
|
||
| `reason` | `session_restoration`, `tab_crash`, `extension_reload`, `other` (as in the episode pixel). |
|
||
| `background` | `woken` (a background view was created during this navigation), `warm` (views were seen, none created for it), `unknown` (no background view seen yet). From the manager's `didCreateBackgroundWebView` callback. |
|
||
|
||
`web_extension_cpm_breakage_persistent` has no parameters. `last_stage` is added by the companion design.
|
||
|
||
Dashboard: persistent installs / DAU from `breakage_persistent_daily` (the success metric); any-breakage installs / DAU from `breakage_daily`; breakages per day from `breakage_count` by `reason` and `background`.
|
||
|
||
### Removed in the same PR
|
||
|
||
The episode pixel `web_extension_cpm_episode` with its timers and episode state, `web_extension_cpm_background_load_restart` (and its observation), and `web_extension_cpm_network_process_recovery_applied`: definitions, events and firing code. The recovery behavior (error-6 restart, network recovery) stays; only the pixels go.
|
||
|
||
### First attempt (`e6419bb910`): defects fixed here
|
||
|
||
Measurements older than five minutes dropped; no minimum lifetime; tab process crash counted; `background` from load duration and one app-wide signal; macOS definition mismatch; `navigationWaitContexts` leak (the wait-context set is deleted); no `cpmStage` test.
|
||
|
||
### PR stack
|
||
|
||
| PR | Content | State |
|
||
|---|---|---|
|
||
| 1, [#8025](https://github.com/duckduckgo/apple-browsers/pull/8025) | Remove ineffective recovery fixes and flags | pushed, in review |
|
||
| 2, `alex/cpm-pixel-cleanup` | Remove diagnostics, trim the episode pixel | local, tests pass, not pushed |
|
||
| 3, `alex/cpm-breakage-metric` (`b8a4361c26`) | This design: new pixels and removal of the old ones, one commit | local; package tests 1013 pass; both pixel validators pass; macOS app and tests build; iOS build running; not pushed |
|
||
| 4, `alex/cpm-init-marks` (`7bb259e9cf`) | Marks and `last_stage` ([[cpm-wake-init-detection-tech-design-2026-11-10]]) | local; package tests 1023 pass; validators pass; not pushed. Extension: `alex/cpm-init-progress` has the new commit `908f9a62`, not pushed |
|
||
|
||
### Known limits
|
||
|
||
- A page that never reaches `navigationFinished`, or a document alive when the app is killed, is not judged.
|
||
- If a new navigation fails after `navigationStarted`, the old document was already judged and reset; its late answer is not attributed.
|
||
- A mark or answer for a reloaded page with the same URL cannot be told from the old document's.
|
||
- The dashboard refresh is sent independently of `initResp`, so a page whose `initResp` was never delivered counts as answered unless the extension reports `response_failed`.
|
||
- `page_init_sent` travels through the same background, so `last_stage=none` cannot tell a lost mark from an init that never started.
|
||
|
||
### Cons
|
||
|
||
- Documents alive when the app exits or suspends are never judged; breakage on pages left open is missed.
|
||
- 30 s hides breakage on short pages.
|
||
- The series is new: no overlap with the episode series, so comparison is by app version against the previous derived number.
|
||
- Streak resets on relaunch.
|
||
|
||
### Risks
|
||
|
||
- A page that works but whose answer is not attributed counts as breakage (`other_tab_response` was 37–53% of outcomes). Documents with an unattributable answer are not judged.
|
||
- Pixel volume for a persistently broken install: one count request per page.
|
||
- The Debug "Simulate CPM Messaging Breakage" switch must keep hiding answers.
|
||
|
||
## Testing
|
||
|
||
- Unit tests per row of the table, streak at 3 once and reset, `reason` values, `background` values, simulation switch, `cpmStage` parsing, extension reload.
|
||
- Pixel definitions pass both validators; parameters match what is sent.
|
||
- Manual: [[CPM Breakage Testing Steps]] with the Debug switch.
|
||
- Real WebKit (CPM repro harness, [PR #7378](https://github.com/duckduckgo/apple-browsers/pull/7378)): not run.
|
||
|
||
## Additional Considerations (if applicable)
|
||
|
||
**Privacy:** two new pixel names with enumerated parameters; no URLs, identifiers or page content. Privacy Triage required, not requested yet.
|
||
|
||
**Security:** no new private WebKit selectors.
|
||
|
||
**Site Breakage:** none; the detector only observes events and recovery is unchanged.
|
||
|
||
**Experimentation:** none.
|
||
|
||
**Operational:** pixel expiry 2027-01-15 as existing definitions. Danger label for the removed pixels. Data checks: `metrics.pixels_aggregation` and `metrics._pixels_new` (`pixel_id = 'web'`).
|
||
|
||
**Localization / Internationalization:** none.
|