diff --git a/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md b/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md index 71441c69..994583c6 100644 --- a/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md +++ b/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md @@ -40,13 +40,13 @@ The monitor keeps, per tab: the time of the latest eligible commit (until a mess | Event | Effect | |---|---| -| Commit of an http(s) page, extension loaded, not an error page, not a history restore, not a restored session | The tab waits for a message. A message up to 2 s before the commit answers it (the extension can message a tab before the app reports the commit), also for a tab whose extension tab is not learned yet. | +| Commit of an http(s) page, extension loaded, not an error page, not a history restore, not a restored session (tabs send the kind of navigation, the monitor decides) | The tab waits for a message. A message up to 2 s before the commit answers it (the extension can message a tab before the app reports the commit), also for a tab whose extension tab is not learned yet. | | Message from the current extension (dashboard refresh) | Answers the tab's page, even when late. Ends the series of silent tabs. | | `config_unavailable` or `settings_missing` refresh | Not an answer: the extension said it cannot answer the page's init. It still shows the extension is alive (restart budget reset). | | Any event, with a commit older than 30 s and no message | The tab is silent: reported once. Tabs found silent in the same check are all `many`. | | New commit or tab close | A page that already waited 30 s is reported first; a younger one is dropped. | | Web content process terminated | The tab's page is dropped, not reported. | -| App suspended (iOS background, macOS sleep) | Nothing is checked, and the suspended time is added back to every wait; a page committed while suspended starts waiting at resume. The clock is time since boot, so a wall-clock change cannot make tabs overdue. | +| App suspended (iOS background, macOS sleep) | Silent tabs are checked as the suspension begins, so a tab already past 30 s is reported when iOS goes to the background. While suspended nothing is checked, and the suspended time is added back to every wait; a page committed while suspended starts waiting at resume. The clock is time since boot, so a wall-clock change cannot make tabs overdue. | | Extension reload | All waits, mappings and series are forgotten; the open pages stay, so tabs are learned again. | ### Tying a message to a tab @@ -87,20 +87,46 @@ The episode pixel `web_extension_cpm_episode` with its timers, episode state and | 3, `alex/cpm-breakage-metric` (`a696254ca3`) | Per-tab silence, the two pixels, `tabs`, definitions, adapters | local, one commit on PR 2; 1048 package tests pass; validators pass; not pushed | | 4, `alex/cpm-init-marks` (`8751a745ab`) | Init progress marks and `last_stage`; extension #3727 | local, one commit on PR 3; 1071 package tests pass; not pushed | +### Reporting cases + +Each case was run against the monitor (PR 3, 30 s window, 2 s tolerance) with an injected clock on 11 Nov; the outcome column is what the code did. "Hidden" is a missed breakage, "wrong" is a report for a working extension. + +| # | Case | Outcome | +|---|---|---| +| 1 | Page commits, the extension answers within 30 s | Nothing. | +| 2 | Answer arrives after 30 s with no event in between | Nothing: the answer is applied before the check. | +| 3 | Answer arrives after 30 s but another event came first | Tab reported `one`; the late answer then ends the series. | +| 4 | One page silent, the user stays on it, then any event | `one`. | +| 5 | Same tab, three silent pages in a row (30 s+ each) | `one`, `one`, `one`; never persistent (one tab). | +| 6 | Two tabs silent one after the other | `one`, then `many` and persistent once. | +| 7 | Two tabs silent in the same check | `many`, `many` and persistent once. | +| 8 | A third tab gets an answer, then two more tabs go silent | `many`, `many`, persistent again (the answer ended the series). | +| 9 | Dead extension, the user navigates every 20 s | Nothing: pages younger than 30 s are dropped. | +| 10 | iOS goes to the background with a tab already past 30 s | Reported at the suspension. | +| 11 | Suspended for 100 s with 10 s already waited | Not silent at 29 s of app time, silent at 31 s: suspended time does not count. | +| 12 | `config_unavailable` / `settings_missing` refresh | Page reported `one`: not an answer. | +| 13 | Two tabs on the same host and path (for example two `duckduckgo.com/?q=` result pages), one message | Both answered. Hidden if one was silent. | +| 14 | Late refresh of the previous page on a mapped tab, after the next commit | Answers the new page. Hidden. | +| 15 | Page path changes (single-page app) before the first message | No tab matches, nothing answered, tab reported `one`. Wrong. | +| 16 | Message from tab C (back/forward commit, not tracked) while tab A waits on the same host and path | C's extension tab is mapped to A. When C then loads an eligible page and the extension answers, C is reported `one`. Wrong. | + +Cases 15 and 16 are the only paths that report a working extension; the design's "ambiguity answers every candidate" does not cover them. A possible fix, not implemented: treat a mapping as invalid when the message's host and path match a different committed page than the mapped tab's, and learn again. + ### Known limits -- Evaluation happens only on events. A silent tab in an app that then sits idle until it is killed is not reported. -- The first message of a tab can only be tied to a tab by URL. Two tabs on the same URL, or a page whose path changed before the first message, stay unmapped; their messages answer every candidate tab, which hides some real silence. -- A single-page app changing its path can break the host and path match for a tab that is not mapped yet. -- Pages the extension does not inject into (some non-document types) send no message and look silent. The MIME type is not available at commit, so these are not filtered. -- A message within 2 s before a commit is counted as that commit's answer even when it belonged to the previous page. -- A message from an extension tab that already belongs to another app tab is never counted as an early answer (or mark) for a new tab's commit. A late refresh from the previous page of a mapped tab, arriving after the next commit, still answers the new page; marks are checked against the current page, answers are not. +- Evaluation happens only on events. A silent tab in an app that then sits idle until it is killed is not reported; iOS reports it at the background transition, macOS at sleep. +- The page match ignores the query, so every result page of a search site, or every watch page of a video site, has the same identity. Tabs on such pages stay unmapped and their messages answer every candidate (case 13). +- A page whose path changed before the first message matches no tab and is reported silent although the extension answered (case 15). +- A message from an untracked tab (back/forward or restored page) can teach the wrong tab when exactly one other tab waits on the same host and path; that tab's later commits then count as silent (case 16). Low likelihood, not guarded. +- Pages the extension does not inject into (some non-document types) send no message and look silent. The MIME type is not available at commit, so these are not filtered. Whether the extension messages on every page the app counts as eligible (search result pages, `duck.ai`, non-document types) is not verified: the extension source is not in this repo. +- A message within 2 s before a commit is counted as that commit's answer even when it belonged to the previous page, in the same tab or another one that is not mapped yet. +- A message from an extension tab that already belongs to another app tab is never counted as an early answer (or mark) for a new tab's commit. A late refresh from the previous page of a mapped tab, arriving after the next commit, still answers the new page (case 14); marks are checked against the current page, answers are not. - A reload of the same page within 2 s of its last mark starts with that mark's stage; `last_stage` can then show the previous load's furthest stage instead of `none`. -- A first message can teach the wrong tab when another tab sits on the same host and path but is not in the committed pages (back/forward or restored page) inside the 2 s window; that tab's later commits then count as silent. Low likelihood, not guarded. - The dashboard refresh is sent independently of `initResp`, so a page whose `initResp` was never delivered counts as answered. - The first series has no overlap with the episode series: comparison is by app version against the previous derived number. - The restart budget now resets on any dashboard refresh from the current embedded extension, not on a message matched to a page, and without the old http(s) and generation filters. -- A page that navigates before the 30 s window ends is dropped, so a user who navigates every 20 s with a dead extension is never reported. +- A page that navigates before the 30 s window ends is dropped, so a user who navigates every 20 s with a dead extension is never reported (case 9). +- A single tab can never make the persistent pixel: it needs two distinct tabs. A user with a dead extension who browses in one tab sends `breakage` only. - If the native channel itself is lost, every tab looks silent and `tabs=many` follows: the metric cannot tell a lost channel from a broken extension. ### Cons @@ -108,11 +134,12 @@ The episode pixel `web_extension_cpm_episode` with its timers, episode state and - Silence of tabs left open until the app exits is never reported. - 30 s hides breakage on short pages. - Streak and series reset on relaunch. -- The tab mapping is a heuristic bootstrap; its error direction is missing a breakage, not inventing one. +- The tab mapping is a heuristic bootstrap. It mostly errs towards missing a breakage, but two cases report a working extension (cases 15 and 16). ### Risks -- A page that works but whose message is not attributed counts as silent when the mapping is wrong. Mitigation: ambiguous messages answer all candidates. +- A page that works but whose message is not attributed counts as silent when the mapping is wrong. Mitigation: ambiguous messages answer all candidates; an unambiguous wrong match (case 16) and a path change (case 15) are not covered. +- Fire (burner) tabs are measured: the embedded extension is trusted and gets private browsing access, so it runs in them (checked in the loader and the permission controller). - Pixel volume for a persistently broken install: one count request per silent tab. - The Debug "Simulate CPM Messaging Breakage" switch must keep hiding messages (it does; tests cover it). - Both definitions have no `expires`: they are the permanent metric (`REVIEW-pixels.md`). @@ -121,9 +148,9 @@ The episode pixel `web_extension_cpm_episode` with its timers, episode state and - Unit tests per row of the table (window boundary, no timer, late answer, tolerance before commit, suspension, ineligible commits, crash, close, reload, mapping learned, kept and ambiguous, simulation, old extension, `tabs` one/many, persistent once and again after a message, closed silent tab). Mutation checks: answer applied after the check, no suspension shift, persistent on one tab, no tolerance each make a test fail. - Pixel definitions pass both validators; adapter tests check names, frequency and parameters. -- App-hosted macOS and iOS tests: not run in this stack; package tests, validators, SwiftLint and the macOS `build-for-testing` are the checks. +- Package tests, validators, SwiftLint and `build-for-testing` on both platforms. App-hosted, signed: iOS (simulator) `TabTests` and `WebExtensionPixelFiringTests` 61 tests, macOS unit tests 16 (`MacOSWebExtensionPixelFiringTests`, `WebExtensionsDebugMenuTests`, `NavigationPixelNavigationResponderTests`) and the integration `CPMErrorPageMonitoringTests` 3 tests pass. +- The reporting cases above were run once as a throwaway test, which is not kept. Existing unit tests cover cases 1, 2, 4 to 13; cases 3, 14, 15 and 16 have no test of their own. - Manual: [[CPM Breakage Testing Steps]] with the Debug switch. -- iOS `build-for-testing` (iOS Unit Tests, simulator) passes on PR 4. - Real WebKit (CPM repro harness, [PR #7378](https://github.com/duckduckgo/apple-browsers/pull/7378)): not run. ## Additional Considerations (if applicable)