diff --git a/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md b/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md index 607ee3d0..d091a966 100644 --- a/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md +++ b/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md @@ -9,205 +9,114 @@ status: draft --- # TD: Pixel Definition: CPM Permanent Breakage Metric (macOS/iOS) -> Internal metrics, keep out of public PRs. Companion: [[cpm-wake-init-detection-tech-design-2026-11-10]] (adds `last_stage` to the pixel defined here). Where we are: [[cpm-project-summary-2026-11-03]] (section "Состояние на 10 ноября"). Review of the earlier attempt that this design replaces: same section. +> Internal metrics, keep out of public PRs. Companion: [[cpm-wake-init-detection-tech-design-2026-11-10]] (adds `last_stage`). Where we are: [[cpm-project-summary-2026-11-03]]. **Author:** Alex M **Reviewer:** TBD (previous CPM pixel designs: Michal) **Stakeholders:** Michal, Konrad, Russell (from the previous Tech Designs; confirm) **Project:** [CPM: root-cause the stuck embedded extension messaging (macOS/iOS)](https://app.asana.com/1/137249556945/task/1216761517055116) -**Task:** to create ("Permanent breakage metrics", from the plan in [[cpm-project-summary-2026-11-03]]) -**Previous Tech Designs:** [[cpm-readiness-telemetry-tech-design-2026-10-15]], [[cpm-m3-telemetry-privacy-triage]] (both describe pixels this design supersedes) -**Privacy Triage:** to request (new pixel names, see Additional Considerations) +**Task:** to create ("Permanent breakage metrics") +**Previous Tech Designs:** [[cpm-readiness-telemetry-tech-design-2026-10-15]], [[cpm-m3-telemetry-privacy-triage]] +**Privacy Triage:** to request (new pixel names) ## Background & Requirements -The project goal is permanent CPM failures below 1% of DAU (0.2–0.5% counts as success). Today that number does not exist as a pixel. It is derived from `web_extension_cpm_episode` with a long SQL: a four-second navigation timeout opens an episode, a later timeout or reload confirms it, a five-minute deadline or a response closes it. Three problems: +The goal is permanent CPM failures below 1% of DAU. There is no pixel for it; the number is derived from `web_extension_cpm_episode` with a long SQL. The episode is wrong in two ways: -- **The timer measures slowness, not breakage.** On iOS 7.244, 6.0% of episodes closed with a late answer on the same page, 42% of those later than 30 s. A page that answers after 40 s is counted as a failure. -- **Episodes are lost.** An unfinished episode lives in memory; closing or killing the app drops it. The pages that stay broken longest are the ones that end with the app. -- **The pixel carries what the investigation needed** (diagnostic booleans, `bg_events`, load fields) and is being trimmed to five parameters (PR 2 in the stack below). What remains cannot be turned into a small, fixed dashboard set. +- **A timer decides.** A four-second navigation timeout opens an episode. On iOS 7.244, 6.0% of episodes closed with a late answer on the same page, 42% of those later than 30 s. Slow answers count as failures. +- **Episodes are lost.** An open episode lives in memory and is dropped when the app exits. -Requirements: - -- A small fixed set of metrics, as rates against DAU, for the health indicators dashboard. -- Detection that does not depend on a timer and does not depend on reaching a deadline before the app exits. -- No page content, URLs or persistent identifiers. -- Runs next to the episode pixel for one release, so the two series can be compared before the old one is removed. -- End state: the only CPM pixels are `web_extension_cpm_breakage` and `web_extension_cpm_breakage_persistent`. +Requirements: a metric without a timer, as a rate against DAU; no page content, URLs or persistent identifiers. End state: the only CPM pixels are the two defined here. ## Problem Statement -Define and measure, per platform and version: - -1. **Breakage:** an http(s) page that finished loading with the extension loaded, and never got the extension's answer while its document lived. -2. **Persistent breakage:** several documents in a row with no answer in between, meaning the extension is broken for the user, not for one page. -3. How many installs per day are affected, and how many breakages happen. - -The wake investigation (which init step stops) is separate: [[cpm-wake-init-detection-tech-design-2026-11-10]]. +Report pages that really never got the extension's answer, and tell persistent breakage from a single page. ## Recommended Approach -A page is **healthy** as soon as an answer for its document arrives, at any time while the document lives (a late answer is not a failure). A page that never got the answer is **judged when its document ends**. There are no navigation timers. The approach is additive: the existing monitor, its timers and the episode pixel stay unchanged; the new logic is a second consumer of the same events. - -### What was wrong with the first attempt - -The first attempt (branch `alex/cpm-breakage-pixel`, commit `e6419bb910`) already used page-end judging, but combined it with deleting the episode pixel and had these defects, all addressed below: - -1. Unanswered measurements older than five minutes were dropped, so the longest-silent pages were never reported. -2. No minimum lifetime: a page left 200 ms after loading counted as broken. -3. A web content process crash counted as extension breakage. -4. `background` was derived from "background load took at least one second" and from one app-wide signal, not from the page's navigation. -5. Pages alive when the app exits were never judged. -6. The macOS pixel definition omitted `pixelSource` and `channel` and used iOS-only suffixes. -7. `navigationWaitContexts` leaked an entry when the context was replaced, and `cpmStage` parsing had no test. - -### Detection rules - -Per eligible document: http(s), not a browser error page, not a history restore from cache (existing eligibility rules, unchanged). +**A page is healthy when an answer reaches it at any time while its document lives. A page that never got an answer is judged when its document ends. There are no navigation timers.** | Event | Effect | |---|---| -| Navigation finished, extension loaded | Document becomes **judgeable** and its clock starts. If the extension is not loaded yet, it becomes judgeable when the extension loads while the page lives. | -| Attributed dashboard response (later also `response_sent`, see the companion design) | Healthy. Ends judging for this document. Resets the streak. | -| Response that cannot be tied to one document | The document is not judgeable (neither outcome provable). | -| Document ends by a new navigation or tab close | Judge now. | -| App suspends (iOS to background, macOS sleep) or macOS terminates | Judge every pending judgeable document now, once. The processes still run at this moment; nothing is judged while suspended. | -| Web content process terminated | **Not judged.** The silence is the crash's, not the extension's. The tab's next navigation is reported with `reason=tab_crash`. | -| Extension signals `cpmStage` = `config_unavailable` or `settings_missing` | Breakage at once (`ended_by=explicit_signal`), no lifetime limit. | -| Extension reload | Existing rule: evidence from the replaced context is discarded. | +| Navigation finished, extension loaded | The document becomes judgeable and its clock starts. If the extension loads later, the clock starts then. | +| Attributed answer (dashboard refresh) | Healthy. Resets the streak. | +| Document ends: new navigation or tab close | If unanswered and alive at least **30 s**: breakage. Shorter silence is not judged (it may be a slow wake). | +| Web content process terminated | Not judged. The tab's next navigation gets `reason=tab_crash`. | +| Extension signals `cpmStage` = `config_unavailable` or `settings_missing` | Breakage at once, no lifetime limit. | +| Answer that cannot be tied to one document | That document is not judged. | +| App suspended | Nothing is judged while suspended. | -A document is reported only if it lived at least **30 s** after it became judgeable (`minimum lifetime`). A shorter silence is ambiguous: it may be a slow wake. The 30 s value follows WebKit's unload timer and the late-answer data above; it is a constant in the first release. Shorter and longer lifetimes are told apart in analysis by the `lifetime` parameter. +Judged documents are never re-judged. Measurement state never expires (the old five-minute drop is gone). -A judged document is never judged again and never retracted: an answer after the report changes nothing. Documents whose process or app ends without a suspension event (kill, crash of the app) are lost; this is the known bias, now limited to the time between the last suspension and the kill. - -Measurement state does not expire after five minutes. The old five-minute lifetime stays only for the restart observation, which is unrelated. - -### Persistence - -The monitor keeps a streak: the number of consecutive judged-broken documents since the last healthy one (any tab). When it reaches **3** the persistent pixel fires once for that streak. A healthy answer resets the streak. The streak lives in memory for the process lifetime. A relaunch resets it (see open questions). - -`tabs` on the persistent pixel records whether the streak's documents were in one tab or several. This keeps the cross-tab distinction the earlier analysis found important (cross-tab permanent was the figure used against the 1% target). +**Streak:** consecutive judged breakages since the last healthy answer, any tab, in memory. At 3 the persistent pixel fires once for that streak. ### Pixels | Pixel | Trigger | Frequency | |---|---|---| -| `web_extension_cpm_breakage` | A document is judged broken. Sent at that moment, once per document. | Daily + count (`dailyAndCount`) | -| `web_extension_cpm_breakage_persistent` | A streak reaches 3 documents. Once per streak. | Daily + count (`dailyAndCount`) | +| `web_extension_cpm_breakage` | A document is judged broken. | Daily + count | +| `web_extension_cpm_breakage_persistent` | The streak reaches 3. Once per streak. | Daily + count | -`_daily` estimates installs per day, `_count` counts events. Both metadata sets follow the existing episode pixel on each platform: `appVersion` on both; macOS also `pixelSource` and `channel`, suffix `daily_count`; iOS keeps its existing suffix set. The macOS definition must match what is sent (item 6 above). +`_daily` is installs per day; `_count` is events. Metadata follows the existing episode pixel on each platform: `appVersion` on both; macOS also `pixelSource`, `channel`, suffix `daily_count`; iOS keeps its suffix set. The macOS definition must match what is sent. `web_extension_cpm_breakage` parameters: | Parameter | Values | |---|---| -| `reason` | `session_restoration`, `tab_crash`, `extension_reload`, `other`. Context of the document's navigation. Same values as the episode pixel. | -| `background` | `woken` (the extension's background view was created during this navigation), `warm` (a background view existed and none was created), `unknown` (no background view known). Comes from the manager's `didCreateBackgroundWebView` delegate callback, correlated to the navigation, not from load duration. | -| `ended_by` | `navigation`, `tab_closed`, `app_suspended`, `explicit_signal`. | -| `lifetime` | Time from judgeable to judged: `30s_to_2m`, `2m_to_10m`, `over_10m`. Omitted for `explicit_signal`. | -| `streak` | Breakages in a row including this one: `1`, `2`, `3_plus`. | +| `reason` | `session_restoration`, `tab_crash`, `extension_reload`, `other` (as in the episode pixel). | +| `background` | `woken` (a background view was created during this navigation), `warm` (views were seen, none created for it), `unknown` (no background view seen yet). From the manager's `didCreateBackgroundWebView` callback. | -`web_extension_cpm_breakage_persistent` parameters: +`web_extension_cpm_breakage_persistent` has no parameters. `last_stage` is added by the companion design. -| Parameter | Values | -|---|---| -| `tabs` | `one` or `many`: whether the streak's documents were in one tab or several. | +Dashboard: persistent installs / DAU from `breakage_persistent_daily` (the success metric); any-breakage installs / DAU from `breakage_daily`; breakages per day from `breakage_count` by `reason` and `background`. -The `last_stage` parameter for wake diagnosis is added by [[cpm-wake-init-detection-tech-design-2026-11-10]], not here. No URL, document, tab or process identifier is sent. +### Removed in the same PR -### The metric set for the dashboard +The episode pixel `web_extension_cpm_episode` with its timers and episode state, `web_extension_cpm_background_load_restart` (and its observation), and `web_extension_cpm_network_process_recovery_applied`: definitions, events and firing code. The recovery behavior (error-6 restart, network recovery) stays; only the pixels go. -| Metric | Source | Meaning | -|---|---|---| -| Installs with persistent breakage / DAU | `breakage_persistent_daily`, split by `tabs` | The success metric (target below 1%). | -| Installs with any breakage / DAU | `breakage_daily` | Upper bound; includes single-page failures. | -| Breakages per day | `breakage_count`, split by `reason`, `background`, `ended_by` | Volume and mix, for diagnosis. | +### First attempt (`e6419bb910`): defects fixed here -Everything is per platform and app version. Rates use the existing DAU series. - -### Where it lives in the code - -- `CPMMessagingHealthMonitor` (existing, per-tab document state already tracks response, judgeable and crash state) gains a per-document record independent of the timeout measurement: when it became judgeable, whether it was judged, the navigation kind. Judging and the streak are small additions to existing handlers; the correlation logic is reused, not duplicated. -- New event `CPMMessagingHealthEvent.backgroundViewCreated`, sent by the manager from `didCreateBackgroundWebView`. It replaces `backgroundLoaded(wasWoken:)` from the first attempt. -- macOS also sends the existing suspension event on termination (`applicationWillTerminate`); iOS and macOS sleep already send `interruptionBegan`. -- `AutoconsentWebExtensionMessageHandler` already separates the `cpmStage` signals; they get a test. -- `CPMMessagingBreakageReport` and the two pixel events in `WebExtensionPixelFiring`. Definitions in both platforms' `web_extensions.json5` and `params_dictionary.json5`. +Measurements older than five minutes dropped; no minimum lifetime; tab process crash counted; `background` from load duration and one app-wide signal; macOS definition mismatch; `navigationWaitContexts` leak (the wait-context set is deleted); no `cpmStage` test. ### PR stack -| PR | Branch | Content | State, 10 Nov | -|---|---|---|---| -| 1 | `alex/cpm-episode-metrics`, [#8025](https://github.com/duckduckgo/apple-browsers/pull/8025) | Remove ineffective recovery fixes, their flags and pixels | pushed, in review | -| 2 | `alex/cpm-pixel-cleanup` (`f04c93730f`) | Remove the diagnostics, trim the episode pixel to five parameters. No new pixel, no detection change. Needs the Danger label (existing pixel parameters change). | local; 1104 package tests pass, SwiftLint clean on the changed core files; app builds and pixel validation not run; not pushed | -| 3 | new, on top of 2 | This design: two pixels, document-end judging, streak | not started | -| 4 | new, on top of 3 | Init progress marks, `last_stage` ([[cpm-wake-init-detection-tech-design-2026-11-10]]) plus extension [#3727](https://github.com/duckduckgo/duckduckgo-privacy-extension/pull/3727) | native part drafted in `alex/cpm-init-diagnostics` (`adb28e643d`), needs the same fixes | -| 5 | later | One release after PR 3: remove the episode pixel with its timers and states, and the pixels `web_extension_cpm_background_load_restart` and `web_extension_cpm_network_process_recovery_applied` (their definitions and firing code; the recovery behavior stays). After this the only CPM pixels are the two new ones. | after comparison | - -Old branches `alex/cpm-breakage-pixel` and `alex/cpm-init-diagnostics` stay as sources only. - -### Pros - -- Measures the user-facing fact (this page never got an answer), not an elapsed time. -- A late answer is never a failure, which removes the 6% late-answer error class and its tail. -- Detected breakages are sent at once, so closing the app loses only documents that were never judged. -- Persistent breakage has its own daily pixel: the success metric is one series, not a join. -- Additive: the episode pixel and the recovery logic keep working, so the new numbers can be validated against the old. +| PR | Content | State | +|---|---|---| +| 1, [#8025](https://github.com/duckduckgo/apple-browsers/pull/8025) | Remove ineffective recovery fixes and flags | pushed, in review | +| 2, `alex/cpm-pixel-cleanup` | Remove diagnostics, trim the episode pixel | local, tests pass, not pushed | +| 3, `alex/cpm-breakage-metric` | This design, two commits: new pixels; removal of the old pixels | in progress | +| 4 | Marks and `last_stage` ([[cpm-wake-init-detection-tech-design-2026-11-10]]) plus extension #3727 | not started | ### Cons -- Pages that live until a kill are not judged. Persistent breakage that a user never ends (leaves the page open for hours) appears only at suspension. -- 30 s minimum lifetime hides breakage on short pages. If most breakage is on short pages the metric underestimates; `lifetime` shows how much sits near the limit. -- A page-based streak does not equal the old "no answer for five minutes". The two series will differ, and the comparison needs interpretation. -- The streak resets on relaunch; iOS terminates apps often, so persistent breakage across launches is undercounted. +- Documents alive when the app exits or suspends are never judged; breakage on pages left open is missed. +- 30 s hides breakage on short pages. +- The series is new: no overlap with the episode series, so comparison is by app version against the previous derived number. +- Streak resets on relaunch. ### Risks -- **Pixel volume.** A user with persistent breakage fires one count request per page. Mitigation: soft cap of 20 `breakage` count requests per process; the persistent pixel is unaffected by the cap. Volume to be estimated from `web_extension_cpm_episode` counts before release. -- **Healthy evidence that is missing in a working page** (dashboard refresh not delivered, wrong attribution). Counted as breakage. The earlier analysis found `other_tab_response` at 37–53% of outcomes, an attribution problem. Mitigation: documents with an unattributable response are not judged, and `unknown` attribution is visible only locally for now. Compare with the episode series. -- **Judging at suspension** assumes silence for 30 s before suspension means breakage. A wake that began just before suspension could be reported. Mitigation: the lifetime limit; the suspension case is separated by `ended_by`. -- **Simulation switch.** The Debug "Simulate CPM Messaging Breakage" switch hides dashboard responses from the monitor and must keep doing so for both detectors. +- A page that works but whose answer is not attributed counts as breakage (`other_tab_response` was 37–53% of outcomes). Documents with an unattributable answer are not judged. +- Pixel volume for a persistently broken install: one count request per page. +- The Debug "Simulate CPM Messaging Breakage" switch must keep hiding answers. ## Testing -- Unit tests for each rule in the table: late answer before document end is healthy; unanswered document shorter than 30 s is not reported and does not touch the streak; unanswered longer document is reported on navigation, tab close, suspension; process termination is not judged and tags the next navigation; judged once; extension not loaded at finish becomes judgeable on load; unattributable response; explicit signals; streak reaches 3 once, resets on a healthy answer; `tabs` value; `background` from the view-created event inside and outside the navigation window; simulation switch hides answers; extension reload evidence. -- Handler test for `cpmStage` parsing; test for `navigationWaitContexts` after a context replacement. -- Pixel definitions: schema validators of both platforms in CI; parameter lists match what is sent (compare the macOS definition against the event). -- Manual: updated steps in [[CPM Breakage Testing Steps]] (Debug simulation switch: load a page for 35 s without an answer, navigate away, expect one `breakage`; three in a row, expect `persistent`). -- Real WebKit: the CPM repro harness ([PR #7378](https://github.com/duckduckgo/apple-browsers/pull/7378)) with the injected error 6 and idle-wake fixtures to see both pixels with the expected values. Not run yet. +- Unit tests per row of the table, streak at 3 once and reset, `reason` values, `background` values, simulation switch, `cpmStage` parsing, extension reload. +- Pixel definitions pass both validators; parameters match what is sent. +- Manual: [[CPM Breakage Testing Steps]] with the Debug switch. +- Real WebKit (CPM repro harness, [PR #7378](https://github.com/duckduckgo/apple-browsers/pull/7378)): not run. ## Additional Considerations (if applicable) -**Privacy** +**Privacy:** two new pixel names with enumerated parameters; no URLs, identifiers or page content. Privacy Triage required, not requested yet. -Two new pixel names with enumerated, bucketed parameters. No URLs, page content, tab or document identifiers, or process IDs. The streak is a local integer that is never sent; only `streak` as a bucket. Because the pixel names are new, Privacy Triage is required before release. Not requested yet. Combined distinguishability of `reason`, `background`, `ended_by`, `lifetime`, `streak` and the platform metadata is small (five low-cardinality enums). +**Security:** no new private WebKit selectors. -**Security** +**Site Breakage:** none; the detector only observes events and recovery is unchanged. -No new private WebKit selectors. `didCreateBackgroundWebView` is already implemented in the manager. +**Experimentation:** none. -**Site Breakage** +**Operational:** pixel expiry 2027-01-15 as existing definitions. Danger label for the removed pixels. Data checks: `metrics.pixels_aggregation` and `metrics._pixels_new` (`pixel_id = 'web'`). -None intended. The detector only observes events; it sends no messages and changes no recovery behavior. Existing recovery (error-6 restart, network recovery, upgrade before load) is unchanged. - -**Experimentation** - -No experiment. The comparison is observational: the new and old series for the same versions during the overlap release. - -**Operational** - -- Overlap: ship PR 3 next to the episode pixel for one release; compare "installs with persistent breakage / DAU" with the existing permanent-failure series by platform. Expected: new is lower where late answers dominated, higher where the episode never reached its deadline. Remove the episode pixel (PR 5) when the difference is explained. -- Pixel expiry follows the existing definitions (2027-01-15). -- Add the three metrics to the health indicators dashboard. Data checks use ClickHouse: pixel counts from `metrics.pixels_aggregation` (`pixel_id = 'web'`), parameters from `metrics._pixels_new` (`pixel_id = 'web'`, one day). -- No remote kill switch is proposed. If volume surprises, the soft cap and the next release are the lever. **Question for reviewers:** is a feature flag required for a new pixel here? - -**Localization / Internationalization** - -None. - -## Open questions - -1. Minimum lifetime 30 s and streak 3: acceptable as first constants, or tune from the episode data first? -2. Persist the streak across launches (a local counter)? It makes persistent breakage visible across iOS terminations at the price of local state. -3. Flag or no flag for a new pixel. -4. Judge at suspension: accept, or only at document end? -5. Soft cap value. +**Localization / Internationalization:** none. diff --git a/work/projects/cpm-project-summary-2026-11-03.md b/work/projects/cpm-project-summary-2026-11-03.md index a2f94b74..a31d45f1 100644 --- a/work/projects/cpm-project-summary-2026-11-03.md +++ b/work/projects/cpm-project-summary-2026-11-03.md @@ -110,7 +110,7 @@ Open: **Что неверно в `e6419bb910`** (проверено по коду): тихие замеры старше 5 минут сбрасываются (`discardExpiredState`); нет минимального времени жизни страницы; падение процесса вкладки засчитывается поломкой; `background` берётся из «загрузка ≥ 1 с» и одного общего сигнала; страницы, живущие до выхода приложения, не оцениваются; определение macOS без `pixelSource` и `channel`, суффиксы от iOS; утечка `navigationWaitContexts`; нет теста на разбор `cpmStage`. **Tech design (черновики, ревьюер не назначен):** -- [[cpm-permanent-breakage-metric-tech-design-2026-11-10]]: два пикселя (`web_extension_cpm_breakage`, `web_extension_cpm_breakage_persistent`), оценка страницы по концу документа, минимум 30 с жизни, серия из 3 подряд, суждение при приостановке приложения, `background` по `didCreateBackgroundWebView`. Работает рядом с эпизодным пикселем один релиз. Конечное состояние: из CPM пикселей остаются только эти два; эпизодный, `background_load_restart` и `network_process_recovery_applied` удаляются в PR 5. +- [[cpm-permanent-breakage-metric-tech-design-2026-11-10]]: два пикселя (`web_extension_cpm_breakage` с `reason` и `background`, `web_extension_cpm_breakage_persistent` без параметров), оценка страницы по концу документа, минимум 30 с жизни, серия из 3 подряд. Старые пиксели (эпизодный, `background_load_restart`, `network_process_recovery_applied`) удаляются в том же PR 3 вторым коммитом, без задержки на релиз; остаются только эти два. - [[cpm-wake-init-detection-tech-design-2026-11-10]]: метки `cpmInitProgress` из расширения, `last_stage` в пикселе, `response_sent` как здоровый сигнал, изменение расширения (PR #3727, ветка `alex/cpm-init-progress`; метки `page_init_sent`, `config_read`, `settings_checked`, `site_checked` в рабочем дереве без коммита и без запуска тестов). Открытые вопросы для ревью: порог 30 с и серия 3; сохранять серию между запусками; считать ли `response_sent` здоровым до проверки без слушателя; шесть меток или меньше; нужен ли флаг для нового пикселя. diff --git a/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md b/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md index 772aa4a9..e7f6bc5b 100644 --- a/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md +++ b/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md @@ -74,7 +74,7 @@ Marks are best effort: they carry `{ tabId, url, stage }`, never wait and never - `AutoconsentWebExtensionMessageHandler` gets a `cpmInitProgress` method. Unknown stage names are ignored without an error, so a newer extension cannot make an older app answer with failures. - `CPMMessagingHealthEvent.initProgress` carries the mark; the monitor stores `lastStage` per document and clears it on a new navigation, a cancelled navigation and an extension reload. -- `response_sent` counts as healthy evidence for the document, like a dashboard response. `response_failed` judges a pending document at once (`ended_by=explicit_signal`). Healthy evidence is either the dashboard refresh or `response_sent`; the open question below asks whether `response_sent` should count before it is verified. +- `response_sent` counts as healthy evidence for the document, like a dashboard response. `response_failed` judges a pending document at once. Healthy evidence is either the dashboard refresh or `response_sent`; the open question below asks whether `response_sent` should count before it is verified. - The Debug "Simulate CPM Messaging Breakage" switch hides marks as well as dashboard responses. ### Pixel change