From c7184e855dbb6cf47af44628896d1c2949662044 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Wed, 11 Nov 2026 18:56:41 +0600 Subject: [PATCH] [2026-11-11] eagle: work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md --- ...-breakage-metric-tech-design-2026-11-10.md | 59 +++++----------- ...e-init-detection-tech-design-2026-11-10.md | 67 ++++++------------- 2 files changed, 38 insertions(+), 88 deletions(-) diff --git a/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md b/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md index 38113b21..d6e1cfb3 100644 --- a/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md +++ b/work/projects/cpm-permanent-breakage-metric-tech-design-2026-11-10.md @@ -19,12 +19,9 @@ status: draft ## Background & Requirements -The goal is permanent CPM failures below 1% of DAU. There is no pixel for it: the number is derived from `web_extension_cpm_episode` with a long SQL query, and the episode is a poor basis: +The goal is permanent CPM failures below 1% of DAU, but no pixel measures it: the number comes from `web_extension_cpm_episode` through a long query, and the episode is a poor basis. A four-second navigation timer opens it (on iOS 7.244, 6.0% of episodes closed with a late answer on the same page, 42% of those after 30 s), and an open episode is lost when the app exits. -- **A timer decides.** A four-second navigation timeout opens an episode. On iOS 7.244, 6.0% of episodes closed with a late answer on the same page, 42% of those later than 30 s. Slow answers are counted as failures. -- **Episodes are lost.** An open episode lives in memory and is dropped when the app exits. - -Requirements: a metric without a navigation timer, expressed as a rate against DAU; no page content, URLs or persistent identifiers; a small layer next to the product code (no private WebKit selectors). End state: the only CPM pixels are the two below. +Requirements: no navigation timer, a rate against DAU, no URLs, page content or persistent identifiers. ## Problem Statement @@ -32,56 +29,32 @@ Report the tabs whose pages never hear from the extension, and tell one broken p ## Recommended Approach -While a page initializes, the extension sends the app a message from that tab. The app treats a tab as **silent** when it committed a page that should get a message and heard nothing for 30 seconds of app time. A message that arrives later still counts as an answer. There is no timer: the check runs whenever any other event arrives (a page commit, a tab closing, a message, the app going to the background or waking). +While a page initializes, the extension sends the app a message from that tab. A tab is **silent** when it loaded a page that should get a message and heard nothing for 30 s of app time. A late message still counts as an answer. There is no timer: the check runs when another event arrives (page load, tab close, message, app to background or wake). -- **Pages that count:** http(s), extension loaded, not the browser's error page, not a back/forward restore, not a restored session. -- **Time that counts:** time the app is suspended is not counted. A page that is replaced or closed before 30 s, and a page whose process crashed, is not reported. -- **Which tab a message belongs to:** the message carries the extension's tab number and the page URL. The app learns the tab number from the first message that matches exactly one open page (host and path). A message that cannot be tied to one tab answers every tab it could belong to, so the metric prefers missing a breakage to reporting a working tab. -- **Not an answer:** a refresh saying the extension has no configuration or no settings. It shows the extension is alive, but the page stays unanswered. +- Pages that count: http(s), extension loaded, not an error page, not a back/forward restore, not a restored session. Suspended time is not counted. A page replaced or closed within 30 s is not reported. +- A message carries the extension's tab number and the page URL. The app learns which tab it is from the first message that matches exactly one open page (host and path). A message it cannot place answers every tab it could belong to: the metric prefers missing a breakage to reporting a working tab. | Pixel | Trigger | Frequency | |---|---|---| -| `web_extension_cpm_breakage` | A tab is silent. | Daily + count | -| `web_extension_cpm_breakage_persistent` | A second tab is silent and no tab heard from the extension in between. Once per series. | Daily + count | +| `web_extension_cpm_breakage` | A tab is silent. Parameter `tabs`: `one` (no other tab was silent since the extension last answered: likely the page) or `many` (another tab was too: likely the extension). | Daily + count | +| `web_extension_cpm_breakage_persistent` | A second tab is silent and the extension answered nothing in between. Once per series. | Daily + count | -`web_extension_cpm_breakage` has the parameter `tabs`: `one` (no other tab was silent since the extension last sent a message: the page is the likely cause) or `many` (another tab was silent too: the extension is the likely cause). The persistent pixel has no parameters. A series ends when the extension sends any message from any tab. A second parameter, `last_stage`, is defined in the wake init detection design. +Success metric: installs with the persistent pixel / DAU. The episode pixel, `web_extension_cpm_background_load_restart` and `web_extension_cpm_network_process_recovery_applied` are removed with the code that only fed them; the recoveries themselves stay. -Dashboard: installs per day with the persistent pixel / DAU is the success metric; installs with any breakage / DAU and breakages per day by `tabs` complete it. +## Pros and Cons -The episode pixel (`web_extension_cpm_episode`), `web_extension_cpm_background_load_restart` and `web_extension_cpm_network_process_recovery_applied` are removed together with the code that only fed them. The recoveries themselves are unchanged. - -## Pros - -- No navigation timer, and a slow answer is no longer a failure. -- A rate against DAU straight from the pixel, without a query chain. -- Small: one monitor that counts events, no per-navigation state machine. - -## Cons and known limits - -- A tab left open and silent until the app exits is not reported (iOS reports it when going to the background). -- A page left before 30 s is not reported, so a user who moves on every 20 s with a broken extension is never seen. -- Tabs on the same host and path (for example two search result pages) cannot be told apart: a silent one can hide behind its twin. -- The tab mapping is a heuristic. In rare cases it attributes a message to the wrong tab and a working extension is reported. -- Pages the extension does not inject into look silent. The page type is not known at commit, so they are not filtered. -- One tab can never make the persistent pixel: it needs two distinct tabs. If the native channel itself is lost, every tab looks silent and `tabs=many` follows. +- Pros: no timer, a slow answer is not a failure, a rate straight from the pixel. +- Cons: a silent tab left open until exit is not reported (iOS reports it on going to the background); pages left within 30 s are never reported; tabs on the same host and path hide each other; the tab mapping is a heuristic and rarely blames a working extension; one tab alone never triggers the persistent pixel. ## Testing -- Unit tests for each rule above, including the 30 s boundary, late answers, suspension, crashes, reload, tab mapping and the `one`/`many`/persistent logic. -- Both platforms' pixel definition validators and the adapter tests for names, frequency and parameters. -- App-hosted macOS and iOS tests for the touched code, including an integration test that the browser's error page is not measured. -- Not done: a real-WebKit run, and a check that the extension messages on every page counted as eligible (search result pages, `duck.ai`, non-document types). +Unit tests for each rule, pixel definition validators, adapter tests, app-hosted macOS and iOS tests. Not done: a real-WebKit run, and a check that the extension messages on every eligible page (search results, `duck.ai`, non-document types). ## Additional Considerations -**Privacy:** two new pixel names with an enumerated parameter; no URLs, identifiers or page content. Privacy Triage is required and not requested yet. - +**Privacy:** two new pixel names, one enumerated parameter, no URLs or identifiers. Privacy Triage not requested yet. **Security:** no new private WebKit selectors. - -**Site Breakage:** none. The monitor only observes events; recovery is unchanged. - +**Site Breakage:** none. **Experimentation:** none. - -**Operational:** the work is a stack of PRs: removal of the old diagnostics and pixels ([#8054](https://github.com/duckduckgo/apple-browsers/pull/8054), on top of [#8025](https://github.com/duckduckgo/apple-browsers/pull/8025)), the metric, and the `last_stage` addition. The removal and the metric must ship in the same release, otherwise no CPM breakage pixel exists in between. The new definitions have no expiry date (permanent metric). The first persistent series cannot be compared with the old episode numbers directly: compare by app version against the previous derived number. - -**Localization / Internationalization:** none. +**Operational:** a stack of PRs ([#8054](https://github.com/duckduckgo/apple-browsers/pull/8054) removal on top of [#8025](https://github.com/duckduckgo/apple-browsers/pull/8025), then the metric). The removal and the metric ship in the same release. No expiry date: permanent metric. +**Localization:** none. diff --git a/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md b/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md index 84fbf0bf..eb7a2427 100644 --- a/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md +++ b/work/projects/cpm-wake-init-detection-tech-design-2026-11-10.md @@ -21,75 +21,52 @@ status: draft ## Background & Requirements -Failures after a background wake are the largest group with no fix: on macOS 1.214 they are 39% of all failures and 36% of permanent ones (0.68% of DAU); on iOS 7.244, 22% and 15% (0.06% of DAU). The usual causes are ruled out: the background is ready (under one second in 93% of macOS wake episodes), and a plain idle-unload and wake loses no messages in the real-WebKit harness. +Failures after a background wake are the largest group with no fix: on macOS 1.214, 39% of all failures (0.68% of DAU); on iOS 7.244, 22% (0.06% of DAU). The background is ready and a plain idle-unload and wake loses no messages in the real-WebKit harness, so the usual causes are out. -What the app cannot see is where the startup message stops. For one page the chain is: the page's script sends `init` → the background receives it → reads the configuration → reads the user setting → checks the site → answers the page and refreshes the dashboard in the app. The app sees only the last step, so every failure looks the same: no answer. +The app cannot see where the startup message stops. For one page: page script sends `init` → background receives it → reads configuration → reads the user setting → checks the site → answers the page and refreshes the app's dashboard. The app sees only the last step, so every failure looks the same: no answer. -Requirements: tell where the chain stopped for a page that got no answer, and whether the page really received its answer; no change to what the extension does for the page; no URLs, page content or persistent identifiers collected; an older app must tolerate a newer extension and the other way round. +Requirements: find where the chain stopped, without changing what the extension does for the page, without collecting URLs or persistent identifiers, and with older and newer app and extension tolerating each other. ## Problem Statement -For every tab the permanent breakage metric reports as silent, record the furthest init step the extension reported for its page, so the next fix is picked from data. +For every tab the permanent breakage metric reports as silent, record the furthest init step the extension reported, so the next fix is picked from data. ## Recommended Approach -The extension sends a small fire-and-forget notification to the app at each step of init. The app keeps the furthest step per tab and adds it as `last_stage` to the `web_extension_cpm_breakage` pixel. A notification never counts as an answer: only the dashboard refresh does. Notifications never create a report; they only describe a tab that is already silent. +The extension sends the app a fire-and-forget notification at each init step. The app keeps the furthest step per tab (for its current page only) and adds it as `last_stage` to `web_extension_cpm_breakage`. A notification is never an answer and never creates a report. Unknown step names are ignored. | `last_stage` | The chain stopped | |---|---| -| `none` | nothing reached the app (script not injected, or the message or notification was lost) | -| `page_init_sent` | the page's script sent `init`; the background never processed it | -| `received` | `init` arrived; the configuration or storage read did not finish | +| `none` | nothing reached the app (script not injected, or a message was lost) | +| `page_init_sent` | the page script sent `init`; the background never processed it | +| `received` | `init` arrived; configuration or storage read did not finish | | `config_read` | waiting for the user setting from the app | | `settings_checked` | waiting for the site check | -| `site_checked` | all checks passed; no answer and no dashboard refresh arrived | +| `site_checked` | all checks passed; no answer arrived | | `response_sent` | the answer was sent to the page, but no dashboard refresh followed | | `response_failed` | sending the answer to the page failed | -The app keeps only the highest step, so a late or out-of-order notification never lowers it, and only notifications for the tab's current page count. Unknown step names are ignored, so a newer extension cannot break an older app. The Debug "Simulate CPM Messaging Breakage" switch hides these notifications as well. +The extension change only adds these notifications inside the existing `init` handling; they never wait, throw or change what the page receives. The extension ships inside the app, so it reaches users with an app release; the app side can ship first (then `last_stage` is `none`). -**Extension change.** Notifications are sent inside the existing `init` handling: before any wait for the first step, after each check for the next, and around the answer to the page. They never wait, never throw and never change what the page receives; a failure to answer is still reported as before. +## Pros and Cons -**Compatibility.** The extension ships inside the app, so it reaches users with an app release after its PR merges. The app side can ship first: with no notifications, `last_stage` is `none`. A newer extension on an older app is expected to be harmless; this is to be confirmed against the real message router. The page's script is one bundle for every extension build, so `page_init_sent` also reaches builds that ignore it, at the cost of one extra message per page. - -## Pros - -- Turns "no answer" into a position in a known chain, which maps to the open hypotheses and to concrete fixes. -- Shows cases the dashboard refresh hides, such as an answer that was sent but not received. -- Out of the page's way: no waits, no state changes. -- Both sides tolerate the other being older. - -## Cons and risks - -- Notifications travel through the same background whose failure we study, so the earliest failures end up as `none`. -- Up to seven extra notifications per page on a path suspected to be fragile. The extra load has not been measured. Mitigation: ship to internal builds first, compare breakage rates and the `last_stage` mix with and without notifications, and drop `config_read` and `site_checked` if the cost shows. -- `response_sent` can be misleading: WebKit may report a send as done when no one was listening. It never answers a page, so the metric is not affected, only how `last_stage=response_sent` is read. -- The extension part reaches users only with an app release, so the first data comes about two release cycles after the work. -- A page cannot forge notifications: the script runs in an isolated world, and the background accepts a page-side notification only for `page_init_sent` and only from the main frame. +- Pros: turns "no answer" into a position in a known chain that maps to concrete fixes; stays out of the page's way. +- Cons: notifications pass through the background we study, so the earliest failures end as `none`; up to seven extra notifications per page, load not measured (internal builds first); first data comes about two release cycles after the work. ## Testing -- App: the pixel carries the furthest step of the silent tab and not of another tab; no notifications give `none`; a notification is not an answer; unknown steps are ignored; stale notifications are forgotten at the next page; the parameter appears in both platforms' pixel definitions. -- Extension: the existing unit suite for the cookie prompt management, extended for the order of notifications, a site that disables CPM ending at `site_checked`, and sub-frame and unknown-step notifications being ignored. Part of the extension change has not been run yet. -- Not done: a real-WebKit run showing every step in order on a healthy page, the matching `last_stage` for an injected stall at each step, and a page without the script not yielding `response_sent`. +App unit tests for the furthest step per tab, `none` without notifications, ignored unknown steps and the pixel definitions. Extension unit tests for the order of notifications (part not run yet). Not done: a real-WebKit run showing every step in order and the right `last_stage` for a stall at each step. ## Additional Considerations -**Privacy:** one new enumerated parameter on a pixel that is new itself. The notifications carry the tab number and the page URL from the extension to the app, inside the app (the dashboard refresh already does); neither enters a pixel or a log. Privacy Triage is to be requested once for both designs. - -**Security:** no new private WebKit selectors. The new message is read-only: the app parses a step name and a tab and answers with nothing. - -**Site Breakage:** no intended change for pages. - -**Experimentation:** none. Rollout is by release channel, internal builds first. - -**Operational:** the app side lands first, the extension after its review and the usual update of the embedded files. After the first public release with notifications, look at `last_stage` by platform and wake state over the breakage count and pick the fix from the table above. The notifications and `last_stage` are investigation tools: remove them when the wake group has a fix or is closed. Not in scope: per-step durations and the background's own lifecycle metrics. - -**Localization / Internationalization:** none. +**Privacy:** one new enumerated parameter. Tab number and URL travel inside the app only; neither enters a pixel or a log. Privacy Triage is requested once for both designs. +**Security:** no new private WebKit selectors; the new message is read-only. +**Site Breakage:** no intended change. +**Experimentation:** none; rollout by release channel. +**Operational:** app side first, extension after review and the embedded-files update. Removed when the wake group has a fix or is closed. +**Localization:** none. ## Open Questions -1. Seven notifications per page or fewer: are `config_read` and `site_checked` worth it for the first read? -2. Send `page_init_sent` only in the embedded build, or accept the extra message everywhere? -3. Is a switch inside the extension needed, or is release-channel rollout enough? -4. Who reviews the extension change, and can the update of the embedded files be timed with the app release that carries the app side? +1. Are `config_read` and `site_checked` worth the extra notifications for the first read? +2. Send `page_init_sent` only in the embedded build, or in every build?