TD: Pixel Definition: CPM Permanent Breakage Metric (macOS/iOS)

Author: Alex M Reviewer: TBD (previous CPM pixel designs: Michal) Stakeholders: Michal, Konrad, Russell (from the previous Tech Designs; confirm) Project: CPM: root-cause the stuck embedded extension messaging (macOS/iOS) Task: to create ("Permanent breakage metrics") Previous Tech Designs: CPM breakage pixels, CPM failure diagnostics Privacy Triage: to request (new pixel names)

Background & Requirements

The goal is permanent CPM failures below 1% of DAU. There is no pixel for it: the number is derived from web_extension_cpm_episode with a long SQL query, and the episode is a poor basis:

A timer decides. A four-second navigation timeout opens an episode. On iOS 7.244, 6.0% of episodes closed with a late answer on the same page, 42% of those later than 30 s. Slow answers are counted as failures.
Episodes are lost. An open episode lives in memory and is dropped when the app exits.

Requirements: a metric without a navigation timer, expressed as a rate against DAU; no page content, URLs or persistent identifiers; a small layer next to the product code (no private WebKit selectors). End state: the only CPM pixels are the two below.

Problem Statement

Report the tabs whose pages never hear from the extension, and tell one broken page from a broken extension.

Recommended Approach

While a page initializes, the extension sends the app a message from that tab. The app treats a tab as silent when it committed a page that should get a message and heard nothing for 30 seconds of app time. A message that arrives later still counts as an answer. There is no timer: the check runs whenever any other event arrives (a page commit, a tab closing, a message, the app going to the background or waking).

Pages that count: http(s), extension loaded, not the browser's error page, not a back/forward restore, not a restored session.
Time that counts: time the app is suspended is not counted. A page that is replaced or closed before 30 s, and a page whose process crashed, is not reported.
Which tab a message belongs to: the message carries the extension's tab number and the page URL. The app learns the tab number from the first message that matches exactly one open page (host and path). A message that cannot be tied to one tab answers every tab it could belong to, so the metric prefers missing a breakage to reporting a working tab.
Not an answer: a refresh saying the extension has no configuration or no settings. It shows the extension is alive, but the page stays unanswered.

Pixel	Trigger	Frequency
`web_extension_cpm_breakage`	A tab is silent.	Daily + count
`web_extension_cpm_breakage_persistent`	A second tab is silent and no tab heard from the extension in between. Once per series.	Daily + count

web_extension_cpm_breakage has the parameter tabs: one (no other tab was silent since the extension last sent a message: the page is the likely cause) or many (another tab was silent too: the extension is the likely cause). The persistent pixel has no parameters. A series ends when the extension sends any message from any tab. A second parameter, last_stage, is defined in the wake init detection design.

Dashboard: installs per day with the persistent pixel / DAU is the success metric; installs with any breakage / DAU and breakages per day by tabs complete it.

The episode pixel (web_extension_cpm_episode), web_extension_cpm_background_load_restart and web_extension_cpm_network_process_recovery_applied are removed together with the code that only fed them. The recoveries themselves are unchanged.

Pros

No navigation timer, and a slow answer is no longer a failure.
A rate against DAU straight from the pixel, without a query chain.
Small: one monitor that counts events, no per-navigation state machine.

Cons and known limits

A tab left open and silent until the app exits is not reported (iOS reports it when going to the background).
A page left before 30 s is not reported, so a user who moves on every 20 s with a broken extension is never seen.
Tabs on the same host and path (for example two search result pages) cannot be told apart: a silent one can hide behind its twin.
The tab mapping is a heuristic. In rare cases it attributes a message to the wrong tab and a working extension is reported.
Pages the extension does not inject into look silent. The page type is not known at commit, so they are not filtered.
One tab can never make the persistent pixel: it needs two distinct tabs. If the native channel itself is lost, every tab looks silent and tabs=many follows.

Testing

Unit tests for each rule above, including the 30 s boundary, late answers, suspension, crashes, reload, tab mapping and the one/many/persistent logic.
Both platforms' pixel definition validators and the adapter tests for names, frequency and parameters.
App-hosted macOS and iOS tests for the touched code, including an integration test that the browser's error page is not measured.
Not done: a real-WebKit run, and a check that the extension messages on every page counted as eligible (search result pages, duck.ai, non-document types).

Additional Considerations

Privacy: two new pixel names with an enumerated parameter; no URLs, identifiers or page content. Privacy Triage is required and not requested yet.

Security: no new private WebKit selectors.

Site Breakage: none. The monitor only observes events; recovery is unchanged.

Experimentation: none.

Operational: the work is a stack of PRs: removal of the old diagnostics and pixels (#8054, on top of #8025), the metric, and the last_stage addition. The removal and the metric must ship in the same release, otherwise no CPM breakage pixel exists in between. The new definitions have no expiry date (permanent metric). The first persistent series cannot be compared with the old episode numbers directly: compare by app version against the previous derived number.

Localization / Internationalization: none.

6.4 KiB Raw Blame History