183 lines
8.5 KiB
Markdown
Executable File
183 lines
8.5 KiB
Markdown
Executable File
---
|
|
tags:
|
|
- cpm
|
|
- experiment
|
|
- ab-testing
|
|
status: ready-for-review
|
|
updated: '2026-09-22'
|
|
---
|
|
# CPM Background Graveyard — Experiment Definition
|
|
|
|
> Status: **ready for review; not posted to Asana**
|
|
> Updated: 2026-09-22
|
|
> Code milestone: M6 `alex/cpm-graveyard-ab`
|
|
> Depends on: M3 telemetry → M5 mitigation
|
|
|
|
## Asana task fields
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Name — iOS | `iOS Mobile Experiment: [control, treatment] CPM background graveyard` |
|
|
| Name — macOS | `macOS Desktop Experiment: [control, treatment] CPM background graveyard` |
|
|
| Platform | Separate experiment-log task for iOS and macOS |
|
|
| Experiment Name | `cpmBackgroundGraveyardExperiment` |
|
|
| Cohorts | `control`, `treatment` |
|
|
| Start / end | TBD after sample-size review |
|
|
| Status | Draft until Privacy Config and privacy/schema approvals are complete |
|
|
| Project | CPM Web Extension breakage |
|
|
| Related CPM task | [1216761517055116](https://app.asana.com/1/137249556945/project/1211834678943996/task/1216761517055116) |
|
|
| Technical design | [1218005985662983](https://app.asana.com/1/137249556945/project/1211834678943996/task/1218005985662983) |
|
|
| A/B framework | [1208680035673270](https://app.asana.com/1/137249556945/project/1208889145294658/task/1208680035673270) |
|
|
|
|
## Background
|
|
|
|
A forced termination of the Web Extension background WebContent process reproduces a persistent all-tabs CPM outage. Reloading a tab or opening a new tab does not recover it; reloading the embedded extension does.
|
|
|
|
M5 retains at most one terminated background `WKWebView` for 3 seconds. The hold keeps the dead page/process proxy reachable while WebKit establishes a replacement service-worker context connection. The mitigation is disabled by default and has a separate runtime kill switch.
|
|
|
|
Production incidence of natural process termination and its causal connection to CPM outage must first be confirmed by M3 telemetry.
|
|
|
|
## Hypothesis
|
|
|
|
For eligible users whose CPM background WebContent process terminates, the treatment reduces the proportion of users with a subsequent CPM `initialization_failed` outcome, without increasing crashes, hangs, or unmeasured outcomes.
|
|
|
|
## Eligibility and enrollment
|
|
|
|
A user is enrolled only when all conditions are true:
|
|
|
|
1. Web Extensions are available: iOS 18.4+ or macOS 15.4+.
|
|
2. M5 kill switch `cpmBackgroundGraveyard` is enabled.
|
|
3. Experiment subfeature `cpmBackgroundGraveyardExperiment` is enabled and its target/rollout matches.
|
|
4. The current CPM background WebContent process actually terminates.
|
|
5. `resolveCohort` returns `control` or `treatment`.
|
|
|
|
Enrollment occurs at the first real exposure event—the process termination—not at app startup or privacy-config parsing. Assignment is stable through the native A/B framework.
|
|
|
|
## Variants
|
|
|
|
| Cohort | Behavior |
|
|
|---|---|
|
|
| `control` | Enroll and emit the same denominator/outcome telemetry; do not retain the terminated view. |
|
|
| `treatment` | Retain at most one terminated background view for 3 seconds; replace an older held view; release on timeout, context load/unload, or runtime kill-switch disable. |
|
|
|
|
Both cohorts share all code except the bounded hold. Pending outcome correlation is capped at 10 entries; overflow is finalized as `not_measured`.
|
|
|
|
## Metrics
|
|
|
|
### Primary experiment metric
|
|
|
|
Framework pixel:
|
|
|
|
- Experiment: `cpmBackgroundGraveyardExperiment`
|
|
- Metric: `cpmBackgroundGraveyardOutcome`
|
|
- Conversion window: `0...1` days from enrollment
|
|
- Predefined values: `healthy`, `initialization_failed`, `not_measured`
|
|
- Primary comparison: share of enrolled users with value `initialization_failed`, treatment vs control
|
|
|
|
The framework makes this user-level and unique by experiment/cohort/metric/window/value. A user may legitimately emit both `healthy` and `initialization_failed`; analysis must treat the primary result as “ever failed in the conversion window,” not an event rate.
|
|
|
|
### Operational diagnostics
|
|
|
|
- `debug_web_extension_cpm_background_graveyard_termination`: countable denominator by bounded termination reason and cohort.
|
|
- `debug_web_extension_cpm_background_graveyard_outcome`: countable per-termination outcome by reason, cohort, and outcome.
|
|
- Existing initialization-failure and stuck pixels add `background_graveyard_cohort`.
|
|
- Standard framework enrollment, app-use retention, and search retention pixels are automatic.
|
|
|
|
Operational pixels are required for event-rate and correlation diagnosis; the framework metric is required for the standard experiment analysis pipeline.
|
|
|
|
### Guardrails
|
|
|
|
1. `not_measured` coverage and termination/outcome mismatch.
|
|
2. Crash and hang rate by platform/cohort.
|
|
3. Treatment-only graveyard lifecycle: one held view maximum and bounded release.
|
|
4. Cohort balance and enrollment volume.
|
|
5. Standard app-use/search retention metrics.
|
|
|
|
## Success and stop criteria
|
|
|
|
Success criteria must be frozen before rollout:
|
|
|
|
- statistically meaningful reduction in user-level `initialization_failed`;
|
|
- no material regression in crash/hang or app-use/search retention;
|
|
- no unexplained cohort imbalance;
|
|
- acceptable `not_measured` rate;
|
|
- operational denominator/outcome coverage is sufficient for interpretation.
|
|
|
|
Data Science must set minimum sample size, minimum detectable effect, significance/power, and observation duration before public rollout.
|
|
|
|
To stop new enrollment, set both cohort weights to `0`. Do not delete a cohort unless reassignment is intentional.
|
|
|
|
Emergency rollback requires disabling **both**:
|
|
|
|
1. `cpmBackgroundGraveyardExperiment` — stops experiment activity/retention reporting.
|
|
2. `cpmBackgroundGraveyard` — immediately releases a held view and finalizes pending outcomes as `not_measured`.
|
|
|
|
Disabling only the M5 kill switch prevents new exposure enrollment but previously assigned users can remain active in the framework while the experiment subfeature is enabled.
|
|
|
|
## Privacy Config draft
|
|
|
|
Names are framework-compliant: alphanumeric camelCase, no underscores. Version and rollout values remain explicit TODOs; do not invent them in the config PR.
|
|
|
|
```json
|
|
{
|
|
"features": {
|
|
"cpmBackgroundGraveyard": {
|
|
"state": "enabled",
|
|
"description": "Emergency kill switch for bounded CPM background-view retention",
|
|
"minSupportedVersion": "<platform-specific release version>"
|
|
},
|
|
"cpmBackgroundGraveyardExperiment": {
|
|
"state": "internal",
|
|
"description": "Control/treatment experiment for CPM recovery after background process termination",
|
|
"minSupportedVersion": "<platform-specific release version>",
|
|
"rollout": {
|
|
"steps": [
|
|
{
|
|
"percent": "<approved enrollment percent>"
|
|
}
|
|
]
|
|
},
|
|
"cohorts": [
|
|
{
|
|
"name": "control",
|
|
"weight": 1
|
|
},
|
|
{
|
|
"name": "treatment",
|
|
"weight": 1
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Create separate iOS and macOS config/review paths because release versions, rollout timing, and analysis are platform-specific.
|
|
|
|
## Rollout
|
|
|
|
1. Ship M3 and validate natural termination → CPM failure correlation for one production release.
|
|
2. Land M5/M6 with both runtime flags off; verify pixels and forced-kill recovery internally.
|
|
3. Enable experiment for internal/review users at 50/50; verify assignment, enrollment, outcome coverage, and rollback.
|
|
4. Start public rollout only after Data Science freezes sample size/duration and privacy/schema review is approved.
|
|
5. At target sample, set cohort weights to 0; keep configuration active through the observation window, then decide treatment rollout or removal.
|
|
|
|
## Tasks to create — do not create from this note automatically
|
|
|
|
1. iOS experiment-log task from template [1209007672204094](https://app.asana.com/1/137249556945/project/1208889145294658/task/1209007672204094).
|
|
2. macOS experiment-log task from the same template.
|
|
3. Privacy Configuration change/review for the M5 kill switch and M6 experiment subfeature.
|
|
4. Data Science sample-size/design review plus Native Apps Experiment Analysis request in project [1209801922076585](https://app.asana.com/1/137249556945/project/1209801922076585).
|
|
5. Privacy triage/schema review for operational CPM pixels and the custom framework metric.
|
|
|
|
## Implementation readiness
|
|
|
|
- Lazy enrollment at actual exposure: implemented.
|
|
- Stable native cohort assignment: implemented.
|
|
- Control/treatment behavior isolation: implemented.
|
|
- Standard framework outcome metric: implemented.
|
|
- Countable operational diagnostics: implemented.
|
|
- Runtime kill switch and bounded cleanup: implemented.
|
|
- iOS/macOS compile, WebExtensions tests, and pixel schema validation: passed locally.
|
|
- Remaining before rollout: PRs, the five tasks above, config values, approvals, and production M3 evidence.
|