[2026-09-22] eagle: personal/tech/lampac-deployment.md work/projects/cpm-background-graveyard-experiment-definition.md
This commit is contained in:
Alexey Martemyanov committed
1 parent
5e389f93d7
commit
37fca3688d
2 files changed
+220
No files matched your search
@@ -0,0 +1,182 @@
|
||||
---
|
||||
tags:
|
||||
- cpm
|
||||
- experiment
|
||||
- ab-testing
|
||||
status: ready-for-review
|
||||
updated: '2026-09-22'
|
||||
---
|
||||
# CPM Background Graveyard — Experiment Definition
|
||||
|
||||
> Status: **ready for review; not posted to Asana**
|
||||
> Updated: 2026-09-22
|
||||
> Code milestone: M6 `alex/cpm-graveyard-ab`
|
||||
> Depends on: M3 telemetry → M5 mitigation
|
||||
|
||||
## Asana task fields
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Name — iOS | `iOS Mobile Experiment: [control, treatment] CPM background graveyard` |
|
||||
| Name — macOS | `macOS Desktop Experiment: [control, treatment] CPM background graveyard` |
|
||||
| Platform | Separate experiment-log task for iOS and macOS |
|
||||
| Experiment Name | `cpmBackgroundGraveyardExperiment` |
|
||||
| Cohorts | `control`, `treatment` |
|
||||
| Start / end | TBD after sample-size review |
|
||||
| Status | Draft until Privacy Config and privacy/schema approvals are complete |
|
||||
| Project | CPM Web Extension breakage |
|
||||
| Related CPM task | [1216761517055116](https://app.asana.com/1/137249556945/project/1211834678943996/task/1216761517055116) |
|
||||
| Technical design | [1218005985662983](https://app.asana.com/1/137249556945/project/1211834678943996/task/1218005985662983) |
|
||||
| A/B framework | [1208680035673270](https://app.asana.com/1/137249556945/project/1208889145294658/task/1208680035673270) |
|
||||
|
||||
## Background
|
||||
|
||||
A forced termination of the Web Extension background WebContent process reproduces a persistent all-tabs CPM outage. Reloading a tab or opening a new tab does not recover it; reloading the embedded extension does.
|
||||
|
||||
M5 retains at most one terminated background `WKWebView` for 3 seconds. The hold keeps the dead page/process proxy reachable while WebKit establishes a replacement service-worker context connection. The mitigation is disabled by default and has a separate runtime kill switch.
|
||||
|
||||
Production incidence of natural process termination and its causal connection to CPM outage must first be confirmed by M3 telemetry.
|
||||
|
||||
## Hypothesis
|
||||
|
||||
For eligible users whose CPM background WebContent process terminates, the treatment reduces the proportion of users with a subsequent CPM `initialization_failed` outcome, without increasing crashes, hangs, or unmeasured outcomes.
|
||||
|
||||
## Eligibility and enrollment
|
||||
|
||||
A user is enrolled only when all conditions are true:
|
||||
|
||||
1. Web Extensions are available: iOS 18.4+ or macOS 15.4+.
|
||||
2. M5 kill switch `cpmBackgroundGraveyard` is enabled.
|
||||
3. Experiment subfeature `cpmBackgroundGraveyardExperiment` is enabled and its target/rollout matches.
|
||||
4. The current CPM background WebContent process actually terminates.
|
||||
5. `resolveCohort` returns `control` or `treatment`.
|
||||
|
||||
Enrollment occurs at the first real exposure event—the process termination—not at app startup or privacy-config parsing. Assignment is stable through the native A/B framework.
|
||||
|
||||
## Variants
|
||||
|
||||
| Cohort | Behavior |
|
||||
|---|---|
|
||||
| `control` | Enroll and emit the same denominator/outcome telemetry; do not retain the terminated view. |
|
||||
| `treatment` | Retain at most one terminated background view for 3 seconds; replace an older held view; release on timeout, context load/unload, or runtime kill-switch disable. |
|
||||
|
||||
Both cohorts share all code except the bounded hold. Pending outcome correlation is capped at 10 entries; overflow is finalized as `not_measured`.
|
||||
|
||||
## Metrics
|
||||
|
||||
### Primary experiment metric
|
||||
|
||||
Framework pixel:
|
||||
|
||||
- Experiment: `cpmBackgroundGraveyardExperiment`
|
||||
- Metric: `cpmBackgroundGraveyardOutcome`
|
||||
- Conversion window: `0...1` days from enrollment
|
||||
- Predefined values: `healthy`, `initialization_failed`, `not_measured`
|
||||
- Primary comparison: share of enrolled users with value `initialization_failed`, treatment vs control
|
||||
|
||||
The framework makes this user-level and unique by experiment/cohort/metric/window/value. A user may legitimately emit both `healthy` and `initialization_failed`; analysis must treat the primary result as “ever failed in the conversion window,” not an event rate.
|
||||
|
||||
### Operational diagnostics
|
||||
|
||||
- `debug_web_extension_cpm_background_graveyard_termination`: countable denominator by bounded termination reason and cohort.
|
||||
- `debug_web_extension_cpm_background_graveyard_outcome`: countable per-termination outcome by reason, cohort, and outcome.
|
||||
- Existing initialization-failure and stuck pixels add `background_graveyard_cohort`.
|
||||
- Standard framework enrollment, app-use retention, and search retention pixels are automatic.
|
||||
|
||||
Operational pixels are required for event-rate and correlation diagnosis; the framework metric is required for the standard experiment analysis pipeline.
|
||||
|
||||
### Guardrails
|
||||
|
||||
1. `not_measured` coverage and termination/outcome mismatch.
|
||||
2. Crash and hang rate by platform/cohort.
|
||||
3. Treatment-only graveyard lifecycle: one held view maximum and bounded release.
|
||||
4. Cohort balance and enrollment volume.
|
||||
5. Standard app-use/search retention metrics.
|
||||
|
||||
## Success and stop criteria
|
||||
|
||||
Success criteria must be frozen before rollout:
|
||||
|
||||
- statistically meaningful reduction in user-level `initialization_failed`;
|
||||
- no material regression in crash/hang or app-use/search retention;
|
||||
- no unexplained cohort imbalance;
|
||||
- acceptable `not_measured` rate;
|
||||
- operational denominator/outcome coverage is sufficient for interpretation.
|
||||
|
||||
Data Science must set minimum sample size, minimum detectable effect, significance/power, and observation duration before public rollout.
|
||||
|
||||
To stop new enrollment, set both cohort weights to `0`. Do not delete a cohort unless reassignment is intentional.
|
||||
|
||||
Emergency rollback requires disabling **both**:
|
||||
|
||||
1. `cpmBackgroundGraveyardExperiment` — stops experiment activity/retention reporting.
|
||||
2. `cpmBackgroundGraveyard` — immediately releases a held view and finalizes pending outcomes as `not_measured`.
|
||||
|
||||
Disabling only the M5 kill switch prevents new exposure enrollment but previously assigned users can remain active in the framework while the experiment subfeature is enabled.
|
||||
|
||||
## Privacy Config draft
|
||||
|
||||
Names are framework-compliant: alphanumeric camelCase, no underscores. Version and rollout values remain explicit TODOs; do not invent them in the config PR.
|
||||
|
||||
```json
|
||||
{
|
||||
"features": {
|
||||
"cpmBackgroundGraveyard": {
|
||||
"state": "enabled",
|
||||
"description": "Emergency kill switch for bounded CPM background-view retention",
|
||||
"minSupportedVersion": "<platform-specific release version>"
|
||||
},
|
||||
"cpmBackgroundGraveyardExperiment": {
|
||||
"state": "internal",
|
||||
"description": "Control/treatment experiment for CPM recovery after background process termination",
|
||||
"minSupportedVersion": "<platform-specific release version>",
|
||||
"rollout": {
|
||||
"steps": [
|
||||
{
|
||||
"percent": "<approved enrollment percent>"
|
||||
}
|
||||
]
|
||||
},
|
||||
"cohorts": [
|
||||
{
|
||||
"name": "control",
|
||||
"weight": 1
|
||||
},
|
||||
{
|
||||
"name": "treatment",
|
||||
"weight": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Create separate iOS and macOS config/review paths because release versions, rollout timing, and analysis are platform-specific.
|
||||
|
||||
## Rollout
|
||||
|
||||
1. Ship M3 and validate natural termination → CPM failure correlation for one production release.
|
||||
2. Land M5/M6 with both runtime flags off; verify pixels and forced-kill recovery internally.
|
||||
3. Enable experiment for internal/review users at 50/50; verify assignment, enrollment, outcome coverage, and rollback.
|
||||
4. Start public rollout only after Data Science freezes sample size/duration and privacy/schema review is approved.
|
||||
5. At target sample, set cohort weights to 0; keep configuration active through the observation window, then decide treatment rollout or removal.
|
||||
|
||||
## Tasks to create — do not create from this note automatically
|
||||
|
||||
1. iOS experiment-log task from template [1209007672204094](https://app.asana.com/1/137249556945/project/1208889145294658/task/1209007672204094).
|
||||
2. macOS experiment-log task from the same template.
|
||||
3. Privacy Configuration change/review for the M5 kill switch and M6 experiment subfeature.
|
||||
4. Data Science sample-size/design review plus Native Apps Experiment Analysis request in project [1209801922076585](https://app.asana.com/1/137249556945/project/1209801922076585).
|
||||
5. Privacy triage/schema review for operational CPM pixels and the custom framework metric.
|
||||
|
||||
## Implementation readiness
|
||||
|
||||
- Lazy enrollment at actual exposure: implemented.
|
||||
- Stable native cohort assignment: implemented.
|
||||
- Control/treatment behavior isolation: implemented.
|
||||
- Standard framework outcome metric: implemented.
|
||||
- Countable operational diagnostics: implemented.
|
||||
- Runtime kill switch and bounded cleanup: implemented.
|
||||
- iOS/macOS compile, WebExtensions tests, and pixel schema validation: passed locally.
|
||||
- Remaining before rollout: PRs, the five tasks above, config values, approvals, and production M3 evidence.
|
||||
Reference in new issue
Block a user