obsidian-vault/work/projects/cpm-background-graveyard-experiment-definition.md

tags, status, updated

tags	status	updated
cpm experiment ab-testing	historical-proposal	2026-10-20

CPM Background Graveyard — Experiment Definition

Статус на 20 октября 2026: историческое предложение эксперимента, не текущий план запуска. background_graveyard_enabled удалён из episode report по решению пользователя; сама mitigation не менялась, её local default в проверенном коде OFF. Наличие старой реализации/merged mitigation PR не доказывает активный remote rollout или подтверждённую пользу этого treatment.

Последняя telemetry implementation — cpm-pr0-episode-telemetry-summary-2026-10-16, #7376 5d1cb4abd2 (закрыт без merge); результаты и privacy gaps — cpm-status-2026-10-15#6. Телеметрия. P5/P6 evidence не обосновывает graveyard. Cohort/enrollment/success criteria и «ready» ниже — архив 22 сентября; approval/config/задачи здесь заново не проверялись.

История предложения и контекста — сентябрь / 15 октября

Контекст актуализирован 15 октября 2026: cpm-status-2026-10-15; архитектурные ограничения — cpm-service-worker-architecture-and-diagnostics. Определение эксперимента, задачи и readiness ниже сохранены как документ от 22 сентября; их текущий rollout/approval статус этим проходом не переутверждался.

Новые P5/P6 проверки не являются доказательством пользы graveyard: P5/shared-window теряет tab routing после нормального idle, P6 сохраняет неправильный controller после main-flag cycle. Оба механизма отличаются от terminated-background retention. Для пустых listener metadata естественный production-trigger всё ещё не найден. Полная матрица, scopes и отрицательные контроли — основной статус §10–12. Эксперимент, cohorts, enrollment, metrics и flags этим обновлением не менялись.

Status: ready for review; not posted to Asana
Updated: 2026-09-22
Code milestone: M6 alex/cpm-graveyard-ab
Depends on: M3 telemetry → M5 mitigation

Asana task fields

Field	Value
Name — iOS	iOS Mobile Experiment: [control, treatment] CPM background graveyard
Name — macOS	macOS Desktop Experiment: [control, treatment] CPM background graveyard
Platform	Separate experiment-log task for iOS and macOS
Experiment Name	cpmBackgroundGraveyardExperiment
Cohorts	control, treatment
Start / end	TBD after sample-size review
Status	Draft until Privacy Config and privacy/schema approvals are complete
Project	CPM Web Extension breakage
Related CPM task	1216761517055116
Technical design	1218005985662983
A/B framework	1208680035673270

Background

A forced termination of the Web Extension background WebContent process reproduces a persistent all-tabs CPM outage. Reloading a tab or opening a new tab does not recover it; reloading the embedded extension does.

M5 retains at most one terminated background WKWebView for 3 seconds. The hold keeps the dead page/process proxy reachable while WebKit establishes a replacement service-worker context connection. The mitigation is disabled by default and has a separate runtime kill switch.

Production incidence of natural process termination and its causal connection to CPM outage must first be confirmed by M3 telemetry.

Hypothesis

For eligible users whose CPM background WebContent process terminates, the treatment reduces the proportion of users with a subsequent CPM initialization_failed outcome, without increasing crashes, hangs, or unmeasured outcomes.

Eligibility and enrollment

A user is enrolled only when all conditions are true:

1. Web Extensions are available: iOS 18.4+ or macOS 15.4+.
2. M5 kill switch cpmBackgroundGraveyard is enabled.
3. Experiment subfeature cpmBackgroundGraveyardExperiment is enabled and its target/rollout matches.
4. The current CPM background WebContent process actually terminates.
5. resolveCohort returns control or treatment.

Enrollment occurs at the first real exposure event—the process termination—not at app startup or privacy-config parsing. Assignment is stable through the native A/B framework.

Variants

Cohort	Behavior
control	Enroll and emit the same denominator/outcome telemetry; do not retain the terminated view.
treatment	Retain at most one terminated background view for 3 seconds; replace an older held view; release on timeout, context load/unload, or runtime kill-switch disable.

Both cohorts share all code except the bounded hold. Pending outcome correlation is capped at 10 entries; overflow is finalized as not_measured.

Metrics

Primary experiment metric

Framework pixel:

• Experiment: cpmBackgroundGraveyardExperiment
• Metric: cpmBackgroundGraveyardOutcome
• Conversion window: 0...1 days from enrollment
• Predefined values: healthy, initialization_failed, not_measured
• Primary comparison: share of enrolled users with value initialization_failed, treatment vs control

The framework makes this user-level and unique by experiment/cohort/metric/window/value. A user may legitimately emit both healthy and initialization_failed; analysis must treat the primary result as “ever failed in the conversion window,” not an event rate.

Operational diagnostics

• debug_web_extension_cpm_background_graveyard_termination: countable denominator by bounded termination reason and cohort.
• debug_web_extension_cpm_background_graveyard_outcome: countable per-termination outcome by reason, cohort, and outcome.
• Existing initialization-failure and stuck pixels add background_graveyard_cohort.
• Standard framework enrollment, app-use retention, and search retention pixels are automatic.

Operational pixels are required for event-rate and correlation diagnosis; the framework metric is required for the standard experiment analysis pipeline.

Guardrails

1. not_measured coverage and termination/outcome mismatch.
2. Crash and hang rate by platform/cohort.
3. Treatment-only graveyard lifecycle: one held view maximum and bounded release.
4. Cohort balance and enrollment volume.
5. Standard app-use/search retention metrics.

Success and stop criteria

Success criteria must be frozen before rollout:

• statistically meaningful reduction in user-level initialization_failed;
• no material regression in crash/hang or app-use/search retention;
• no unexplained cohort imbalance;
• acceptable not_measured rate;
• operational denominator/outcome coverage is sufficient for interpretation.

Data Science must set minimum sample size, minimum detectable effect, significance/power, and observation duration before public rollout.

To stop new enrollment, set both cohort weights to 0. Do not delete a cohort unless reassignment is intentional.

Emergency rollback requires disabling both:

1. cpmBackgroundGraveyardExperiment — stops experiment activity/retention reporting.
2. cpmBackgroundGraveyard — immediately releases a held view and finalizes pending outcomes as not_measured.

Disabling only the M5 kill switch prevents new exposure enrollment but previously assigned users can remain active in the framework while the experiment subfeature is enabled.

Privacy Config draft

Names are framework-compliant: alphanumeric camelCase, no underscores. Version and rollout values remain explicit TODOs; do not invent them in the config PR.

{
  "features": {
    "cpmBackgroundGraveyard": {
      "state": "enabled",
      "description": "Emergency kill switch for bounded CPM background-view retention",
      "minSupportedVersion": "<platform-specific release version>"
    },
    "cpmBackgroundGraveyardExperiment": {
      "state": "internal",
      "description": "Control/treatment experiment for CPM recovery after background process termination",
      "minSupportedVersion": "<platform-specific release version>",
      "rollout": {
        "steps": [
          {
            "percent": "<approved enrollment percent>"
          }
        ]
      },
      "cohorts": [
        {
          "name": "control",
          "weight": 1
        },
        {
          "name": "treatment",
          "weight": 1
        }
      ]
    }
  }
}

Create separate iOS and macOS config/review paths because release versions, rollout timing, and analysis are platform-specific.

Rollout

1. Ship M3 and validate natural termination → CPM failure correlation for one production release.
2. Land M5/M6 with both runtime flags off; verify pixels and forced-kill recovery internally.
3. Enable experiment for internal/review users at 50/50; verify assignment, enrollment, outcome coverage, and rollback.
4. Start public rollout only after Data Science freezes sample size/duration and privacy/schema review is approved.
5. At target sample, set cohort weights to 0; keep configuration active through the observation window, then decide treatment rollout or removal.

Tasks to create — do not create from this note automatically

1. iOS experiment-log task from template 1209007672204094.
2. macOS experiment-log task from the same template.
3. Privacy Configuration change/review for the M5 kill switch and M6 experiment subfeature.
4. Data Science sample-size/design review plus Native Apps Experiment Analysis request in project 1209801922076585.
5. Privacy triage/schema review for operational CPM pixels and the custom framework metric.

Implementation readiness

• Lazy enrollment at actual exposure: implemented.
• Stable native cohort assignment: implemented.
• Control/treatment behavior isolation: implemented.
• Standard framework outcome metric: implemented.
• Countable operational diagnostics: implemented.
• Runtime kill switch and bounded cleanup: implemented.
• iOS/macOS compile, WebExtensions tests, and pixel schema validation: passed locally.
• Remaining before rollout: PRs, the five tasks above, config values, approvals, and production M3 evidence.