[2026-10-20] eagle: work/projects/CPM Breakage Testing Steps.md work/projects/cpm-background-graveyard-experiment-definition.md work/projects/cpm-extension-health-pixels-privacy-triage.md work/projects/cpm-m3-telemetry-privacy-triage.md work/projects/cpm-metrics-causes-and-mitigations-2026-10-16.md work/projects/cpm-pr0-episode-telemetry-summary-2026-10-16.md work/projects/cpm-readiness-telemetry-tech-design-2026-10-15.md work/projects/cpm-service-worker-architecture-and-diagnostics.md work/projects/cpm-telemetry-data-requirements-2026-10-15.md work/projects/cpm-web-extension-breakage-findings.md
This commit is contained in:
Alexey Martemyanov committed
1 parent
5b8f6bc510
commit
8f39ed45d1
17 files changed
+256
-42
No files matched your search
@@ -1,3 +1,70 @@
|
||||
# CPM episode telemetry — testing steps
|
||||
|
||||
Verified against #7376 **`5d1cb4abd2`**, 20 October 2026. These expectations apply to that published branch; the PR is closed without merge. The macOS counterpart #7484 still has the earlier `was_stuck` contract. Full values and limitations: [[cpm-pr0-episode-telemetry-summary-2026-10-16]].
|
||||
|
||||
Use an isolated test profile and filter the Xcode console by `PixelKit` and `CPM Health Monitor`. Finish each episode before starting the next case. In production, the first completed episode per day sends `web_extension_cpm_episode_daily` and `_count` with the same payload; subsequent episodes send count only, across every reason/outcome. PixelKit dry-run logs instead of sending and compresses daily deduplication to two minutes, so a five-minute case may cross that boundary.
|
||||
|
||||
The Debug breakage switch hides native dashboard responses from the detector. It tests monitor/reporting behavior, **not actual loss of page initResp**. A working page can intentionally look broken to the monitor.
|
||||
|
||||
## Healthy navigation
|
||||
|
||||
1. Keep Simulate CPM Messaging Breakage OFF; load an HTTP(S) page with CPM enabled.
|
||||
2. Confirm a normal dashboard response and no episode report after the grace period.
|
||||
3. With simulation ON, visit internal New Tab, Settings or Bookmarks pages: they must not start an eligible HTTP(S) failure measurement.
|
||||
|
||||
## Unconfirmed one-tab failure, then response
|
||||
|
||||
1. Enable simulation; load an HTTP(S) page in one tab.
|
||||
2. Wait for the four-second timeout: an episode opens locally; **no pixel is sent**.
|
||||
3. Disable simulation and reload the same tab before the five-minute deadline.
|
||||
4. On a qualifying response, expect one terminal report: `stuck=false-one`, normally `outcome=failed_tab_response` for the later measurement, `reload_state=none`, bucketed `response_after`.
|
||||
5. No separate init_failed, stuck or recovered event is emitted; the cancelled deadline must not send a second report.
|
||||
|
||||
A late response to the original timed-out measurement is covered by automated tests and yields `failed_measurement_response`; a page reload creates a different measurement.
|
||||
|
||||
## Confirmed persistence: one tab or several
|
||||
|
||||
1. With simulation ON, let tab A time out.
|
||||
2. After that first failure, begin another eligible regular navigation in A and let it time out: persistence becomes true, scope stays one.
|
||||
3. End the episode by a qualifying response after disabling simulation, or leave the app running until the deadline. Expect `stuck=true-one`.
|
||||
4. Repeat with the second qualifying timeout in another tab: expect `stuck=true-many`.
|
||||
5. Automatic extension reload may change `reload_state`; do not assume none. Loaded is not proof that messaging recovered.
|
||||
|
||||
one/many counts distinct tabs with observed timeouts; it does not actively check every tab or determine whether all tabs in one window died.
|
||||
|
||||
## Concurrent failures and session restoration
|
||||
|
||||
1. Start pages in two tabs before either first timeout. Let both time out: scope is many, but those already-in-flight attempts must not confirm persistence.
|
||||
2. Disable simulation and obtain a qualifying response before a further timeout, or wait for the deadline: expect `stuck=false-many`.
|
||||
3. For restoration, use genuinely lazy-restored tabs and verify `session_restoration` in the monitor log. Timeouts in additional restored tabs increase scope but do not confirm persistence.
|
||||
4. A later ordinary navigation started after the first failure can confirm stuck if it times out. `reason` remains the first failure's context.
|
||||
|
||||
## Deadline without another navigation
|
||||
|
||||
1. With simulation ON, cause one eligible timeout.
|
||||
2. Do not open or reload another page; keep the process/monitor alive through five minutes after the first failure.
|
||||
3. Expect `outcome=no_response_before_deadline`, `stuck=false-one`, and **no `response_after`**.
|
||||
4. This is an observation deadline, not proof of permanent failure. Destroying the monitor/process before closure loses the unfinished report.
|
||||
|
||||
## Reload state
|
||||
|
||||
Within an open episode, inspect lifecycle transitions: none → pending → loaded or failed; the first eligible post-reload measurement can set response_timed_out. These transitions do not send separate CPM events. A terminal report carries the observed state. The general web-extension reload-error pixel remains separate. Precise reload/error-generation cases are in `CPMMessagingHealthMonitorTests`.
|
||||
|
||||
## Background history and load attribution
|
||||
|
||||
- All eight booleans and four load fields describe the first failure; a later error must not overwrite them.
|
||||
- Non-stuck episodes have only initial history. Stuck episodes use `initial;events-until-first-stuck`, frozen before its automatic recovery.
|
||||
- Split tokens on commas and semicolons; ignore empty entries and standalone `...`. The marker identifies dropped oldest entries.
|
||||
- The combined value must stay within 12 events and 255 characters, including markers/separators. A long continuation can evict the entire initial phase; no reservation is implemented.
|
||||
- New repeated lifecycle records are retained; only consecutive identical load errors within one second are suppressed.
|
||||
- Missing optional values must be omitted. No URL, document/tab/context ID or PID may be added.
|
||||
|
||||
Deterministic coverage: `testStuckParameterDistinguishesPersistenceAndDistinctFailedTabs`, `testDeadlineReportsQuietEpisodeWithoutAnotherEvent`, `testStuckTimelinePreservesEventsAcrossContextResetAndExcludesRecovery`, `testStuckTimelineMarksOverflowOfTheLiveContinuation` and `CPMMessagingDiagnosticsTests`.
|
||||
|
||||
## Historical checks for the previous separate-pixel contract
|
||||
|
||||
The steps below are retained for the old implementation; their event names and immediate-send expectations **do not apply to #7376**.
|
||||
|
||||
### Testing Steps
|
||||
|
||||
Run each case with fresh app data so daily pixel deduplication does not hide results. In Xcode’s console, filter by `PixelKit` Category.
|
||||
@@ -47,4 +114,3 @@ Run each case with fresh app data so daily pixel deduplication does not hide res
|
||||
- `debug_web_extension_cpm_messaging_extension_reload_failed`
|
||||
4. Navigate again and wait 5 seconds → `debug_web_extension_cpm_messaging_stuck_extension_reload` is sent.
|
||||
5. Select **Simulate CPM Messaging Breakage (Off)** and reload → `debug_web_extension_cpm_messaging_recovered_after_extension_reload` is sent.
|
||||
|
||||
Reference in new issue
Block a user