159 lines
12 KiB
Markdown
Executable File
159 lines
12 KiB
Markdown
Executable File
---
|
||
updated: '2026-10-30'
|
||
---
|
||
# CPM episode telemetry — testing steps
|
||
|
||
## Error-6 restart — PR 7730 (30 October 2026)
|
||
|
||
Current production PR: [#7730](https://github.com/duckduckgo/apple-browsers/pull/7730), **`cf668e2938`**, OPEN and not shipped. Review harness: `codex/cpm-error6-restart-harness` at **`5348c7254e`**, production code identical to the PR; its extra fixtures are outside main and production CI. Use a clean, isolated macOS checkout with the repository's Xcode toolchain.
|
||
|
||
1. Fetch and select the harness:
|
||
```sh
|
||
git fetch origin
|
||
git switch --detach origin/codex/cpm-error6-restart-harness
|
||
```
|
||
2. Disable only retained-view restart:
|
||
```sh
|
||
CPM_REPRO=1 CPM_REPRO_RESTART=0 swift test \
|
||
--package-path SharedPackages/WebExtensions \
|
||
--filter CPMReproError6RestartTests/testRetainedError6RestartRestoresMessaging
|
||
```
|
||
Expected **functional FAIL**: `enabled=false error6=true retained=true`, original document timeout, nonempty `lost`, `restarts=0 responses=0`; existing M4 eventually repairs messaging. Setup errors, skips or absence of the injected real error are not valid negative controls.
|
||
3. Enable restart on the same binary:
|
||
```sh
|
||
CPM_REPRO=1 CPM_REPRO_RESTART=1 swift test \
|
||
--package-path SharedPackages/WebExtensions \
|
||
--skip-build \
|
||
--filter CPMReproError6RestartTests/testRetainedError6RestartRestoresMessaging
|
||
```
|
||
Expected **PASS**: `enabled=true error6=true retained=true`, original document gets an attributed dashboard response **and actual page initResp**, `lost=[] restarts=1 responses=1`, two later ordinary idle/wake cycles answer.
|
||
4. Return to production:
|
||
```sh
|
||
git switch --detach origin/alex/cpm-error6-restart
|
||
```
|
||
|
||
The last completed same-binary real-WebKit OFF/ON used **production `d1a2e1de65`**, not the latest head. The fixture branch is now synchronized for reviewers to rerun. This injected worker-start exception validates the retained-view mechanism; it does not establish the natural production trigger or physical iOS suspension behavior.
|
||
|
||
Permanent tests in the PR cover new/historical/equal errors, healthy replacements, weak ownership, foreground/wake, runtime kill switches, queued newer errors, automatic cooldown expiry and expiry in background. Telemetry tests cover success vs background_ready, measured/unobserved deadlines, suspension, supersession, reload failure/timeout and embedded upgrades. Same-identifier loaded and another extension type must not supersede an active observation. These tests run in ordinary PR CI; the optional slow fault-injection harness does not.
|
||
|
||
Latest local package result: **1064 PASS**; final source review with eight temporary probes: **1072 PASS**, no important findings. The temporary probes were removed. Treat `background_ready` only as a stage, not messaging success. Evaluate `web_extension_cpm_background_load_restart` using **count** rows; daily deduplication is by name. No response by deadline or a missing terminal row after process exit does not prove permanent breakage.
|
||
|
||
Full contract and evidence: [[cpm-status-2026-10-15#Error-6 restart — PR 7730]].
|
||
|
||
## Historical episode-telemetry scope — 20 October 2026
|
||
|
||
Verified against #7376 **`5d1cb4abd2`**, 20 October 2026. These expectations apply to that published branch; the PR is closed without merge. The macOS counterpart #7484 still has the earlier `was_stuck` contract. Full values and limitations: [[cpm-pr0-episode-telemetry-summary-2026-10-16]].
|
||
|
||
Use an isolated test profile and filter the Xcode console by `PixelKit` and `CPM Health Monitor`. Finish each episode before starting the next case. In production, the first completed episode per day sends `web_extension_cpm_episode_daily` and `_count` with the same payload; subsequent episodes send count only, across every reason/outcome. PixelKit dry-run logs instead of sending and compresses daily deduplication to two minutes, so a five-minute case may cross that boundary.
|
||
|
||
The Debug breakage switch hides native dashboard responses from the detector. It tests monitor/reporting behavior, **not actual loss of page initResp**. A working page can intentionally look broken to the monitor.
|
||
|
||
## Healthy navigation
|
||
|
||
1. Keep Simulate CPM Messaging Breakage OFF; load an HTTP(S) page with CPM enabled.
|
||
2. Confirm a normal dashboard response and no episode report after the grace period.
|
||
3. With simulation ON, visit internal New Tab, Settings or Bookmarks pages: they must not start an eligible HTTP(S) failure measurement.
|
||
|
||
## Unconfirmed one-tab failure, then response
|
||
|
||
1. Enable simulation; load an HTTP(S) page in one tab.
|
||
2. Wait for the four-second timeout: an episode opens locally; **no pixel is sent**.
|
||
3. Disable simulation and reload the same tab before the five-minute deadline.
|
||
4. On a qualifying response, expect one terminal report: `stuck=false-one`, normally `outcome=failed_tab_response` for the later measurement, `reload_state=none`, bucketed `response_after`.
|
||
5. No separate init_failed, stuck or recovered event is emitted; the cancelled deadline must not send a second report.
|
||
|
||
A late response to the original timed-out measurement is covered by automated tests and yields `failed_measurement_response`; a page reload creates a different measurement.
|
||
|
||
## Confirmed persistence: one tab or several
|
||
|
||
1. With simulation ON, let tab A time out.
|
||
2. After that first failure, begin another eligible regular navigation in A and let it time out: persistence becomes true, scope stays one.
|
||
3. End the episode by a qualifying response after disabling simulation, or leave the app running until the deadline. Expect `stuck=true-one`.
|
||
4. Repeat with the second qualifying timeout in another tab: expect `stuck=true-many`.
|
||
5. Automatic extension reload may change `reload_state`; do not assume none. Loaded is not proof that messaging recovered.
|
||
|
||
one/many counts distinct tabs with observed timeouts; it does not actively check every tab or determine whether all tabs in one window died.
|
||
|
||
## Concurrent failures and session restoration
|
||
|
||
1. Start pages in two tabs before either first timeout. Let both time out: scope is many, but those already-in-flight attempts must not confirm persistence.
|
||
2. Disable simulation and obtain a qualifying response before a further timeout, or wait for the deadline: expect `stuck=false-many`.
|
||
3. For restoration, use genuinely lazy-restored tabs and verify `session_restoration` in the monitor log. Timeouts in additional restored tabs increase scope but do not confirm persistence.
|
||
4. A later ordinary navigation started after the first failure can confirm stuck if it times out. `reason` remains the first failure's context.
|
||
|
||
## Deadline without another navigation
|
||
|
||
1. With simulation ON, cause one eligible timeout.
|
||
2. Do not open or reload another page; keep the process/monitor alive through five minutes after the first failure.
|
||
3. Expect `outcome=no_response_before_deadline`, `stuck=false-one`, and **no `response_after`**.
|
||
4. This is an observation deadline, not proof of permanent failure. Destroying the monitor/process before closure loses the unfinished report.
|
||
|
||
## Reload state
|
||
|
||
Within an open episode, inspect lifecycle transitions: none → pending → loaded or failed; the first eligible post-reload measurement can set response_timed_out. These transitions do not send separate CPM events. A terminal report carries the observed state. The general web-extension reload-error pixel remains separate. Precise reload/error-generation cases are in `CPMMessagingHealthMonitorTests`.
|
||
|
||
## Background history and load attribution
|
||
|
||
- All eight booleans and four load fields describe the first failure; a later error must not overwrite them.
|
||
- Non-stuck episodes have only initial history. Stuck episodes use `initial;events-until-first-stuck`, frozen before its automatic recovery.
|
||
- Split tokens on commas and semicolons; ignore empty entries and standalone `...`. The marker identifies dropped oldest entries.
|
||
- The combined value must stay within 12 events and 255 characters, including markers/separators. A long continuation can evict the entire initial phase; no reservation is implemented.
|
||
- New repeated lifecycle records are retained; only consecutive identical load errors within one second are suppressed.
|
||
- Missing optional values must be omitted. No URL, document/tab/context ID or PID may be added.
|
||
|
||
Deterministic coverage: `testStuckParameterDistinguishesPersistenceAndDistinctFailedTabs`, `testDeadlineReportsQuietEpisodeWithoutAnotherEvent`, `testStuckTimelinePreservesEventsAcrossContextResetAndExcludesRecovery`, `testStuckTimelineMarksOverflowOfTheLiveContinuation` and `CPMMessagingDiagnosticsTests`.
|
||
|
||
## Historical checks for the previous separate-pixel contract
|
||
|
||
The steps below are retained for the old implementation; their event names and immediate-send expectations **do not apply to #7376**.
|
||
|
||
### Testing Steps
|
||
|
||
Run each case with fresh app data so daily pixel deduplication does not hide results. In Xcode’s console, filter by `PixelKit` Category.
|
||
|
||
#### Healthy navigation
|
||
|
||
1. Open a web page.
|
||
2. Wait at least 5 seconds → no `debug_web_extension_cpm_*` pixel is sent.
|
||
3. Open **Debug → Web Extensions → Simulate CPM Messaging Breakage (On)**.
|
||
4. Open New Tab, Settings, Bookmarks (internal pages), validate no `debug_web_extension_cpm_*` pixel is sent
|
||
|
||
#### Failure, stuck, and recovery
|
||
|
||
1. Open **Debug → Web Extensions → Simulate CPM Messaging Breakage (On)**.
|
||
2. Load a web page and wait 5 seconds → `debug_web_extension_cpm_initialization_failed_after_other` is sent; no `stuck` pixel is sent.
|
||
3. Navigate to another HTTP(S) page and wait 5 seconds → `debug_web_extension_cpm_messaging_stuck_other` is sent.
|
||
4. Select **Simulate CPM Messaging Breakage (Off)**.
|
||
5. Reload the page → `debug_web_extension_cpm_messaging_recovered_without_extension_reload` is sent.
|
||
6. Navigate again → no additional CPM failure or recovery pixel is sent.
|
||
|
||
#### Session restoration
|
||
1. Open a web page, e.g. https://privacy-test-pages.site/features/autoconsent/banner.html
|
||
2. Duplicate the tab twice (3 tabs are open)
|
||
3. Open Settings → General, activate Reopen all windows and tabs from last session
|
||
4. Restart the browser
|
||
5. Open **Debug → Web Extensions → Simulate CPM Messaging Breakage (On)**
|
||
6. Switch to another tab
|
||
7. wait 5 seconds → `debug_web_extension_cpm_initialization_failed_after_session_restoration` is sent
|
||
8. Switch to the remaining restored tab
|
||
9. wait 5 seconds → `debug_web_extension_cpm_initialization_failed_after_session_restoration` is **skipped** (as it‘s daily): no `debug_web_extension_cpm_messaging_stuck…` is sent
|
||
10. Reload the tab or navigate somewhere else
|
||
11. wait 5 seconds → `debug_web_extension_cpm_initialization_failed_after_other` is sent; `debug_web_extension_cpm_messaging_stuck_session_restoration` is sent
|
||
|
||
#### Concurrent navigations
|
||
|
||
1. Select **Simulate CPM Messaging Breakage (On)**.
|
||
2. Load HTTP(S) pages in two tabs before either 4-second timeout expires.
|
||
3. Wait 5 seconds → initialization failures are observed, but no `debug_web_extension_cpm_messaging_stuck_*` pixel is sent.
|
||
4. Start another navigation after those failures and wait 5 seconds → `debug_web_extension_cpm_messaging_stuck_other` is sent.
|
||
|
||
#### Extension reload
|
||
|
||
1. Start with **Simulate CPM Messaging Breakage (Off)** and use the Fire button to trigger extension reloading.
|
||
2. After data clearing completes, select **Simulate CPM Messaging Breakage (On)**.
|
||
3. Load a web page and wait 5 seconds → these pixels are sent:
|
||
- `debug_web_extension_cpm_initialization_failed_after_extension_reload`
|
||
- `debug_web_extension_cpm_messaging_extension_reload_failed`
|
||
4. Navigate again and wait 5 seconds → `debug_web_extension_cpm_messaging_stuck_extension_reload` is sent.
|
||
5. Select **Simulate CPM Messaging Breakage (Off)** and reload → `debug_web_extension_cpm_messaging_recovered_after_extension_reload` is sent.
|