obsidian-vault/work/projects/CPM Breakage Testing Steps.md

updated

updated
2026-10-30

CPM episode telemetry — testing steps

Error-6 restart — PR 7730 (30 October 2026)

Current production PR: #7730, cf668e2938, OPEN and not shipped. Review harness: codex/cpm-error6-restart-harness at 5348c7254e, production code identical to the PR; its extra fixtures are outside main and production CI. Use a clean, isolated macOS checkout with the repository's Xcode toolchain.

1. Fetch and select the harness:

   git fetch origin
   git switch --detach origin/codex/cpm-error6-restart-harness
   

2. Disable only retained-view restart:

   CPM_REPRO=1 CPM_REPRO_RESTART=0 swift test \
     --package-path SharedPackages/WebExtensions \
     --filter CPMReproError6RestartTests/testRetainedError6RestartRestoresMessaging
   

   Expected functional FAIL: enabled=false error6=true retained=true, original document timeout, nonempty lost, restarts=0 responses=0; existing M4 eventually repairs messaging. Setup errors, skips or absence of the injected real error are not valid negative controls.
3. Enable restart on the same binary:

   CPM_REPRO=1 CPM_REPRO_RESTART=1 swift test \
     --package-path SharedPackages/WebExtensions \
     --skip-build \
     --filter CPMReproError6RestartTests/testRetainedError6RestartRestoresMessaging
   

   Expected PASS: enabled=true error6=true retained=true, original document gets an attributed dashboard response and actual page initResp, lost=[] restarts=1 responses=1, two later ordinary idle/wake cycles answer.
4. Return to production:

   git switch --detach origin/alex/cpm-error6-restart
   

The last completed same-binary real-WebKit OFF/ON used production d1a2e1de65, not the latest head. The fixture branch is now synchronized for reviewers to rerun. This injected worker-start exception validates the retained-view mechanism; it does not establish the natural production trigger or physical iOS suspension behavior.

Permanent tests in the PR cover new/historical/equal errors, healthy replacements, weak ownership, foreground/wake, runtime kill switches, queued newer errors, automatic cooldown expiry and expiry in background. Telemetry tests cover success vs background_ready, measured/unobserved deadlines, suspension, supersession, reload failure/timeout and embedded upgrades. Same-identifier loaded and another extension type must not supersede an active observation. These tests run in ordinary PR CI; the optional slow fault-injection harness does not.

Latest local package result: 1064 PASS; final source review with eight temporary probes: 1072 PASS, no important findings. The temporary probes were removed. Treat background_ready only as a stage, not messaging success. Evaluate web_extension_cpm_background_load_restart using count rows; daily deduplication is by name. No response by deadline or a missing terminal row after process exit does not prove permanent breakage.

Full contract and evidence: cpm-status-2026-10-15#Error-6 restart — PR 7730.

Historical episode-telemetry scope — 20 October 2026

Verified against #7376 5d1cb4abd2, 20 October 2026. These expectations apply to that published branch; the PR is closed without merge. The macOS counterpart #7484 still has the earlier was_stuck contract. Full values and limitations: cpm-pr0-episode-telemetry-summary-2026-10-16.

Use an isolated test profile and filter the Xcode console by PixelKit and CPM Health Monitor. Finish each episode before starting the next case. In production, the first completed episode per day sends web_extension_cpm_episode_daily and _count with the same payload; subsequent episodes send count only, across every reason/outcome. PixelKit dry-run logs instead of sending and compresses daily deduplication to two minutes, so a five-minute case may cross that boundary.

The Debug breakage switch hides native dashboard responses from the detector. It tests monitor/reporting behavior, not actual loss of page initResp. A working page can intentionally look broken to the monitor.

Healthy navigation

1. Keep Simulate CPM Messaging Breakage OFF; load an HTTP(S) page with CPM enabled.
2. Confirm a normal dashboard response and no episode report after the grace period.
3. With simulation ON, visit internal New Tab, Settings or Bookmarks pages: they must not start an eligible HTTP(S) failure measurement.

Unconfirmed one-tab failure, then response

1. Enable simulation; load an HTTP(S) page in one tab.
2. Wait for the four-second timeout: an episode opens locally; no pixel is sent.
3. Disable simulation and reload the same tab before the five-minute deadline.
4. On a qualifying response, expect one terminal report: stuck=false-one, normally outcome=failed_tab_response for the later measurement, reload_state=none, bucketed response_after.
5. No separate init_failed, stuck or recovered event is emitted; the cancelled deadline must not send a second report.

A late response to the original timed-out measurement is covered by automated tests and yields failed_measurement_response; a page reload creates a different measurement.

Confirmed persistence: one tab or several

1. With simulation ON, let tab A time out.
2. After that first failure, begin another eligible regular navigation in A and let it time out: persistence becomes true, scope stays one.
3. End the episode by a qualifying response after disabling simulation, or leave the app running until the deadline. Expect stuck=true-one.
4. Repeat with the second qualifying timeout in another tab: expect stuck=true-many.
5. Automatic extension reload may change reload_state; do not assume none. Loaded is not proof that messaging recovered.

one/many counts distinct tabs with observed timeouts; it does not actively check every tab or determine whether all tabs in one window died.

Concurrent failures and session restoration

1. Start pages in two tabs before either first timeout. Let both time out: scope is many, but those already-in-flight attempts must not confirm persistence.
2. Disable simulation and obtain a qualifying response before a further timeout, or wait for the deadline: expect stuck=false-many.
3. For restoration, use genuinely lazy-restored tabs and verify session_restoration in the monitor log. Timeouts in additional restored tabs increase scope but do not confirm persistence.
4. A later ordinary navigation started after the first failure can confirm stuck if it times out. reason remains the first failure's context.

Deadline without another navigation

1. With simulation ON, cause one eligible timeout.
2. Do not open or reload another page; keep the process/monitor alive through five minutes after the first failure.
3. Expect outcome=no_response_before_deadline, stuck=false-one, and no response_after.
4. This is an observation deadline, not proof of permanent failure. Destroying the monitor/process before closure loses the unfinished report.

Reload state

Within an open episode, inspect lifecycle transitions: none → pending → loaded or failed; the first eligible post-reload measurement can set response_timed_out. These transitions do not send separate CPM events. A terminal report carries the observed state. The general web-extension reload-error pixel remains separate. Precise reload/error-generation cases are in CPMMessagingHealthMonitorTests.

Background history and load attribution

• All eight booleans and four load fields describe the first failure; a later error must not overwrite them.
• Non-stuck episodes have only initial history. Stuck episodes use initial;events-until-first-stuck, frozen before its automatic recovery.
• Split tokens on commas and semicolons; ignore empty entries and standalone .... The marker identifies dropped oldest entries.
• The combined value must stay within 12 events and 255 characters, including markers/separators. A long continuation can evict the entire initial phase; no reservation is implemented.
• New repeated lifecycle records are retained; only consecutive identical load errors within one second are suppressed.
• Missing optional values must be omitted. No URL, document/tab/context ID or PID may be added.

Deterministic coverage: testStuckParameterDistinguishesPersistenceAndDistinctFailedTabs, testDeadlineReportsQuietEpisodeWithoutAnotherEvent, testStuckTimelinePreservesEventsAcrossContextResetAndExcludesRecovery, testStuckTimelineMarksOverflowOfTheLiveContinuation and CPMMessagingDiagnosticsTests.

Historical checks for the previous separate-pixel contract

The steps below are retained for the old implementation; their event names and immediate-send expectations do not apply to #7376.

Testing Steps

Run each case with fresh app data so daily pixel deduplication does not hide results. In Xcode’s console, filter by PixelKit Category.

Healthy navigation

1. Open a web page.
2. Wait at least 5 seconds → no debug_web_extension_cpm_* pixel is sent.
3. Open Debug → Web Extensions → Simulate CPM Messaging Breakage (On).
4. Open New Tab, Settings, Bookmarks (internal pages), validate no debug_web_extension_cpm_* pixel is sent

Failure, stuck, and recovery

1. Open Debug → Web Extensions → Simulate CPM Messaging Breakage (On).
2. Load a web page and wait 5 seconds → debug_web_extension_cpm_initialization_failed_after_other is sent; no stuck pixel is sent.
3. Navigate to another HTTP(S) page and wait 5 seconds → debug_web_extension_cpm_messaging_stuck_other is sent.
4. Select Simulate CPM Messaging Breakage (Off).
5. Reload the page → debug_web_extension_cpm_messaging_recovered_without_extension_reload is sent.
6. Navigate again → no additional CPM failure or recovery pixel is sent.

Session restoration

1. Open a web page, e.g. https://privacy-test-pages.site/features/autoconsent/banner.html
2. Duplicate the tab twice (3 tabs are open)
3. Open Settings → General, activate Reopen all windows and tabs from last session
4. Restart the browser
5. Open Debug → Web Extensions → Simulate CPM Messaging Breakage (On)
6. Switch to another tab
7. wait 5 seconds → debug_web_extension_cpm_initialization_failed_after_session_restoration is sent
8. Switch to the remaining restored tab
9. wait 5 seconds → debug_web_extension_cpm_initialization_failed_after_session_restoration is skipped (as it‘s daily): no debug_web_extension_cpm_messaging_stuck… is sent
10. Reload the tab or navigate somewhere else
11. wait 5 seconds → debug_web_extension_cpm_initialization_failed_after_other is sent; debug_web_extension_cpm_messaging_stuck_session_restoration is sent

Concurrent navigations

1. Select Simulate CPM Messaging Breakage (On).
2. Load HTTP(S) pages in two tabs before either 4-second timeout expires.
3. Wait 5 seconds → initialization failures are observed, but no debug_web_extension_cpm_messaging_stuck_* pixel is sent.
4. Start another navigation after those failures and wait 5 seconds → debug_web_extension_cpm_messaging_stuck_other is sent.

Extension reload

1. Start with Simulate CPM Messaging Breakage (Off) and use the Fire button to trigger extension reloading.
2. After data clearing completes, select Simulate CPM Messaging Breakage (On).
3. Load a web page and wait 5 seconds → these pixels are sent:
   • debug_web_extension_cpm_initialization_failed_after_extension_reload
   • debug_web_extension_cpm_messaging_extension_reload_failed
4. Navigate again and wait 5 seconds → debug_web_extension_cpm_messaging_stuck_extension_reload is sent.
5. Select Simulate CPM Messaging Breakage (Off) and reload → debug_web_extension_cpm_messaging_recovered_after_extension_reload is sent.