From b819e3d23ddde6ab05b28974d608da3ccec16888 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Fri, 25 Sep 2026 17:57:08 +0600 Subject: [PATCH 1/5] [2026-09-25] eagle: work/tech-design/coredata-code-defined-models.md --- .../coredata-code-defined-models.md | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 work/tech-design/coredata-code-defined-models.md diff --git a/work/tech-design/coredata-code-defined-models.md b/work/tech-design/coredata-code-defined-models.md new file mode 100644 index 00000000..0f25f509 --- /dev/null +++ b/work/tech-design/coredata-code-defined-models.md @@ -0,0 +1,95 @@ +--- +title: 'Tech Design: Code-defined Core Data models' +created: '2026-09-25' +tags: + - tech-design + - apple + - core-data + - persistence + - spm + - build +status: draft +chosen-approach: 'Schemas defined in Swift with full version history; explicit lightweight migration' +asana: TODO +--- +**Author:** Alexey Martemyanov +**Reviewer:** Sam Symons +**Stakeholders:** Dominik Kapusta (Bookmarks, Sync), Federico Cappelli (Persistence, Remote Messaging), Michal Smaga (History) +**Project:** Build the apps and packages outside Xcode (VS Code / `swift build`) + +--- + +## Background & Requirements + +The iOS and macOS browsers keep local data in Core Data: history, bookmarks, sync metadata, privacy stats, remote messages, downloads and a few smaller stores. Each store's schema lives in an Xcode model file (`.xcdatamodeld`). At build time Xcode compiles it into a binary model (`.momd`), and the app loads that at runtime. + +A model file holds every schema version that has shipped, not just the current one. When an app update changes the schema, Core Data opens the existing store, finds the old version among the compiled models, and upgrades the store in place ("lightweight migration"). + +There are 10 production models; 7 of them have more than one version (History has 7, Bookmarks 6). Two stores, the main `Database` on iOS and macOS, each combine several models into one file. + +We are moving development and CI to builds that don't go through Xcode. Swift Package Manager doesn't compile model files outside Xcode. Today this is patched with a debug-only hack that runs Xcode's model compiler at runtime from inside tests. That only works on a machine with Xcode installed, and only on macOS. + +**Requirements** + +- Apps and packages build and test without Xcode's model compiler. +- On-disk data is unchanged: existing stores open as they do today. +- Users upgrading from **any** older app version keep their data. +- Schema changes are readable in code review. +- No change to minimum OS versions (iOS 15, macOS 12.3). + +## Problem Statement + +Core Data schemas are defined in Xcode-only model files, so any build that doesn't go through Xcode is missing them. This blocks moving packages and the apps off Xcode and keeps a fragile runtime workaround in place. + +## Recommended Approach + +Define every schema in Swift, one file per model. Each file contains **all versions the model has ever shipped with**[1], and each version is written as a change to the previous one. A small helper in the `Persistence` package builds the model at runtime and, when a store was created by an older version, upgrades it with the same lightweight migration Core Data uses today[2]. + +Safety rests on one property: Core Data decides whether a store matches a model by comparing per-entity **version hashes**. These are derived from the schema's structure, not from how the model was loaded. A code-defined version that is structurally identical to the shipped one opens existing stores untouched. Each version is: + +- compared field by field against the compiled Xcode model before the model file is deleted, and +- pinned by its hashes in unit tests, so any accidental change to a shipped version fails CI. + +Implementation, one PR per step. Steps 2–4 are stacked on step 1. + +1. **Foundation + pilots.** Schema description types and versioned-model support in `Persistence`, with migration integrated into `CoreDataDatabase`. Convert **PrivacyStats** (1 version) and **BrowsingHistory** (3 versions); each has its own store. *(PR ready for review.)* +2. **Bookmarks + SyncMetadata** (6 and 4 versions), including the custom bookmarks form-factor migration and the test database builders[3]. +3. **Combined stores.** Allow one store to be composed of several models, then convert **HTTPSUpgrade**, **RemoteMessaging** and **AppRatingPrompt** (the iOS main `Database`)[4]. +4. **macOS app models.** Convert **History** (7 versions), **Downloads**, legacy **Bookmark**, and the macOS main `Database`, including its encrypted value transformers. +5. **Cleanup.** Remove model loading from bundles and the runtime model-compiler workaround. + +Every step can be undone by reverting it: the on-disk format never changes, so a store written by a converted build still opens with the model-file build and vice versa. + +## Notes + +[1] **Why keep old versions in code.** The source model for a migration used to come from the compiled model files. Without them there is nothing for Core Data to search. Core Data also keeps a copy of the model inside each SQLite store and can migrate from that; I confirmed this experimentally, but it is not a contract we should rely on. Keeping versions explicit makes migration deterministic, and it's needed for the custom bookmarks migration and for building test fixtures. The store-embedded copy stays as a fallback for stores that match none of the listed versions. + +[2] **Migration mechanics are unchanged.** We upgrade in one step from the store's version to the current one, using an inferred mapping, exactly as automatic migration does today. It runs only when a store is outdated, and writes to a copy that replaces the original only on success. `NSStagedMigrationManager`, Apple's built-in support for code-defined versions, requires iOS 17 / macOS 14, so it isn't an option. + +[3] Bookmarks has a custom migration that inspects the store's current version. It moves from reading compiled model files to the in-code version list. + +[4] Combined stores are why HTTPSUpgrade isn't in step 1. Migrating a combined store needs the old version of every model in it, so all its models have to be converted together. + +**Alternatives considered** + +- *Drop only the generated classes and keep the model files* — the model compiler is still required, so the problem isn't solved. +- *Commit precompiled `.momd` files* — binary artifacts in the repo that can silently drift from the source model files. +- *Keep only the current version* — works today only because of the store-embedded model copy [1]. It breaks the custom bookmarks migration and loses the ability to build old-version fixtures. + +**Trade-off:** we lose Xcode's visual model editor. In exchange, schema changes are plain Swift diffs. + +## Testing + +- **Schema fidelity:** each shipped version's entity hashes are pinned in unit tests; they were captured from the Xcode-compiled models. +- **Migration:** existing fixture stores from old versions (e.g. BrowsingHistory v1) are migrated to the current version and their data verified. `Persistence` unit tests cover migration, no-op load of an up-to-date store, a new store, an unknown version, and cleanup of temporary files. +- **Outside Xcode:** affected package tests run with plain `swift test` (verified for step 1: 102 tests). +- **Manual, per PR:** install the current App Store build on iOS and macOS, create data, upgrade to the branch build, and confirm the data is intact. + +## Additional Considerations + +- **Privacy** — No change to what data is collected, used or retained. Same stores, same locations. +- **Security** — No new dependencies or entitlements. The encrypted value transformers on macOS keep their current registration, verified in step 4. +- **Site Breakage** — None; there are no web-facing changes. +- **Experimentation** — Not applicable. There's no user-visible change, and a store schema can't sit behind a remote feature flag because the model is needed before configuration loads. Rollback is a revert (see Recommended Approach). +- **Operational** — Removes Xcode's model compiler as a build requirement for packages; no CI or release changes are needed. Migration runs as rarely as today (only for outdated stores). Failures surface through the existing store-loading error handling and pixels. +- **Localization / Internationalization** — Not applicable. From a6baec7ead107abc4a770eb48a18d6044b6c86a2 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Fri, 25 Sep 2026 18:32:44 +0600 Subject: [PATCH 2/5] [2026-09-25] eagle: work/tech-design/coredata-code-defined-models.md --- .../coredata-code-defined-models.md | 82 ++++++++----------- 1 file changed, 33 insertions(+), 49 deletions(-) diff --git a/work/tech-design/coredata-code-defined-models.md b/work/tech-design/coredata-code-defined-models.md index 0f25f509..aba6f818 100644 --- a/work/tech-design/coredata-code-defined-models.md +++ b/work/tech-design/coredata-code-defined-models.md @@ -14,82 +14,66 @@ asana: TODO --- **Author:** Alexey Martemyanov **Reviewer:** Sam Symons -**Stakeholders:** Dominik Kapusta (Bookmarks, Sync), Federico Cappelli (Persistence, Remote Messaging), Michal Smaga (History) +**Stakeholders:** Bartek, Dominik, Federico Cappelli, Michal Smaga **Project:** Build the apps and packages outside Xcode (VS Code / `swift build`) --- ## Background & Requirements -The iOS and macOS browsers keep local data in Core Data: history, bookmarks, sync metadata, privacy stats, remote messages, downloads and a few smaller stores. Each store's schema lives in an Xcode model file (`.xcdatamodeld`). At build time Xcode compiles it into a binary model (`.momd`), and the app loads that at runtime. +We're moving development and CI to VS Code and SwiftPM builds that don't go through Xcode. -A model file holds every schema version that has shipped, not just the current one. When an app update changes the schema, Core Data opens the existing store, finds the old version among the compiled models, and upgrades the store in place ("lightweight migration"). +Core Data schemas live in `.xcdatamodeld` files. Only Xcode compiles them into the `.momd` the app loads, so SwiftPM builds and tests outside Xcode have no model. BSK tests work around this by running `momc` at runtime. -There are 10 production models; 7 of them have more than one version (History has 7, Bookmarks 6). Two stores, the main `Database` on iOS and macOS, each combine several models into one file. - -We are moving development and CI to builds that don't go through Xcode. Swift Package Manager doesn't compile model files outside Xcode. Today this is patched with a debug-only hack that runs Xcode's model compiler at runtime from inside tests. That only works on a machine with Xcode installed, and only on macOS. - -**Requirements** - -- Apps and packages build and test without Xcode's model compiler. -- On-disk data is unchanged: existing stores open as they do today. -- Users upgrading from **any** older app version keep their data. -- Schema changes are readable in code review. -- No change to minimum OS versions (iOS 15, macOS 12.3). +Requirements: +- existing user stores open unchanged +- upgrades from any older version keep data +- minimum OS stays at iOS 15 / macOS 12.3. ## Problem Statement -Core Data schemas are defined in Xcode-only model files, so any build that doesn't go through Xcode is missing them. This blocks moving packages and the apps off Xcode and keeps a fragile runtime workaround in place. +Core Data models can't be built outside Xcode. ## Recommended Approach -Define every schema in Swift, one file per model. Each file contains **all versions the model has ever shipped with**[1], and each version is written as a change to the previous one. A small helper in the `Persistence` package builds the model at runtime and, when a store was created by an older version, upgrades it with the same lightweight migration Core Data uses today[2]. +Define each model in Swift instead of `.xcdatamodeld`, one file per model, containing every shipped schema version. A small `Persistence` helper builds the model at runtime and migrates stores created by older versions.[1] -Safety rests on one property: Core Data decides whether a store matches a model by comparing per-entity **version hashes**. These are derived from the schema's structure, not from how the model was loaded. A code-defined version that is structurally identical to the shipped one opens existing stores untouched. Each version is: +Existing stores are unaffected: Core Data matches a store to a model by schema hashes, and the code-defined versions produce the same hashes as the compiled `.momd`. The hashes are pinned in tests. -- compared field by field against the compiled Xcode model before the model file is deleted, and -- pinned by its hashes in unit tests, so any accidental change to a shipped version fails CI. +Steps (one PR each): -Implementation, one PR per step. Steps 2–4 are stacked on step 1. +1. `Persistence` support + PrivacyStats and BrowsingHistory. *(PR ready.)* +2. Bookmarks + SyncMetadata. +3. Stores combining several models: HTTPSUpgrade, RemoteMessaging, AppRatingPrompt.[2] +4. macOS app models: History, Downloads, legacy Bookmark. +5. Remove the `momc` workaround. -1. **Foundation + pilots.** Schema description types and versioned-model support in `Persistence`, with migration integrated into `CoreDataDatabase`. Convert **PrivacyStats** (1 version) and **BrowsingHistory** (3 versions); each has its own store. *(PR ready for review.)* -2. **Bookmarks + SyncMetadata** (6 and 4 versions), including the custom bookmarks form-factor migration and the test database builders[3]. -3. **Combined stores.** Allow one store to be composed of several models, then convert **HTTPSUpgrade**, **RemoteMessaging** and **AppRatingPrompt** (the iOS main `Database`)[4]. -4. **macOS app models.** Convert **History** (7 versions), **Downloads**, legacy **Bookmark**, and the macOS main `Database`, including its encrypted value transformers. -5. **Cleanup.** Remove model loading from bundles and the runtime model-compiler workaround. - -Every step can be undone by reverting it: the on-disk format never changes, so a store written by a converted build still opens with the model-file build and vice versa. +Each step is revertible: the on-disk format doesn't change. ## Notes -[1] **Why keep old versions in code.** The source model for a migration used to come from the compiled model files. Without them there is nothing for Core Data to search. Core Data also keeps a copy of the model inside each SQLite store and can migrate from that; I confirmed this experimentally, but it is not a contract we should rely on. Keeping versions explicit makes migration deterministic, and it's needed for the custom bookmarks migration and for building test fixtures. The store-embedded copy stays as a fallback for stores that match none of the listed versions. +[1] Old versions are needed because migration looks for the old schema in the compiled models, which will no longer exist. `NSStagedMigrationManager` handles code-defined versions but requires iOS 17 / macOS 14. -[2] **Migration mechanics are unchanged.** We upgrade in one step from the store's version to the current one, using an inferred mapping, exactly as automatic migration does today. It runs only when a store is outdated, and writes to a copy that replaces the original only on success. `NSStagedMigrationManager`, Apple's built-in support for code-defined versions, requires iOS 17 / macOS 14, so it isn't an option. +[2] A combined store needs the old versions of all its models, so they're converted together. -[3] Bookmarks has a custom migration that inspects the store's current version. It moves from reading compiled model files to the in-code version list. +Alternatives: +- Remove only the generated classes: `momc` is still needed. +- Commit precompiled `.momd` files: binary artifacts that can drift from the source. -[4] Combined stores are why HTTPSUpgrade isn't in step 1. Migrating a combined store needs the old version of every model in it, so all its models have to be converted together. - -**Alternatives considered** - -- *Drop only the generated classes and keep the model files* — the model compiler is still required, so the problem isn't solved. -- *Commit precompiled `.momd` files* — binary artifacts in the repo that can silently drift from the source model files. -- *Keep only the current version* — works today only because of the store-embedded model copy [1]. It breaks the custom bookmarks migration and loses the ability to build old-version fixtures. - -**Trade-off:** we lose Xcode's visual model editor. In exchange, schema changes are plain Swift diffs. +Trade-off: we lose Xcode's visual model editor. ## Testing -- **Schema fidelity:** each shipped version's entity hashes are pinned in unit tests; they were captured from the Xcode-compiled models. -- **Migration:** existing fixture stores from old versions (e.g. BrowsingHistory v1) are migrated to the current version and their data verified. `Persistence` unit tests cover migration, no-op load of an up-to-date store, a new store, an unknown version, and cleanup of temporary files. -- **Outside Xcode:** affected package tests run with plain `swift test` (verified for step 1: 102 tests). -- **Manual, per PR:** install the current App Store build on iOS and macOS, create data, upgrade to the branch build, and confirm the data is intact. +- Unit tests pin the schema hashes of every shipped version. +- Migration tests upgrade old-version stores and check the data. +- Package tests run with plain `swift test`. +- Manual check on iOS and macOS: upgrade from the App Store build and confirm the data is intact. ## Additional Considerations -- **Privacy** — No change to what data is collected, used or retained. Same stores, same locations. -- **Security** — No new dependencies or entitlements. The encrypted value transformers on macOS keep their current registration, verified in step 4. -- **Site Breakage** — None; there are no web-facing changes. -- **Experimentation** — Not applicable. There's no user-visible change, and a store schema can't sit behind a remote feature flag because the model is needed before configuration loads. Rollback is a revert (see Recommended Approach). -- **Operational** — Removes Xcode's model compiler as a build requirement for packages; no CI or release changes are needed. Migration runs as rarely as today (only for outdated stores). Failures surface through the existing store-loading error handling and pixels. -- **Localization / Internationalization** — Not applicable. +- **Privacy** — No change. +- **Security** — No change. +- **Site Breakage** — N/A. +- **Experimentation** — N/A. The schema can't be feature-flagged; rollback is a revert. +- **Operational** — Package builds no longer need Xcode. Migration failures are reported through the existing store-loading errors. +- **Localization / Internationalization** — N/A. From 470a8a8208a64953a7cea4bdb794f4b48e5d56bf Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Fri, 25 Sep 2026 18:37:47 +0600 Subject: [PATCH 3/5] [2026-09-25] eagle: work/tech-design/coredata-code-defined-models.md --- work/tech-design/coredata-code-defined-models.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/work/tech-design/coredata-code-defined-models.md b/work/tech-design/coredata-code-defined-models.md index aba6f818..2377bd9d 100644 --- a/work/tech-design/coredata-code-defined-models.md +++ b/work/tech-design/coredata-code-defined-models.md @@ -42,7 +42,7 @@ Existing stores are unaffected: Core Data matches a store to a model by schema h Steps (one PR each): -1. `Persistence` support + PrivacyStats and BrowsingHistory. *(PR ready.)* +1. `Persistence` support + PrivacyStats and BrowsingHistory. 2. Bookmarks + SyncMetadata. 3. Stores combining several models: HTTPSUpgrade, RemoteMessaging, AppRatingPrompt.[2] 4. macOS app models: History, Downloads, legacy Bookmark. @@ -57,10 +57,10 @@ Each step is revertible: the on-disk format doesn't change. [2] A combined store needs the old versions of all its models, so they're converted together. Alternatives: -- Remove only the generated classes: `momc` is still needed. +- Remove only the generated dynamic CoreData property accessors: `momc` is still needed and compiled as a separate Swift Build System Plugin step. - Commit precompiled `.momd` files: binary artifacts that can drift from the source. -Trade-off: we lose Xcode's visual model editor. +Trade-off: we lose Xcode's visual model editor (but we get standardized, structured, great&shiny code definitions!) ## Testing @@ -75,5 +75,5 @@ Trade-off: we lose Xcode's visual model editor. - **Security** — No change. - **Site Breakage** — N/A. - **Experimentation** — N/A. The schema can't be feature-flagged; rollback is a revert. -- **Operational** — Package builds no longer need Xcode. Migration failures are reported through the existing store-loading errors. +- **Operational** — Package builds no longer need Xcode‘s momd compiler. Migration failures are reported through the existing store-loading errors. - **Localization / Internationalization** — N/A. From 3e9a6f281ebc74f1a4e5e872d8edec225a4d56c7 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Fri, 25 Sep 2026 19:13:22 +0600 Subject: [PATCH 4/5] [2026-09-25] eagle: work/projects/cpm-web-extension-breakage-validation.md --- work/projects/cpm-web-extension-breakage-validation.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/work/projects/cpm-web-extension-breakage-validation.md b/work/projects/cpm-web-extension-breakage-validation.md index 0e2d54eb..d45e9ade 100644 --- a/work/projects/cpm-web-extension-breakage-validation.md +++ b/work/projects/cpm-web-extension-breakage-validation.md @@ -37,7 +37,7 @@ M4 перезагружает Web Extensions при первом сигнале | M3 attribution telemetry | `alex/cpm-failure-telemetry` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-failure-telemetry` | **Merged 2026-09-23:** [PR #6844](https://github.com/duckduckgo/apple-browsers/pull/6844), CI green. Bucketed diagnostics, app launch/version attribution, `extension_updated`, `base_url_changed`, `recovery_from`, reload-error pixel; recorder и proxy имеют отдельные flags. [[cpm-m3-telemetry-privacy-triage]]. | | M4 hung recovery | `codex/cpm-hung-recovery` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-hung-recovery` | **In progress:** первый `messaging init failure` открывает diagnostic episode; первый последующий `hung` перезагружает Web Extensions. | | M5+M6 graveyard mitigation + A/B | `alex/cpm-graveyard-mitigation` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-graveyard-mitigation` | **Production-ready, default off; one PR over merged M3:** one retained view maximum, 3-second bounded hold, weak scheduled release, runtime cleanup, lazy stable cohort enrollment, control/treatment isolation, bounded pending outcomes, operational and experiment pixels. 431/431 WebExtensions tests; iOS/macOS builds and pixel validators pass; earlier real forced-kill iOS E2E recovered with release at ≈3.15 s. | -| M7 state-restoration readiness gate | `alex/cpm-state-restoration-race` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-state-restoration-race` | **Production-ready locally from `origin/main` (`a18fca47c3`):** restored main-frame HTTP(S) navigation now waits for initial Web Extension `loadAndSync`; Swift package and full `macOS Browser` builds pass. Targeted tests pass: 19/19 coordinator suite (including 3 new M7 cases) and 3/3 navigation-gate tests. | +| M7 state-restoration readiness gate | `alex/cpm-state-restoration-race` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-state-restoration-race` | **Production-ready locally from current `origin/main` (`b93091e769`):** restored main-frame HTTP(S) navigation waits for initial Web Extension `loadAndSync`, bounded by `withTimeout(3)` in `decidePolicy`. Timeout cancels only the cancellation-aware waiter; the underlying load continues and its retained task is cleared on completion/cancellation. Swift package and `macOS Browser` test builds pass. Targeted tests: 20/20 coordinator and 4/4 navigation-gate, including a non-cooperative load hang, waiter timeout, and late load completion. | All milestone diffs pass `git diff --check`; changed Swift lines are within 150 columns. @@ -45,11 +45,11 @@ All milestone diffs pass `git diff --check`; changed Swift lines are within 150 **Confirmed race:** `setupWebExtensions()` creates the manager/controller before state restoration, but starts `WebExtensionLifecycleCoordinator.loadAndSync()` without awaiting it. `stateRestorationManager.applicationDidFinishLaunching()` runs immediately afterward. A restored tab can therefore begin its main-frame navigation while the CPM `WKWebExtensionContext` is not loaded, so WebKit cannot inject the CPM content script at document start. The existing Content Blocking navigation responder already waits for internal user scripts and rule lists; M7 adds the equivalent readiness boundary for Web Extensions. -1. Add an awaitable initial-load readiness API to `WebExtensionLifecycleCoordinator`; it must reuse the existing coalesced `loadAndSync` task and complete on success, empty/disabled embedded set, or cancellation. -2. Inject a narrow readiness waiter into tab extensions and await it from the navigation policy path before main-frame HTTP(S) navigation proceeds. Do not poll `isAutoconsentExtensionLoaded`, because a disabled/failed CPM extension must not deadlock navigation. +1. Retain only the first coalesced `loadAndSync` task in `WebExtensionLifecycleCoordinator`, clear it on completion/cancellation, and expose a cancellation-aware waiter that never cancels the underlying load. +2. Inject that waiter into tab extensions. In the `decidePolicy` path, wrap it in the project `withTimeout(3)` helper before allowing main-frame HTTP(S) navigation to proceed. Timeout is fail-open and cancels only the waiter. 3. Keep the wait startup-only: once initial `loadAndSync` finishes, later navigations pass immediately. Non-web and subframe navigations do not wait. -4. Cover coordinator blocking/release/cancellation and navigation-gate scope with unit tests; run targeted WebExtensions/macOS tests after explicit approval. -5. Acceptance: a restored page cannot start before the initial extension load/sync settles; disabled Web Extensions and cancelled startup work never hold a tab; normal post-launch navigation has no added asynchronous work. +4. Cover blocking/release/cancellation, a non-cooperative underlying load hang, late load completion, the three-second default, and navigation scope with unit tests. +5. Acceptance: restored page loading normally waits for extension context readiness; a hung load cannot hold `decidePolicy` beyond three seconds; disabled/cancelled startup work and normal post-launch navigation do not add a persistent wait. ## Stack and Git state From a58f0b395a8736dd0cb844fb1345fc1edc82917b Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Fri, 25 Sep 2026 19:49:08 +0600 Subject: [PATCH 5/5] [2026-09-25] eagle: work/projects/cpm-web-extension-breakage-validation.md --- .../cpm-web-extension-breakage-validation.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/work/projects/cpm-web-extension-breakage-validation.md b/work/projects/cpm-web-extension-breakage-validation.md index d45e9ade..64ebbc87 100644 --- a/work/projects/cpm-web-extension-breakage-validation.md +++ b/work/projects/cpm-web-extension-breakage-validation.md @@ -37,7 +37,7 @@ M4 перезагружает Web Extensions при первом сигнале | M3 attribution telemetry | `alex/cpm-failure-telemetry` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-failure-telemetry` | **Merged 2026-09-23:** [PR #6844](https://github.com/duckduckgo/apple-browsers/pull/6844), CI green. Bucketed diagnostics, app launch/version attribution, `extension_updated`, `base_url_changed`, `recovery_from`, reload-error pixel; recorder и proxy имеют отдельные flags. [[cpm-m3-telemetry-privacy-triage]]. | | M4 hung recovery | `codex/cpm-hung-recovery` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-hung-recovery` | **In progress:** первый `messaging init failure` открывает diagnostic episode; первый последующий `hung` перезагружает Web Extensions. | | M5+M6 graveyard mitigation + A/B | `alex/cpm-graveyard-mitigation` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-graveyard-mitigation` | **Production-ready, default off; one PR over merged M3:** one retained view maximum, 3-second bounded hold, weak scheduled release, runtime cleanup, lazy stable cohort enrollment, control/treatment isolation, bounded pending outcomes, operational and experiment pixels. 431/431 WebExtensions tests; iOS/macOS builds and pixel validators pass; earlier real forced-kill iOS E2E recovered with release at ≈3.15 s. | -| M7 state-restoration readiness gate | `alex/cpm-state-restoration-race` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-state-restoration-race` | **Production-ready locally from current `origin/main` (`b93091e769`):** restored main-frame HTTP(S) navigation waits for initial Web Extension `loadAndSync`, bounded by `withTimeout(3)` in `decidePolicy`. Timeout cancels only the cancellation-aware waiter; the underlying load continues and its retained task is cleared on completion/cancellation. Swift package and `macOS Browser` test builds pass. Targeted tests: 20/20 coordinator and 4/4 navigation-gate, including a non-cooperative load hang, waiter timeout, and late load completion. | +| M7 state-restoration readiness gate | `alex/cpm-state-restoration-race` — `/Users/admin/DuckDuckGo/apple-browsers.git/cpm-state-restoration-race` | **Implemented on macOS and iOS from current `origin/main` (`b93091e769`):** initial main-frame HTTP(S) navigation waits for Web Extension `loadAndSync`, bounded by shared `withTimeout(3)`. Timeout stops only the cancellation-aware waiter; the underlying load continues and its retained task is cleared on completion/cancellation. WebExtensions lifecycle suite: 20/20. iOS Swift compilation reached the app target successfully; the generic-simulator build then stopped in existing environment-dependent scripts because `TARGET_DEVICE_OS_VERSION` and a signing identity were unavailable. The 100-cycle macOS app-restart/session-restore UI regression was imported from the WIP worktree; its run was skipped by request. SwiftLint and diff/project-file checks pass. | All milestone diffs pass `git diff --check`; changed Swift lines are within 150 columns. @@ -45,11 +45,13 @@ All milestone diffs pass `git diff --check`; changed Swift lines are within 150 **Confirmed race:** `setupWebExtensions()` creates the manager/controller before state restoration, but starts `WebExtensionLifecycleCoordinator.loadAndSync()` without awaiting it. `stateRestorationManager.applicationDidFinishLaunching()` runs immediately afterward. A restored tab can therefore begin its main-frame navigation while the CPM `WKWebExtensionContext` is not loaded, so WebKit cannot inject the CPM content script at document start. The existing Content Blocking navigation responder already waits for internal user scripts and rule lists; M7 adds the equivalent readiness boundary for Web Extensions. -1. Retain only the first coalesced `loadAndSync` task in `WebExtensionLifecycleCoordinator`, clear it on completion/cancellation, and expose a cancellation-aware waiter that never cancels the underlying load. -2. Inject that waiter into tab extensions. In the `decidePolicy` path, wrap it in the project `withTimeout(3)` helper before allowing main-frame HTTP(S) navigation to proceed. Timeout is fail-open and cancels only the waiter. -3. Keep the wait startup-only: once initial `loadAndSync` finishes, later navigations pass immediately. Non-web and subframe navigations do not wait. -4. Cover blocking/release/cancellation, a non-cooperative underlying load hang, late load completion, the three-second default, and navigation scope with unit tests. -5. Acceptance: restored page loading normally waits for extension context readiness; a hung load cannot hold `decidePolicy` beyond three seconds; disabled/cancelled startup work and normal post-launch navigation do not add a persistent wait. +1. Retain only the first coalesced `loadAndSync` task in `WebExtensionLifecycleCoordinator`, clear it on completion/cancellation, and expose a cancellation-aware waiter that never cancels the underlying load. **Done.** +2. Inject the waiter into both macOS `PrivacyDashboardTabExtension` and iOS `TabViewController`. Gate initial main-frame HTTP(S) navigation with the shared `WebExtensionNavigationGate`, which uses project `withTimeout(3)`. **Done.** +3. Keep the wait startup-only: after initial `loadAndSync`, the provider returns `nil`, so later navigations avoid an async hop. Non-web and subframe navigations do not wait. **Done.** +4. Cover blocking/release/cancellation, timeout without cancellation of the underlying load, late completion, default timeout, and navigation scope. WebExtensions lifecycle suite is 20/20; SwiftLint and structural checks pass. **Done.** +5. Add the existing WIP UI regression that restarts the macOS app 100 times, restores the Autoconsent page, and verifies CPM injection after every restart. **Included; execution intentionally skipped by request.** + +Technical decision: awaiting `initialLoadAndSync.value` directly inside structured `withTimeout` was hang-tested and rejected. The timeout cancels its child operation, but a child suspended on `Task.value` does not finish from that cancellation, so the task group cannot return. The small cancellation-aware waiter gives `withTimeout` a cancellable wait without cancelling the retained load task. ## Stack and Git state