diff --git a/work/tech-design/coredata-code-defined-models.md b/work/tech-design/coredata-code-defined-models.md index 0f25f509..aba6f818 100644 --- a/work/tech-design/coredata-code-defined-models.md +++ b/work/tech-design/coredata-code-defined-models.md @@ -14,82 +14,66 @@ asana: TODO --- **Author:** Alexey Martemyanov **Reviewer:** Sam Symons -**Stakeholders:** Dominik Kapusta (Bookmarks, Sync), Federico Cappelli (Persistence, Remote Messaging), Michal Smaga (History) +**Stakeholders:** Bartek, Dominik, Federico Cappelli, Michal Smaga **Project:** Build the apps and packages outside Xcode (VS Code / `swift build`) --- ## Background & Requirements -The iOS and macOS browsers keep local data in Core Data: history, bookmarks, sync metadata, privacy stats, remote messages, downloads and a few smaller stores. Each store's schema lives in an Xcode model file (`.xcdatamodeld`). At build time Xcode compiles it into a binary model (`.momd`), and the app loads that at runtime. +We're moving development and CI to VS Code and SwiftPM builds that don't go through Xcode. -A model file holds every schema version that has shipped, not just the current one. When an app update changes the schema, Core Data opens the existing store, finds the old version among the compiled models, and upgrades the store in place ("lightweight migration"). +Core Data schemas live in `.xcdatamodeld` files. Only Xcode compiles them into the `.momd` the app loads, so SwiftPM builds and tests outside Xcode have no model. BSK tests work around this by running `momc` at runtime. -There are 10 production models; 7 of them have more than one version (History has 7, Bookmarks 6). Two stores, the main `Database` on iOS and macOS, each combine several models into one file. - -We are moving development and CI to builds that don't go through Xcode. Swift Package Manager doesn't compile model files outside Xcode. Today this is patched with a debug-only hack that runs Xcode's model compiler at runtime from inside tests. That only works on a machine with Xcode installed, and only on macOS. - -**Requirements** - -- Apps and packages build and test without Xcode's model compiler. -- On-disk data is unchanged: existing stores open as they do today. -- Users upgrading from **any** older app version keep their data. -- Schema changes are readable in code review. -- No change to minimum OS versions (iOS 15, macOS 12.3). +Requirements: +- existing user stores open unchanged +- upgrades from any older version keep data +- minimum OS stays at iOS 15 / macOS 12.3. ## Problem Statement -Core Data schemas are defined in Xcode-only model files, so any build that doesn't go through Xcode is missing them. This blocks moving packages and the apps off Xcode and keeps a fragile runtime workaround in place. +Core Data models can't be built outside Xcode. ## Recommended Approach -Define every schema in Swift, one file per model. Each file contains **all versions the model has ever shipped with**[1], and each version is written as a change to the previous one. A small helper in the `Persistence` package builds the model at runtime and, when a store was created by an older version, upgrades it with the same lightweight migration Core Data uses today[2]. +Define each model in Swift instead of `.xcdatamodeld`, one file per model, containing every shipped schema version. A small `Persistence` helper builds the model at runtime and migrates stores created by older versions.[1] -Safety rests on one property: Core Data decides whether a store matches a model by comparing per-entity **version hashes**. These are derived from the schema's structure, not from how the model was loaded. A code-defined version that is structurally identical to the shipped one opens existing stores untouched. Each version is: +Existing stores are unaffected: Core Data matches a store to a model by schema hashes, and the code-defined versions produce the same hashes as the compiled `.momd`. The hashes are pinned in tests. -- compared field by field against the compiled Xcode model before the model file is deleted, and -- pinned by its hashes in unit tests, so any accidental change to a shipped version fails CI. +Steps (one PR each): -Implementation, one PR per step. Steps 2–4 are stacked on step 1. +1. `Persistence` support + PrivacyStats and BrowsingHistory. *(PR ready.)* +2. Bookmarks + SyncMetadata. +3. Stores combining several models: HTTPSUpgrade, RemoteMessaging, AppRatingPrompt.[2] +4. macOS app models: History, Downloads, legacy Bookmark. +5. Remove the `momc` workaround. -1. **Foundation + pilots.** Schema description types and versioned-model support in `Persistence`, with migration integrated into `CoreDataDatabase`. Convert **PrivacyStats** (1 version) and **BrowsingHistory** (3 versions); each has its own store. *(PR ready for review.)* -2. **Bookmarks + SyncMetadata** (6 and 4 versions), including the custom bookmarks form-factor migration and the test database builders[3]. -3. **Combined stores.** Allow one store to be composed of several models, then convert **HTTPSUpgrade**, **RemoteMessaging** and **AppRatingPrompt** (the iOS main `Database`)[4]. -4. **macOS app models.** Convert **History** (7 versions), **Downloads**, legacy **Bookmark**, and the macOS main `Database`, including its encrypted value transformers. -5. **Cleanup.** Remove model loading from bundles and the runtime model-compiler workaround. - -Every step can be undone by reverting it: the on-disk format never changes, so a store written by a converted build still opens with the model-file build and vice versa. +Each step is revertible: the on-disk format doesn't change. ## Notes -[1] **Why keep old versions in code.** The source model for a migration used to come from the compiled model files. Without them there is nothing for Core Data to search. Core Data also keeps a copy of the model inside each SQLite store and can migrate from that; I confirmed this experimentally, but it is not a contract we should rely on. Keeping versions explicit makes migration deterministic, and it's needed for the custom bookmarks migration and for building test fixtures. The store-embedded copy stays as a fallback for stores that match none of the listed versions. +[1] Old versions are needed because migration looks for the old schema in the compiled models, which will no longer exist. `NSStagedMigrationManager` handles code-defined versions but requires iOS 17 / macOS 14. -[2] **Migration mechanics are unchanged.** We upgrade in one step from the store's version to the current one, using an inferred mapping, exactly as automatic migration does today. It runs only when a store is outdated, and writes to a copy that replaces the original only on success. `NSStagedMigrationManager`, Apple's built-in support for code-defined versions, requires iOS 17 / macOS 14, so it isn't an option. +[2] A combined store needs the old versions of all its models, so they're converted together. -[3] Bookmarks has a custom migration that inspects the store's current version. It moves from reading compiled model files to the in-code version list. +Alternatives: +- Remove only the generated classes: `momc` is still needed. +- Commit precompiled `.momd` files: binary artifacts that can drift from the source. -[4] Combined stores are why HTTPSUpgrade isn't in step 1. Migrating a combined store needs the old version of every model in it, so all its models have to be converted together. - -**Alternatives considered** - -- *Drop only the generated classes and keep the model files* — the model compiler is still required, so the problem isn't solved. -- *Commit precompiled `.momd` files* — binary artifacts in the repo that can silently drift from the source model files. -- *Keep only the current version* — works today only because of the store-embedded model copy [1]. It breaks the custom bookmarks migration and loses the ability to build old-version fixtures. - -**Trade-off:** we lose Xcode's visual model editor. In exchange, schema changes are plain Swift diffs. +Trade-off: we lose Xcode's visual model editor. ## Testing -- **Schema fidelity:** each shipped version's entity hashes are pinned in unit tests; they were captured from the Xcode-compiled models. -- **Migration:** existing fixture stores from old versions (e.g. BrowsingHistory v1) are migrated to the current version and their data verified. `Persistence` unit tests cover migration, no-op load of an up-to-date store, a new store, an unknown version, and cleanup of temporary files. -- **Outside Xcode:** affected package tests run with plain `swift test` (verified for step 1: 102 tests). -- **Manual, per PR:** install the current App Store build on iOS and macOS, create data, upgrade to the branch build, and confirm the data is intact. +- Unit tests pin the schema hashes of every shipped version. +- Migration tests upgrade old-version stores and check the data. +- Package tests run with plain `swift test`. +- Manual check on iOS and macOS: upgrade from the App Store build and confirm the data is intact. ## Additional Considerations -- **Privacy** — No change to what data is collected, used or retained. Same stores, same locations. -- **Security** — No new dependencies or entitlements. The encrypted value transformers on macOS keep their current registration, verified in step 4. -- **Site Breakage** — None; there are no web-facing changes. -- **Experimentation** — Not applicable. There's no user-visible change, and a store schema can't sit behind a remote feature flag because the model is needed before configuration loads. Rollback is a revert (see Recommended Approach). -- **Operational** — Removes Xcode's model compiler as a build requirement for packages; no CI or release changes are needed. Migration runs as rarely as today (only for outdated stores). Failures surface through the existing store-loading error handling and pixels. -- **Localization / Internationalization** — Not applicable. +- **Privacy** — No change. +- **Security** — No change. +- **Site Breakage** — N/A. +- **Experimentation** — N/A. The schema can't be feature-flagged; rollback is a revert. +- **Operational** — Package builds no longer need Xcode. Migration failures are reported through the existing store-loading errors. +- **Localization / Internationalization** — N/A.