132 lines
11 KiB
Markdown
132 lines
11 KiB
Markdown
# Tech Design: VS Code and Cursor on the Xcode projects through a build server
|
||
|
||
**Author:** Alex M
|
||
**Reviewer:** TBD
|
||
**Stakeholders:** macOS and iOS teams
|
||
**Project:** [Migration to fully VSCode-supported builds&debugging](https://app.asana.com/1/137249556945/project/1203301625297703/task/1209808486380252) / https://app.asana.com/1/137249556945/project/1203301625297703/task/1219079700626622?focus=true
|
||
|
||
## Background & Requirements
|
||
|
||
- Developers increasingly work in VS Code and Cursor (AI agents, multiple worktrees), but code intelligence for our apps only exists in Xcode.
|
||
- The Swift extension's language server, sourcekit-lsp, understands SwiftPM packages natively. For other build systems it talks to a *build server* through the Build Server Protocol (BSP), configured by a `buildServer.json` in the folder.
|
||
- The current plan makes the macOS browser code a SwiftPM package ([[projects/vscode-migration/tech-design-spm-browser-package|tech design]]). That is a big change to the project, and macOS only.
|
||
- [sourcekit-xcode-bsp](https://github.com/slime-studio/sourcekit-xcode-bsp) (Apache 2.0, early-stage, version 0.1.0) is a build server for `.xcodeproj`/`.xcworkspace`. It uses swift-build, the open-source build engine of Xcode, so it reads the projects the way Xcode does.
|
||
- **Requirements:**
|
||
- Code intelligence (completion, diagnostics, jump to definition, references) for app, test and package code, macOS and iOS.
|
||
- Build, run, test and debug from the editor, including the iOS simulator.
|
||
- No change to the Xcode projects, schemes, CI or release.
|
||
- Developers who also use Xcode don't pay twice: builds and the index are shared with Xcode.
|
||
- Works in many worktrees without hours of indexing per worktree.
|
||
|
||
## Problem Statement
|
||
|
||
Out of the box, sourcekit-xcode-bsp 0.1.0 on our workspace:
|
||
- gives app code intelligence after ~3 minutes;
|
||
- gives **nothing in test files**;
|
||
- produces build failures and drops ~250 object files into the repository root;
|
||
- needs an estimated **2 hours** to index a worktree;
|
||
- keeps its own build and index, separate from Xcode's.
|
||
|
||
## Recommended Approach
|
||
|
||
Use sourcekit-xcode-bsp with four changes to it, plus a small setup in the repository.
|
||
|
||
**Changes to sourcekit-xcode-bsp** (`scripts/vscode/sourcekit-xcode-bsp.patch`, to be upstreamed)
|
||
1. **`configuration`, default `Debug`.** Without it swift-build uses the projects' default configuration, Release, which has no testability. Every `@testable import` failed, so test files had no results.
|
||
2. **`targets` include/exclude patterns** on `<project>/<target>`, plus a `list-targets` command to check them. The server put all 977 workspace targets into one build: DMG and App Store variants, both platforms, and 3–4 variants of each package library. They build into the same paths, so swift-build dropped tasks. Xcode never builds them together; a scheme picks one variant and package targets come in as dependencies. We configure the same way: 86 targets, which pull in 288.
|
||
3. **A per-target cache of Swift compiler arguments.** swift-build computes them for every file from scratch (~0.8 s each, one at a time). Within a Swift target they differ only in the file's `-index-unit-output-path`, which the server already reports. The first request of a target is computed, and concurrent requests wait for it. Indexing a fresh worktree went from ~2 hours to **~5.5 minutes** on an M4 Max.
|
||
4. **`derivedDataLayout: xcode`.** The server works in the worktree's Xcode DerivedData folder with Xcode's layout:
|
||
- the index store is Xcode's;
|
||
- index records name the regular build's object files, so a file indexed or built by Xcode or `xcodebuild` is up to date for the editor, and the other way round;
|
||
- preparation builds go into a folder of their own, so the server never shares a build database with an open Xcode.
|
||
|
||
Also added: `buildSettings` overrides.
|
||
|
||
**In the repository** (`DuckDuckGo.code-workspace`, `scripts/vscode/`)
|
||
- **Language server config.** `setup-build-server.py [macos|ios]` writes `buildServer.json` on folder open. It sets Debug, one platform (kept until switched) and the target filter. The filter keeps the app project and package test products, and drops App Store variants and the other platform's app and local packages.
|
||
- **Builds.** `xcodebuild.sh` builds into Xcode's DerivedData folder with Xcode's settings, so Xcode and VS Code don't rebuild after each other. It also feeds the shared index. Tasks:
|
||
- build a macOS scheme;
|
||
- run unit tests with `-only-testing`; failures show in the Problems panel;
|
||
- build and run the iOS app on a simulator.
|
||
- **Debugging** uses lldb-dap.
|
||
- macOS: launch, with or without building first.
|
||
- iOS simulator: `simulator.sh` builds for the chosen simulator, installs the app and launches it waiting for the debugger. The launch configuration then attaches by pid; the process picker lists newest first.
|
||
- **Opt-in cache across worktrees.** `DDG_WORKTREE_CACHE=1` builds in the worktree instead. The worktree and build folder are mapped to `/^wt` and `/^dd`, so Xcode's compilation cache is shared between worktrees: a clean build in a new worktree takes 90–120 s instead of 179 s. Launch configurations map the paths back for the debugger. These builds aren't shared with Xcode.
|
||
- **Background indexing.** `index-in-background.py` indexes worktrees headless while no editor uses them; an editor that opens stops it. `background-indexing.sh install` runs it every 30 minutes while Xcode, VS Code and Cursor are closed (opt-in).
|
||
- **Install.** `install-build-server.sh` builds the patched server from a pinned upstream commit, with pinned dependencies, into `~/.local/opt`. The setup task warns when the patch changes.
|
||
|
||
**Steps**
|
||
1. Land the setup and the patch in the repository. Xcode users aren't affected.
|
||
2. Early adopters on macOS and iOS; then a short how-to in the team docs.
|
||
3. Open upstream PRs for the server changes. Drop the patch when a release has them, or move it to a fork under the DuckDuckGo organization.
|
||
|
||
### Pros
|
||
|
||
- **No project changes.** The Xcode projects, schemes and CI stay the source of truth; nothing to keep in sync.
|
||
- **Same build settings as Xcode.** swift-build plans the build, so flags, macros, plugins, resources and per-configuration settings match Xcode. Hand-written SwiftPM manifests can drift from them.
|
||
- **Works together with Xcode.** One build and one index per worktree, whichever tool made them.
|
||
- **Both platforms and every target type:** apps, extensions, app tests, package tests.
|
||
- **Fast enough for many worktrees.**
|
||
- ~5.5 min first index, less after a build.
|
||
- Background indexing.
|
||
- Optional shared compile cache across worktrees.
|
||
- Complements the SwiftPM package work rather than replacing it: modularization still helps build times and testing.
|
||
|
||
### Cons
|
||
|
||
- **A young third-party tool** (0.1.0, "early-stage" by its authors) and our patch on top until upstream merges it.
|
||
- **One platform per worktree at a time;** switching runs the setup again and reindexes the other platform's files.
|
||
- **Files outside the configured targets have no results,** for example App Store-only files (none today) or packages only the other platform uses.
|
||
- **iOS debugging takes two steps:** run the task, then attach. VS Code resolves the process picker before a pre-launch task, so the two can't be one action. Attaching with a launch-time command doesn't stop at breakpoints with lldb-dap today.
|
||
- **The cross-worktree cache and sharing with Xcode exclude each other.** Prefix-mapped paths in debug info need a source map, which we can only set up in VS Code.
|
||
- **xcodebuild builds take as long as Xcode's.** Don't build one worktree from both at the same time.
|
||
|
||
### Risks
|
||
|
||
- **swift-build changes with Xcode.** The server bundles its own swift-build, pinned by upstream, while the compiler comes from the selected Xcode. Mitigation: pinned commit and dependencies in `install-build-server.sh`; check after each Xcode update.
|
||
- **Upstream doesn't take the patches** or the project stalls. Mitigation: a fork under the DuckDuckGo organization. The patch is ~550 lines of code plus ~450 lines of tests.
|
||
- **Shared index store with Xcode.** Xcode and sourcekit-lsp both write records into it; the index store format is designed for several writers, but we haven't seen it under load. Mitigation: the server never shares Xcode's build database, and deleting the DerivedData folder resets both.
|
||
- **Background indexing costs CPU and disk.** Mitigation: it's opt-in, runs only while editors are closed, at low priority, and skips worktrees that haven't changed.
|
||
|
||
## Notes
|
||
|
||
- [1] Measured on an M4 Max with Xcode 27.1 on `main` at 3d850d74dc.
|
||
- Fresh-worktree index: 323–347 s at low load. That is ~1 min to load the workspace and resolve packages, then ~3.5 min to prepare and index, running up to 11 compilers in parallel.
|
||
- Before the arguments cache: 6,000 of ~8,000 index records after 48 min.
|
||
- After a `macOS Browser` build into the shared folder, indexing skipped every app file: 2,150 files left instead of 3,972.
|
||
- [2] The compilation cache doesn't speed up the language server, because swift-build doesn't use it for index preparation. A cold and a warm cache gave the same time.
|
||
- [3] Sharing the index between worktrees was rejected. Index records hold absolute paths, and sourcekit-lsp reindexes any file newer than its record, which is every file in a fresh checkout.
|
||
- [4] Alternatives considered:
|
||
- **xcode-build-server** (parses Xcode build logs): needs a full Xcode build first and goes stale until the next build.
|
||
- **SwiftPM package for the app code** (current plan): the package work stays useful, but the editor setup doesn't have to wait for it.
|
||
- **SweetPad** extension: wraps xcodebuild and xcode-build-server; same staleness, plus another extension to maintain.
|
||
|
||
## Testing
|
||
|
||
- Code intelligence checked with a headless LSP client (hover, definition, completion, a diagnostic for an injected type error), in both layouts, on:
|
||
- a macOS app file;
|
||
- a macOS unit test, including a jump into app code;
|
||
- a package test;
|
||
- an iOS app file in iOS mode.
|
||
- Tasks:
|
||
- "Test macOS Unit Tests": 33 tests pass.
|
||
- Builds don't rebuild after Xcode-style `xcodebuild` builds: both are no-ops in ~30 s.
|
||
- iOS simulator: build, install, launch, then lldb-dap attaches by pid and stops at a breakpoint in `AppDelegate`.
|
||
- macOS: lldb stops at a breakpoint set by full path, with and without the cross-worktree path mapping.
|
||
- Background indexer:
|
||
- full run;
|
||
- skipping an unchanged worktree;
|
||
- an editor taking over mid-run, with no orphaned processes.
|
||
- sourcekit-xcode-bsp: 58 unit tests. They cover the filter, the cache (single flight, failures, purposes, invalidation, stale results), the index-path override and the layouts.
|
||
- Independent code review; its findings are fixed (see notes).
|
||
- To do before rollout:
|
||
- a pass through the VS Code and Cursor UI by two people on fresh machines;
|
||
- checking that the Xcode IDE picks up a worktree built and indexed from VS Code without rebuilding.
|
||
|
||
## Additional Considerations
|
||
|
||
- **Privacy / Security:** No change to the apps. The install script downloads and builds sourcekit-xcode-bsp from its public repository at a pinned commit with pinned dependencies. Everything runs locally.
|
||
- **Site Breakage / Experimentation:** Not applicable.
|
||
- **Operational:** No CI or release change. Opting in means running the installer and opening the workspace file. Xcode users aren't affected.
|
||
- **Localization:** Not applicable.
|