git_admin/obsidian-vault
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[2026-10-15] eagle: work/projects/vscode-migration/tech-design-xcode-build-server.md

Browse Source
This commit is contained in:
Alexey Martemyanov committed 2026-10-15 12:16:30 +06:00
1 parent dca7034922
commit 32bbc89143
1 file changed
+64 -95
Download Patch File Download Diff File Expand all files Collapse all files
work/projects/vscode-migration/tech-design-xcode-build-server.md
+64 -95
View File
@@ -1,4 +1,4 @@
# Tech Design: VS Code and Cursor on the Xcode projects through a build server
# Tech Design: VS Code and Cursor support through an Xcode build server
**Author:** Alex M
**Reviewer:** TBD
@@ -7,125 +7,94 @@
## Background & Requirements
- Developers increasingly work in VS Code and Cursor (AI agents, multiple worktrees), but code intelligence for our apps only exists in Xcode.
- The Swift extension's language server, sourcekit-lsp, understands SwiftPM packages natively. For other build systems it talks to a *build server* through the Build Server Protocol (BSP), configured by a `buildServer.json` in the folder.
- The current plan makes the macOS browser code a SwiftPM package ([[projects/vscode-migration/tech-design-spm-browser-package|tech design]]). That is a big change to the project, and macOS only.
- [sourcekit-xcode-bsp](https://github.com/slime-studio/sourcekit-xcode-bsp) (Apache 2.0, early-stage, version 0.1.0) is a build server for `.xcodeproj`/`.xcworkspace`. It uses swift-build, the open-source build engine of Xcode, so it reads the projects the way Xcode does.
- Some of us (would prefer to) spend much of the day outside Xcode: in VS Code, reviewing changes, working with AI agents, or across several worktrees at once. There, the Swift language server has no build information for our Xcode projects. So there's no direct build&run support, no jump to definition, no completion and no errors as you type, and no debugger.
- The [previous tech design](https://app.asana.com/1/137249556945/project/1202406491309510/task/1219304032604521) tried to solve this by making the macOS browser code a Swift package. The team decided not to do it at this stage. The concerns were:
- a more custom, more fragile setup;
- resources that can fail silently at runtime;
- no good way to have per-configuration dependencies;
- macOS and iOS drifting apart;
- an unclear number of users and an unclear owner.
- The Swift language server (sourcekit-lsp) can get build information from a *build server* instead of a Swift package. [sourcekit-xcode-bsp](https://github.com/slime-studio/sourcekit-xcode-bsp) is an open-source build server that reads `.xcodeproj` / `.xcworkspace` with swift-build, the build engine Apple open-sourced from Xcode.
- **Requirements:**
- Code intelligence (completion, diagnostics, jump to definition, references) for app, test and package code, macOS and iOS.
- Build, run, test and debug from the editor, including the iOS simulator.
- No change to the Xcode projects, schemes, CI or release.
- Developers who also use Xcode don't pay twice: builds and the index are shared with Xcode.
- Works in many worktrees without hours of indexing per worktree.
- No change to the Xcode projects, schemes, CI or release, and nothing changes for people who only use Xcode.
- Opt-in and easy to remove.
- macOS and iOS, including tests and packages.
- People who use both Xcode and VS Code don't build or index everything twice.
## Problem Statement
Out of the box, sourcekit-xcode-bsp 0.1.0 on our workspace:
- gives app code intelligence after ~3 minutes;
- gives **nothing in test files**;
- produces build failures and drops ~250 object files into the repository root;
- needs an estimated **2 hours** to index a worktree;
- keeps its own build and index, separate from Xcode's.
Code intelligence and debugging in VS Code and Cursor need build information that today only Xcode has. We want to provide it from the Xcode projects as they are, without restructuring the codebase around it.
## Recommended Approach
Use sourcekit-xcode-bsp with four changes to it, plus a small setup in the repository.
Add an opt-in VS Code setup to the repository: a workspace file and a few scripts in `scripts/vscode/`. Nothing else in the repository changes.
**Changes to sourcekit-xcode-bsp** (`scripts/vscode/sourcekit-xcode-bsp.patch`, to be upstreamed)
1. **`configuration`, default `Debug`.** Without it swift-build uses the projects' default configuration, Release, which has no testability. Every `@testable import` failed, so test files had no results.
2. **`targets` include/exclude patterns** on `<project>/<target>`, plus a `list-targets` command to check them. The server put all 977 workspace targets into one build: DMG and App Store variants, both platforms, and 3–4 variants of each package library. They build into the same paths, so swift-build dropped tasks. Xcode never builds them together; a scheme picks one variant and package targets come in as dependencies. We configure the same way: 86 targets, which pull in 288.
3. **A per-target cache of Swift compiler arguments.** swift-build computes them for every file from scratch (~0.8 s each, one at a time). Within a Swift target they differ only in the file's `-index-unit-output-path`, which the server already reports. The first request of a target is computed, and concurrent requests wait for it. Indexing a fresh worktree went from ~2 hours to **~5.5 minutes** on an M4 Max.
4. **`derivedDataLayout: xcode`.** The server works in the worktree's Xcode DerivedData folder with Xcode's layout:
- the index store is Xcode's;
- index records name the regular build's object files, so a file indexed or built by Xcode or `xcodebuild` is up to date for the editor, and the other way round;
- preparation builds go into a folder of their own, so the server never shares a build database with an open Xcode.
**What a developer gets**
- Completion, jump to definition, references and live errors for app, test and package code, on macOS or iOS. One platform is chosen per worktree and can be switched.
- Tasks to build any scheme and run unit tests; errors and failed tests show in the Problems panel.
- Debugging with breakpoints:
- the macOS app, launched from the editor;
- the iOS app on a simulator, by running the app and then attaching.
- **Shared with Xcode.** Builds and the index live in the worktree's usual Xcode DerivedData folder. Whatever Xcode built or indexed is ready in VS Code, and the other way round.
- Optional background indexing of worktrees while no editor is open.
Also added: `buildSettings` overrides.
**How it works**
- The workspace generates the build server's config on folder open. The config chooses the platform and the targets to index, the same way an Xcode scheme does.
- Builds and tests go through `xcodebuild` with the same settings Xcode uses.
- The build server is installed locally by a script, from a pinned version.
**In the repository** (`DuckDuckGo.code-workspace`, `scripts/vscode/`)
- **Language server config.** `setup-build-server.py [macos|ios]` writes `buildServer.json` on folder open. It sets Debug, one platform (kept until switched) and the target filter. The filter keeps the app project and package test products, and drops App Store variants and the other platform's app and local packages.
- **Builds.** `xcodebuild.sh` builds into Xcode's DerivedData folder with Xcode's settings, so Xcode and VS Code don't rebuild after each other. It also feeds the shared index. Tasks:
- build a macOS scheme;
- run unit tests with `-only-testing`; failures show in the Problems panel;
- build and run the iOS app on a simulator.
- **Debugging** uses lldb-dap.
- macOS: launch, with or without building first.
- iOS simulator: `simulator.sh` builds for the chosen simulator, installs the app and launches it waiting for the debugger. The launch configuration then attaches by pid; the process picker lists newest first.
- **Opt-in cache across worktrees.** `DDG_WORKTREE_CACHE=1` builds in the worktree instead. The worktree and build folder are mapped to `/^wt` and `/^dd`, so Xcode's compilation cache is shared between worktrees: a clean build in a new worktree takes 90–120 s instead of 179 s. Launch configurations map the paths back for the debugger. These builds aren't shared with Xcode.
- **Background indexing.** `index-in-background.py` indexes worktrees headless while no editor uses them; an editor that opens stops it. `background-indexing.sh install` runs it every 30 minutes while Xcode, VS Code and Cursor are closed (opt-in).
- **Install.** `install-build-server.sh` builds the patched server from a pinned upstream commit, with pinned dependencies, into `~/.local/opt`. The setup task warns when the patch changes.
**The build server needed fixes.** On our workspace it gave no results in test files, broke on the DMG/App Store variants and took hours to index. Four small changes fix this (see Appendix), and a worktree now indexes in about 5 minutes. They're general improvements, so the plan is to contribute them upstream. Until they're merged, the repository carries them as a patch that the install script applies.
**Steps**
1. Land the setup and the patch in the repository. Xcode users aren't affected.
2. Early adopters on macOS and iOS; then a short how-to in the team docs.
3. Open upstream PRs for the server changes. Drop the patch when a release has them, or move it to a fork under the DuckDuckGo organization.
1. Land the setup in the repository, plus a short README.
2. Open PRs for the build server changes in its repository.
3. A few early adopters on macOS and iOS try it for a couple of weeks; then we decide whether to announce it more widely.
### Pros
- **No project changes.** The Xcode projects, schemes and CI stay the source of truth; nothing to keep in sync.
- **Same build settings as Xcode.** swift-build plans the build, so flags, macros, plugins, resources and per-configuration settings match Xcode. Hand-written SwiftPM manifests can drift from them.
- **Works together with Xcode.** One build and one index per worktree, whichever tool made them.
- **Both platforms and every target type:** apps, extensions, app tests, package tests.
- **Fast enough for many worktrees.**
- ~5.5 min first index, less after a build.
- Background indexing.
- Optional shared compile cache across worktrees.
- Complements the SwiftPM package work rather than replacing it: modularization still helps build times and testing.
- **No project changes.** The Xcode projects stay the only source of truth. Removing the setup means deleting `scripts/vscode/` and the workspace file.
- **Same build as Xcode.** swift-build reads the projects, so per-configuration settings and dependencies, resources, macros and plugins behave like in Xcode.
- **Nothing to maintain in the app code.** No lint rules or checks to keep the setup working.
- **Both platforms**, so it doesn't add a gap between macOS and iOS.
- **No cost for Xcode users**, and those who use both tools share one build and one index.
### Cons
- **A young third-party tool** (0.1.0, "early-stage" by its authors) and our patch on top until upstream merges it.
- **One platform per worktree at a time;** switching runs the setup again and reindexes the other platform's files.
- **Files outside the configured targets have no results,** for example App Store-only files (none today) or packages only the other platform uses.
- **iOS debugging takes two steps:** run the task, then attach. VS Code resolves the process picker before a pre-launch task, so the two can't be one action. Attaching with a launch-time command doesn't stop at breakpoints with lldb-dap today.
- **The cross-worktree cache and sharing with Xcode exclude each other.** Prefix-mapped paths in debug info need a source map, which we can only set up in VS Code.
- **xcodebuild builds take as long as Xcode's.** Don't build one worktree from both at the same time.
- **A young third-party tool** (version 0.1.0, "early-stage" by its authors), and our patch on top of it until upstream accepts the changes.
- **Some rough edges:**
- one platform per worktree at a time;
- iOS debugging is two steps (run, then attach);
- the first index of a new worktree takes ~5 minutes.
- **Someone has to own it.** I will; the setup is small (a workspace file and ~6 scripts), and it's documented.
### Risks
- **swift-build changes with Xcode.** The server bundles its own swift-build, pinned by upstream, while the compiler comes from the selected Xcode. Mitigation: pinned commit and dependencies in `install-build-server.sh`; check after each Xcode update.
- **Upstream doesn't take the patches** or the project stalls. Mitigation: a fork under the DuckDuckGo organization. The patch is ~550 lines of code plus ~450 lines of tests.
- **Shared index store with Xcode.** Xcode and sourcekit-lsp both write records into it; the index store format is designed for several writers, but we haven't seen it under load. Mitigation: the server never shares Xcode's build database, and deleting the DerivedData folder resets both.
- **Background indexing costs CPU and disk.** Mitigation: it's opt-in, runs only while editors are closed, at low priority, and skips worktrees that haven't changed.
## Notes
- [1] Measured on an M4 Max with Xcode 27.1 on `main` at 3d850d74dc.
- Fresh-worktree index: 323–347 s at low load. That is ~1 min to load the workspace and resolve packages, then ~3.5 min to prepare and index, running up to 11 compilers in parallel.
- Before the arguments cache: 6,000 of ~8,000 index records after 48 min.
- After a `macOS Browser` build into the shared folder, indexing skipped every app file: 2,150 files left instead of 3,972.
- [2] The compilation cache doesn't speed up the language server, because swift-build doesn't use it for index preparation. A cold and a warm cache gave the same time.
- [3] Sharing the index between worktrees was rejected. Index records hold absolute paths, and sourcekit-lsp reindexes any file newer than its record, which is every file in a fresh checkout.
- [4] Alternatives considered:
- **xcode-build-server** (parses Xcode build logs): needs a full Xcode build first and goes stale until the next build.
- **SwiftPM package for the app code** (current plan): the package work stays useful, but the editor setup doesn't have to wait for it.
- **SweetPad** extension: wraps xcodebuild and xcode-build-server; same staleness, plus another extension to maintain.
- **Xcode updates can break the build server**, since it bundles its own swift-build. Mitigation: a pinned version and a check after each Xcode update. The worst case is that the setup stops working for the people who opted in; Xcode and CI aren't affected.
- **Upstream doesn't accept the changes** or the project is abandoned. Mitigation: keep a fork under the DuckDuckGo organization (the changes are ~550 lines plus tests), or drop the setup.
- **Xcode and the editor share the index.** Both write to Xcode's index store; deleting the DerivedData folder resets it, as today.
## Testing
- Code intelligence checked with a headless LSP client (hover, definition, completion, a diagnostic for an injected type error), in both layouts, on:
- a macOS app file;
- a macOS unit test, including a jump into app code;
- a package test;
- an iOS app file in iOS mode.
- Code intelligence: macOS and iOS app files, app unit tests (including jumping into app code), package tests.
- Tasks:
- "Test macOS Unit Tests": 33 tests pass.
- Builds don't rebuild after Xcode-style `xcodebuild` builds: both are no-ops in ~30 s.
- iOS simulator: build, install, launch, then lldb-dap attaches by pid and stops at a breakpoint in `AppDelegate`.
- macOS: lldb stops at a breakpoint set by full path, with and without the cross-worktree path mapping.
- Background indexer:
- full run;
- skipping an unchanged worktree;
- an editor taking over mid-run, with no orphaned processes.
- sourcekit-xcode-bsp: 58 unit tests. They cover the filter, the cache (single flight, failures, purposes, invalidation, stale results), the index-path override and the layouts.
- Independent code review; its findings are fixed (see notes).
- To do before rollout:
- a pass through the VS Code and Cursor UI by two people on fresh machines;
- checking that the Xcode IDE picks up a worktree built and indexed from VS Code without rebuilding.
- building macOS and iOS schemes;
- running unit tests;
- builds from Xcode and VS Code don't rebuild each other's work.
- Debugging: breakpoints in the macOS app and in the iOS app on a simulator.
- The build server changes have unit tests.
- Before announcing: two people try it on fresh machines in VS Code and Cursor, and we check that Xcode picks up a worktree built and indexed from VS Code without rebuilding.
## Additional Considerations
- **Privacy / Security:** No change to the apps. The install script downloads and builds sourcekit-xcode-bsp from its public repository at a pinned commit with pinned dependencies. Everything runs locally.
- **Site Breakage / Experimentation:** Not applicable.
- **Operational:** No CI or release change. Opting in means running the installer and opening the workspace file. Xcode users aren't affected.
- **Localization:** Not applicable.
- **Privacy / Security:** No change to the apps. The install script builds the build server from its public repository at a pinned version, and everything runs locally.
- **Site Breakage / Experimentation / Localization:** Not applicable.
- **Operational:** No CI or release change.
## Appendix: changes to the build server
1. **Debug configuration by default.** Without one, swift-build used the projects' default configuration, Release, which can't import app modules into tests, so test files had no results.
2. **Choosing targets.** It configured all ~980 workspace targets in one build, including DMG and App Store variants of the same products, which write to the same paths. It now takes include/exclude patterns, so we configure what a scheme would.
3. **Caching compiler arguments per target.** It computed them separately for every file, ~1 s each, one at a time, which made indexing take hours. With the cache, a fresh worktree indexes in ~5 minutes on an M4 Max.
4. **Xcode's DerivedData layout**, so the index and builds are shared with Xcode.
Research notes with measurements: [[projects/vscode-migration/xcode-build-server]].