194 lines
10 KiB
Markdown
194 lines
10 KiB
Markdown
---
|
|
title: 'Tech Design: Opaque URL Fragment Detection for Same-Document Navigation'
|
|
created: '2026-06-24'
|
|
tags:
|
|
- tech-design
|
|
- apple
|
|
- navigation
|
|
- webkit
|
|
- url
|
|
- crash
|
|
status: in-progress
|
|
chosen-approach: '_web_originalDataAsString via URLComponents(webKitUrl:)'
|
|
asana: >-
|
|
https://app.asana.com/1/137249556945/project/1202406491309510/task/1215403820548743
|
|
---
|
|
**Author:** Apple team
|
|
**Project:** fix-fragment-crash (macOS / iOS browser)
|
|
**Sentry:** [APPLE-IOS-DY5A](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DY5A/?project=8) · [DZ58](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ58/?project=8) · [DZ5C](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ5C/?project=8)
|
|
|
|
---
|
|
|
|
## Background
|
|
|
|
Every navigation through `WKWebView` goes through `DistributedNavigationDelegate`, which classifies each `WKNavigationAction` as a **same-document navigation** (fragment-only change, e.g. `#section`) via `isSameDocumentNavigation`. Misclassification corrupts the navigation state machine.
|
|
|
|
`isSameDocumentNavigation` relies on `hasFragment` and `.sameDocument` URL equality, both of which must correctly extract the fragment — which is where the bug lives.
|
|
|
|
### Two URL parsers in WebKit
|
|
|
|
| Parser | Used where | Fragment in opaque URLs (`data:`, `about:`, `blob:`) |
|
|
|--------|-----------|------------------------------------------------------|
|
|
| **WTF::URLParser** | WebKit internals, navigation decisions | ✅ Correctly split at `#` |
|
|
| **NSURL / CFURL** | Foundation APIs, Swift `URL` | ❌ Encodes `#` as `%23`; `.fragment` returns `nil` |
|
|
|
|
When WebKit bridges a `WTF::URL` to Objective-C it calls:
|
|
|
|
```c
|
|
CFURLCreateAbsoluteURLWithBytes(alloc, rawBytes, len, kCFStringEncodingUTF8, nil, true)
|
|
```
|
|
|
|
The raw bytes contain the literal `#`, but CFURL's old-RFC parser does not recognise `#` as a fragment delimiter in opaque-scheme URLs — it encodes it as `%23` and stores the whole thing as the opaque path. Result for `data:text/html,hello#anchor`:
|
|
|
|
| Property | WTF::URL | NSURL / Swift URL |
|
|
|----------|----------|-------------------|
|
|
| `.fragment` | `"anchor"` | `nil` |
|
|
| `.absoluteString` | `data:text/html,hello#anchor` | `data:text/html,hello%23anchor` |
|
|
|
|
---
|
|
|
|
## Problem
|
|
|
|
A multi-MB `data:` URL reaches `isSameDocumentNavigation` on the main thread. The old `hashedSuffix` implementation called `String.firstIndex(of: "#")`, iterating grapheme-by-grapheme — O(n) per navigation decision on the **main thread**. For a large enough URL this blocks the thread >10 s → RunningBoard watchdog SIGKILL (`0x8BADF00D`).
|
|
|
|
The fix must:
|
|
1. Make fragment detection fast (no O(n) grapheme scan).
|
|
2. Correctly detect fragments in opaque URLs where NSURL has already encoded `#` as `%23`.
|
|
|
|
---
|
|
|
|
## Information-Loss Chain
|
|
|
|
Fragment information is irreversibly lost between WebKit and Swift:
|
|
|
|
```
|
|
WTF::URL (m_string = "data:…#anchor")
|
|
↓ createCFURL()
|
|
↓ CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor") ← '#' visible here
|
|
↓ CFURL re-parses; '#' → '%23'
|
|
NSURL (.absoluteString = "…%23anchor", .fragment = nil)
|
|
↓
|
|
WKNavigationAction.request.URL ← what Swift sees
|
|
```
|
|
|
|
Every downstream Foundation API (`URL.fragment`, `URLComponents`, `NSURLComponents`) operates on the already-broken NSURL — they all return `nil` for the fragment. No public or private WebKit API exposes the raw `WTF::URL` string … except one (see Chosen Approach).
|
|
|
|
---
|
|
|
|
## Approaches Considered
|
|
|
|
### 1. Scan `absoluteString` for `%23` ❌
|
|
|
|
Since NSURL encodes `#` as `%23`, scanning for it recovers the fragment in many cases. Partly working, but fundamentally ambiguous: `%23` is also valid payload in `data:` URLs (SVG fill colors, anchor hrefs, CSS). The two cases produce identical `absoluteString` and cannot be distinguished from the string alone. A size guard avoids scanning large payloads but causes incorrect equality results.
|
|
|
|
### 2. `CFURLCreateAbsoluteURLWithBytes` GOT-patch hook
|
|
|
|
WebKit calls `CFURLCreateAbsoluteURLWithBytes()` at the exact moment it converts a `WTF::URL` to a `CFURL`. The raw byte buffer at that point still contains the literal `#`. The hook intercepts this call using **runtime GOT patching** (same technique as `CxaThrowSwapper` / fishhook) — not a link-time `__DATA,__interpose` section, which would not reach WebKit's internal call from `WebKit.framework` into `CoreFoundation.framework`.
|
|
|
|
Mechanism:
|
|
1. `_dyld_register_func_for_add_image` registers a callback that fires for every Mach-O image loaded into the process (including already-loaded images at registration time).
|
|
2. The callback walks the image's **indirect symbol table** (`ImageMap.rebindSymbol`) targeting only the `WebKit` dylib, finds its GOT entry for `CFURLCreateAbsoluteURLWithBytes`, and replaces it with the hook pointer. ARM64e pointer authentication is handled via `BSKStripFunctionPointer` / `BSKSignFunctionPointer`.
|
|
3. The hook calls the original via the saved pointer, then scans the raw byte buffer with `memchr` (SIMD-vectorised on ARM64/x86) for `':'` (opaque check) then `'#'`, and stores the resulting `NSRange` as an associated object on the `CFURL` result.
|
|
4. `URL.opaqueFragmentAnnotation` reads/writes that associated object; `URL.isOpaqueFragmentScanned` guards against double-scanning the same instance.
|
|
5. `uninstallCFURLSwapper()` walks all loaded images and restores the saved original pointer.
|
|
|
|
```swift
|
|
// install: register dyld callback → patch WebKit's GOT entry at runtime
|
|
_dyld_register_func_for_add_image(_cfURLProcessMachHeader)
|
|
|
|
// hook: memchr scan + associated-object annotation
|
|
if let hashRaw = memchr(searchBase, Int32(UInt8(ascii: "#")), searchLen) {
|
|
url.opaqueFragmentAnnotation = NSRange(location: hashOff, length: len - hashOff)
|
|
}
|
|
```
|
|
|
|
**Downside:** couples to WebKit's dylib name and a private CFURL symbol; requires bypassing page protection to write into a read-only GOT page; depends on dyld image-load ordering; massive and fragile.
|
|
|
|
### 2a. `__DATA,__interpose` (static link-time interpose) ⚠️ (option, not chosen)
|
|
|
|
A related but distinct mechanism: a `__DATA,__interpose` section in the binary redirects calls to the named C function to a replacement at load time, before any code runs. Simpler than GOT patching (no dyld callback, no image walk), and dyld applies it process-wide so it does intercept calls from other frameworks.
|
|
|
|
**Downside:** the `__DATA,__interpose` section is a well-known jailbreak / hooking fingerprint; App Store review tooling flags it, making approval unlikely.
|
|
|
|
---
|
|
|
|
## Chosen Approach: Private WebKit URL String (`_web_originalDataAsString`)
|
|
|
|
NSURL instances that originate from WebKit (via `createNSURL()`) carry a private property `_web_originalDataAsString` which holds the original `WTF::URL` byte string — **before** CFURL re-encoded `#` as `%23`. Parsing that string with Swift's native `URLComponents(string:)` gives correct path/query/fragment decomposition for all opaque URLs.
|
|
|
|
```swift
|
|
// URLComponentsExtension.swift
|
|
extension URLComponents {
|
|
/// Parses a WebKit-origin URL using the original WTF::URL string.
|
|
/// Falls back to URLComponents(url:) for non-WebKit or hierarchical URLs.
|
|
init?(webKitUrl: URL) {
|
|
guard webKitUrl.isOpaque,
|
|
let originalString = (webKitUrl as NSURL).value(forKey: "_web_originalDataAsString") as? String,
|
|
let swiftNativeURLComponents = URLComponents(string: originalString) else {
|
|
self.init(url: webKitUrl, resolvingAgainstBaseURL: false)
|
|
return
|
|
}
|
|
self = swiftNativeURLComponents
|
|
}
|
|
}
|
|
```
|
|
|
|
```swift
|
|
// URLExtension.swift
|
|
public var originalWebKitString: String? {
|
|
(self as NSURL).value(forKey: "_web_originalDataAsString") as? String
|
|
}
|
|
|
|
public var hasFragment: Bool {
|
|
guard let components = URLComponents(webKitUrl: self),
|
|
let fragment = components.fragment else { return false }
|
|
return !fragment.isEmpty
|
|
}
|
|
|
|
public func equals(_ other: URL, by components: EqualityComponents) -> Bool {
|
|
guard let selfParsed = URLComponents(webKitUrl: self),
|
|
let otherParsed = URLComponents(webKitUrl: other) else { return false }
|
|
// component-by-component comparison …
|
|
}
|
|
```
|
|
|
|
### Why this is correct
|
|
|
|
- `_web_originalDataAsString` holds the `WTF::URL`'s `m_string` verbatim — `#` is still a literal character, not `%23`.
|
|
- Swift's `URLComponents(string:)` is WHATWG-aligned and correctly splits at `#`, so `.fragment` returns the real fragment.
|
|
- No ambiguity with `%23` payload: we parse the original string, not the NSURL's re-encoded `absoluteString`.
|
|
- The property is absent on non-WebKit URLs (`URL(string:)`-created values) — the initialiser falls back to `URLComponents(url:)`, which is correct for hierarchical URLs.
|
|
- Pure Swift/Obj-C KVC access; no interpose, no associated objects, no extra files.
|
|
|
|
### Call flow
|
|
|
|
```
|
|
WTF::URL (m_string = "data:…#anchor")
|
|
↓ createNSURL()
|
|
NSURL (.absoluteString = "…%23anchor", ._web_originalDataAsString = "data:…#anchor")
|
|
↓
|
|
URLComponents(webKitUrl: url)
|
|
→ value(forKey: "_web_originalDataAsString") → "data:…#anchor"
|
|
→ URLComponents(string: "data:…#anchor")
|
|
→ .fragment = "anchor" ✅
|
|
```
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
- **Unit tests** (`URLExtensionTests.swift`):
|
|
- `hasFragment` and `equals(by: .sameDocument / .fuzzyIdentity)` on `about:blank#section`, `data:text/html,…#anchor`, opaque URLs with query AND fragment.
|
|
- Large (20 MB) `data:` URL: comparison must complete in < 0.1 s.
|
|
- Diagnostic timing test (`tempOpaqueNSURLComponentTimings`) for all Foundation URL APIs on a 20 MB `data:` URL.
|
|
- **Regression**: navigation tests in `DistributedNavigationDelegateTests.swift` updated to use `.equals(by: .fuzzyIdentity)` in place of the removed `.matches()` API.
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- **WebKit bug:** [bugs.webkit.org/68089](https://bugs.webkit.org/show_bug.cgi?id=68089). Chromium fixed `#` handling in M72 (2019). `WTF::URLParser` is correct; only the CFURL bridge is broken.
|
|
- **Primary safety fix** is the O(n) crash: `isSameDocumentNavigation` now uses `URLComponents(webKitUrl:)` which is constant-time relative to URL length (KVC + a single string parse), replacing the old grapheme-scan loop.
|
|
- **`%23` ambiguity** only affects `data:` / `about:`. For `blob:` and `javascript:` the opaque path never legitimately contains `%23` as encoded payload.
|
|
- No user data collected or transmitted; all processing is local and in-process.
|