186 lines
9.2 KiB
Markdown
186 lines
9.2 KiB
Markdown
---
|
|
title: 'Tech Design: Opaque URL Fragment Detection for Same-Document Navigation'
|
|
created: '2026-06-24'
|
|
tags:
|
|
- tech-design
|
|
- apple
|
|
- navigation
|
|
- webkit
|
|
- url
|
|
- crash
|
|
status: in-progress
|
|
chosen-approach: '_web_originalDataAsString via URLComponents(webKitUrl:)'
|
|
asana: >-
|
|
https://app.asana.com/1/137249556945/project/1202406491309510/task/1215403820548743
|
|
---
|
|
**Author:** Apple team
|
|
**Project:** fix-fragment-crash (macOS / iOS browser)
|
|
**Sentry:** [APPLE-IOS-DY5A](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DY5A/?project=8) · [DZ58](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ58/?project=8) · [DZ5C](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ5C/?project=8)
|
|
|
|
---
|
|
|
|
## Background
|
|
|
|
Every navigation through `WKWebView` goes through `DistributedNavigationDelegate`, which classifies each `WKNavigationAction` as a **same-document navigation** (fragment-only change, e.g. `#section`) via `isSameDocumentNavigation`. Misclassification corrupts the navigation state machine.
|
|
|
|
`isSameDocumentNavigation` relies on `hasFragment` and `.sameDocument` URL equality, both of which must correctly extract the fragment — which is where the bug lives.
|
|
|
|
### Two URL parsers in WebKit
|
|
|
|
| Parser | Used where | Fragment in opaque URLs (`data:`, `about:`, `blob:`) |
|
|
|--------|-----------|------------------------------------------------------|
|
|
| **WTF::URLParser** | WebKit internals, navigation decisions | ✅ Correctly split at `#` |
|
|
| **NSURL / CFURL** | Foundation APIs, Swift `URL` | ❌ Encodes `#` as `%23`; `.fragment` returns `nil` |
|
|
|
|
When WebKit bridges a `WTF::URL` to Objective-C it calls:
|
|
|
|
```c
|
|
CFURLCreateAbsoluteURLWithBytes(alloc, rawBytes, len, kCFStringEncodingUTF8, nil, true)
|
|
```
|
|
|
|
The raw bytes contain the literal `#`, but CFURL's old-RFC parser does not recognise `#` as a fragment delimiter in opaque-scheme URLs — it encodes it as `%23` and stores the whole thing as the opaque path. Result for `data:text/html,hello#anchor`:
|
|
|
|
| Property | WTF::URL | NSURL / Swift URL |
|
|
|----------|----------|-------------------|
|
|
| `.fragment` | `"anchor"` | `nil` |
|
|
| `.absoluteString` | `data:text/html,hello#anchor` | `data:text/html,hello%23anchor` |
|
|
|
|
---
|
|
|
|
## Problem
|
|
|
|
A multi-MB `data:` URL reaches `isSameDocumentNavigation` on the main thread. The old `hashedSuffix` implementation called `String.firstIndex(of: "#")`, iterating grapheme-by-grapheme — O(n) per navigation decision on the **main thread**. For a large enough URL this blocks the thread >10 s → RunningBoard watchdog SIGKILL (`0x8BADF00D`).
|
|
|
|
The fix must:
|
|
1. Make fragment detection fast (no O(n) grapheme scan).
|
|
2. Correctly detect fragments in opaque URLs where NSURL has already encoded `#` as `%23`.
|
|
|
|
---
|
|
|
|
## Information-Loss Chain
|
|
|
|
Fragment information is irreversibly lost between WebKit and Swift:
|
|
|
|
```
|
|
WTF::URL (m_string = "data:…#anchor")
|
|
↓ createCFURL()
|
|
↓ CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor") ← '#' visible here
|
|
↓ CFURL re-parses; '#' → '%23'
|
|
NSURL (.absoluteString = "…%23anchor", .fragment = nil)
|
|
↓
|
|
WKNavigationAction.request.URL ← what Swift sees
|
|
```
|
|
|
|
Every downstream Foundation API (`URL.fragment`, `URLComponents`, `NSURLComponents`) operates on the already-broken NSURL — they all return `nil` for the fragment. No public or private WebKit API exposes the raw `WTF::URL` string … except one (see Chosen Approach).
|
|
|
|
---
|
|
|
|
## Approaches Considered
|
|
|
|
### 1. Scan `absoluteString` for `%23` ❌
|
|
|
|
Since NSURL encodes `#` as `%23`, scanning for it recovers the fragment in many cases. Partly working, but fundamentally ambiguous: `%23` is also valid payload in `data:` URLs (SVG fill colors, anchor hrefs, CSS). The two cases produce identical `absoluteString` and cannot be distinguished from the string alone. A size guard avoids scanning large payloads but causes incorrect equality results.
|
|
|
|
### 2. `CFURLCreateAbsoluteURLWithBytes` hook
|
|
|
|
WebKit calls `CFURLCreateAbsoluteURLWithBytes()` at the exact moment it converts a `WTF::URL` to a `CFURL`. The raw byte buffer at that call site still contains the literal `#`. Intercepting this C function via the Mach-O `__DATA,__interpose` section (without method swizzling) gives unambiguous access to the original bytes — scan for `0x23` in a simple C loop, store the byte range as an associated object on the bridged `NSURL`.
|
|
|
|
```c
|
|
#define DYLD_INTERPOSE(_replacement, _replacee) \
|
|
__attribute__((used)) \
|
|
static struct { const void *replacement; const void *replacee; } \
|
|
_interpose_##_replacee \
|
|
__attribute__((section("__DATA,__interpose"))) = \
|
|
{ (const void *)&(_replacement), (const void *)&(_replacee) }
|
|
```
|
|
|
|
Callers read `NSURL.ddg_fragmentByteRange` for the unambiguous `#` position. Could be limited to `#ifdef DEBUG` so there is zero production overhead.
|
|
|
|
**Downside:** adds Objective-C interpose files (`CFURLCreateAbsoluteURLWithBytesInterpose.m`, `NSURL+FragmentByteRange.h/.m`), couples to a private CFURL entry point, and requires the associated-object write to happen before any Swift code reads the URL — fragile under future WebKit changes.
|
|
|
|
### 2a. `dyld_interpose` at the dyld level ⚠️ (option, not chosen)
|
|
|
|
Same concept as above but applied via `DYLD_INSERT_LIBRARIES` + `DYLD_INTERPOSE` at the dynamic linker level, outside the process binary entirely. More powerful (intercepts calls from any dylib), but more invasive, not available in App Store binaries, and harder to confine to DEBUG.
|
|
|
|
---
|
|
|
|
## Chosen Approach: Private WebKit URL String (`_web_originalDataAsString`)
|
|
|
|
NSURL instances that originate from WebKit (via `createNSURL()`) carry a private property `_web_originalDataAsString` which holds the original `WTF::URL` byte string — **before** CFURL re-encoded `#` as `%23`. Parsing that string with Swift's native `URLComponents(string:)` gives correct path/query/fragment decomposition for all opaque URLs.
|
|
|
|
```swift
|
|
// URLComponentsExtension.swift
|
|
extension URLComponents {
|
|
/// Parses a WebKit-origin URL using the original WTF::URL string.
|
|
/// Falls back to URLComponents(url:) for non-WebKit or hierarchical URLs.
|
|
init?(webKitUrl: URL) {
|
|
guard webKitUrl.isOpaque,
|
|
let originalString = (webKitUrl as NSURL).value(forKey: "_web_originalDataAsString") as? String,
|
|
let swiftNativeURLComponents = URLComponents(string: originalString) else {
|
|
self.init(url: webKitUrl, resolvingAgainstBaseURL: false)
|
|
return
|
|
}
|
|
self = swiftNativeURLComponents
|
|
}
|
|
}
|
|
```
|
|
|
|
```swift
|
|
// URLExtension.swift
|
|
public var originalWebKitString: String? {
|
|
(self as NSURL).value(forKey: "_web_originalDataAsString") as? String
|
|
}
|
|
|
|
public var hasFragment: Bool {
|
|
guard let components = URLComponents(webKitUrl: self),
|
|
let fragment = components.fragment else { return false }
|
|
return !fragment.isEmpty
|
|
}
|
|
|
|
public func equals(_ other: URL, by components: EqualityComponents) -> Bool {
|
|
guard let selfParsed = URLComponents(webKitUrl: self),
|
|
let otherParsed = URLComponents(webKitUrl: other) else { return false }
|
|
// component-by-component comparison …
|
|
}
|
|
```
|
|
|
|
### Why this is correct
|
|
|
|
- `_web_originalDataAsString` holds the `WTF::URL`'s `m_string` verbatim — `#` is still a literal character, not `%23`.
|
|
- Swift's `URLComponents(string:)` is WHATWG-aligned and correctly splits at `#`, so `.fragment` returns the real fragment.
|
|
- No ambiguity with `%23` payload: we parse the original string, not the NSURL's re-encoded `absoluteString`.
|
|
- The property is absent on non-WebKit URLs (`URL(string:)`-created values) — the initialiser falls back to `URLComponents(url:)`, which is correct for hierarchical URLs.
|
|
- Pure Swift/Obj-C KVC access; no interpose, no associated objects, no extra files.
|
|
|
|
### Call flow
|
|
|
|
```
|
|
WTF::URL (m_string = "data:…#anchor")
|
|
↓ createNSURL()
|
|
NSURL (.absoluteString = "…%23anchor", ._web_originalDataAsString = "data:…#anchor")
|
|
↓
|
|
URLComponents(webKitUrl: url)
|
|
→ value(forKey: "_web_originalDataAsString") → "data:…#anchor"
|
|
→ URLComponents(string: "data:…#anchor")
|
|
→ .fragment = "anchor" ✅
|
|
```
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
- **Unit tests** (`URLExtensionTests.swift`):
|
|
- `hasFragment` and `equals(by: .sameDocument / .fuzzyIdentity)` on `about:blank#section`, `data:text/html,…#anchor`, opaque URLs with query AND fragment.
|
|
- Large (20 MB) `data:` URL: comparison must complete in < 0.1 s.
|
|
- Diagnostic timing test (`tempOpaqueNSURLComponentTimings`) for all Foundation URL APIs on a 20 MB `data:` URL.
|
|
- **Regression**: navigation tests in `DistributedNavigationDelegateTests.swift` updated to use `.equals(by: .fuzzyIdentity)` in place of the removed `.matches()` API.
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- **WebKit bug:** [bugs.webkit.org/68089](https://bugs.webkit.org/show_bug.cgi?id=68089). Chromium fixed `#` handling in M72 (2019). `WTF::URLParser` is correct; only the CFURL bridge is broken.
|
|
- **Primary safety fix** is the O(n) crash: `isSameDocumentNavigation` now uses `URLComponents(webKitUrl:)` which is constant-time relative to URL length (KVC + a single string parse), replacing the old grapheme-scan loop.
|
|
- **`%23` ambiguity** only affects `data:` / `about:`. For `blob:` and `javascript:` the opaque path never legitimately contains `%23` as encoded payload.
|
|
- No user data collected or transmitted; all processing is local and in-process.
|