234 lines
12 KiB
Markdown
234 lines
12 KiB
Markdown
---
|
|
title: 'Tech Design: Opaque URL Fragment Detection for Same-Document Navigation'
|
|
created: '2026-06-24'
|
|
tags:
|
|
- tech-design
|
|
- apple
|
|
- navigation
|
|
- webkit
|
|
- url
|
|
- crash
|
|
status: draft
|
|
asana: >-
|
|
https://app.asana.com/1/137249556945/project/1202406491309510/task/1215403820548743
|
|
---
|
|
**Author:** Apple team
|
|
**Project:** fix-fragment-crash (macOS / iOS browser)
|
|
**Sentry:** [APPLE-IOS-DY5A](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DY5A/?project=8) · [DZ58](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ58/?project=8) · [DZ5C](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ5C/?project=8)
|
|
|
|
---
|
|
|
|
## Background & Requirements
|
|
|
|
The DuckDuckGo browser uses `WKWebView` for all page rendering. Every navigation goes through the `decidePolicyFor` delegate chain managed by `DistributedNavigationDelegate` (BrowserServicesKit). A central piece of that chain is classifying each `WKNavigationAction` as a **same-document navigation** (fragment-only change within the current page, e.g. `#section`) via:
|
|
|
|
```swift
|
|
// WKNavigationActionExtension.swift:184
|
|
public var isSameDocumentNavigation: Bool {
|
|
switch navigationType {
|
|
case .linkActivated, .other:
|
|
return !isRedirect && newURL.hasFragment && currentURL.equals(newURL, by: .sameDocument)
|
|
case .backForward:
|
|
return (newURL.hasFragment || currentURL.hasFragment) && currentURL.equals(newURL, by: .sameDocument)
|
|
...
|
|
}
|
|
}
|
|
```
|
|
|
|
`isSameDocumentNavigation` drives the `DistributedNavigationDelegate` state machine: same-document navigations reuse the existing `Navigation` object and trigger client-redirect handling rather than starting a fresh navigation, while non-same-document navigations allocate a new `Navigation`. Misclassification corrupts navigation state.
|
|
|
|
`hasFragment` and the `.sameDocument` equality both depend on extracting the URL fragment—which is where the bug lives.
|
|
|
|
### Two URL parsers coexist in WebKit on Apple platforms
|
|
|
|
| Parser | Used where | Standard | Fragment in opaque URLs (`data:`, `about:`, `blob:`) |
|
|
|--------|-----------|----------|------------------------------------------------------|
|
|
| **WTF::URLParser** | WebKit rendering, WKWebView navigation decisions | WHATWG URL | ✅ Correctly split at `#` |
|
|
| **NSURL / CFURL** | Foundation APIs, Swift `URL` | Old RFC 1808/2396 | ❌ Encodes `#` as `%23`, returns `nil` for `.fragment` |
|
|
|
|
When WebKit bridges its internal `WTF::URL` to Objective-C it calls:
|
|
|
|
```c
|
|
CFURLCreateAbsoluteURLWithBytes(alloc, rawBytes, len, kCFStringEncodingUTF8, nil, true)
|
|
```
|
|
|
|
The raw bytes at this point still contain the literal `#`. CFURL re-parses them with its old-RFC parser, which does **not** recognise `#` as a fragment delimiter in opaque-scheme URLs. It encodes `#` → `%23` and stores the whole thing as the opaque path.
|
|
|
|
Result for `data:text/html,hello#anchor`:
|
|
|
|
| Property | WTF::URL | NSURL / Swift URL |
|
|
|----------|----------|-------------------|
|
|
| `.path` | `text/html,hello` | `""` (empty) |
|
|
| `.query` | nil | nil |
|
|
| `.fragment` | `"anchor"` | nil |
|
|
| `.absoluteString` | `data:text/html,hello#anchor` | `data:text/html,hello%23anchor` |
|
|
|
|
---
|
|
|
|
## Problem Statement
|
|
|
|
A pathological URL—a multi-MB `data:` URL or attacker-crafted href—reaches `WKNavigationAction.isSameDocumentNavigation` on the main thread. The call chain:
|
|
|
|
1. `WebKit::WebPageProxy::decidePolicyForNavigationAction` dispatches to the main-thread delegate.
|
|
2. `DistributedNavigationDelegate` reads `wkNavigationAction.isSameDocumentNavigation`.
|
|
3. `isSameDocumentNavigation` calls `newURL.absoluteString.hashedSuffix` (and for `.backForward`, also `currentURL.absoluteString.hashedSuffix`).
|
|
4. `hashedSuffix` → `hashedSuffixRange` → `String.firstIndex(of: "#")` (`StringExtension.swift:67`) iterates grapheme-by-grapheme over `absoluteString`.
|
|
5. For a multi-MB `data:` URL this is O(n) UTF-8 grapheme decoding on the **main thread**, per navigation decision.
|
|
6. Main thread blocked >10 s under thermal-state `serious` at 100% CPU → RunningBoard scene-update watchdog kills the app: **SIGKILL, terminationReason `0x8BADF00D`**.
|
|
|
|
The fix must:
|
|
1. Make `hashedSuffixRange` / `isSameDocumentNavigation` fast for arbitrarily large URLs (no O(n) grapheme scan).
|
|
2. Correctly detect fragments in opaque URLs where NSURL has already encoded `#` as `%23`.
|
|
|
|
---
|
|
|
|
## Why Standard APIs Can't Help: The Information-Loss Chain
|
|
|
|
Fragment information is **irreversibly lost** between WebKit's internal representation and what Swift code receives in `decidePolicyFor`:
|
|
|
|
```
|
|
WTF::URL (WHATWG-compliant; m_string = "data:…#anchor", m_queryEnd before '#')
|
|
↓ URL::createNSURL() → createCFURL()
|
|
↓ CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor") ← '#' still visible here
|
|
↓ CFURL re-parses with old-RFC parser; encodes '#' → '%23'
|
|
NSURL (.absoluteString = "…%23anchor", .fragment = nil, .path = "")
|
|
↓ [NSMutableURLRequest initWithURL: url().createNSURL()] (ResourceRequestCocoa.mm)
|
|
NSURLRequest (same NSURL)
|
|
↓
|
|
WKNavigationAction.request.URL ← what Swift sees
|
|
WKNavigationAction._originalURL ← same path, same loss (createNSURL() again)
|
|
```
|
|
|
|
Every Foundation/Swift API is downstream of the encoding step:
|
|
|
|
- **`URL.fragment` / `.path` / `.query`** — return `nil` / `""` for all opaque URLs.
|
|
- **`URLComponents(url:)` / `(string:)` and `NSURLComponents` equivalents** — either use NSURL's already-broken cached parse tree, or re-parse `absoluteString` where `#` is already `%23`. RFC 3986 treats `%23` as a literal character, not a delimiter. Fragment stays nil in all four variants.
|
|
- **`WKNavigationActionPrivate.h`** (`_originalURL`, `_isRedirect`, `_hitTestResult`, …) — all URL-returning properties go through `WTF::URL::createNSURL()` → same `createCFURL()` path. No private API exposes the raw WTF string.
|
|
|
|
There is **no Foundation or WebKit API**, public or private, that recovers the original `#` once the NSURL has been constructed.
|
|
|
|
---
|
|
|
|
## Approaches Tried
|
|
|
|
### 1. Scan `absoluteString` for `%23`
|
|
|
|
Since NSURL encodes the original `#` as `%23` in `absoluteString`, scanning for `%23` as a fragment delimiter recovers the fragment in many cases. Implemented and partly working.
|
|
|
|
**Fundamental problem:** `%23` is also valid payload data in `data:` URLs:
|
|
- SVG fill colors: `data:image/svg+xml,<rect fill="%23ff0000"/>` — `%23` is `#` in the SVG
|
|
- HTML anchor hrefs: `data:text/html,<a href="%23section">`
|
|
- CSS: `data:text/css,color:%23333`
|
|
|
|
Both cases produce identical `absoluteString`. They are **indistinguishable from the string alone**. A 4 KB size guard was added to avoid scanning large `data:` URLs, but that makes two different large data: URLs compare as equal — semantically wrong.
|
|
|
|
### 2. Private API / `WKNavigationAction._originalURL`
|
|
|
|
See information-loss chain above. No private API bypasses it; all paths go through `createCFURL()`.
|
|
|
|
---
|
|
|
|
## Recommended Approach: `CFURLCreateAbsoluteURLWithBytes` Interpose
|
|
|
|
### Key Insight
|
|
|
|
WebKit calls `CFURLCreateAbsoluteURLWithBytes()` at the **exact** moment it converts a `WTF::URL` to a `CFURL`/`NSURL`. The raw byte buffer at that call site still contains the literal `#`, before CFURL encodes it. Intercepting this C function gives unambiguous access to the original bytes.
|
|
|
|
### Mechanism: `DYLD_INTERPOSE`
|
|
|
|
The `__DATA,__interpose` Mach-O section replaces any C function in the same dylib at load time, without method swizzling or private SPI:
|
|
|
|
```c
|
|
#define DYLD_INTERPOSE(_replacement, _replacee) \
|
|
__attribute__((used)) \
|
|
static struct { const void *replacement; const void *replacee; } \
|
|
_interpose_##_replacee \
|
|
__attribute__((section("__DATA,__interpose"))) = \
|
|
{ (const void *)&(_replacement), (const void *)&(_replacee) }
|
|
```
|
|
|
|
### Implementation
|
|
|
|
Files:
|
|
- `CFURLCreateAbsoluteURLWithBytesInterpose.m` — DEBUG-only (`#ifdef DEBUG`)
|
|
- `NSURL+FragmentByteRange.h` / `.m` — category exposing the stored range
|
|
|
|
**Interpose logic:**
|
|
|
|
1. Call the real `CFURLCreateAbsoluteURLWithBytes` to get the `CFURLRef`.
|
|
2. For opaque URLs (scheme present, no `//` authority), scan the raw byte buffer for `'#'` (byte `0x23`) with a simple C loop—O(n) but done once at NSURL construction, not per navigation decision.
|
|
3. If found, store the byte range as an `NSValue` associated object on the bridged `NSURL` via `objc_setAssociatedObject`.
|
|
4. Callers read `NSURL.ddg_fragmentByteRange` for the unambiguous `#` position.
|
|
|
|
```objc
|
|
if (relativeURLBytes[i] == '#') {
|
|
NSRange fragmentRange = NSMakeRange(i, length - i);
|
|
NSURL *nsurl = (__bridge NSURL *)result;
|
|
objc_setAssociatedObject(nsurl, &kFragmentByteRangeKey,
|
|
[NSValue valueWithRange:fragmentRange],
|
|
OBJC_ASSOCIATION_RETAIN_NONATOMIC);
|
|
}
|
|
```
|
|
|
|
### Why This Is Correct
|
|
|
|
- Bytes are scanned **before** CFURL encodes `#` → `%23` — no ambiguity with payload `%23`.
|
|
- Associated object is set on the **exact NSURL instance** for that WTF::URL.
|
|
- `ddg_fragmentByteRange.location` matches WTF::URL's internal `m_queryEnd + 1`.
|
|
- Works for all opaque schemes: `data:`, `about:`, `blob:`, `javascript:`.
|
|
- DEBUG-only: zero production overhead; interpose section absent from release builds.
|
|
|
|
### Complete Call Flow with Interpose
|
|
|
|
```
|
|
WTF::URL (m_string = "data:…#anchor", m_queryEnd before '#')
|
|
↓ URL::createCFURL()
|
|
↓ CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor")
|
|
↑↑ INTERPOSED: scan rawBytes → store NSRange on result NSURL
|
|
↓ NSURL (.absoluteString = "…%23anchor", .fragment = nil)
|
|
↑ .ddg_fragmentByteRange → {location=N} — unambiguous '#' position
|
|
Swift URLExtension.opaqueComponents:
|
|
DEBUG → use ddg_fragmentByteRange (exact)
|
|
release → NSString.range(of: "#") fallback (unambiguous for non-data: schemes)
|
|
```
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
[1] The `%23` ambiguity only affects `data:` or `about:` URLs. For `blob:`, `javascript:` the opaque path never legitimately contains `%23` as encoded payload.
|
|
|
|
[2] The primary safety fix is the O(n) crash: replace `String.firstIndex(of: "#")` with a byte-level scan and short-circuit on `data:` scheme or length threshold. The interpose is the accuracy improvement layered on top for DEBUG validation.
|
|
|
|
[3] WebKit bug for NSURL not handling `#` in opaque URLs: [bugs.webkit.org/68089](https://bugs.webkit.org/show_bug.cgi?id=68089). Chromium fixed this in M72 (2019). WebKit's WTF::URLParser is correct; only the Foundation/CFURL bridge is broken.
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
- **Unit tests** (`URLExtensionTests.swift`):
|
|
- `equals(by: .sameDocument)` and `equals(by: .fuzzyIdentity)` on opaque URLs with fragment: `about:blank#section`, `data:text/html,…#anchor`
|
|
- Opaque URLs with query AND fragment: `data:text/html,…?key=value#anchor`
|
|
- Large (20 MB) `data:` URL performance test: comparison must complete in < 0.1 s
|
|
- Diagnostic timing test (`tempOpaqueNSURLComponentTimings`) printing per-operation durations for all Foundation URL APIs on a 20 MB `data:` URL with query and fragment
|
|
|
|
- **Manual test** (DEBUG build): verify `ddg_fragmentByteRange` is set on `WKNavigationAction.request.URL` when navigating to a `data:` URL with a `#` fragment via `WKWebView`.
|
|
|
|
- **Regression**: navigation tests in `DistributedNavigationDelegateTests.swift` and `NavigationTestHelpers.swift` updated to use `.equals(by: .fuzzyIdentity)` in place of removed `.matches()` API.
|
|
|
|
---
|
|
|
|
## Additional Considerations
|
|
|
|
### Privacy
|
|
- No user data collected or transmitted. All processing is local and in-process.
|
|
|
|
### Security
|
|
- No extra data is stored or passed anywhere
|
|
- The approach injects into the existing API and stores a helper pointer
|
|
### Site Breakage
|
|
- The O(n) fix prevents crashes on large URLs. Fragment-detection correctness improves navigation state machine accuracy but does not change whether a navigation is allowed or blocked.
|
|
|
|
### Operational
|
|
- No new infrastructure, network calls, or persisted state. The crash was a SIGKILL watchdog kill visible in Sentry; the fix eliminates the `0x8BADF00D` terminations for this call chain.
|