---
title: 'Tech Design: Opaque URL Fragment Detection for Same-Document Navigation'
created: '2026-06-24'
tags:
  - tech-design
  - apple
  - navigation
  - webkit
  - url
  - crash
status: draft
asana: >-
  https://app.asana.com/1/137249556945/project/1202406491309510/task/1215403820548743
---
**Author:** Apple team
**Project:** fix-fragment-crash (macOS / iOS browser)
**Sentry:** [APPLE-IOS-DY5A](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DY5A/?project=8) · [DZ58](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ58/?project=8) · [DZ5C](https://errors.duckduckgo.com/organizations/ddg/issues/APPLE-IOS-DZ5C/?project=8)

---

## Background & Requirements

The DuckDuckGo browser uses `WKWebView` for all page rendering. Every navigation goes through the `decidePolicyFor` delegate chain managed by `DistributedNavigationDelegate` (BrowserServicesKit). A central piece of that chain is classifying each `WKNavigationAction` as a **same-document navigation** (fragment-only change within the current page, e.g. `#section`) via:

```swift
// WKNavigationActionExtension.swift:184
public var isSameDocumentNavigation: Bool {
    switch navigationType {
    case .linkActivated, .other:
        return !isRedirect && newURL.hasFragment && currentURL.equals(newURL, by: .sameDocument)
    case .backForward:
        return (newURL.hasFragment || currentURL.hasFragment) && currentURL.equals(newURL, by: .sameDocument)
    ...
    }
}
```

`isSameDocumentNavigation` drives the `DistributedNavigationDelegate` state machine: same-document navigations reuse the existing `Navigation` object and trigger client-redirect handling rather than starting a fresh navigation, while non-same-document navigations allocate a new `Navigation`. Misclassification corrupts navigation state.

`hasFragment` and the `.sameDocument` equality both depend on extracting the URL fragment—which is where the bug lives.

### Two URL parsers coexist in WebKit on Apple platforms

| Parser | Used where | Standard | Fragment in opaque URLs (`data:`, `about:`, `blob:`) |
|--------|-----------|----------|------------------------------------------------------|
| **WTF::URLParser** | WebKit rendering, WKWebView navigation decisions | WHATWG URL | ✅ Correctly split at `#` |
| **NSURL / CFURL** | Foundation APIs, Swift `URL` | Old RFC 1808/2396 | ❌ Encodes `#` as `%23`, returns `nil` for `.fragment` |

When WebKit bridges its internal `WTF::URL` to Objective-C it calls:

```c
CFURLCreateAbsoluteURLWithBytes(alloc, rawBytes, len, kCFStringEncodingUTF8, nil, true)
```

The raw bytes at this point still contain the literal `#`. CFURL re-parses them with its old-RFC parser, which does **not** recognise `#` as a fragment delimiter in opaque-scheme URLs. It encodes `#` → `%23` and stores the whole thing as the opaque path.

Result for `data:text/html,hello#anchor`:

| Property | WTF::URL | NSURL / Swift URL |
|----------|----------|-------------------|
| `.path` | `text/html,hello` | `""` (empty) |
| `.query` | nil | nil |
| `.fragment` | `"anchor"` | nil |
| `.absoluteString` | `data:text/html,hello#anchor` | `data:text/html,hello%23anchor` |

---

## Problem Statement

A pathological URL—a multi-MB `data:` URL or attacker-crafted href—reaches `WKNavigationAction.isSameDocumentNavigation` on the main thread. The call chain:

1. `WebKit::WebPageProxy::decidePolicyForNavigationAction` dispatches to the main-thread delegate.
2. `DistributedNavigationDelegate` reads `wkNavigationAction.isSameDocumentNavigation`.
3. `isSameDocumentNavigation` calls `newURL.absoluteString.hashedSuffix` (and for `.backForward`, also `currentURL.absoluteString.hashedSuffix`).
4. `hashedSuffix` → `hashedSuffixRange` → `String.firstIndex(of: "#")` (`StringExtension.swift:67`) iterates grapheme-by-grapheme over `absoluteString`.
5. For a multi-MB `data:` URL this is O(n) UTF-8 grapheme decoding on the **main thread**, per navigation decision.
6. Main thread blocked >10 s under thermal-state `serious` at 100% CPU → RunningBoard scene-update watchdog kills the app: **SIGKILL, terminationReason `0x8BADF00D`**.

The fix must:
1. Make `hashedSuffixRange` / `isSameDocumentNavigation` fast for arbitrarily large URLs (no O(n) grapheme scan).
2. Correctly detect fragments in opaque URLs where NSURL has already encoded `#` as `%23`.

---

## Why Standard APIs Can't Help: The Information-Loss Chain

Fragment information is **irreversibly lost** between WebKit's internal representation and what Swift code receives in `decidePolicyFor`:

```
WTF::URL  (WHATWG-compliant; m_string = "data:…#anchor", m_queryEnd before '#')
    ↓  URL::createNSURL() → createCFURL()
    ↓  CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor")   ← '#' still visible here
    ↓  CFURL re-parses with old-RFC parser; encodes '#' → '%23'
NSURL  (.absoluteString = "…%23anchor",  .fragment = nil,  .path = "")
    ↓  [NSMutableURLRequest initWithURL: url().createNSURL()]   (ResourceRequestCocoa.mm)
NSURLRequest  (same NSURL)
    ↓
WKNavigationAction.request.URL          ← what Swift sees
WKNavigationAction._originalURL         ← same path, same loss (createNSURL() again)
```

Every Foundation/Swift API is downstream of the encoding step:

- **`URL.fragment` / `.path` / `.query`** — return `nil` / `""` for all opaque URLs.
- **`URLComponents(url:)` / `(string:)` and `NSURLComponents` equivalents** — either use NSURL's already-broken cached parse tree, or re-parse `absoluteString` where `#` is already `%23`. RFC 3986 treats `%23` as a literal character, not a delimiter. Fragment stays nil in all four variants.
- **`WKNavigationActionPrivate.h`** (`_originalURL`, `_isRedirect`, `_hitTestResult`, …) — all URL-returning properties go through `WTF::URL::createNSURL()` → same `createCFURL()` path. No private API exposes the raw WTF string.

There is **no Foundation or WebKit API**, public or private, that recovers the original `#` once the NSURL has been constructed.

---

## Approaches Tried

### 1. Scan `absoluteString` for `%23`

Since NSURL encodes the original `#` as `%23` in `absoluteString`, scanning for `%23` as a fragment delimiter recovers the fragment in many cases. Implemented and partly working.

**Fundamental problem:** `%23` is also valid payload data in `data:` URLs:
- SVG fill colors: `data:image/svg+xml,<rect fill="%23ff0000"/>` — `%23` is `#` in the SVG
- HTML anchor hrefs: `data:text/html,<a href="%23section">`
- CSS: `data:text/css,color:%23333`

Both cases produce identical `absoluteString`. They are **indistinguishable from the string alone**. A 4 KB size guard was added to avoid scanning large `data:` URLs, but that makes two different large data: URLs compare as equal — semantically wrong.

### 2. Private API / `WKNavigationAction._originalURL`

See information-loss chain above. No private API bypasses it; all paths go through `createCFURL()`.

---

## Recommended Approach: `CFURLCreateAbsoluteURLWithBytes` Interpose

### Key Insight

WebKit calls `CFURLCreateAbsoluteURLWithBytes()` at the **exact** moment it converts a `WTF::URL` to a `CFURL`/`NSURL`. The raw byte buffer at that call site still contains the literal `#`, before CFURL encodes it. Intercepting this C function gives unambiguous access to the original bytes.

### Mechanism: `DYLD_INTERPOSE`

The `__DATA,__interpose` Mach-O section replaces any C function in the same dylib at load time, without method swizzling or private SPI:

```c
#define DYLD_INTERPOSE(_replacement, _replacee) \
    __attribute__((used)) \
    static struct { const void *replacement; const void *replacee; } \
    _interpose_##_replacee \
    __attribute__((section("__DATA,__interpose"))) = \
    { (const void *)&(_replacement), (const void *)&(_replacee) }
```

### Implementation

Files:
- `CFURLCreateAbsoluteURLWithBytesInterpose.m` — DEBUG-only (`#ifdef DEBUG`)
- `NSURL+FragmentByteRange.h` / `.m` — category exposing the stored range

**Interpose logic:**

1. Call the real `CFURLCreateAbsoluteURLWithBytes` to get the `CFURLRef`.
2. For opaque URLs (scheme present, no `//` authority), scan the raw byte buffer for `'#'` (byte `0x23`) with a simple C loop—O(n) but done once at NSURL construction, not per navigation decision.
3. If found, store the byte range as an `NSValue` associated object on the bridged `NSURL` via `objc_setAssociatedObject`.
4. Callers read `NSURL.ddg_fragmentByteRange` for the unambiguous `#` position.

```objc
if (relativeURLBytes[i] == '#') {
    NSRange fragmentRange = NSMakeRange(i, length - i);
    NSURL *nsurl = (__bridge NSURL *)result;
    objc_setAssociatedObject(nsurl, &kFragmentByteRangeKey,
                             [NSValue valueWithRange:fragmentRange],
                             OBJC_ASSOCIATION_RETAIN_NONATOMIC);
}
```

### Why This Is Correct

- Bytes are scanned **before** CFURL encodes `#` → `%23` — no ambiguity with payload `%23`.
- Associated object is set on the **exact NSURL instance** for that WTF::URL.
- `ddg_fragmentByteRange.location` matches WTF::URL's internal `m_queryEnd + 1`.
- Works for all opaque schemes: `data:`, `about:`, `blob:`, `javascript:`.
- DEBUG-only: zero production overhead; interpose section absent from release builds.

### Complete Call Flow with Interpose

```
WTF::URL  (m_string = "data:…#anchor",  m_queryEnd before '#')
    ↓  URL::createCFURL()
    ↓  CFURLCreateAbsoluteURLWithBytes(rawBytes = "…#anchor")
         ↑↑ INTERPOSED: scan rawBytes → store NSRange on result NSURL
    ↓  NSURL  (.absoluteString = "…%23anchor",  .fragment = nil)
         ↑ .ddg_fragmentByteRange → {location=N}  — unambiguous '#' position
Swift URLExtension.opaqueComponents:
    DEBUG   → use ddg_fragmentByteRange (exact)
    release → NSString.range(of: "#") fallback (unambiguous for non-data: schemes)
```

---

## Notes

[1] The `%23` ambiguity only affects `data:` or `about:` URLs. For `blob:`, `javascript:` the opaque path never legitimately contains `%23` as encoded payload.

[2] The primary safety fix is the O(n) crash: replace `String.firstIndex(of: "#")` with a byte-level scan and short-circuit on `data:` scheme or length threshold. The interpose is the accuracy improvement layered on top for DEBUG validation.

[3] WebKit bug for NSURL not handling `#` in opaque URLs: [bugs.webkit.org/68089](https://bugs.webkit.org/show_bug.cgi?id=68089). Chromium fixed this in M72 (2019). WebKit's WTF::URLParser is correct; only the Foundation/CFURL bridge is broken.

---

## Testing

- **Unit tests** (`URLExtensionTests.swift`):
  - `equals(by: .sameDocument)` and `equals(by: .fuzzyIdentity)` on opaque URLs with fragment: `about:blank#section`, `data:text/html,…#anchor`
  - Opaque URLs with query AND fragment: `data:text/html,…?key=value#anchor`
  - Large (20 MB) `data:` URL performance test: comparison must complete in < 0.1 s
  - Diagnostic timing test (`tempOpaqueNSURLComponentTimings`) printing per-operation durations for all Foundation URL APIs on a 20 MB `data:` URL with query and fragment

- **Manual test** (DEBUG build): verify `ddg_fragmentByteRange` is set on `WKNavigationAction.request.URL` when navigating to a `data:` URL with a `#` fragment via `WKWebView`.

- **Regression**: navigation tests in `DistributedNavigationDelegateTests.swift` and `NavigationTestHelpers.swift` updated to use `.equals(by: .fuzzyIdentity)` in place of removed `.matches()` API.

---

## Additional Considerations

### Privacy
- No user data collected or transmitted. All processing is local and in-process.

### Security
- No extra data is stored or passed anywhere
- The approach injects into the existing API and stores a helper pointer
### Site Breakage
- The O(n) fix prevents crashes on large URLs. Fragment-detection correctness improves navigation state machine accuracy but does not change whether a navigation is allowed or blocked.

### Operational
- No new infrastructure, network calls, or persisted state. The crash was a SIGKILL watchdog kill visible in Sentry; the fix eliminates the `0x8BADF00D` terminations for this call chain.