---
source: ~/DuckDuckGo/apple-browsers.git/main/.cursor/rules/abn-experiment-framework.mdc
confidence: 0.9
namespace: work
last_synced: 2026-04-28
alwaysApply: false
---

# A/B/N Experiment Framework

## Overview

The DuckDuckGo browser includes a comprehensive A/B/N experiment framework that enables data-driven feature testing across iOS and macOS platforms. This framework allows you to safely experiment with new ideas while maintaining control groups and measuring impact.

**Reference**: Video of knowledge sharing session: ✓ A/B/N Experiment Framework

## What is A/B/N Testing?

An **A/B/N test** is a method of experimenting with multiple variants of a feature (A, B, ... N) to determine which performs better. It's like traditional A/B testing but scaled up to support more than two groups.

### Use Cases

You can use this framework to explore:
- **UI/UX variations**: Whether blue buttons outperform green buttons
- **Content experiments**: If showing pictures of cats 🐱 or dogs 🐶 boosts user retention
- **Feature comparisons**: Different implementations of the same functionality
- **Performance optimization**: Testing various algorithms or approaches

### When to Use A/B/N Testing

✅ **Use for**:
- Comparing user behavior between two or more feature variants
- Validating hypotheses before rolling out changes to all users
- Safely experimenting with new ideas while maintaining control groups
- Measurable, impactful decisions with clear success metrics

❌ **Don't use for**:
- Simple bug fixes or obvious improvements
- Changes without measurable impact
- Features that can't be easily reversed

⚠️ **Note**: Not every change needs a test—reserve it for measurable, impactful decisions. This is typically decided in collaboration with ODRIs and Data Science.

## Framework Architecture

### Remote Configuration System

A/B/N experiments are supported via **remote configuration** on both macOS and iOS:

- **Sub-features** are used for experiments
- **Parent features** group related experiments
- **Cohorts** define the different variants
- **Weights** control user distribution
- **Targets** allow locale-based segmentation

## Configuration Setup

### 1. Privacy Config Structure

Experiments are defined in the Privacy Configuration with this structure:

```json
"amazingMacroFeature": {
  "state": "enabled",
  "features": {
    "petsPictures": {
      "state": "enabled",
      "description": "This feature shows users pictures of cute pets",
      "targets": [
        { "localeLanguage": "en", "localeCountry": "US" },
        { "localeLanguage": "fr", "localeCountry": "CA" }
      ],
      "cohorts": [
        { "name": "cats", "weight": 1 },
        { "name": "dogs", "weight": 1 }
      ]
    }
  }
}
```

### Configuration Elements

#### **State Options**
- `enabled`: Visible to all users
- `internal`: Visible only to internal users  
- `disabled`: Hidden from all users

#### **Description**
Explains the experiment's purpose for team reference.

#### **Targets** (Optional)
Specify user segments based on locale:
```json
"targets": [
  { "localeLanguage": "en", "localeCountry": "US" },
  { "localeLanguage": "fr", "localeCountry": "CA" }
]
```

#### **Cohorts**
Define experiment variants:
- `name`: Cohort identifier (e.g., "cats", "dogs")
- `weight`: Probability of assignment (normally 1 or 0)

## Client Implementation

### Step 1: Add Feature to PrivacyFeature (BSK)

#### Check for Existing Features

```swift
// In PrivacyFeature enum, check if parent feature exists
public enum PrivacyFeature: String, CaseIterable {
    case amazingMacroFeature
    // ... other features
}

// Add sub-feature to existing enum or create new one
public enum AmazingMacroFeatureSubfeatures: String, CaseIterable {
    case petsPictures
    // ... other sub-features
}
```

#### Add New Features

If the parent feature doesn't exist:
1. Add it to the `PrivacyFeature` enum
2. Create a new sub-features enum
3. Add your sub-feature to the enum

### Step 2: Define Feature Flag

Add your experiment to the local `FeatureFlag` enum:

```swift
public enum FeatureFlag: String, CaseIterable {
    case debugMenu
    case sslCertificatesBypass
    case maliciousSiteProtection
    // ... existing flags
    case petsPictures

    public var cohortType: (any FeatureFlagCohortDescribing.Type)? {
        switch self {
        case .petsPictures:
            return PetsPicturesCohort.self
        default:
            return nil
        }
    }

    public enum PetsPicturesCohort: String, FeatureFlagCohortDescribing {
        case cats
        case dogs
    }

    public var source: FeatureFlagSource {
        switch self {
        // ... other cases
        case .petsPictures:
            return .remoteReleasable(.subfeature(AmazingMacroFeatureSubfeatures.petsPictures))
        }
    }

    public var supportsLocalOverriding: Bool {
        switch self {
        // ... other cases
        case .petsPictures: 
            return true
        }
    }
}
```

#### Key Properties

**`cohortType`**: Links to experiment cohorts enum
- Must conform to `String, FeatureFlagCohortDescribing`
- Defines available variants (cats, dogs)

**`source`**: Defines feature flag toggle location
- `.disabled`: Feature is off
- `.internalOnly`: Internal users only
- `.remoteDevelopment`: Development testing
- `.remoteReleasable`: Production experiments

**`supportsLocalOverriding`**: Enables debug menu overrides
- `true`: Internal users can override cohort assignment
- `false`: No local overrides allowed

### Step 3: Implement Cohort Decision Logic

Request cohort assignment when needed:

```swift
// ✅ CORRECT: Request cohort only when decision is needed
guard let petsCohort = Application.appDelegate.featureFlagger.resolveCohort(for: .petsPictures) as? FeatureFlag.PetsPicturesCohort else { 
    return 
}

switch petsCohort {
case .cats:
    showCatPics()
case .dogs:
    showDogPics()
}
```

⚠️ **Important**: Only request the cohort at the moment it's needed! This ensures accurate assignment and avoids data dilution.

### Step 4: Handle Dynamic Cohort Changes

For features that need to respond to runtime cohort changes:

```swift
private func subscribeToPetsExperimentFeatureFlagChanges() {
    guard let overridesHandler = Application.appDelegate.featureFlagger.localOverrides?.actionHandler as? FeatureFlagOverridesPublishingHandler<FeatureFlag> else {
        return
    }

    overridesHandler.experimentFlagDidChangePublisher
        .filter { $0.0 == .petsPictures }
        .sink { (_, cohort) in
            guard let newCohort = FeatureFlag.PetsPicturesCohort.cohort(for: cohort) else { return }
            switch newCohort {
            case .cats:
                // IMMEDIATELY SHOW A CUTE CAT
            case .dogs:
                // IMMEDIATELY SHOW A CUTE DOG
            }
        }
        .store(in: &cancellables)
}
```

## Metrics and Analytics

### Default Retention Metrics

The framework automatically tracks core engagement metrics without additional configuration:

#### 1. Enrollment Pixel

Tracks when users join experiments:

```
Pixel Name: experiment_enroll_{experimentName}_{cohortName}
Parameters:
- enrollmentDate: Date in ET (YYYY-MM-DD format)
```

#### 2. Search Activity Pixels

Monitors search behavior post-enrollment:

```
Pixel Name: experiment_metrics_{experimentName}_{cohortName}
Parameters:
- metric: "search"
- conversionWindowDays: Time frame (e.g., "1", "5-7")
- value: Number of searches performed
- enrollmentDate: Enrollment date (YYYY-MM-DD)
```

**Predefined Tracking Windows**:
- **Value 1**: Conversion windows [1, 2, 3, 4, 5, 6, 7, 5-7]
- **Values 4, 6, 11, 21, 30**: Conversion windows [5-7, 8-15]

#### 3. App Usage Pixels

Tracks app engagement (launches, foregrounds, etc.):

```
Pixel Name: experiment_metrics_{experimentName}_{cohortName}
Parameters:
- metric: "app_use"  
- conversionWindowDays: Time frame (e.g., "0", "1", "5-7")
- value: Number of app usage events
- enrollmentDate: Enrollment date (YYYY-MM-DD)
```

**Predefined Tracking Windows**:
- **Value 1**: Conversion windows [0, 1, 2, 3, 4, 5, 6, 7, 5-7]
- **Values 4, 6, 11, 21, 30**: Conversion windows [5-7, 8-15]

### Custom Metrics

Track experiment-specific behaviors using `PixelExperimentKit`:

#### Import Required Framework

```swift
import PixelExperimentKit
```

#### Fire Custom Metric Pixels

```swift
// Method 1: Direct pixel firing
func fireExperimentPixel(
    for subfeatureID: SubfeatureID,
    metric: String,
    conversionWindowDays: ConversionWindow,
    value: String
)

// Method 2: Threshold-based pixel firing
func fireExperimentPixelIfThresholdReached(
    for subfeatureID: SubfeatureID,
    metric: String,
    conversionWindowDays: ConversionWindow,
    threshold: NumberOfCalls
)
```

#### Example: Button Click Tracking

```swift
// Track immediate button clicks
PixelKit.fireExperimentPixel(
    for: "petsPictures",
    metric: "adopt_button_clicks",
    conversionWindowDays: 1...1,
    value: "1"
)

// Track threshold-based clicks (fires after 5 clicks)
PixelKit.fireExperimentPixelIfThresholdReached(
    for: "petsPictures",
    metric: "button_clicks",
    conversionWindowDays: 1...7,
    threshold: 5
)
```

#### Custom Metric Examples

```swift
// Form completion tracking
PixelKit.fireExperimentPixel(
    for: "petsPictures",
    metric: "form_completed",
    conversionWindowDays: 1...1,
    value: "true"
)

// Feature adoption tracking
PixelKit.fireExperimentPixel(
    for: "petsPictures", 
    metric: "set_as_default",
    conversionWindowDays: 1...7,
    value: "true"
)

// Error tracking
PixelKit.fireExperimentPixel(
    for: "petsPictures",
    metric: "error_occurred",
    conversionWindowDays: 1...1,
    value: "network_timeout"
)
```

## Experiment Management

### Stop Accepting New Users to a Cohort

To prevent new enrollments while maintaining existing users:

```json
"cohorts": [
  { "name": "cats", "weight": 0 },  // No new users
  { "name": "dogs", "weight": 1 }   // All new users go here
]
```

**Behavior**:
- **Existing users**: Remain in their assigned cohorts
- **New users**: Only assigned to cohorts with weight > 0
- **No enrollment**: Set all weights to 0

### Remove Users from a Cohort

To completely remove a cohort and reassign users:

```json
// Before: Two cohorts
"cohorts": [
  { "name": "cats", "weight": 1 },
  { "name": "dogs", "weight": 1 }
]

// After: Cats cohort removed
"cohorts": [
  { "name": "dogs", "weight": 1 }
]
```

**Behavior**:
- **Existing users**: Automatically reassigned to remaining cohorts
- **New users**: Assigned to available cohorts
- **No cohorts**: No users enrolled if all cohorts removed

### Stop an Experiment

Complete cleanup requires both code and configuration changes:

#### 1. Clean Up Code

```swift
// ❌ Remove experiment-specific logic
// switch petsCohort {
// case .cats:
//     showCatPics()
// case .dogs:
//     showDogPics()
// }

// ✅ Implement final chosen behavior
showDogPics() // Or whatever was determined to be the winner
```

#### 2. Update Configuration

```json
// Remove cohorts or entire sub-feature
"amazingMacroFeature": {
  "state": "enabled",
  "features": {
    // "petsPictures": { ... } // Remove entire sub-feature
  }
}
```

⚠️ **Critical**: Always remove code before removing configuration to prevent runtime errors.

## Development and Testing

### Feature Flag Sources for Development

Control experiment access during development:

#### Internal-Only Testing

```swift
public var source: FeatureFlagSource {
    switch self {
    case .petsPictures:
        // Only internal users see this experiment
        return .internalOnly(.subfeature(AmazingMacroFeatureSubfeatures.petsPictures))
    }
}
```

#### Development Random Assignment

```swift
public var source: FeatureFlagSource {
    switch self {
    case .petsPictures:
        // Internal users get random assignment based on remote config
        return .remoteDevelopment(.subfeature(AmazingMacroFeatureSubfeatures.petsPictures))
    }
}
```

#### Production Release

```swift
public var source: FeatureFlagSource {
    switch self {
    case .petsPictures:
        // All users participate based on remote config
        return .remoteReleasable(.subfeature(AmazingMacroFeatureSubfeatures.petsPictures))
    }
}
```

### Local Overrides for Testing

Enable debug menu overrides for internal testing:

```swift
public var supportsLocalOverriding: Bool {
    switch self {
    case .petsPictures:
        return true  // Enables debug menu cohort selection
    }
}
```

**Usage**:
1. Open debug menu in internal builds
2. Navigate to "Feature Flag Overrides"
3. Select specific cohort for testing
4. App immediately reflects cohort change

## Best Practices

### ✅ DO

```swift
// Request cohort only when needed
guard let cohort = featureFlagger.resolveCohort(for: .petsPictures) as? FeatureFlag.PetsPicturesCohort else { return }

// Use meaningful cohort names
public enum PetsPicturesCohort: String, FeatureFlagCohortDescribing {
    case cats           // Clear, descriptive names
    case dogs
}

// Track relevant custom metrics
PixelKit.fireExperimentPixel(
    for: "petsPictures",
    metric: "adoption_success",
    conversionWindowDays: 1...7,
    value: "true"
)

// Clean up after experiments
// Remove cohort logic and implement winning variant
```

### ❌ DON'T

```swift
// Don't request cohorts unnecessarily
let cohort = featureFlagger.resolveCohort(for: .petsPictures) // ❌ Called too early

// Don't use unclear cohort names  
public enum TestCohort: String {
    case a              // ❌ Unclear what this represents
    case b
}

// Don't forget to clean up
// Leaving experiment code after completion ❌

// Don't remove config before code
// Can cause runtime crashes ❌
```

### 🔒 Security and Privacy

```swift
// Don't log sensitive cohort information
Logger.debug("User assigned to cohort: \(cohort)") // ❌ Potential privacy issue

// Use privacy-safe logging
Logger.debug("Experiment cohort assigned") // ✅ Safe

// Don't store cohort assignments locally
UserDefaults.standard.set(cohort.rawValue, forKey: "cohort") // ❌ Privacy risk
```

## Troubleshooting

### Common Issues

#### Cohort Assignment Not Working
```swift
// Check feature flag setup
guard let cohort = featureFlagger.resolveCohort(for: .petsPictures) else {
    Logger.error("Failed to resolve cohort for petsPictures")
    return
}
```

#### Metrics Not Appearing
```swift
// Verify PixelExperimentKit import
import PixelExperimentKit

// Check subfeature ID matches config
PixelKit.fireExperimentPixel(
    for: "petsPictures", // Must match config exactly
    metric: "test_metric",
    conversionWindowDays: 1...1,
    value: "1"
)
```

#### Debug Menu Not Showing Experiment
```swift
public var supportsLocalOverriding: Bool {
    switch self {
    case .petsPictures:
        return true  // Must be true for debug menu
    }
}
```

### Validation Checklist

- [ ] Privacy config includes correct cohort names and weights
- [ ] FeatureFlag enum properly configured with cohort type
- [ ] Feature flag source matches intended audience
- [ ] Custom metrics fire at appropriate times
- [ ] Local overrides work in debug builds
- [ ] Experiment cleanup plan documented

---

This A/B/N experiment framework provides a robust, scalable solution for data-driven feature development in the DuckDuckGo browser, enabling safe experimentation while maintaining user privacy and providing comprehensive analytics.