Noise Infusion Robustness Testing¶

This notebook demonstrates using Monte Carlo noise infusion to test strategy robustness and detect overfitting to specific historical price patterns.

Concept¶

Noise infusion tests if a strategy is overfit to noise-free historical data by:

1. Adding synthetic noise to price data (perturbing OHLCV)
2. Re-running the backtest on noisy data
3. Repeating N times with different noise realizations
4. Measuring performance degradation

Interpretation:

• Robust strategy: Small degradation (< 20%) → Generalizes well
• Fragile strategy: Large degradation (> 50%) → Overfit to specific patterns

This is analogous to regularization in machine learning - testing generalization beyond training data.

In [ ]:

from decimal import Decimal

import matplotlib.pyplot as plt
import numpy as np
import polars as pl

from rustybt.optimization import NoiseInfusionSimulator

# Set random seed for reproducibility
np.random.seed(42)

from decimal import Decimal import matplotlib.pyplot as plt import numpy as np import polars as pl from rustybt.optimization import NoiseInfusionSimulator # Set random seed for reproducibility np.random.seed(42)

1. Generate Sample OHLCV Data¶

Create synthetic price data with realistic properties.

In [ ]:

# Generate 500 bars of daily price data
n_bars = 500

# Generate price series with trend + noise
returns = np.random.normal(0.0005, 0.02, n_bars)  # Daily returns
close_prices = 100 * np.exp(np.cumsum(returns))

# Generate OHLC with proper relationships
high_prices = close_prices * np.random.uniform(1.001, 1.02, n_bars)
low_prices = close_prices * np.random.uniform(0.98, 0.999, n_bars)
open_prices = close_prices * np.random.uniform(0.99, 1.01, n_bars)

# Ensure OHLCV constraints
high_prices = np.maximum.reduce([high_prices, open_prices, close_prices])
low_prices = np.minimum.reduce([low_prices, open_prices, close_prices])

volume = np.random.uniform(1_000_000, 5_000_000, n_bars)

# Create DataFrame
data = pl.DataFrame(
    {
        "timestamp": pl.datetime_range(
            start=pl.datetime(2022, 1, 1),
            end=pl.datetime(2023, 5, 15),
            interval="1d",
            eager=True,
        ),
        "open": open_prices,
        "high": high_prices,
        "low": low_prices,
        "close": close_prices,
        "volume": volume,
    }
)

# Generate 500 bars of daily price data n_bars = 500 # Generate price series with trend + noise returns = np.random.normal(0.0005, 0.02, n_bars) # Daily returns close_prices = 100 * np.exp(np.cumsum(returns)) # Generate OHLC with proper relationships high_prices = close_prices * np.random.uniform(1.001, 1.02, n_bars) low_prices = close_prices * np.random.uniform(0.98, 0.999, n_bars) open_prices = close_prices * np.random.uniform(0.99, 1.01, n_bars) # Ensure OHLCV constraints high_prices = np.maximum.reduce([high_prices, open_prices, close_prices]) low_prices = np.minimum.reduce([low_prices, open_prices, close_prices]) volume = np.random.uniform(1_000_000, 5_000_000, n_bars) # Create DataFrame data = pl.DataFrame( { "timestamp": pl.datetime_range( start=pl.datetime(2022, 1, 1), end=pl.datetime(2023, 5, 15), interval="1d", eager=True, ), "open": open_prices, "high": high_prices, "low": low_prices, "close": close_prices, "volume": volume, } )

In [ ]:

# Visualize price data
plt.figure(figsize=(12, 6))
plt.plot(data["close"].to_numpy(), linewidth=1.5)
plt.title("Synthetic Price Data (500 Days)", fontsize=14, fontweight="bold")
plt.xlabel("Day", fontsize=12)
plt.ylabel("Price ($)", fontsize=12)
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

# Visualize price data plt.figure(figsize=(12, 6)) plt.plot(data["close"].to_numpy(), linewidth=1.5) plt.title("Synthetic Price Data (500 Days)", fontsize=14, fontweight="bold") plt.xlabel("Day", fontsize=12) plt.ylabel("Price ($)", fontsize=12) plt.grid(True, alpha=0.3) plt.tight_layout() plt.show()

2. Define Strategies¶

We'll test two strategies:

A. Robust Strategy (Trend Following)¶

Simple moving average crossover - captures broad trends, not sensitive to noise.

B. Fragile Strategy (Pattern Matching)¶

Over-tuned pattern detector - overfit to specific price sequences, highly noise-sensitive.

In [ ]:

def robust_trend_following_backtest(data: pl.DataFrame) -> dict[str, Decimal]:
    """
    Robust trend-following strategy: Simple moving average crossover.

    Signal: Buy when fast MA crosses above slow MA, sell when crosses below.
    This strategy captures broad trends and is not sensitive to minor price noise.
    """
    # Calculate moving averages
    fast_window = 20
    slow_window = 50

    close = data["close"].to_numpy()

    # Calculate MAs
    fast_ma = np.convolve(close, np.ones(fast_window) / fast_window, mode="valid")
    slow_ma = np.convolve(close, np.ones(slow_window) / slow_window, mode="valid")

    # Align arrays
    alignment_offset = slow_window - fast_window
    fast_ma = fast_ma[alignment_offset:]

    # Generate signals
    signals = np.where(fast_ma > slow_ma, 1, -1)  # 1 = long, -1 = short

    # Calculate returns
    price_returns = np.diff(close[-len(signals) :]) / close[-len(signals) : -1]
    strategy_returns = signals[:-1] * price_returns

    # Calculate metrics
    if len(strategy_returns) > 1:
        mean_return = np.mean(strategy_returns)
        std_return = np.std(strategy_returns, ddof=1)
        sharpe = mean_return / std_return * np.sqrt(252) if std_return > 0 else 0.0
    else:
        sharpe = 0.0

    total_return = np.prod(1 + strategy_returns) - 1

    return {
        "sharpe_ratio": Decimal(str(sharpe)),
        "total_return": Decimal(str(total_return)),
    }


def fragile_pattern_matching_backtest(data: pl.DataFrame) -> dict[str, Decimal]:
    """
    Fragile pattern-matching strategy: Over-tuned to specific sequences.

    This strategy looks for very specific price patterns that are likely
    overfit to historical noise. It will fail when noise is added.
    """
    close = data["close"].to_numpy()

    # Look for very specific pattern: 3-day sequence with exact relationships
    # This is intentionally overfit!
    signals = []
    for i in range(3, len(close)):
        # Check if last 3 days match a specific pattern
        r1 = (close[i - 2] - close[i - 3]) / close[i - 3]
        r2 = (close[i - 1] - close[i - 2]) / close[i - 2]
        r3 = (close[i] - close[i - 1]) / close[i - 1]

        # Overfit condition: very specific thresholds
        if 0.001 < r1 < 0.003 and -0.002 < r2 < 0.001 and 0.002 < r3 < 0.005:
            signals.append(1)  # Buy
        elif r1 < -0.001 and r2 > 0.002:
            signals.append(-1)  # Sell
        else:
            signals.append(0)  # Hold

    # Calculate returns when signal is non-zero
    price_returns = np.diff(close[3:]) / close[3:-1]
    signals = np.array(signals)

    # Only take positions when signal is non-zero
    mask = signals[:-1] != 0
    if mask.sum() > 0:
        strategy_returns = signals[:-1][mask] * price_returns[mask]
    else:
        strategy_returns = np.array([0.0])

    # Calculate metrics
    if len(strategy_returns) > 1:
        mean_return = np.mean(strategy_returns)
        std_return = np.std(strategy_returns, ddof=1)
        sharpe = mean_return / std_return * np.sqrt(252) if std_return > 0 else 0.0
    else:
        sharpe = 0.0

    total_return = np.prod(1 + strategy_returns) - 1 if len(strategy_returns) > 0 else 0.0

    return {
        "sharpe_ratio": Decimal(str(sharpe)),
        "total_return": Decimal(str(total_return)),
    }

def robust_trend_following_backtest(data: pl.DataFrame) -> dict[str, Decimal]: """ Robust trend-following strategy: Simple moving average crossover. Signal: Buy when fast MA crosses above slow MA, sell when crosses below. This strategy captures broad trends and is not sensitive to minor price noise. """ # Calculate moving averages fast_window = 20 slow_window = 50 close = data["close"].to_numpy() # Calculate MAs fast_ma = np.convolve(close, np.ones(fast_window) / fast_window, mode="valid") slow_ma = np.convolve(close, np.ones(slow_window) / slow_window, mode="valid") # Align arrays alignment_offset = slow_window - fast_window fast_ma = fast_ma[alignment_offset:] # Generate signals signals = np.where(fast_ma > slow_ma, 1, -1) # 1 = long, -1 = short # Calculate returns price_returns = np.diff(close[-len(signals) :]) / close[-len(signals) : -1] strategy_returns = signals[:-1] * price_returns # Calculate metrics if len(strategy_returns) > 1: mean_return = np.mean(strategy_returns) std_return = np.std(strategy_returns, ddof=1) sharpe = mean_return / std_return * np.sqrt(252) if std_return > 0 else 0.0 else: sharpe = 0.0 total_return = np.prod(1 + strategy_returns) - 1 return { "sharpe_ratio": Decimal(str(sharpe)), "total_return": Decimal(str(total_return)), } def fragile_pattern_matching_backtest(data: pl.DataFrame) -> dict[str, Decimal]: """ Fragile pattern-matching strategy: Over-tuned to specific sequences. This strategy looks for very specific price patterns that are likely overfit to historical noise. It will fail when noise is added. """ close = data["close"].to_numpy() # Look for very specific pattern: 3-day sequence with exact relationships # This is intentionally overfit! signals = [] for i in range(3, len(close)): # Check if last 3 days match a specific pattern r1 = (close[i - 2] - close[i - 3]) / close[i - 3] r2 = (close[i - 1] - close[i - 2]) / close[i - 2] r3 = (close[i] - close[i - 1]) / close[i - 1] # Overfit condition: very specific thresholds if 0.001 < r1 < 0.003 and -0.002 < r2 < 0.001 and 0.002 < r3 < 0.005: signals.append(1) # Buy elif r1 < -0.001 and r2 > 0.002: signals.append(-1) # Sell else: signals.append(0) # Hold # Calculate returns when signal is non-zero price_returns = np.diff(close[3:]) / close[3:-1] signals = np.array(signals) # Only take positions when signal is non-zero mask = signals[:-1] != 0 if mask.sum() > 0: strategy_returns = signals[:-1][mask] * price_returns[mask] else: strategy_returns = np.array([0.0]) # Calculate metrics if len(strategy_returns) > 1: mean_return = np.mean(strategy_returns) std_return = np.std(strategy_returns, ddof=1) sharpe = mean_return / std_return * np.sqrt(252) if std_return > 0 else 0.0 else: sharpe = 0.0 total_return = np.prod(1 + strategy_returns) - 1 if len(strategy_returns) > 0 else 0.0 return { "sharpe_ratio": Decimal(str(sharpe)), "total_return": Decimal(str(total_return)), }

3. Run Original Backtests (Noise-Free)¶

First, evaluate both strategies on the original data.

In [ ]:

# Test both strategies on original data
robust_result = robust_trend_following_backtest(data)
fragile_result = fragile_pattern_matching_backtest(data)

# Test both strategies on original data robust_result = robust_trend_following_backtest(data) fragile_result = fragile_pattern_matching_backtest(data)

4. Noise Infusion Test: Robust Strategy¶

Test if the trend-following strategy maintains performance with noisy data.

In [ ]:

# Initialize noise infusion simulator
simulator = NoiseInfusionSimulator(
    n_simulations=1000,  # 1000 noise realizations
    std_pct=0.01,  # 1% noise amplitude
    noise_model="gaussian",  # Gaussian noise
    seed=42,  # Reproducibility
)


# Run simulation
robust_noise_result = simulator.run(data, robust_trend_following_backtest)

# Display summary

# Initialize noise infusion simulator simulator = NoiseInfusionSimulator( n_simulations=1000, # 1000 noise realizations std_pct=0.01, # 1% noise amplitude noise_model="gaussian", # Gaussian noise seed=42, # Reproducibility ) # Run simulation robust_noise_result = simulator.run(data, robust_trend_following_backtest) # Display summary

In [ ]:

# Visualize distribution
robust_noise_result.plot_distribution("sharpe_ratio", show=True)

# Visualize distribution robust_noise_result.plot_distribution("sharpe_ratio", show=True)

5. Noise Infusion Test: Fragile Strategy¶

Test if the pattern-matching strategy fails with noisy data.

In [ ]:

# Run simulation
fragile_noise_result = simulator.run(data, fragile_pattern_matching_backtest)

# Display summary

# Run simulation fragile_noise_result = simulator.run(data, fragile_pattern_matching_backtest) # Display summary

In [ ]:

# Visualize distribution
fragile_noise_result.plot_distribution("sharpe_ratio", show=True)

# Visualize distribution fragile_noise_result.plot_distribution("sharpe_ratio", show=True)

6. Comparison: Robust vs Fragile¶

Compare degradation between the two strategies.

In [ ]:

# Compare degradation

# Compare degradation

7. Test Different Noise Levels¶

How does degradation change with noise amplitude?

In [ ]:

# Test multiple noise levels
noise_levels = [0.005, 0.01, 0.02, 0.03]
robust_degradations = []
fragile_degradations = []


for std_pct in noise_levels:
    sim = NoiseInfusionSimulator(
        n_simulations=500,  # Fewer sims for speed
        std_pct=std_pct,
        seed=42,
    )

    robust_res = sim.run(data, robust_trend_following_backtest)
    fragile_res = sim.run(data, fragile_pattern_matching_backtest)

    robust_degradations.append(float(robust_res.degradation_pct["sharpe_ratio"]))
    fragile_degradations.append(float(fragile_res.degradation_pct["sharpe_ratio"]))

# Test multiple noise levels noise_levels = [0.005, 0.01, 0.02, 0.03] robust_degradations = [] fragile_degradations = [] for std_pct in noise_levels: sim = NoiseInfusionSimulator( n_simulations=500, # Fewer sims for speed std_pct=std_pct, seed=42, ) robust_res = sim.run(data, robust_trend_following_backtest) fragile_res = sim.run(data, fragile_pattern_matching_backtest) robust_degradations.append(float(robust_res.degradation_pct["sharpe_ratio"])) fragile_degradations.append(float(fragile_res.degradation_pct["sharpe_ratio"]))

In [ ]:

# Plot degradation vs noise level
fig, ax = plt.subplots(figsize=(10, 6))

noise_pct = [n * 100 for n in noise_levels]

ax.plot(
    noise_pct, robust_degradations, marker="o", linewidth=2, label="Robust Strategy", color="green"
)
ax.plot(
    noise_pct, fragile_degradations, marker="s", linewidth=2, label="Fragile Strategy", color="red"
)

# Add threshold lines
ax.axhline(20, color="orange", linestyle="--", alpha=0.5, label="20% Threshold (Robust/Moderate)")
ax.axhline(50, color="red", linestyle="--", alpha=0.5, label="50% Threshold (Moderate/Fragile)")

ax.set_xlabel("Noise Amplitude (%)", fontsize=12)
ax.set_ylabel("Performance Degradation (%)", fontsize=12)
ax.set_title("Strategy Robustness: Degradation vs Noise Level", fontsize=14, fontweight="bold")
ax.legend(loc="upper left", fontsize=10)
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Plot degradation vs noise level fig, ax = plt.subplots(figsize=(10, 6)) noise_pct = [n * 100 for n in noise_levels] ax.plot( noise_pct, robust_degradations, marker="o", linewidth=2, label="Robust Strategy", color="green" ) ax.plot( noise_pct, fragile_degradations, marker="s", linewidth=2, label="Fragile Strategy", color="red" ) # Add threshold lines ax.axhline(20, color="orange", linestyle="--", alpha=0.5, label="20% Threshold (Robust/Moderate)") ax.axhline(50, color="red", linestyle="--", alpha=0.5, label="50% Threshold (Moderate/Fragile)") ax.set_xlabel("Noise Amplitude (%)", fontsize=12) ax.set_ylabel("Performance Degradation (%)", fontsize=12) ax.set_title("Strategy Robustness: Degradation vs Noise Level", fontsize=14, fontweight="bold") ax.legend(loc="upper left", fontsize=10) ax.grid(True, alpha=0.3) plt.tight_layout() plt.show()

8. Bootstrap Noise Model¶

Compare Gaussian noise with bootstrap noise (preserves return distribution).

In [ ]:

# Test robust strategy with bootstrap noise
sim_bootstrap = NoiseInfusionSimulator(
    n_simulations=1000,
    std_pct=0.01,
    noise_model="bootstrap",  # Bootstrap instead of Gaussian
    seed=42,
)

robust_bootstrap_result = sim_bootstrap.run(data, robust_trend_following_backtest)

# Test robust strategy with bootstrap noise sim_bootstrap = NoiseInfusionSimulator( n_simulations=1000, std_pct=0.01, noise_model="bootstrap", # Bootstrap instead of Gaussian seed=42, ) robust_bootstrap_result = sim_bootstrap.run(data, robust_trend_following_backtest)

Key Takeaways¶

1. Robust strategies (trend following) maintain performance with noise

   • Low degradation (< 20%)
   • Capture broad market patterns
   • Generalize beyond historical data

2. Fragile strategies (pattern matching) fail with noise

   • High degradation (> 50%)
   • Overfit to specific price sequences
   • Don't generalize well

3. Use noise infusion before live trading to validate robustness

   • Similar to regularization in ML
   • Tests strategy generalization
   • Identifies overfitting

4. Noise models:

   • Gaussian: Simple, symmetric noise
   • Bootstrap: Preserves empirical return distribution (fat tails)

5. Interpretation guidelines:

   • Degradation < 20%: Robust ✅
   • Degradation 20-50%: Moderate ⚠️
   • Degradation > 50%: Fragile ❌ (likely overfit)