Parameter Sensitivity and Stability Analysis¶

This notebook demonstrates how to use SensitivityAnalyzer to:

1. Identify robust vs. sensitive parameters
2. Detect overfitting to specific parameter values
3. Analyze parameter interactions
4. Generate recommendations for robust parameter selection

Why Sensitivity Analysis?¶

After finding "optimal" parameters through optimization, sensitivity analysis helps answer:

• Are these parameters robust? (flat performance surface)
• Are we overfitting? (sharp performance cliffs)
• Do parameters interact? (optimize jointly vs. independently)
• What's a safe range? (stable regions)

In [ ]:

import matplotlib.pyplot as plt
import numpy as np

from rustybt.optimization import SensitivityAnalyzer

%matplotlib inline
plt.style.use("seaborn-v0_8-darkgrid")

import matplotlib.pyplot as plt import numpy as np from rustybt.optimization import SensitivityAnalyzer %matplotlib inline plt.style.use("seaborn-v0_8-darkgrid")

Example 1: Moving Average Crossover Strategy¶

We'll analyze a simple moving average crossover strategy with two parameters:

• fast_period: Fast MA period (days)
• slow_period: Slow MA period (days)

In [ ]:

def moving_average_strategy(params: dict[str, float]) -> float:
    """
    Simulate backtest results for MA crossover strategy.

    In practice, this would run a full backtest.
    Here we use a synthetic function that mimics typical behavior:
    - fast_period: Relatively robust (broad optimum)
    - slow_period: Moderately sensitive (narrower optimum)
    """
    fast = params["fast_period"]
    slow = params["slow_period"]

    # Simulate: optimal around fast=10, slow=50
    # fast_period has gentle surface (robust)
    fast_component = -0.001 * (fast - 10) ** 2

    # slow_period has sharper surface (more sensitive)
    slow_component = -0.005 * (slow - 50) ** 2

    # Add some interaction
    interaction = -0.0001 * (fast - 10) * (slow - 50)

    # Simulate Sharpe ratio
    sharpe = 1.5 + fast_component + slow_component + interaction

    return sharpe


# Test the objective function

def moving_average_strategy(params: dict[str, float]) -> float: """ Simulate backtest results for MA crossover strategy. In practice, this would run a full backtest. Here we use a synthetic function that mimics typical behavior: - fast_period: Relatively robust (broad optimum) - slow_period: Moderately sensitive (narrower optimum) """ fast = params["fast_period"] slow = params["slow_period"] # Simulate: optimal around fast=10, slow=50 # fast_period has gentle surface (robust) fast_component = -0.001 * (fast - 10) ** 2 # slow_period has sharper surface (more sensitive) slow_component = -0.005 * (slow - 50) ** 2 # Add some interaction interaction = -0.0001 * (fast - 10) * (slow - 50) # Simulate Sharpe ratio sharpe = 1.5 + fast_component + slow_component + interaction return sharpe # Test the objective function

Step 1: Initialize Sensitivity Analyzer¶

We'll analyze sensitivity around the "optimal" parameters found by optimization.

In [ ]:

# Suppose optimization found these "optimal" parameters
base_params = {
    "fast_period": 10.0,
    "slow_period": 50.0,
}

# Initialize analyzer
analyzer = SensitivityAnalyzer(
    base_params=base_params,
    n_points=30,  # Sample 30 points per parameter
    perturbation_pct=0.5,  # Test ±50% around base
    n_bootstrap=100,  # Bootstrap iterations for CI
    interaction_threshold=0.05,  # Interaction detection threshold
    random_seed=42,  # Reproducibility
)

# Suppose optimization found these "optimal" parameters base_params = { "fast_period": 10.0, "slow_period": 50.0, } # Initialize analyzer analyzer = SensitivityAnalyzer( base_params=base_params, n_points=30, # Sample 30 points per parameter perturbation_pct=0.5, # Test ±50% around base n_bootstrap=100, # Bootstrap iterations for CI interaction_threshold=0.05, # Interaction detection threshold random_seed=42, # Reproducibility )

Step 2: Run Sensitivity Analysis¶

Analyze each parameter independently (varying one while holding others constant).

In [ ]:

# Perform sensitivity analysis
results = analyzer.analyze(
    objective=moving_average_strategy,
    calculate_ci=True,  # Calculate confidence intervals
)

# Display results
for _param_name, result in results.items():
    if result.confidence_lower and result.confidence_upper:
        pass

# Perform sensitivity analysis results = analyzer.analyze( objective=moving_average_strategy, calculate_ci=True, # Calculate confidence intervals ) # Display results for _param_name, result in results.items(): if result.confidence_lower and result.confidence_upper: pass

Step 3: Visualize Sensitivity Curves¶

1D plots show how performance varies with each parameter.

Interpretation:

• Flat line → Robust parameter (performance insensitive to changes)
• Steep slope → Sensitive parameter (small change = big impact)
• Cliff edge → Overfit risk (optimal on unstable boundary)

In [ ]:

# Plot fast_period sensitivity
fig1 = analyzer.plot_sensitivity("fast_period", show_ci=True)
plt.show()

# Plot slow_period sensitivity
fig2 = analyzer.plot_sensitivity("slow_period", show_ci=True)
plt.show()

# Plot fast_period sensitivity fig1 = analyzer.plot_sensitivity("fast_period", show_ci=True) plt.show() # Plot slow_period sensitivity fig2 = analyzer.plot_sensitivity("slow_period", show_ci=True) plt.show()

Step 4: Analyze Parameter Interactions¶

Do parameters interact? (Does optimizing one depend on the value of the other?)

Interpretation:

• Diagonal bands → Interaction present (optimize jointly)
• Horizontal/vertical bands → No interaction (optimize independently)

In [ ]:

# Analyze interaction between fast_period and slow_period
interaction = analyzer.analyze_interaction(
    param1="fast_period",
    param2="slow_period",
    objective=moving_average_strategy,
)


if interaction.has_interaction:
    pass
else:
    pass

# Plot interaction heatmap
fig3 = analyzer.plot_interaction("fast_period", "slow_period")
plt.show()

# Analyze interaction between fast_period and slow_period interaction = analyzer.analyze_interaction( param1="fast_period", param2="slow_period", objective=moving_average_strategy, ) if interaction.has_interaction: pass else: pass # Plot interaction heatmap fig3 = analyzer.plot_interaction("fast_period", "slow_period") plt.show()

Step 5: Generate Analysis Report¶

Get a comprehensive markdown report with recommendations.

In [ ]:

report = analyzer.generate_report()

report = analyzer.generate_report()

Example 2: Identifying Overfitting¶

Let's create a scenario with an overfit parameter (sharp peak).

In [ ]:

def overfit_strategy(params: dict[str, float]) -> float:
    """Strategy with one robust and one overfit parameter."""
    robust_param = params["robust"]
    overfit_param = params["overfit"]

    # Robust: gentle quadratic
    robust_component = -0.01 * (robust_param - 20) ** 2

    # Overfit: sharp Gaussian peak
    overfit_component = -np.exp(-50 * (overfit_param - 0.05) ** 2)

    return 1.0 + robust_component + overfit_component


# Analyze
overfit_analyzer = SensitivityAnalyzer(
    base_params={"robust": 20.0, "overfit": 0.05},
    n_points=30,
    random_seed=42,
)

overfit_results = overfit_analyzer.analyze(
    objective=overfit_strategy,
    param_ranges={"robust": (10, 30), "overfit": (0.01, 0.10)},
    calculate_ci=False,
)

# Compare stability scores
for _param, result in overfit_results.items():
    pass

# Visualize
fig4 = overfit_analyzer.plot_sensitivity("robust")
plt.title("Robust Parameter (Stable)")
plt.show()

fig5 = overfit_analyzer.plot_sensitivity("overfit")
plt.title("Overfit Parameter (Sensitive - Sharp Peak)")
plt.show()

# Generate report
overfit_report = overfit_analyzer.generate_report()

def overfit_strategy(params: dict[str, float]) -> float: """Strategy with one robust and one overfit parameter.""" robust_param = params["robust"] overfit_param = params["overfit"] # Robust: gentle quadratic robust_component = -0.01 * (robust_param - 20) ** 2 # Overfit: sharp Gaussian peak overfit_component = -np.exp(-50 * (overfit_param - 0.05) ** 2) return 1.0 + robust_component + overfit_component # Analyze overfit_analyzer = SensitivityAnalyzer( base_params={"robust": 20.0, "overfit": 0.05}, n_points=30, random_seed=42, ) overfit_results = overfit_analyzer.analyze( objective=overfit_strategy, param_ranges={"robust": (10, 30), "overfit": (0.01, 0.10)}, calculate_ci=False, ) # Compare stability scores for _param, result in overfit_results.items(): pass # Visualize fig4 = overfit_analyzer.plot_sensitivity("robust") plt.title("Robust Parameter (Stable)") plt.show() fig5 = overfit_analyzer.plot_sensitivity("overfit") plt.title("Overfit Parameter (Sensitive - Sharp Peak)") plt.show() # Generate report overfit_report = overfit_analyzer.generate_report()

Key Takeaways¶

Interpreting Stability Scores¶

Score Range	Classification	Interpretation
> 0.8	Robust	Safe to use, performance stable across range
0.5 - 0.8	Moderate	Use with caution, monitor if parameter changes
< 0.5	Sensitive	Overfit risk, consider alternatives

Overfitting Red Flags¶

1. Low stability score (< 0.5)
2. Sharp performance cliff near optimal
3. High variance across parameter range
4. Steep gradient (rapid performance change)

Recommended Workflow¶

# 1. Optimize parameters
optimal_params = optimizer.optimize(objective, param_space)

# 2. Validate with sensitivity analysis
analyzer = SensitivityAnalyzer(base_params=optimal_params)
results = analyzer.analyze(objective)

# 3. Check for overfitting
sensitive_params = [
    name for name, r in results.items() 
    if r.classification == 'sensitive'
]

if sensitive_params:
    print(f"⚠ Warning: Sensitive parameters: {sensitive_params}")
    # Consider:
    # - Using parameters from stable region
    # - Widening search space
    # - Walk-forward validation

# 4. Analyze interactions
for p1, p2 in combinations(optimal_params.keys(), 2):
    interaction = analyzer.analyze_interaction(p1, p2, objective)
    if interaction.has_interaction:
        print(f"Optimize {p1} and {p2} jointly")

Save Report and Plots¶

Export results for documentation.

In [ ]:

from pathlib import Path

# Create output directory
output_dir = Path("sensitivity_analysis_results")
output_dir.mkdir(exist_ok=True)

# Save report
report_path = output_dir / "sensitivity_report.md"
with open(report_path, "w") as f:
    f.write(report)

# Save plots
analyzer.plot_sensitivity("fast_period", output_path=output_dir / "fast_period.png")
analyzer.plot_sensitivity("slow_period", output_path=output_dir / "slow_period.png")
analyzer.plot_interaction("fast_period", "slow_period", output_path=output_dir / "interaction.png")

from pathlib import Path # Create output directory output_dir = Path("sensitivity_analysis_results") output_dir.mkdir(exist_ok=True) # Save report report_path = output_dir / "sensitivity_report.md" with open(report_path, "w") as f: f.write(report) # Save plots analyzer.plot_sensitivity("fast_period", output_path=output_dir / "fast_period.png") analyzer.plot_sensitivity("slow_period", output_path=output_dir / "slow_period.png") analyzer.plot_interaction("fast_period", "slow_period", output_path=output_dir / "interaction.png")