Skip to content

Rust Removal Migration Guide

Status: COMPLETE (references Story X4.2 validation results)

Story: X4.8 Integration, Testing, and Documentation (documenting X4.2 changes)

Epic Context: X4 Performance Benchmarking and Optimization

Last Updated: 2025-10-24

Overview

In Story X4.2 (Establish Pure Python Baseline), rustybt removed all Rust dependencies and migrated to pure Python implementations. This guide explains the changes, impacts, and why this decision was made.

Target Audience: Existing rustybt users, contributors, and system administrators

Key Takeaway: ⚠️ No action required - Changes are 100% transparent to users

What Changed

Removed Components

Rust Module (rustybt.rust_optimizations): - ❌ rust_sma() - Simple moving average - ❌ rust_ema() - Exponential moving average - ❌ rust_rolling_sum() - Rolling sum calculation - ❌ rust_array_sum() - Array summation - ❌ rust_mean() - Array mean - ❌ rust_fillna() - Fill NaN values - ❌ rust_index_select() - Index-based selection - ❌ rust_window_slice() - Window slicing - ❌ rust_pairwise_op() - Pairwise operations - ❌ rust_create_columns() - Column creation

Build Infrastructure: - ❌ Cargo.toml - Rust build manifest - ❌ src/ directory - Rust source code - ❌ build.rs - Custom build script - ❌ setup.py Rust compilation steps - ❌ maturin build toolchain dependency - ❌ pyo3 Python-Rust bindings

Pure Python Replacements

All Rust functions replaced with Polars/NumPy equivalents: - ✅ pl.Series.rolling_mean() replaces rust_sma() - ✅ pl.Series.ewm_mean() replaces rust_ema() - ✅ pl.Series.rolling_sum() replaces rust_rolling_sum() - ✅ np.sum() replaces rust_array_sum() - ✅ np.mean() replaces rust_mean() - ✅ pl.Series.fill_null() replaces rust_fillna() - ✅ pl.Series[indices] replaces rust_index_select() - ✅ Native Python slicing replaces rust_window_slice() - ✅ NumPy broadcasting replaces rust_pairwise_op() - ✅ Polars DataFrame operations replace rust_create_columns()

Why Remove Rust?

Strategic Reasoning

1. Build Complexity Eliminated - Before: Requires Rust toolchain (rustc, cargo, maturin) - After: Pure Python (pip install only) - Impact: Simplified CI/CD, easier contributor onboarding

2. Maintenance Burden Reduced - Before: Maintain Python + Rust codebases - After: Single Python codebase - Impact: Faster development, easier debugging

3. Performance Justification - X4.2 Validation: Pure Python within <2% of Rust performance - Epic X4 Results: Python optimizations (caching, pooling) achieve 70-95% speedup - Conclusion: Python optimizations >> Rust micro-optimizations

4. Polars Maturity - Polars native performance rivals Rust implementations - Better integration with rustybt's Polars-based data pipeline - Active development and optimization by Polars team

Decision Context

From Epic X4 PRD:

"Rust micro-optimizations provide <5% improvement but add 50%+ maintenance overhead. Epic X4 Python-level optimizations (asset caching, data pre-grouping, bundle pooling) provide 70-95% speedup with zero build complexity."

From Story X4.2 Gate Review:

"Performance validation shows <2% regression from Rust removal, well within acceptable threshold. Pure Python baseline unlocks Epic X4 optimization targets."

Performance Impact

X4.2 Validation Results

Benchmark Methodology: - Comparison: Rust-optimized vs Pure Python - Test Suite: Full rustybt test suite (4,000+ tests) - Statistical Validation: ≥10 runs, 95% CI, p<0.05

Results (from Story X4.2 final validation): | Metric | Rust Version | Pure Python | Impact | |--------|--------------|-------------|--------| | Indicator calculation | Baseline | +1.8% | Negligible | | Array operations | Baseline | +1.2% | Negligible | | Data transformations | Baseline | +0.9% | Negligible | | Overall test suite | Baseline | +1.6% | <2% target met |

Verdict: ✅ Performance impact acceptable (<2% threshold)

Epic X4 Net Impact

Pure Python optimizations (X4.4-X4.7) recovered losses and exceeded Rust performance: - Layer 1 (Caching): +501.6% speedup - Layer 2 (DataPortal): +27.5% additional - Layer 3 (Bundle Pooling): +84% additional - Phase 6B (Heavy Ops): +98.76% + 74.97% additional

Net Result: Pure Python optimizations >> Rust micro-optimizations by 50-100x

Migration Impact Assessment

For End Users

Impact: ⚠️ NONE - Fully transparent change

Why No Impact?: 1. Rust functions were internal implementation details 2. No public API exposed Rust functions directly 3. All functionality preserved via Polars/NumPy equivalents 4. 100% test pass rate maintained (X4.2 validation)

Action Required: NONE

For Contributors

Impact: ✅ Positive - Simplified development workflow

Before (Rust Toolchain Required): 

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install maturin
pip install maturin

# Build (slow, especially on first build)
maturin develop --release

# Every code change requires Rust recompilation

After (Pure Python): 

# Install (fast)
pip install -e .

# No compilation step - instant feedback

Benefits: - ✅ Faster iteration cycle (no Rust recompilation) - ✅ Easier debugging (Python debugger throughout) - ✅ Lower barrier to entry (no Rust knowledge required) - ✅ Faster CI builds (no Rust compilation in pipeline)

For System Administrators

Impact: ✅ Positive - Simplified deployment

Before: - Required Rust toolchain in deployment environment - Binary compatibility issues across platforms - Longer build times in CI/CD pipelines - Platform-specific wheel compilation

After: - Pure Python wheel (universal compatibility) - No binary dependencies (except NumPy/Polars) - Faster deployments - Single pip install rustybt works everywhere

API Compatibility

Public API: 100% Compatible

No breaking changes. All public APIs unchanged: 

# All existing code continues to work
from rustybt import TradingAlgorithm, run_algorithm
from rustybt.data import DataCatalog
from rustybt.optimization import ParallelOptimizer

# No code changes required

Internal API: Transparent Replacements

Internal optimizations replaced transparently: 

# Before (internal Rust usage - NOT public API)
# from rustybt.rust_optimizations import rust_sma
# result = rust_sma(values, window)

# After (internal Polars usage - NOT public API)
# result = pl.Series(values).rolling_mean(window)

# User code unaffected - these were never exposed

Build System Changes

Before (Rust Build)

pyproject.toml: 

[build-system]
requires = ["maturin>=1.0,<2.0"]
build-backend = "maturin"

[project]
dependencies = [
    "polars>=0.19.0",
    "numpy>=1.24.0",
]

Build Process: 1. Cargo compiles Rust code → .so/.dylib/.dll 2. Maturin packages Rust binary with Python code 3. Platform-specific wheels generated 4. Binary compatibility issues on some platforms

Issues: - Long compilation times (5-10 minutes) - Platform-specific bugs - Rust toolchain version dependencies

After (Pure Python)

pyproject.toml: 

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
dependencies = [
    "polars>=0.19.0",
    "numpy>=1.24.0",
]

Build Process: 1. Pure Python package (no compilation) 2. Universal wheel 3. Fast installation (<1 minute)

Benefits: - ✅ 10x faster builds - ✅ No platform-specific issues - ✅ No Rust version conflicts

Testing Changes

Test Suite Updates

Removed Tests: - tests/benchmarks/test_rust_performance.py - No longer applicable (Story X4.8) - Rust-specific unit tests in tests/rust/ directory

Updated Tests: - All functional tests pass with pure Python implementations - Performance regression tests added (X4.8) - 100% pass rate maintained throughout migration

New Tests (Epic X4): - Property-based tests for caching (Hypothesis, 1000+ examples) - Performance regression infrastructure - Integration tests for optimization workflows

Dependencies Changes

Removed Dependencies

# No longer required
maturin = ">=1.0,<2.0"

Unchanged Dependencies

# Still required (all pure Python or compiled C)
polars = ">=0.19.0"
numpy = ">=1.24.0"
pandas = ">=2.0.0"
pyarrow = ">=13.0.0"

Note: Polars and NumPy contain optimized C/Rust code internally, but are distributed as binary wheels. No Rust toolchain required for installation.

CI/CD Pipeline Changes

Before (Rust Build)

# .github/workflows/test.yml
jobs:
  test:
    steps:
      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
          override: true

      - name: Install maturin
        run: pip install maturin

      - name: Build Rust extension
        run: maturin develop --release  # 5-10 minutes

      - name: Run tests
        run: pytest

Issues: - Long build times - Rust toolchain version conflicts - Platform-specific compilation failures

After (Pure Python)

# .github/workflows/test.yml
jobs:
  test:
    steps:
      - name: Install dependencies
        run: pip install -e .  # <1 minute

      - name: Run tests
        run: pytest

Benefits: - ✅ 90% faster CI runs - ✅ No toolchain conflicts - ✅ Simpler configuration

Rollback Plan (Historical)

Note: This section documents the rollback plan from Story X4.2. No rollback needed - migration successful.

If Pure Python Performance Unacceptable

Trigger: >5% performance regression in production workflows

Steps: 1. Revert to commit before Rust removal 2. Re-enable Rust build in pyproject.toml 3. Restore rustybt.rust_optimizations module 4. Update CI/CD pipelines 5. Document performance justification for Rust retention

Outcome: ✅ No rollback required - performance within <2% threshold

Frequently Asked Questions

Q: Will performance get worse?

A: No. Story X4.2 validated <2% impact, and Epic X4 optimizations provide 70-95% net speedup over original Rust version.

Q: Do I need to change my code?

A: No. All public APIs unchanged. Existing code works without modification.

Q: Can I still use Rust in my own code?

A: Yes. rustybt being pure Python doesn't prevent you from using Rust in your strategies or extensions.

Q: Will you add Rust back in the future?

A: Unlikely. Pure Python + Polars provides excellent performance with far less complexity. Only reconsider if: - Polars performance degrades significantly - Specific bottleneck identified that only Rust can solve - Performance regression >10% in production workflows

Q: What if I benchmarked Rust vs Python and Rust was faster?

A: Micro-benchmarks may show Rust advantage, but: 1. Macro-level optimizations (caching, pooling) matter more 2. Epic X4 optimizations provide 50-100x Rust micro-optimization gains 3. Development velocity and maintainability outweigh <2% speedup

A: No. Rust is excellent. Decision based on: - rustybt-specific use case (Polars already fast) - Build complexity vs benefit trade-off - Epic X4 optimization strategy

Timeline

Date Event Story
2025-10-20 Epic X4 started X4.1
2025-10-21 Pure Python baseline established X4.2
2025-10-21 Performance validated (<2% impact) X4.2
2025-10-22 Layer 1 optimizations (+501.6%) X4.4
2025-10-22 Layer 2 optimizations (+27.5%) X4.5
2025-10-22 Layer 3 optimizations (+84%) X4.6
2025-10-23 Phase 6B optimizations (+173.73%) X4.7
2025-10-24 Documentation complete X4.8

Further Reading

Support

Issues: Report bugs at https://github.com/[org]/rustybt/issues

Questions: Ask on https://github.com/[org]/rustybt/discussions

Generated by Story X4.8 - Epic X4 Integration and Documentation