tensortrade-rl

USE FOR:

• "train RL agent to trade"
• "reinforcement learning for algorithmic trading"
• "PPO / DQN / A3C trading agent"
• "custom trading environment (gym-style)"
• "backtesting with RL agent"
• "distributed RL training with Ray"
• "position-based reward function trading"

tags: [RL, reinforcement-learning, trading, PPO, DQN, Ray, RLlib, backtesting, gym, TensorTrade, algorithmic] kind: framework category: quant-ml-trading

What Is TensorTrade?

Open-source Python RL trading framework. Builds composable TradingEnv environments to train and evaluate RL agents against market data.

• Repo: https://github.com/tensortrade-org/tensortrade
• Docs: tensortrade.org
• License: Apache 2.0
• Python: 3.12+

Research finding: PPO on BTC/USD achieves directional prediction at zero commission, but trading frequency → commissions exceed profits in live conditions.

Core Architecture

┌─────────────── TradingEnv ────────────────────┐
│                                               │
│  Observer      → feature engineering          │
│  ↓                                            │
│  Agent         → RL policy (PPO/DQN/etc.)     │
│  ↓                                            │
│  ActionScheme  → BUY / SELL / HOLD orders     │
│  ↓                                            │
│  Exchange      → simulated execution + fees   │
│  ↓                                            │
│  Portfolio     → wallet + position tracking   │
│  ↓                                            │
│  RewardScheme  → position-based returns       │
└───────────────────────────────────────────────┘

Installation

python3.12 -m venv tensortrade-env
source tensortrade-env/bin/activate
pip install -e .
pip install -r examples/requirements.txt

Basic Usage

import tensortrade.env.default as default
from tensortrade.feed.core import Stream, DataFeed
from tensortrade.oms.exchanges import Exchange
from tensortrade.oms.services.execution.simulated import execute_order
from tensortrade.oms.wallets import Wallet, Portfolio

# Define exchange and portfolio
coinbase = Exchange("coinbase", service=execute_order)(
    Stream.source(price_data, dtype="float").rename("USD-BTC")
)
portfolio = Portfolio(USD, [Wallet(coinbase, 10000 * USD), Wallet(coinbase, 0 * BTC)])

# Build environment
env = default.create(
    portfolio=portfolio,
    action_scheme="managed-risk",     # BUY/SELL/HOLD with risk sizing
    reward_scheme="risk-adjusted",    # Sharpe-ratio based reward
    feed=DataFeed([price_stream, volume_stream, rsi_stream]),
    window_size=20,
)

Training with Ray RLlib

from ray import tune
from ray.rllib.algorithms.ppo import PPOConfig

config = (
    PPOConfig()
    .environment(env_class, env_config={"window_size": 20})
    .training(lr=1e-4, gamma=0.99, train_batch_size=4000)
    .rollouts(num_rollout_workers=4)
)

tune.run("PPO", config=config.to_dict(), stop={"episodes_total": 500})

Hyperparameter Optimization (Optuna)

# examples/training/train_optuna.py
# Searches: lr, gamma, clip_param, window_size, reward_scheme

Action Schemes

Reward Schemes

Key Research Insights

Pre-built Examples

# Simple training run
python examples/training/train_simple.py

# Distributed training (Ray)
python examples/training/train_ray.py

# Hyperparameter search
python examples/training/train_optuna.py

# Use pre-tuned config
python examples/training/train_best.py