Add trade lifecycle management documentation

docs/trade-lifecycle.md

@@ -0,0 +1,205 @@
+# Trade Lifecycle Management
+
+This document explains how the trading bot manages the lifecycle of a trade, from its initiation to its conclusion, including various states, updates, and handling of special conditions like stop losses and external exchange activities.
+
+## Core Components
+
+The trade lifecycle is primarily managed by the interplay of three core components:
+
+1.  **`src/trading/trading_engine.py`**: Responsible for initiating trade actions (placing entry, exit, stop-loss orders) and interacting with the exchange client. It relies on `TradingStats` for state persistence.
+2.  **`src/trading/trading_stats.py`**: The brain for state management. It uses an SQLite database (`data/trading_stats.sqlite`) to store and update the status of orders and trade lifecycles. Key tables are `orders` and `trades`.
+3.  **`src/monitoring/market_monitor.py`**: Continuously monitors the exchange for events like order fills, cancellations, and external activities. It updates the state in `TradingStats` based on real-world exchange events.
+
+## Database Schema for Lifecycle Tracking
+
+Two main tables in `trading_stats.sqlite` are central to lifecycle management:
+
+### 1. `orders` Table
+Stores information about individual orders placed by the bot or detected.
+Key fields:
+*   `id` (INTEGER, PK): Database ID for the order.
+*   `bot_order_ref_id` (TEXT, UNIQUE): A UUID generated by the bot for internal tracking *before* an exchange ID is available.
+*   `exchange_order_id` (TEXT, UNIQUE): The order ID provided by the exchange.
+*   `symbol` (TEXT): Trading pair (e.g., "ETH/USDC:USDC").
+*   `side` (TEXT): 'buy' or 'sell'.
+*   `type` (TEXT): Order type (e.g., 'market', 'limit', 'stopMarket', 'stopLimit', 'stop_loss_trigger').
+*   `status` (TEXT): Critical for lifecycle. Examples:
+    *   `pending_submission`: Bot intends to submit, not yet confirmed by exchange.
+    *   `open`: Actively working on the exchange.
+    *   `filled`: Order fully executed.
+    *   `partially_filled`: Order partially executed.
+    *   `cancelled`: Order cancelled (by bot, user, or system).
+    *   `rejected`: Exchange rejected the order.
+    *   `expired`: Order expired.
+    *   `pending_trigger`: A conditional order (like a stop loss) whose trigger conditions have not yet been met.
+    *   `failed_submission`: Bot failed to submit the order to the exchange.
+*   `parent_bot_order_ref_id` (TEXT, NULLABLE): Links conditional orders (like SL triggers) to their parent entry order. This helps maintain context, especially for stop losses attached to a specific trade.
+*   `timestamp_created` (TEXT), `timestamp_updated` (TEXT).
+
+### 2. `trades` Table
+Stores information about entire trade lifecycles, which can consist of one or more orders (e.g., entry, TP, SL).
+Key fields:
+*   `id` (INTEGER, PK): Database ID for the trade record.
+*   `trade_lifecycle_id` (TEXT, UNIQUE): A UUID grouping all related activities (entry, exit, adjustments) for a single conceptual trade.
+*   `symbol` (TEXT).
+*   `status` (TEXT): Critical for lifecycle. Examples:
+    *   `pending`: Lifecycle initiated, entry order may be `pending_submission` or `open`.
+    *   `executed`: Entry order has been placed (usually for market orders that are expected to fill quickly).
+    *   `position_opened`: Entry order confirmed filled, position is active on the exchange.
+    *   `position_closed`: Position has been fully closed (via TP, SL, or manual exit).
+    *   `cancelled`: The trade was cancelled before a position was opened, or an open position's management was explicitly cancelled.
+*   `position_side` (TEXT): 'long', 'short'. Becomes 'flat' implicitly when `position_closed`.
+*   `entry_price` (REAL), `current_position_size` (REAL).
+*   `stop_loss_price` (REAL), `take_profit_price` (REAL).
+*   `entry_order_id` (TEXT), `stop_loss_order_id` (TEXT), `take_profit_order_id` (TEXT): Exchange order IDs for the respective parts of the trade.
+*   `realized_pnl` (REAL), `unrealized_pnl` (REAL).
+*   `position_opened_at` (TEXT), `position_closed_at` (TEXT).
+*   `linked_order_table_id` (INTEGER): Foreign key to the `orders` table, linking a specific fill event to this trade record.
+
+## Trade Lifecycle Stages & Status Transitions
+
+Here's a typical flow of a trade and how its status evolves:
+
+**1. Initiation (e.g., User sends `/long ETH 100 sl 1800`)**
+
+*   **`TradingEngine`**:
+    *   Generates a unique `bot_order_ref_id` for the entry order.
+    *   Calls `TradingStats.record_order_placed()`:
+        *   An entry is made in the `orders` table.
+        *   `orders.status` = `pending_submission`.
+        *   `orders.bot_order_ref_id` is set.
+        *   If a stop loss is part of the initial command (`stop_loss_price` provided):
+            *   A separate order may be recorded in `orders` with `type='stop_loss_trigger'`, `status='pending_trigger'`, and `parent_bot_order_ref_id` linking it to the entry order's `bot_order_ref_id`.
+            *   Alternatively, the stop loss might be implicitly tracked against the trade lifecycle.
+    *   Calls `TradingStats.create_trade_lifecycle()`:
+        *   An entry is made in the `trades` table.
+        *   `trades.trade_lifecycle_id` is generated.
+        *   `trades.status` = `pending` (or `executed` if it's a market order).
+        *   `trades.symbol`, `trades.position_side` are set.
+        *   `trades.stop_loss_price` is recorded if provided.
+
+**2. Order Submission to Exchange**
+
+*   **`TradingEngine`**:
+    *   Attempts to place the entry order via `hyperliquid_client`.
+    *   **On Success**:
+        *   Receives `exchange_order_id` from the exchange.
+        *   Calls `TradingStats.update_order_status()`:
+            *   `orders.status` updated to `open` (for limit orders) or a transient status like `submitted_market` (market orders await fill confirmation from `MarketMonitor`).
+            *   `orders.exchange_order_id` is populated.
+        *   The `trades` table entry (created in step 1) remains `pending` or `executed`.
+    *   **On Failure (e.g., insufficient funds, API error)**:
+        *   Calls `TradingStats.update_order_status()`:
+            *   `orders.status` updated to `failed_submission` or `failed_submission_no_data`.
+        *   Calls `TradingStats.update_trade_cancelled()`:
+            *   `trades.status` updated to `cancelled`.
+            *   The lifecycle ends here.
+
+**3. Exchange Acknowledges Order (Order is Live)**
+
+*   **`MarketMonitor`**:
+    *   In its periodic `_update_cached_data()` and `_check_order_fills()` loop, it sees the new order (with `exchange_order_id`) as open on the exchange.
+    *   The `orders.status` is typically `open`.
+
+**4. Entry Order Fill Detection**
+
+*   **`MarketMonitor` (`_check_order_fills`, `_process_filled_orders`)**:
+    *   Detects that the entry `exchange_order_id` is no longer open (implying it was filled or cancelled).
+    *   It verifies against recent fills from the exchange.
+    *   Assuming it was filled:
+        *   Calls `TradingStats.update_order_status()`:
+            *   `orders.status` (for the entry order) updated to `filled`.
+            *   `orders.amount_filled` is updated.
+        *   Calls `TradingStats.update_trade_position_opened()`:
+            *   `trades.status` updated to `position_opened`.
+            *   `trades.entry_price`, `trades.current_position_size`, `trades.position_opened_at` are recorded.
+            *   `trades.entry_order_id` is set to the `exchange_order_id` of the filled entry order.
+            *   If a stop loss was pre-defined (e.g., `stop_loss_price` in the `trades` table), and a `pending_trigger` SL order exists in the `orders` table, it remains in that state, now actively monitored against the open position.
+
+**5. Position is Open and Managed**
+
+*   The trade is now live (`trades.status = 'position_opened'`).
+*   `MarketMonitor` continues its checks:
+    *   **Pending Stop Loss Monitoring (`_activate_pending_stop_losses_from_trades`, `_check_pending_triggers`)**:
+        *   If a `stop_loss_price` is defined in the `trades` table for this `position_opened` lifecycle (or a linked `pending_trigger` order exists in the `orders` table).
+        *   `MarketMonitor` checks the current market price against this `stop_loss_price`.
+        *   **If Triggered**:
+            *   `MarketMonitor` instructs `TradingEngine.execute_triggered_stop_order()` (or a similar method).
+            *   `TradingEngine` places a *new* market or limit stop order on the exchange.
+            *   This new SL order is recorded in the `orders` table by `TradingEngine` (via `TradingStats.record_order_placed()`), likely with `status='open'` and `type='stopMarket'` or `'stopLimit'`.
+            *   `TradingStats.link_stop_loss_to_trade()` is called to update `trades.stop_loss_order_id` with the `exchange_order_id` of this newly placed active SL order.
+            *   The original `pending_trigger` order (if one existed) in the `orders` table is updated to `triggered` or `cancelled_by_activation`.
+
+**6. Position Closure (SL, TP, or Manual Exit)**
+
+*   **Scenario A: Stop Loss Order Fills**
+    *   `MarketMonitor` detects the SL `exchange_order_id` (from `trades.stop_loss_order_id`) has filled.
+    *   Calls `TradingStats.update_order_status()` for the SL order: `orders.status` = `filled`.
+    *   Calls `TradingStats.update_trade_position_closed()`:
+        *   `trades.status` = `position_closed`.
+        *   `trades.realized_pnl`, `trades.position_closed_at`, and exit price details are recorded.
+    *   If a Take Profit order was also active for this lifecycle, `TradingStats.cancel_linked_orders()` is called to cancel it. The TP order's status in `orders` becomes `cancelled_linked_fill` (or similar).
+*   **Scenario B: Take Profit Order Fills**
+    *   Similar to SL fill, but for the TP order. The linked SL order is cancelled.
+*   **Scenario C: Manual Exit Command (e.g., `/exit ETH`)**
+    *   `TradingCommands` calls `TradingEngine.execute_exit_order()`.
+    *   `TradingEngine` places a market order to close the position. This new exit order goes through the `pending_submission` -> `open` (if limit) / `submitted_market` -> `filled` flow in the `orders` table.
+    *   When `MarketMonitor` detects this exit order fill:
+        *   Calls `TradingStats.update_trade_position_closed()`: `trades.status` = `position_closed`.
+        *   `TradingStats.cancel_pending_stop_losses_by_symbol()` or `cancel_linked_orders()` is called to cancel any active SL/TP orders associated with this `trade_lifecycle_id`. Their status in `orders` becomes `cancelled_position_closed` (or similar).
+
+**7. Order Cancellation**
+
+Orders can be cancelled at various stages:
+
+*   **By Bot User (e.g., `/coo` command for "Cancel All Orders" for a symbol)**:
+    *   `TradingCommands` calls `TradingEngine.cancel_all_orders()`.
+    *   `TradingEngine` calls `client.cancel_order()` for each open order.
+    *   `TradingStats.update_order_status()`: `orders.status` becomes `cancelled`.
+    *   If an entry order is cancelled before opening a position, `TradingStats.update_trade_cancelled()` sets `trades.status` to `cancelled`.
+*   **By System (e.g., Orphaned Stop Loss)**:
+    *   `MarketMonitor._cleanup_orphaned_stop_losses()`: If a position (`trades.status = 'position_closed' or 'cancelled'`) has a lingering SL order that is still `open` or `pending_trigger`.
+    *   `MarketMonitor` instructs `TradingEngine` to cancel it on the exchange.
+    *   `TradingStats.update_order_status()` for the SL order: `orders.status` = `cancelled_orphaned`.
+*   **Detected by `MarketMonitor` (e.g., user cancelled on exchange interface)**:
+    *   `MarketMonitor._process_disappeared_orders()`: If an order known to the bot vanishes from the exchange without a fill.
+    *   After checks, if determined to be a cancellation:
+        *   `TradingStats.update_order_status()`: `orders.status` = `cancelled_external` (or a similar status indicating external action).
+        *   If this was a critical entry order, `trades.status` might also become `cancelled`.
+
+## Handling External Trades/Positions (Outside Telegram/Bot Control)
+
+The system is designed to detect and adapt to trading activity that occurs directly on the exchange, not initiated by the bot.
+
+*   **External Fill Detection (`MarketMonitor._check_external_trades`)**:
+    *   `MarketMonitor` fetches recent exchange fills.
+    *   It compares these fills with orders known to the bot (via `bot_order_ref_id` or `exchange_order_id` in the `orders` table).
+    *   **If a fill does not match any bot order**:
+        *   It's treated as an external trade.
+        *   `TradingStats.record_trade()` is used to log this activity. This might:
+            *   Create a new entry in the `trades` table if it opens a new position. The `status` would become `position_opened`, and entry details are estimated from the fill.
+            *   Update an existing `trades` entry if the external fill closes or modifies a bot-known position. For example, if the bot has `trades.status = 'position_opened'` for ETH, and an external sell fill for ETH is detected, `TradingStats.update_trade_position_closed()` would be called.
+*   **Orphaned Position Sync (`MarketMonitor._auto_sync_orphaned_positions`)**:
+    *   **Scenario 1: Exchange has position, Bot doesn't know.**
+        *   `MarketMonitor` sees a position on the exchange that doesn't correspond to any `position_opened` lifecycle in `trades` table.
+        *   It calls `TradingStats.create_trade_lifecycle()` and `TradingStats.update_trade_position_opened()` to create a new record in `trades`, marking it as `position_opened`. Entry price is estimated from recent fills if possible. This ensures the bot is aware of the position for potential future management or correct P&L.
+    *   **Scenario 2: Bot thinks position is open, Exchange doesn't.**
+        *   `MarketMonitor` sees no corresponding position on the exchange for a `trades` entry that has `status = 'position_opened'`.
+        *   It calls `TradingStats.update_trade_position_closed()`, assuming the position was closed externally. Details like exit price might be approximate or marked as "external close."
+
+## Risk Management Integration
+
+While the detailed logic for `_check_automatic_risk_management` in `MarketMonitor` isn't fully elaborated from the snippets, it would interact with this lifecycle:
+
+*   **Modifying Orders**: If risk rules dictate adjusting a stop loss or taking profit, `MarketMonitor` would instruct `TradingEngine` to cancel the existing SL/TP order and place a new one. This involves:
+    *   Cancelling the old order (updating its status in `orders` to `cancelled_risk_adjusted` or similar).
+    *   Placing a new order (new entry in `orders` table, status `open`).
+    *   Updating the relevant `trades` table fields (`stop_loss_price`, `stop_loss_order_id`, etc.).
+*   **Preventing Trades**: If a global risk limit (e.g., max exposure) is hit, `TradingEngine` might be prevented from initiating new trades, or `MarketMonitor` might flag this. The `TradingEngine`'s order placement methods already include checks (e.g., for balance, valid prices) before attempting to submit orders.
+
+## Summary of Key Statuses
+
+*   **`orders.status`**: `pending_submission`, `open`, `filled`, `cancelled` (with variants like `cancelled_external`, `cancelled_linked_fill`, `cancelled_orphaned`, `cancelled_position_closed`), `failed_submission`, `pending_trigger`, `triggered`.
+*   **`trades.status`**: `pending`, `executed`, `position_opened`, `position_closed`, `cancelled`.
+
+This detailed flow ensures that the bot maintains a robust internal state of its trading activities, reconciles with actual exchange events, and handles various scenarios gracefully.