Markov Regime Switching Model for Risk‑On/Risk‑Off Dashboards of Stock Indices

Knowing the market’s current risk state is foundational for portfolio decisions. Basic Markov chain models are a good starting point, but they fall short on three realities of financial data. Using as case study China's CSI300 Index, here’s how our model resolves each one and delivers a robust, reproducible Risk‑On/Risk‑Off signal.

1) Beyond “memoryless”: markets have persistence and aging

• Problem: Standard Markov chains assume memorylessness—the next state depends only on the current state, not on how long we’ve been there. Markets don’t behave that way; regimes “age” and persistence changes with time.
• Our solution: We use a hidden semi‑Markov model (HSMM) with explicit duration modeling. Each regime has a duration distribution (e.g., Negative Binomial), and transitions are allowed to depend on time spent in the state. This breaks the memoryless assumption, curbs flicker, and captures realistic regime persistence.

2) Heavy‑tailed, skewed returns are the norm—not Gaussian

• Problem: Financial returns are fat‑tailed and often skewed, making Gaussian emissions fragile and prone to mislabeling extreme days.
• Our solution: We model returns with robust, flexible emissions (Student‑t or skew‑t, optionally mixtures). This improves state inference under stress, stabilizes the filter against outliers, and yields more reliable Risk‑On/Risk‑Off probabilities.

3) Regime durations aren’t geometric

• Problem: In vanilla HMMs, implied state durations are geometric—too short and unrealistic for markets.
• Our solution: The HSMM’s explicit duration distributions fit the empirical spell lengths for each regime. Long calm periods and clustered turmoil are both captured without forcing ad‑hoc smoothing.

Model overview

• Two hidden regimes
• State 0: Risk-OFF (low mean, high volatility)
• State 1: Risk-ON (higher mean, lower or moderate volatility)
• Observation
• Daily CSI300 log-returns (optionally with a small exogenous feature set)
• Output
• Smoothed posterior regime probability P(state = Risk‑ON | data up to T), then scaled to 0–100 as the Risk-On Index.

Core tools/techniques/algorithms

• Filtering/smoothing
• Hamilton filter (forward pass) and Kim smoother (backward pass) for Markov-switching models.
• For general HMM/HSMM formulation: forward–backward (Baum–Welch) style recursions with modifications for t-emissions and explicit durations.
• Parameter learning
• EM (Expectation–Maximization) with state posteriors from smoothing in E-step and M-step updates for transition matrix, emission parameters, and (if HSMM) duration parameters.
• Visualization
• Smoothed probability series p_t converted to p_mean by optional light smoothing (e.g., short EMA) and re-scaled 0–100.

1) Handling non-Gaussian returns (heavy tails, outliers)

• Emission distribution: Student‑t per regime (robust heavy‑tailed likelihood)
• Each regime r has parameters: mean μ_r, scale σ_r, and degrees of freedom ν_r.
• This reduces outlier influence vs. Gaussian and better fits equity return tails.
• Optional skewness (if needed): skew‑t or a Gaussian scale mixture (practical heavy-tail proxy).
• Practical fitting notes
• E-step: compute posterior responsibilities using t‑pdf for each regime.
• M-step: update μ_r, σ_r, ν_r by maximizing the t‑likelihood (ν_r updated via a 1D numeric step or fixed in a reasonable range).
• Robustness extras (lightweight)
• Winsorize extreme returns at very high quantiles (e.g., 0.1% tails) before fitting to stabilize early iterations.
• Keep ν_r per state bounded (e.g., 3–30) so the model doesn’t collapse to Gaussian or overfit tails.

2) Handling non‑geometric regime durations (breaking memorylessness)

• Issue
• Standard (first‑order) HMM implies geometric state durations via P(stay) = p_ii, which is often unrealistic (clustered/persistent regimes in markets).
• Hidden semi‑Markov Model (HSMM)
• Explicit duration distribution per state (e.g., Negative Binomial or Poisson‑Lognormal) to control dwell time.
• The forward–backward recursions are modified to sum over (state, duration) blocks rather than single steps.
• Result: empirically realistic, fatter dwell-time tails (longer persistence when appropriate).

3) Addressing the memoryless limitation of first‑order chains

• Semi‑Markov solves much of this by letting the probability of staying depend on how long you’ve already been in the state (non‑memoryless dwell times).
• State‑dependent dynamics: Markov‑Switching AR(1)
• In each regime, returns follow r_t = μ_s + φ_s r_{t−1} + ε_t, with ε_t ~ Student‑t(ν_s, 0, σ_s).
• This captures short‑term autocorrelation and volatility structure typical in markets.
• Optional higher‑order structure
• If needed, you can move to a 2nd‑order chain or augment the state with a short history indicator (but MS‑AR + HSMM usually suffices).
• Post‑processing rule for trading clarity (what you saw on the dashboard)
• Require 3 consecutive closes above the 65 threshold before declaring Risk‑ON. This is not part of the Markov model itself; it’s a simple, effective persistence filter to reduce whipsaws.

How p_mean (Risk‑On Index) is produced

• Compute smoothed posterior probability γ_t = P(state_t = ON | r_1:T) via Kim smoother (or HSMM smoother).
• Optionally apply a short EMA to stabilize day‑to‑day noise (light smoothing, not enough to lag materially).
• Scale to 0–100: risk_on_idx_t = 100 × γ_t.
• Threshold of 65 selected for practical signal separation and applied with the 3‑day persistence rule in the dashboard’s guidance text.