Risk & portfolios

Hidden Markov models for market regimes: an introduction

Risk & portfolios · Published 2026-10-27

Markets seem to alternate between calmer periods of steady gains and turbulent periods of falling prices and high volatility. These "regimes" cannot be observed directly; we only see returns. Hidden Markov models are a statistical way of inferring which regime the market is probably in. This article introduces the building blocks and how they are used, and misused, in investment research.

Markov chains

A Markov chain is a system that moves between a finite number of states, where the probability of the next state depends only on the current state, not on the path that led to it. That property is called the Markov property. The probabilities are collected in a transition matrix. For two states, it might say that a calm market stays calm the next month with probability 0.95 and turns turbulent with probability 0.05, while a turbulent market stays turbulent with probability 0.80 and returns to calm with probability 0.20.

From such a matrix one can compute the expected duration of each state, 1 / (1 − pstay): 20 months for the calm state and 5 for the turbulent one in the example, and the long-run share of time spent in each. The numbers here are illustrative.

Making the states hidden

In a hidden Markov model (HMM), the chain's state is not observed. Instead, each state produces observations according to its own probability distribution, called the emission distribution. In a market model, the observation might be the monthly return, and each state might have its own normal distribution: the calm state with a positive mean and low volatility, the turbulent state with a negative or lower mean and high volatility.

An HMM is therefore defined by three sets of parameters: the initial state probabilities, the transition matrix, and the parameters of each emission distribution. Lawrence Rabiner's 1989 tutorial, written for speech recognition, remains the standard introduction to the mathematics. In economics, James Hamilton's 1989 regime-switching model of U.S. output growth brought the same ideas into econometrics.

The three classic problems

  1. Evaluation: given the parameters, how likely is the observed sequence of returns? The forward algorithm computes this efficiently by recursively combining transition and emission probabilities. As a by-product it gives the filtered probability of each state at each date, using only data up to that date.
  2. Decoding: given the parameters and the data, what is the most likely sequence of hidden states? The Viterbi algorithm answers this with dynamic programming.
  3. Learning: given only the data, what parameters best explain it? The Baum-Welch algorithm, a form of the expectation-maximisation method, iteratively improves the parameter estimates until the likelihood stops increasing.

A two-state market example

Fitting a two-state Gaussian HMM to monthly equity index returns typically produces one state with low volatility and positive average returns and another with high volatility and weak or negative average returns, with the high-volatility state appearing around well-known crises. The model has not been told about crises; it infers them from the clustering of large moves. Exact parameter values depend on the index, the period and the estimation, so they should be estimated rather than borrowed.

In Python, the hmmlearn library implements Gaussian HMMs with the forward, Viterbi and Baum-Welch algorithms; statsmodels provides Markov-switching regression and autoregression models in Hamilton's tradition.

Using regimes in research

Regime probabilities can feed into strategies in several ways: reducing exposure when the probability of the turbulent state is high, switching between strategies that suit different regimes, or adjusting risk estimates. Our article on market timing models for a momentum strategy discusses regime filters as a defence against momentum crashes. Ang and Bekaert (2002) used regime-switching models to study international asset allocation, finding that correlations and volatilities were higher in the bear regime.

Pitfalls

Smoothed versus filtered probabilities

Many software packages report smoothed state probabilities, which use the whole sample, including data after each date. They produce beautiful charts in which crises are identified precisely, but they are not available in real time. A trading rule must use filtered probabilities computed only from past data, and the parameters themselves should be re-estimated using only past data at each point. Using smoothed probabilities is one of the most common sources of look-ahead bias in regime research.

Label switching and instability

The algorithm does not know which state is "calm"; the labels can swap between estimations. Parameters can also shift as new data arrive, so that a state's meaning changes. Sorting states by volatility after each estimation, and checking stability over time, helps.

Local optima

Baum-Welch finds a local maximum of the likelihood. Running it from several starting points and keeping the best fit reduces the risk of a poor solution.

Detection lag

Filtered probabilities react only after several observations from the new regime. With monthly data, the model may recognise a turbulent regime only after one or two bad months, which limits its value for avoiding sudden shocks.

How many states?

More states fit the past better but add parameters. Information criteria such as BIC, and out-of-sample tests, help avoid overfitting. Two or three states are typical for market applications.

Takeaway

HMMs give a principled way to describe the regime structure of markets and to quantify uncertainty about the current state. Used with filtered probabilities, rolling estimation and modest expectations about speed, they are a useful research tool. Used with smoothed probabilities on the full sample, they produce the illusion of foresight.

A minimal research workflow

A sound first experiment uses monthly returns of a broad index over several decades. Fit a two-state Gaussian HMM on an initial training window only. Then step forward one month at a time: update the filtered probability of the turbulent state using the parameters estimated so far, record it, and re-estimate the parameters periodically with an expanding window. At the end, the series of filtered probabilities is a genuine real-time record that can be tested as a signal.

Sources