Home » Research Blog » Systematic trading strategies: fooled by live records

Systematic trading strategies: fooled by live records

Allocators to systematic strategies usually trust live records far more than backtests. Given the moral hazard issues of backtesting in the financial industry, this is understandable (view post here). Unfortunately, for many systematic strategies live records can be even more misleading. First, the survivor bias in published live records is worsening as the business has entered the age of mass production. Second, pronounced seasonality is a natural feature of many single-principle trading strategies. This means that even multi-year live records have very wide standard deviations across time depending on the conditions for the strategy principle. If one relies upon a few years’ of live PnL the probability of investing in a losing strategy or discarding a strong long-term value generator is disturbingly high. This suggests that the use of live record as allocation criterion, without sound theoretical reasoning and backtesting, can be highly inefficient.

The below is a summary of proprietary research of Macrosynergy LLP.

What is wrong with live records?

For a trading strategy live record means accounted profit and loss (PnL) accrued by actually managing risk capital according to the rules of this strategy. Unlike backtests, live records realistically measure execution costs and slippage (difference between expected and actually traded prices), and cannot be modified with hindsight. Hence, live records enjoy more credibility with allocators and are arguably the most persuasive argument for putting money into a strategy.

For many types of trading books, particularly those based on short-term trades, a live record of three years or so is indeed a meaningful guide to the quality of the underlying process. Unfortunately, for most single-principle macro trading strategies it is not. Many valid macro strategies focus on a single asset class and a single guiding principle, such as value, trends or risk premia. Core positions can prevail for weeks or months. While these strategies may or may not be legitimate value creators, their medium-term live records are often misleading, due to two common problems:

  • Survivorship bias is “a type of selection bias where the results, or survivors, of a particular outcome are disproportionately evaluated. Those who ‘failed’, or did not survive, might even be ignored” (Williams and Khim). This is a well know issue for trading strategy selection. According to common practice in the financial industry, strategies with strong live performance are more likely to be shown around than those with poor performance.
    Unfortunately, survivorship bias has become a lot more severe since algorithmic strategies have gone into ‘mass production’. These days it is not difficult or expensive to code up a range of different versions for a trading strategy and trade them at least on modest amounts of capital. With larger institutions being able to run 50-100 strategies live at one time, at least a few of them should produce a positive live performance over several years, even if purely by chance.
  • Seasonality is an even greater problem for live records. Seasonality here means fluctuations in strategy performance due to prolonged periods, maybe years, that are favourable or unfavourable for an investment style. This is not to be confused with calendar seasonality. For example, carry strategies can only work well when a significant amount of carry is being offered and when it is actually an indicator of a premium being paid to investors. This, in turn, depends on the global economic environment and monetary policies across countries.
    By nature, most single-principle macro strategies are seasonal, as they require a specific principle to be relevant and to supersede other market drivers. Also, seasonality can often be self-defeating, since as when a trading principle have performed well over a number of years they tend to attract crowds of investors, resulting in over-positioning and erosion of their available premia.

Relying single-mindedly on live records under seasonality means that [1] many strategies with positive long-term expected value are being discarded, [2] many allocations will go in strategies with no positive expected value, and [3] allocations will tend to be poorly timed and probably underperform the average return of all valid systematic strategies on offer. Put simply, reliance on live records alone, ignoring all related backtesting, is a form of information inefficiency.

The importance of seasonality: a practical example

We apply the analysis of seasonality to a (proprietary) global FX strategy that is (broadly speaking) based on economic trend differentials between 28 developed and emerging market currencies. Although we are confident of having a good-quality measurement of economic trends and their relevance for exchange rate trends, we must accept that we are greatly uncertain about the expected return of a simple version of such a strategy. Hence, we apply a Bayesian estimation procedure, which models this uncertainty explicitly, in order to assess the probability distribution for both the mean and standard deviation of the strategy performance over a medium-term (3 years) horizon.

  • First we set some sceptical prior beliefs as to the plausible performance of the strategy. Specifically, here we suppose that for a 7.5% annual volatility target the mean annualized return should be in a range of -3% to 8%.
  • Then we generate 3-year performance data based on backtests of four simple non-optimized versions of the strategy principle with the sample periods 2000-2019.
  • Finally, we use these 3-year performance observations to estimate the posterior distribution (probability distribution under consideration of the data and the sceptical priors) for the mean of the annualized returns and the 3-year mean’s standard deviation across different 3-year periods (seasonality).

This Bayesian estimation strongly suggests that the strategy will deliver a positive average annualized return in the long run, provided some basic structural stability. Put differently, the probability that the strategy has a positive expect value is very high, above 99%. As the returns are above Libor and net of transaction costs and management fee, and given that the strategy’s historical correlation with the S&P500 has been just 10%, the strategy principle is a strong investment proposition.

However, the Bayesian analysis also shows that the strategy has considerable seasonality. This is in the nature of such a strategy as it requires actual economic divergences to occur, which is more likely in times of economic turmoil, after a financial crisis or in the wake of big commodity price changes.

The below “levelplot” visualizes the Bayesian probability distribution of both the mean and the standard deviations of 3-year annualized returns in one single graph. This is maybe the most realistic single graphical representation for characterizing the commercial properties of a strategy. Roughly speaking it suggests the mean of the medium-term return should be between 3.5% and 6.5% and its standard deviation between 3.5% and 5.5%. Note that the latter is the standard deviation of 3-year annualized returns, i.e. the seasonality, not the annualized standard deviation of daily returns that are used for Sharpe ratios.

The main consequence of such seasonality is that medium-term strategy performance of in-season periods is very different from off-season periods. The below chart shows a Bayesian simulation of mean and seasonality for a large number of instances of annualized return over three-year periods. Using a 95% confidence interval we should expect the actual three-year return to be in a range of -6% to 16%. The probability of a disappointing return of less than 2% is over 25%, and even the probability of a negative performance is not negligible, at 16%.

Judging the value of a strategy from a single instance of 3-years returns is like judging the revenues of a seaside resort based on a single month without consideration of calendar seasonality. In our example, there is roughly a 25% risk of discarding an investment technology of considerable long-term value. There is also a high risk of overestimating and over-positioning the strategy. Also, similar simulations for strategies without any positive expected value would show significant risks of allocating to loss-making strategies based on live record alone.

How to use trading records judiciously

Live records can provide helpful information for allocation, even to seasonal single-principle strategies, if they are applied judiciously and in conjunction with honest backtesting, rather than in isolation:

  • Live records can qualify the validity of a backest. For periods in which live trading and backtest overlap the record effectively checks if the assumptions upon which the backtest is built, such as trading cost, slippage, and data availability. If the live record confirms the basic performance features of the backtest, the latter is more credible.
  • Within the space of seasonal strategies, live record should only be considered as a criterion for a strategy that has sound plausibility. A basis for the latter would be any of the value-generating principles as explained on the SRSV website. The combination of conclusive reasoning and data evidence reduces the risks of falling for accidental patterns or strategies that have historically produced returns at the expense of massive implied tail risk.
  • Live records should be qualified by the state of the season of the investment principle. For example, if a strategy’s performance depends on concurrent high volatility there is no point in evaluating it based on performance in a low-volatility environment alone.

Related articles