Back-testing. I hate it - it's just optimizing over history. You never see a bad back-test. Ever. In any strategy. - Josh Diedesch (2014), CalSTRS

Every trading system is in some form an optimization. - Emilio Tomasini (2009)

Many system developers consider

to be adequate.

Instead, strive to:

• understand your business constraints and objectives
• build a hypothesis for the system
• build the system in pieces
• test the system in pieces
• measure how likely it is that you have overfit

• on Windows, you will need Rtools

install.packages('devtools') # if you don't have it installed
install.packages('PerformanceAnalytics')
install.packages('FinancialInstrument')

devtools::install_github('braverock/blotter')
devtools::install_github('braverock/quantstrat')

Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. - John Tukey (1962) p. 13

No matter how beautiful your theory, no matter how clever you are or what your name is, if it disagrees with experiment, it’s wrong. - Richard P. Feynman (1965)

• facts to support or refute

• general tests
  • MSFE/MSFE
  • confusion matrix
  • standard errors
  • Monte Carlo errors
• specific tests
  • tests related to the model
  • tests of the prediction
  • tests of the relationships between variables

• Euclidean Distance
• often squared
• or de-meaned
• or lagged to line things up
• clustering models
• piece-wise linear decomposition
• mean squared forecast error

Signals are often combined:

This is a composite signal, and serves to reduce the dimensionality of the decision space.

A lower dimensioned space is easier to measure, but is at higher risk of overfitting.

Avoid overfitting while combining signals by making sure that your process has a strong economic or theoretical basis before writing code or running tests

Signals make predictions so all the literature on forecasting is applicable:

• mean squared forecast error, BIC, etc.
• box plots or additive models for forward expectations
• "revealed performance" approach of Racine and Parmeter (2012)
• re-evaluate assumptions about the method of action of the strategy
• detect information bias or luck before moving on

add.distribution(strategy.st,
                 paramset.label = 'signal_analysis',
                 component.type = 'indicator',
                 component.label = '_', 
                 variable = list(n = fastMA),
                 label = 'nFAST'
)
#> [1] "macd"

add.distribution(strategy.st,
                 paramset.label = 'signal_analysis',
                 component.type = 'indicator',
                 component.label = '_', 
                 variable = list(n = slowMA),
                 label = 'nSLOW'
)
#> [1] "macd"

start_t<-Sys.time()
out<-applyStrategy(strategy.st , 
                   portfolios=portfolio.st,
                   parameters=list(nFast=fastMA, 
                                   nSlow=slowMA,
                                   nSig=signalMA,
                                   maType=maType),
                   verbose=TRUE)
end_t<-Sys.time()

start_pt<-Sys.time()
updatePortf(Portfolio=portfolio.st)
end_pt<-Sys.time()

#> [1] "Running the backtest (applyStrategy):"
#> Time difference of 0.7516775 secs
#> [1] "trade blotter portfolio update (updatePortf):"
#> Time difference of 0.08920383 secs

chart.Posn(Portfolio=portfolio.st,Symbol=stock.str)
plot(add_MACD(fast=fastMA, slow=slowMA, signal=signalMA,maType="EMA"))

• all strategies have parameters: What are the right ones?
• locate a parameter combination that most closely matches the hypotheses and objectives
• look for stable regions of both in and out of sample performance
• even your initial parameter choices are an optimization, you've chosen values you believe may be optimal
• parameter optimization just adds process and tools to your investment decision

• parameters are all the non-data inputs to your strategy
• they are the knobs and levers that control the model

• good parameters are parsimonious

  • not too many of them
  • not too loosely defined
  • each has a clear impact on the strategy performance
  • each is testable in a backtest

• production strategies have additional parameters that are specific to the production environment

• focus on the major drivers defined by your hypotheses
• if necessary, refine the hypotheses
• use ROC or effective parameter testing (Hastie, Tibshirani, and Friedman 2009) to identify the major drivers in a small training and testing set

• limiting free parameters

  • limits the opportunities to cherry pick lucky combinations
  • reduces the required adjustment for data mining bias (Bailey and López de Prado 2014, Harvey and Liu (2015))
  • reduces number of trials (resulting in faster parameter searches)

• increases your chance of overfitting
• larger numbers of free parameters vastly increase the amount of data you need

• more parameters lowers your degrees of freedom, and

  • increases the chance that your chosen parameters are a false discovery

• goal should be to eliminate free parameters before running parameter optimization

• eliminate parameters where possible
• fix/freeze tertiary parameters next
• can you use production data to set less important parameters empirically?
• is there a model fitting method you could use instead? ( e.g. regression, maximum likelihood, Bayes)

• does this still count as a free parameter?

  • it depends on the complexity of the model/approach used to fit
  • does your fitting model have parameters too?
  • what assumptions are you making?

• small parameter changes lead to small changes in P&L and objective expectations
• out of sample deterioration is not large, on average
• parameter choices have a sound theoretical or economic basis
• parameter variation should produce correlated differences in multiple objectives

• quantstrat parameters are added via add.distribution

  • they are identified with a parameter set (paramset) label via the paramset.label argument
  • the component.type argument tells quantstrat where to look for a variable (inicators, signals, rules, etc.)
  • the component.label argument must match the label used for the component earlier in the strategy
  • the variable argument tells quantstrat what to act on

• relationships (constraints) between parameters are set via add.distribution.constraint

• by default, quantstrat will use a brute force search by combining all parameters
• since the full brute force search space may be too large, you can use the nsamples argument to apply.paramset to sample from the whole parameter space
• more sophisticated sampling may be done using an optimizer (on my todo list to generalize)

• Tomasini and Jaekle (2009) and Pardo (2008) both recommend finding stable parameter regions
• ideally, those regions will be "obvious" from first principles
• you can easily use plot to compare one parameter and one output, less easily 2-3 parameters
• for more complex parameter surfaces, 3-d surfaces or heat maps (e.g. via contourplot) are better

Look Ahead Bias

• directly using knowledge of future events

Data Mining Bias

• caused by testing multiple configurations and parameters over multiple runs, with adjustments between backtest runs
• exhaustive searches may or may not introduce biases

Data Snooping

• knowledge of the data set can contaminate your choices
• making changes after failures without having strong experimental design

Pardo (2008, 130–31) describes the degrees of freedom of a strategy as:

• the number of of observations
• minus all observations used by your indicators

In parameter optimization, we should consider the sum of observations used by all different parameter combinations.

Goal should be to have 95% or more free parameters or 'free observations' even after parameter search.

degrees.of.freedom(strategy = 'macd', portfolios = 'macd', paramset.method='trial')
#> 
#>  Degrees of freedom report for strategy: macd 
#>  Total market observations: 3628 
#>  Degrees of freedom consumed by strategy: 1248 
#>  Total degrees of freedom remaining: 2380 
#>  % Degrees of Freedom:  65.6 %
degrees.of.freedom(strategy = 'macd', portfolios = 'macd', paramset.method='sum')
#> 
#>  Degrees of freedom report for strategy: macd 
#>  Total market observations: 3628 
#>  Degrees of freedom consumed by strategy: 26204 
#>  Total degrees of freedom remaining: -22576 
#>  % Degrees of Freedom:  -622.27 %

• there is a lot more information in the returned object, see ?dof for more details

• to increase degrees of freedom, you may:

  • increase the length of market data consumed by the backtest
  • increase the number of symbols examined
  • decrease the number of free parameters
  • decrease the ranges of the parameters examined
• torture and training data sets should be large enough to still have reasonable statistical confidence when you move to walk forward

• this underlines the difficulty of doing extensive parameter searches on low frequency data

• we do want to test multiple hypotheses
• as we perform more tests, the likelihood of "discovering" an apparently significant result which is actually random increases
• as we perform more tests, this likelihood approaches 100%
• we need to be prepared to detect and correct for this multiple testing bias

Bailey and López de Prado (2014) describes a way of adjusting the observed Sharpe Ratio of a candidate strategy by taking the variance of the trials and the skewness and kurtosis into account.

• corrects for selection bias from multiple testing on the same or related data (the parameter optimization)
• adjusts for non-normality of observed returns

• establishes theoretical maximum Sharpe for a series of related trials

• we have implemented a version with Kipnis (2017) as SharpeRatio.deflated

dsr <- SharpeRatio.deflated(portfolios='macd',strategy='macd', audit=.paramaudit)

• small numbers of trials will result in a small adjustment
• maximum computed Sharpe ratio relies on correlation of trials and moments of the distribution
• p-value is the assumed significance of the adjusted backtest, so low p-value suggests a low probability of overfitting

• Harvey and Liu (2015) proposes another method of correcting for the multiple testing bias
• they propose three methods of adjusting the Sharpe Ratio for multiple related tests
• this research was mostly aimed at the proliferation of equity 'risk premia', but it is applicable to backtests where we have all the information as well
• Jasen Mackie (2016) implemented these methods in R, and
• we have ported those methods to the quantstrat in SharpeRatio.haircut

• estimate the 'true' properties of a distribution from incomplete information
• evaluate the likelihood (test the hypothesis) that a particular result is
  • not the result of chance
  • not overfit
• understand confidence intervals for other descriptive statistics on the backtest
• simulate different paths that the results might have taken, if the ordering had been different

• Laplace was the first to describe the mathematical properties of sampling from a distribution
• Mahalanobis extended this work in the 1930's to describe sampling from dependent distributions, and anticipated the block bootstrap by examining these dependencies
• Monte Carlo simulation was developed by Stan Ulam and John von Neumann (with computation by Françoise Ulam) as part of the hydrogen bomb program in 1946 (Richard Rhodes, Dark Sun, p.304)
• computational implementation of Monte Carlo simulation was constructed by Nicholas Metropolis on the ENIAC and MANIAC machines
• Metropolis was an author in 1953 of the prior distribution sampler extended by W.K Hastings to the modern Metropolis-Hastings form in 1970
• Maurice Quenouille & John Tukey created 'jackknife' simulation in the 1950's
• Bradley Efron described the modern bootstrap in 1979

• White’s Data Mining Reality Check from White (2000) http://www.cristiandima.com/white-s-reality-check-for-data-snooping-in-r/

• bootstrap optimization as an option in LSTM (Vince 2009)

• discussion in Aronson (2006, 230–40)
• using resampled market data, with or without multi-asset dependence, to train or run the system
• k-fold cross validation

• combinatorially symmetric cross-validation (CSCV) and probability of backtest overfitting (PBO) from Bailey et al. (2017)

• resampling entries and exits
  • round turn size, direction, duration are sampled from the trades
  • also resample from any flat periods
  • applied in order to market data as new transactions at the then-prevalent price
• trade expectations in the random-trade model, compared to backtest expectations
  • Drawdowns and tail risk, as in other simulation types

Disadvantages:

• much more complicated to model trade dynamics
• maintaining constraints e.g. max position

Advantages:

• can more closely compare strategy to random entries and exits with same overall dynamic
• creates a distribution around the trading dynamics, not just the daily P&L

• effectively creates simulated traders with the same style as strategy but no skill

• best for modeling "skill vs. luck"

# nrtxsim <- txnsim( Portfolio = "macd"
#                  , n=250
#                  , replacement=FALSE
#                  , tradeDef = 'increased.to.reduced')

wrtxsim <- txnsim( Portfolio = "macd"
                 , n=250
                 , replacement=TRUE
                 , tradeDef = 'increased.to.reduced')

Comments:

• without replacement samples identical number of trades, randomizing start date
• with replacement samples number of trades to get correct total duration
• entry and exit prices are set at the time of each trade, from current market

• the best way to prevent overfitting is via careful experiment design
• if your signal process isn't predictive, you should never get to the point of constructing a full backtest
• avoid free parameters where possible
• use as much data, and as many instruments, as you can

• apply parameter optimization via walk.forward

• consider choice of objective

  • Sharpe ratio
  • minimize drawdown
  • profit to max draw
  • multiple objective optimization
  • default choice of best in-sample generally a bad idea

• be careful about performing walk forward analysis then making changes

• more trials increases bias

#> 
#>  Running the walk forward search: 
#> 
#> Time difference of 28.85891 mins
#>  Total trials: 6939

chart.forward(wfaresults)

• testing period OOS ranks, w/ chosen paramset
• OOS deterioration for best/chosen paramset

Strong hypotheses guard against risk of ruin.

I hypothesize that this strategy idea will make money.

Specifying hypotheses at the beginning reduces the urge to modify them later and: - adjust expectations while testing - revise the objectives - construct ad hoc hypotheses

seek to answer what and why before going too far

• more overfitting work
• modular objective for walk forward and parameter optimization
• optimizer for parameter optimization and walk forward objectives
• more machine learning examples

• backtesting is a process of trial and error (mostly error)
• build a process that is grounded in stating and testing hypotheses
• trials should be kept track of, and counted against your eventual success

• multiple adjustments exist, examine

  • as many as you have data for
  • the ones that make sense for your trade
  • methods that can help you answer questions

• stay skeptical of your results