Cross checks – automated strategy robustness tests
Cross checks are optional additional methods that can be applied to every strategy after it is generated and passes the first filters.
They can check strategy robustness from more points of view – by trading it on additional markets, or by using Monte Carlo methods to simulate 100s different equity curves, or even using Walk-Forward optimization or Matrix.
The important thing is that you can use cross check filters to dismiss the strategy if it doesn’t pass the cross-check test. This allows you to create funnels, where strategy is scrutinized by increasingly advanced (and more time demanding) methods, and strategy that fails is automatically dropped.
An example of filtering funnel using cross checks during build
It is up to you how many cross checks you’ll employ and how you’ll configure their filters.
Cross checks are divided into three groups – Basic, Standard, Extensive – depending on how time consuming they are. They are also applied from the simple ones to the more complicated ones. So, if strategy doesn’t pass Cross check 1, it is dismissed and not tested by Cross check 2.
|Note that running cross check on a strategy can take significant time!|
Some cross-check methods make complicated simulations and hundreds or even thousands of backtests of the strategy with different parameters and take thousands time more time than the initial strategy generation and initial backtest.So strategy without any cross-check can be generated (for example) in 0.2 second, but with some cross checks applied it could easily take 10 – 200 seconds per one strategy!
Cross checks can be used also in Retester (without filtering), so you don’t need to use all cross-checks in build mode.
Possible optimal usage of cross-checks described
The optimal cross-check application could be to use:
- Cross check Retest with higher precision
- Cross check Monte Carlo trades manipulation
- Cross check Retest on additional markets
- Optionally cross check Monte Carlo retest methods
StrategyQuant will then perform the following steps for every generated strategy:
- Strategy is randomly generated and tested with fastest “Selected timeframe” precision – this is dependent on your setting, but it is the default2. Strategy is automatically filtered and thrown away (dismissed) if it doesn’t pass your global filters, for example, if it doesn’t have enough trades or if Net Profit < $10003. Cross check Retest with higher precision will retest this strategy with minute or even real tick precision – to make sure the strategy was reliably backtested using the basic precision. Note that only strategies that pass point 2. will get here. If strategy doesn’t pass this first cross-check, it is dismissed.
4. Cross check Monte Carlo trades manipulation will run number of simulations of different equity curves by manipulating the existing trades – to ensure that the original equity curve wasn’t achieved just a luck. You’ll filter out strategies that don’t pass this Monte Carlo test
5. Cross check Retest on additional markets will test the strategy on additional markets or timeframes. If it isn’t profitable on other markets it is filtered out.
6. Optional cross check Monte Carlo retest methods will run number of simulations where each simulation is a new backtest of the strategy using small variations in strategy indicator parameters, trading options such as spread, slippage, or in history data.
Note that every such simulation is an independent backtest, so if it took 0.2 seconds to backtest the strategy in step 1., it will take 100 x 0.2 s. = 20 seconds to finish this cross check for this one strategy if you’ll use 100 Monte Carlo simulations.
If strategy passes also this cross-checks it is saved into databank and you can have quite good confidence that it is robust enough.
You can find more detailed description of cross checks in the section Cross checks – robustness tests and analysis.