Strategy Building Process (E-mini S&P 500 Futures)
Building a profitable emini strategy for ES / TF / EMD
by Mark Fric
In this article I'll explain the complete step-by-step process of building a profitable, robust strategy for ES (E-mini S&P 500 Futures), including multiple steps of different robustness tests.
This is a variation of my older article about Strategy Building Process for forex.
When using machine learning techniques such as genetic programming it is realtively easy to get strategies with nice looking equity curve. The danger lies in curve fitting, so the most important part of strategy building process is testing strategy for robustness to ensure that it is not curve-fitted to historical data.
In this article I'll explain how I use double OOS filter plus Robustness Tests plus Walk-Forward Matrix test.
This is the result
For motivation I post results of a nice performing strategies for ES / TF / EMD that I found using the process described below.
Strategy 238433 for ES/TF/EMD
The strategy above is published on our forum (only for licensed users of StrategyQuant) here:
available for download.
The only inputs I use are my expectations of the strategy - I want to build a strategy for ES (E-mini S&P 500) on 15 Min timeframe that is profitable and has as little drawdowns as possible. I want the strategy to be robust enough so that it works also on other futures (TF, EMD) and I want it to pass Walk-Forward Matrix test to make sure that reoptimization works on this strategy.
Strategy building process
- Obtaining the data
- Generating a big pool of potential candidates
- First filter - Out of sample (OOS) check
- Second filter - retesting and second OOS check
- Third filter - GBPUSD check
- Fourth filter - Robustness tests
- Fifth filter - Walk-Forward matrix test
1. Obtaining the data
There are some differences between futures and forex. First of all, obtaining the data for futures is little bit more difficult and expensive. There are no free data sources
and most brokers don't give you history longer than few months.
You can get the data from broker who offers them (Tradestation, if you are their client) or you have to sign up for a live data service, such as Kinetick or iqFeed.
There are also some special services that dont't offer live data feed, but that sell historical futures data. To find them just search for "historical intraday furtures data" on Google.
Second difference is that futures contracts have expiration date, contracts are usually traded for 3-4 months only and then they are replaced by newer version of the same future contract.
To be able to use the futures data for strategy development we need to have the at least few years of data in form of continuous contracts. Most of data services offer this option, so it is only a matter of subscribing to the data service and downloading the data to your trading platform.
Exporting the data from NinjaTrader
When you already have data in your NinjaTrader you have to copy them to StrategyQuant too, so it can text the generated strategies. To do that,we have to export data from NinjaTrader and import them to StrategyQuant. To export data we have to open the chart for ES 15 Min. Make sure you set the correct trading session. I use CME US Index Futures RTH session in this example.
When the chart is opened just find SQDataExport indicator and place it on the chart. It will export the data for currently opened chart to a text file.
Repeat the same process also for TF (Mini Russel 2000 Futures) and EMD (E-mini S & P Midcap 400) also on 15 min timeframe with correct trading session.
We'll use these data later to test our strategies on other symbols as a form of robustness test.
Then open StrategyQuant and create new symbols for ES, TF and EMD and import the respective data files. Importing data from NinjaTrader is described in more details also in the Users guide.
2. Generating a big pool of potential candidates
In the first step of generation I simply have to generate large pool of potentially "good" strategies that I'll test for robustness later. I want my all initial strategies to be profitable and robust (to some extent), so I employ several filters also in this first phase.
First of all, I generate all my strategies on multiple symbols. My goal is to find a good strategy for ES, but I want my strategy to be robust - so I want it to be profitable also on EMD. I add EMD to the additional data, so now the strategy will be tested on both symbols.
Image 1: Setting the data
I'll use data from 2.1.2003 to 31.12.2012, which is 10 years. The rest of the data will be left for further OOS testing later.
I'll use Genetic Evolution mode.
The idea is to make a population of 200 strategies, evolve them during 30 generations and then start again from scratch.
This way I'll avoid running into a dead end during evolution and the best strategies are continually stored to Databank.
You can also see that the only condition for initial population is that it must make at least 100 trades. It doesn't need to be profitable - genetic evolution should be able to improve it.
Image 2: Genetic options
We could use also Random generation without evolution, but evolution should find the profitable strategies faster.
The last important piece of setting is Ranking options. I set Databank to store 2000 best strategies, because I want to have a good base for further selection process. I also set the selection criteria to Return / Drawdown ratio - this is my favorite. You can use other selection criteria, perhaps you'll get better results.
Image 3: Ranking options
Othe of the most important things is to set the initial filter criteria for strategies in Databank. I want to consider only strategies that are at least $2000 in profit, have Return/DD ratio > 3, at least 300 trades AND Return / DD ratio of a portfolio to be at least 2.5.
Because I'm testing the strategies on two symbols - ES and EMD, the portfolio results for the strategies will be also computed. Using this condition I simply specify that the portfolio performance won't be much worse than performance on only ES, and the program will dismiss all strategies with bad portfolio performance.
Now we just have to hit the Start button and let the program do the work.
Remember, we want to generate at least 2000 "good" strategies before we'll continue with the filter process.
Depending on the settings and speed of your computer it could take several hours or even days, so be patient. If the program doesn't produce any strategy for a very long time, perhaps we should switch to a higher timeframe - 30 min, 1 Hour, or make the constraints less restrictive.
3. First filter - Out of sample (OOS) check
When I have 2000 potentially good strategies in the Databank, I'll stop the generation and start filtering process.
I'll apply the first filter - by removing all the system that have bad Out of Sample performance. I can do it quickly, just by sorting the strategies in Databank and deleting the ones that have OOS profit smaller than $3000.
Image 4 Databank with pool of strategies sorted by OOS Net Profit
This first step usually removes a big portion of strategies, so from initial 2000 candidates we are down to around 1700.
2. Second filter - retesting and second OOS check
In this step I'll retest all the strategies on the unknown Out of Sample period plus I'll add test on TF data.
Retesting the strategies is simple - I'll just select all the strategies in the Databank and click on Retest button. This will move all the strategies to a Retest tab. I'll also confirm dialogue asking if it should use the build settings for Retest
I'll then extend the data period to the end of available data. Strategies were generated on data from 2.1.2003 to 31.12.2012, I'll now retest the strategies on data until 31.12.2013 (one more year not used during generation)
and set Out of Sample period from 31.12.2012 to 31.12.2012.
Note that this will retest the strategies on the full data, and the OOS part will show the strategy performance during the last year of the previously unused data.
Image 5: Settings for retest
Because I have also historical data for TF I'll add them to additional data to compare the performance on all three eminis.
The test could take some time and after it is done I'll once again remove all the system that have bad Out of Sample performance. Again I can sort the strategies in Databank by Net Profit (OOS) and delete the ones that have OOS profit smaller than $1500.
3. Third filter - EMD, TF check
Third filter is visual - I'll check performance of the strategies on EMD and TF symbols. I'll go to Results -> Equity chart, switch chart to Portfolio and go through strategies one by one looking at the equity curves for EMD and TF.
Image 6: Example of good and bad EMD / TF performance
What are we looking for? Because these eminis are highly correlated, I want the strategy to be profitable on all three symbols, just like on the first example. We can see on the second example that performance on TF is very poor compared to ES and EMD, its equity curve is not grodwing, so I'll dismiss such strategies.
There can also be another extreme - that performance on TF and/or EMD is much better than performance on ES. This is ok, it often happens that strategies perform better on TF than on ES.
We should be not looking only at the final performance, but also to equity curves. We should dismiss all equity curves that have long periods of stagnation, or big drawdowns.
This way we can make the filter very though, we should end up with no more than 10-20 remaining top strategies for the next step.
4. Fourth filter - Robustness tests
After removing all the strategies with bad EMD / TF performance there are less than 20 strategies left that have good IS and OOS performance, as well as satisfactory performance on EMD / TF. I'll now retest strategies again with Robustness tests and Money Management to see how each of the strategies handles small changes in inputs and to be able to compare the strategies to each other.
I'll change money management from fixed size to fixed amount, letting every startegy risk $500 per trade. This allows for better strategies comparison, because they risk the same amount per trade.
Image 7: Setting money management to fixed amount
In Robustness tests I use at least 20 simulations and test the strategy for all types of stress situations.
After configuring the robustness test I retest the strategies again.
This time it will be fast because there are only few strategies left in the Databank.
Image 8: Robustness tests
How to evaluate Robustness tests
Robustness tests show us how the strategy can behave in reality, when there are missed trades, different history data etc. I'm looking for strategies that have acceptable values for Net Profit and Drawdown in 95% confidence level.
Image 9: Robustness tests results
In the example above we can see robustness results for two strategies.
Strategy on the left has profit in acceptable level, but drawdown more than doubled compared to original result.
Strategy on the right has also profit in acceptable level and drawdown was almost unchanged.
In this step I'll choose only 1-3 final strategies that will be subjected to next test of robustness.
These final strategies are selected by the best results in robustness tests, overall profitability and also simplicity - I want the strategy rules to be as simple as possible, and trading rules should make some sense.
5. Fifth filter - Walk-Forward Matrix test
We are left with a few strategies and we can run the ultimate test for robustness - Walk-Forward Matrix test. WF Matrix is simply a matrix of walk-forward optimizations with different number of runs and run periods.
If the strategy passes Walk-Forward Matrix test it means that with the help of parameter reoptimization the strategy is adaptable to a big range of market conditions AND also that the strategy is not curve fitted to particular data - since with reoptimization it works on many different time periods.
In addition to this WF Matrix test also tells us if the strategy should be permanently reoptimized and what the most optimal reoptimization period is.
Walk-Forward Matrix test has to be done for every strategy separately. I'll load my strategy to Optimizer and select Walk-Forward Matrix option. I'll also select steps for runs and OOS percentages. StrategyQuant will go through all these combinations, performing Walk-Forward optimization of the strategy.
Image 10: Setting up Walk-Forward Matrix
Setting parameters for optimization
For optimization to make sense you have to set the strategy parameters that will be optimized. Each strategy uses different logic and has different parameters, so you have to configure the optimization differently.
Image 11: Optimizing parameters
This strategy is relatively simple, so I'll optimize only Stop Loss value and stop trailing coefficient.
It is not necessary to optimize all parameters, only these that have the biggest impact on strategy performance.
Evaluating Walk-Forward Matrix
When optimization finished, I'll click on the Walk-Forward matrix result in Databank to see the details.
I want optimizer to give me clear answer if the strategy passed Walk-Forward optimization test, so I have to set up score criteria.
These are simple criteria that have to be true for strategy to pass the test.
Image 13: Walk-Forward Matrix results
The final result is that the startegy passed Walk Forward matrix test for robustness. The 3D score chart shows us that 19 out of 24 combinations passed our criteria (default settings used).
The strategy doesn't need to pass for every combination, I'm looking for 2x2 or 3x3 area that has most of the combinations passed - this will be the group of best reoptimization combinations. In this case, I can see that 10 runs with 30% Out of Sample is one of the best combinations, because it is surrounded by other combinations that also passed.
When I check the Walk-Forward optimization chart I can see that the strategy remains profitable also during reoptimization. The decrease of profitability is in line with tests we got from Monte Carlo robustness analysis, but the strategy is still profitable.
Image 14: Walk-Forward optimization chart
I described my complete process of working with StrategyQuant, which led to few interesting new strategies.
These strateguies are just samples an dit took me less than 2 days of SQ running and about 1-2 hours of my time to find them with the use of StrategyQuant.
You can try it yourself, take inspiration and possibly improve the process with your own ideas which you can share on our forum.
Possible improvements of the process - you can try to look for strategies separately for long and short direction. Every direction has its own dynamics, and different strategies for long and short could return better results.
I didn't mention Improver - it is a powerful tool that allows you to look for better variation of your existing strategy, if you are still not satisfied with the performance.
Just keep in mind - the point is not to find strategy that is perfect on historical data. This is a recipe for disaster, because overly optimized strategy is guaranteed to fail in real trading.
Our goal should be to find a strategy that is robust across different data and/or symbols, because this means it has real edge over the market.