A simulated forward test method

Hereby I want to propose a validation method called ‘simulated forward test‘. (This is not WF or WFM, WF/WFM is a different topic, this is also not an OOS test as used during strategy generation)

Simulated forward test is basically a ‘walk forward’-like test of your ‘total strategy design workflow/method’.

In the last few months I tried to develop a fully automated workflow using SQ-X ‘custom projects’ module and I got very interesting results (which I will post in another topic). My small breakthrough was based on observation that majority of traders, and even experienced traders giving online courses, just assume they are right about their strategy validation method based on their experience and their past results. There is nothing wrong with that assumption, but it can be killing for new traders or people who are trying to design their own strategy design methods.

What you see on this forum and all other places is the following:

1. All traders have their own ideas or methods for generating strategies and they use those methods to generate their new systems.

2. In 90% of cases new traders generate and run their systems on demo (or small live) accounts to see if their strategies are profitable or not.

3. This results in a ‘generate & hope’ methodology.

In my opinion the proper approach should be as follow:

1. At first you develop or learn a new strategy design workflow (a new system design method including strategy generation/testing/validation and strategy selection).

2. In order to test if your method works you go back in time and pretend to be living in the past, eg :2014. This is your point of reference.

3. Then you develop a sufficient (significant) amount of strategies using your workflow using data <=2014. In this case using data >2014 is forbidden to prevent any type of ‘Data Mining Bias’.

4. Then you test ALL strategies that have passed your workflow and strategy selection on a simulated forward period (your simulated future) >2014 and verify how many of selected strategies have worked for you. You do this without cheating or fooling yourself, so you use ALL strategies that have passed your workflow successfully. You draw your conclusions based several different pass/fail criteria that are important for you e.g.: Max DD, Net Profit, Red/DD, Stagnation etc….and write down the effectiveness percentage of your workflow e.g: 65% of selected strategies were profitable (or did not exceeded Max. expected DD) in the simulated future.

5. You repeat the whole process by selecting a different point of reference in the past e.g.: 2016. And start all over again by generating, testing and selecting new strategies. Then you test ALL of selected strategies in a simulated period >2016.

You can repeat this process many times, by ‘walking forward’ your whole workflow using historical data and see if your strategy generation method really produces robust strategies in a simulated forward period.

An example of time slots as used in a typical workflow:

– Point of reference (in the past) = 2014.12.31 (you are forbidden to use data > 2015.01.01 until you have generated 30 strategies that passed your workflow and selection method for live trading, this can be including portfolio building methods)

– Your first IS/OSS for strategy generation is e.g.: IS: 2008.01.01…2011.06.01 and your first OOS is 2011.06.02…2014.12.31. 

– Your second OOS2 for cure-fitting check is e.g.: 2003.01.01…2007.12.31

– All other validation tests (spread,skipping trades, MC, cross market etc..) are done using data <=2014.12.31

– You select 30 strategies and test them ALL on data >2015.01.01

After you are confident with your workflow and your results you can jump-back to present time and generate/validate and select new strategies and feel confident that selected systems have a significant chance of being profitable in the upcoming future.

Just wanted to share this because this approach has led to my first small breakthrough with my 100% automated workflow.

Conclusion: stop fooling yourself, stop generating strategies based on untested methods and stop hoping those will work out for you well in the future…

Any comments are welcome 🙂

Gr

Chris

 

 

 

 

 

 

 

 

 

 

 

Hi Chris, thank you for the insights, as I told you in person already, that is an interesting (and sensible) approach.

Question: Do you do the simulated forward test on all the strategies you generate and live-trade ONLY after strategies pass this test, or did you do it a significant amount of times with your workflow already, until you managed to adjust your workflow to a satisfactory stage (e.g 70% of final strategies pass SFT), and from that point on your just use your workflow on all data until today, and feel confident in it’s profitability?

 

Regards,

Ilya

Yes. Sound an interesting approach. Please, correct me if I’m wrong; I understand that you work in two steps:

Step 1: IS + OOS. It strikes me that you use IS 3,5 years and OOS 9,5 years divided in two parts. Wow! That’s a lot. I’ve read by ‘trusted’ gurus advice of 50 IS + 50 OOS. I’ve read by other ‘trusted’ people not OOS at all, just IS and strong Robustness tests. Well, I see you on 75% OOS. Anyway, you get to find 30 good strategies with your criteria. Now move to Step 2

Step 2: OOS 4 years; 2015 to now.

If I got you correctly. How many of these 30 strategies survive to the second Step? if I may ask

Enric just a remark, when using OOS while generating, that’s essentially an IS, for the mere fact we run through millions (or tens of millions or hundreds of millions in my case) of strategies, to find those that has a profitable OOS, landing the strategy in the databank. So essentially in the example it’s more like 7 years of IS and 9 OOS. Still not a common combination 🙂

 

Ilya

Ilya wrote:

Hi Chris, thank you for the insights, as I told you in person already, that is an interesting (and sensible) approach. Question: Do you do the simulated forward test on all the strategies you generate and live-trade ONLY after strategies pass this test, or did you do it a significant amount of times with your workflow already, until you managed to adjust your workflow to a satisfactory stage (e.g 70% of final strategies pass SFT), and from that point on your just use your workflow on all data until today, and feel confident in it’s profitability? Regards, Ilya

I use SFT only on ‘workflow level’, so each time I select a new reference point to test my workflow e.g.: 2014,2015,2016 or today, I generate several new strategies from scratch using exactly the same workflow/method.. If I see that majority of strategies generated with a given method/workflow were profitable in simulated forward period (independent from selected reference point), then I know I can trust my method and this gives confidence that strategies generated today have chance being profitable in the future. So the goal of SFT is to test your method not your strategies.

Enric wrote:

Yes. Sound an interesting approach. Please, correct me if I’m wrong; I understand that you work in two steps: Step 1: IS + OOS. It strikes me that you use IS 3,5 years and OOS 9,5 years divided in two parts. Wow! That’s a lot. I’ve read by ‘trusted’ gurus advice of 50 IS + 50 OOS. I’ve read by other ‘trusted’ people not OOS at all, just IS and strong Robustness tests. Well, I see you on 75% OOS. Anyway, you get to find 30 good strategies with your criteria. Now move to Step 2 Step 2: OOS 4 years; 2015 to now. If I got you correctly. How many of these 30 strategies survive to the second Step? if I may ask

Note Step1: that IS/OOS 50/50 ratio is only during strategy generation step in builder. Yes hereby I followed standard teachings of SQ gurus. But also note that due to many many iterations using same building blocks and same strategy generation setting your first OOS becomes IS. That is why there should be always another second OOS period to validate the strategies generated by the builder. In my opinion this is one of the most powerful tests and from my experience I see that 95% of generated strategies will fail at this step.

Note Step2: Please do not label this period as OOS (I know it is OOS, but it only confuses people), it is kind of simulated future, a simulated forward is a much better naming. Taking my automated workflow as example, and assuming there are no bugs affecting my results and I did not make any stupid mistakes, then I get now 100% effectiveness in the first two years of simulated forward period. This means that 100% of strategies that pass my workflow are profitable in this simulated future. After 2 years I see a big decline in effectiveness of strategies. This also means that every 2 years I need to generate new strategies. But I want to leave the discussion regarding my automated workflow for later…

 

 

 

Yes. As you said, once the OOS period is part of the Building process it becomes IS. That’s the main argument against using OOS; better to use only IS because at the end of the day it’s the same. I’m curious to know the behaviour of strategies during the ‘simulate forward’ period as Coesio asks to call it. Particularlly I’m not convinced of this approach, after all it looks to me like growing the OOS more and more so the effect will be like replacing genetic algorithm by random, but it’s a curve fitting method as well, isn’t it?

Enric wrote:

after all it looks to me like growing the OOS more and more

I see you are still confused about one important thing. In SFT, the simulated forward period, which you call additional ‘OOS’ is outside the strategy selection time window. It is a simulation of real strategy performance and is NOT used to select/test strategies, but to prove that your workflow is able to create profitable tradings systems. It has nothing to do with testing/selecting of strategies so it cannot be seen as additional OOS.

Ok, I think I get it. For me that’s an Out of sample data (=OOS) but it’s just a matter of definition, not a big deal

But only if you call the data from 2050 also OOS data, then yes, then in your definition it is also OOS 😉

I made a small overview to make it more clear:

Strategies are generated, tested and selected using A and B, and results are tested using C period. (Period C is not ‘OOS’ since it is not used for strategy generation and selection). After C period the process starts from scratch, new strategies are generated and selected using new A and new B and tested during new C periods..etc..etc…

Hi coesio. First of all thanks for your generosity to share. No doubt it’s a valuable approach and definitelly worth for a try

Enric wrote:

Hi coesio. First of all thanks for your generosity to share. No doubt it’s a valuable approach and definitelly worth for a try

Too bad nobody is sharing this kind of stuff here and nowhere else…

Buy anyhow… check out this topic where I really put this into practice…let’s see what the results will be:

https://strategyquant.com/forum/topic/100-automated-and-100-accurate-sq-workflow-test-case/

 

Thank you Chris.

This is a powerfully idea I also use this process for build and testing my  strategies too.

I have just come across this thread. seen as though it was 2 years ago I wonder if you could up date us on the live results?

It is going quite well, however 2 years in algo-trading, equals (from my perspective) only to 150 trades 🙂 = barely enough to see this result as statistically significant…

My testing in this period has generated me profits around >+150%…so far.

Some would say this is very interesting.

However, the world is changing so fast, that 150% gain seems not that impressive at all ;)..that’s why algo-trading is my lowest priority right now.

I’m considering to publish my work, including my workflow to a limited amount of people.

You can always contact me if you are interested.

You probably know where to find me…

Gr

Chris