100% automated and 100% accurate SQ workflow test case

My first 100% automated and 100% accurate workflow, StrategyQuant ‘custom projects’ test case

DISCLAIMER: The presented results below are still preliminary, there is still a small chance that my positive results are influenced by an undiscovered bug in the current version of SQ-X (build 118.84) or that I’ve just made a stupid mistake somewhere in my workflow resulting in a huge ‘Data Mining Bias’. However I did my best and rechecked everything multiple times…Moreover since this all is based on a relatively new ‘custom projects’ feature of SQ-X, nothing of this has been tested yet on a real account…but I think I have built a strong case supporting I could be right on this one

My claim: It looks like I’ve managed to create a 100% automated and 100% accurate workflow using StrategyQuant feature called ‘custom projects’.

100% automated means: I push 1 button before going to bed, and every morning my workflow automatically generates, validates and selects few new strategies which are ‘ready to go’

‘Ready to go’ means: I can deploy them immediately to my live account. Without a need of further processing.

100% accurate means: Every single strategy that has been selected by this automatic workflow (~50 so far), has been profitable in the 2 years period from the generation date.

To test my workflow I’ve adapted my SFT method as described in this topic: See HERE.

The workflow is based on standard validation test (common knowledge) as shared by SQ team in their free courses, however with a very rigorous settings. The workflow does not use any advanced validation methods like WFA,WFM,OP,SPP. Instead a customized Monte-Carlo test is used to simulate behavior of a SPR method. No portfolio analysis is performed (some systems can be correlated!).

My automatic workflow test case is split into two verification periods:
1. End of year 2014.
2. End of year 2016.

At each point in time 1 and 2, I used my workflow to automatically generate and automatically select 20 NEW strategies (out of several hundreds thousands systems) without ANY manual intervention and then ALL of selected systems where forward tested using SFT (future data). Let me be clear on one thing: I did not cherry picked any strategies.

It seams that every single selected strategy was profitable in the period following the selected generation date. See figures blow:

Test case 1: Strategy creation @ 2014.12.31, Simulated Forward test: 2015.01.01…2016.12.31.
Real-Ticks (Dukascopy data), Real-spread (no commissions)

Test case 2: Strategy creation @ 2016.12.31, Simulated Forward test: 2017.01.01…2018.12.31.
Real-Ticks (Dukascopy data), Real-spread (no commissions)

My conclusions so far:
1. If there are no mistakes, then it seems that it is totally possible to use SQ-X automated ‘custom projects’ to automatically generate and select profitable trading systems.
2. No advanced validation/filtering methods needed. Of course these tests should only improve total result and minimize DD on portfolio level.
3. The results in SFT of >2014 are slightly better than >2016. Workflow is somehow sensitive to used data during strategy generation (due to changing market condition). It seems that years 2017 and 2018 are very difficult years for trading using the selected trading type.
4. It is not 100% proven yet, but it’s a pretty damn good result so far, taking into account it’s based on a simple workflow that is using basic filtering principles.
5. Some of the strategies can be correlated, but for the sake of this investigation no manual correlation filtering has been performed. This would jeopardize the objectivity of this test case.
6. The filtering settings are very rigorous, this workflow filters out only the most robust strategies. According to my statistics only 0.05% of the generated strategies are able to pass this workflow.

TODO:
– Refine the workflow and implement further strategy selection, perform correlation tests, WFM analysis and additional portfolio level related tests.

– I’m also waiting for 01.03.2019 and build 119 with new features 😉

Greets,
Chris

good job so far 🙂

real tick, real spread without commision and slippage could lead to a very optimistic results…did you make any comparation? tick with fixed spread, 1M precision, add 1 pip slippage?

has been your findings proven also on different markets or timeframes? because EU H1 is the easiest market and TF.

can you show us from test case 1 also results for year 2017-2018? if the strategies are dying, or are the same or better as the strategies from test case2?

* Yeap 1M, fixed spread, slippage give very similar results (maybe even slightly worse)…

* All other test diff TF, diff markets are already integrated parts of my workflow. As I said it is based on ABC of common knowledge + SPR like filtering.

* Extended SFT of test case 1 gives mixed results: some strategies are dying or stagnating after 2 years some are not. It seems that this workflow needs periodic strategy replacement (after 2years a NEW set of strategies need to be generated). This could be different when WFM is used, but this is a different discussion for later…see below the results of case 1 with extended SFT from 2015.01.01 up to 2018.12.31 (4 years):

Using Fixed money stop Is seems very curve fitting which I believe You do. I can get 95 %up to 2015-2016 with no Robustness test at all only add commission and real tick test to the mix. However none ,zero, nil , of these strategies work 2017-2018.  Do the same and come back with result for  2018 and I will believe that you have found something.

and no Fuzzy logic  🙂

1) as i said, i dont like real tick real spread, backtest will be very optimistic. Dukascopy data has very low ECN spread, so without commision its not true. If you are using STOP strategies always add some slippage to a backtest, because on real environment you will get it, many times you will be hit with gaps against you. Always use methods which will lead to a realistic backtest, even 1 pip different price could lead from profit trade to a loss trade. Now imagine that you are using real tick real spread.

2) i dont like fixed risk MM. If you are backtesting with settings which you will not be trading (capital, risk value) and in real account you will set different values, the backtest will be different, in strategies with ATR SL.

3) about the “ready to go” – most of us using demo acc or small real acc as an incubator of strategies – but the major purpose for this is not to wait 2 years and come to some conclusion as you do with your method. This is nothing new, many of us using unseen data in our workflows. The main purpose is to make clear that the strategy is trading the same way as in backtest. In SQX there are building blocks which are very price sensitive (pivots, fibo, etc.) – what you see in backtest, you will not get even on demo acc, neither on real.

5) dont listen to anybody and go your own way 🙂

Ok I was hoping for this kind of comment,my goal is to find flaws in these to positive results not to prove those are 100% right, because none of this has been tested on a real account.

So what kind of spread/slippage settings you consider realistic for Dukascopy/other brokers? Spread@? Slippage@? Commissions?.

Hereby the updated results with added 2pips of slippage and $9/Lot commissions..

Test case 1: Simulated Forward test: 2015.01.01…2016.12.31.

Test case 2: Simulated Forward test: 2017.01.01…2018.12.31.

Note that indeed 2017 and 2018 are extremely difficult years for system traders, it’s like the market became immune to ‘simple’ algo-trading. However even with added slippage and commissions in both cases I see a positive total result.

I do not agree with you guys about one thing, which is testing with fixed spread. Testing with fixed spread even as high as 2 pips can lead to very tricky results. Note that all brokers increase their spread dramatically during nightly hours and also during volatile events. I will take my chances with real ticks and real spread option.

Moreover, I will try to verify one another thing:

– I will take my test case 1 strategies and filter them w.r.t correlation and select few uncorrelated to build portfolio.

– Will do WFM for every single year > 2015.01.01 till now

Let’s see if this will help to deal with 2017/2018.

i was doing the same way 3 years ago when i was starting with old SQ, trying to find the best settings for data to match the backtest as much as possible. Using tick backtest with real spread and come very quick to conclusion that in real life i am getting only 50% of results from backtest. You will catch 1 gap against you, some NFP rollercoaster, etc. Every broker differs, you need to know where you want to trade real…

Backtest was very optimistic and if am building portfolio and computing risk level and needed capital from max drawdowns the values differ a lot.

Now i am using only 1M precision at all and i can live with it very well. For EURUSD using 2 pips fixed spread with 1 pip slippage

for now i have 159 strategies from SQX in incubator and for every strategy i have done the comparaion between tick and 1M precision. In 99% the eq curves doesnt differ in the meaning of direction and how the EQ lookslike. But if i look to RDD or max DD i can see very big differences.

and another truth is – in 99% the tick backtes is better than 1M precision

this is comparation of all strategies together

max DD TICK – 3900

max DD 1M – 4500

Ok this is a nice investigation, however the presented curves are still backtested (correct?), where you compare 99% tick BT with a 1M BT, where for 1M BT you used an assumed fixed spread of 2pips with a 1 pip slippage. Thus, no wonder that 1M based result is worse, this is as expected, you can tell this even without doing a BT. If I do the same with assumed spread of 100 pips the resulting difference will be even much higher;)

hankeys wrote:

for now i have 159 strategies from SQX in incubator and for every strategy i have done the comparaion between tick and 1M precision. In 99% the eq curves doesnt differ in the meaning of direction and how the EQ lookslike. But if i look to RDD or max DD i can see very big differences.

Please show us the good stuff, which is based on your life result comparison, not a BT. This would be most interesting to see…

you dont understand me well and you dont read here all the stuff, because many times i was sending here comparation curvers from real acc.

OK…1 example

red line – real acc

black line – 1M bactest dukascopy

green line – real tick, real spread

yellow line – 1M backtest Asirikuy data

i have many times proven what i am saying here…doing optimistic tick backtest is like digging a hole for yourself

after 2 years of live trading – 1M and TICK backtest is still holding, but real EQ curve is already in loss…NFPs, gaps…and they got ya

Ok, I get your point just wanted to see your data, so thanks for sharing (again), maybe it’s a discussion for another topic but:

* I assume this ‘red line – real acc’ is on dukascopy broker? In that case what level of additional spread/slippage would be required to ’emulate’ the same conditions as on real acc (to align ‘black line’ with the ‘red line’)? If this is even possible… (which I doubt) I see this mismatch is not constant, so probably caused by many different factors.

* Did you try to isolate the biggest factor(s) causing this mismatch, by analyzing your data or running additional tests? Gaps, slippage, latency, news,swaps, types of orders, etc…? Or is this already covered in different topic here…

The more comments I read here, the more I realize ‘not sharing’ anything at all could be the best way to go on this forum. I see you guys have figured it all out already, I see a lot of people making big profits continuously, especially all gurus, there is no room and no need for sharing ideas and having discussions… 😉

Hey, don’t be discouraged. I am more than happy to hear everyone’s ideas and points of view, and I am sure many others here appreciate it too.

Ilya