Ebook workflow is brutal on my strategies.

Wondering how people are fairing following Zdenek Zanka’s ebook of strategy testing.  I tested 2000 EURJPY H1’s today, ran on ebook settings.  1400 passed the multiple market test, I deleted 700 because they didn’t make at least $2000.00.  I didn’t want to look through 1400 equity curves.  I had 28 left after the first robustness test and only one made it to WF Matrix only to fail miserably.  Do I just need to hang in there or are these tests to tough?

Thanks,

Kevin

That’s what I normally get, one or two passes  all the way to WFM. Occasionally I ended up with nothing and had to redo the whole exercise all over again. I normally generates 1 to 2 good EAs in one a month….  🙁

The time consuming aspect is during the Robustness test where I have to screen the strategies one by one, i.e. the 95% should not drop below half of the original values, I wish someone can tell me a better way to screen it …

Tomas & Marc, can you make a filter selection for us to shorten the process in this stage…

Ok, Ok, I now the wish list for SQ4 is growing by the day. Lets get SQ 4 out first and then tackle our wish list later..  Keep up the good work guys, greatly appreciate 😀  

I ended up doing it little different for my last generation I let it run for 12 days longs and shorts separately and I only got 160 +170 strategies of  those who past the criteria’s . Of them totally 105  passed the robustness tests which is to many to walk forward test which probably means that I have sort even harder.

Hi Mabi

Interesting, I would like to learn from you, what are the advantages you found in generating the strategies separately for Long and Short ?

I would think that the results will be more or less the same had you run for 12 days the strategies in combining Long and Short, I might be wrong …

Do you monitor your CPU core temperature during that 12 days? Mine heats up to 80 deg C when the CPU is running above 90%, I am not sure that’s acceptable or normal for the hardware…

Thanks

Cheers

Hi CMK,

It is harder with the same settings to find only long or only shorts that passes. I am doing this since I found that many times even if the strategy seems good it would have been better if I had combined it with a short from another strategy and  vise versa . I am a little biased towards separating them because of this and I have some manual made strategies with same similar entries that need different profit tagets and stops depending on if they are short or long to work best. At least 1 well known Algotraders from futures cup claims that he does this as well. The only problem I have is that I run out of memory so 12 days seems to be max out for me. Do not check the CPU temp but it is always above 90 %. The only one that complains is the girl friend who says that it is to noisy. ( Who cares ) 😀

It’s meant to be tough..you are looking for robust, good or decent strategies, after all…and it’s ending up trading real money, so you want to be careful…

I collect strategies form my databanks every Friday, and start retesting/robustness over the weekend. I have a few different OOS tests they have to pass, filtering out most strategies. It’s basically  a bunch of manual walk forward tests. The ones that pass the multiple OOS tests, I put through robustness tests. Any that pass the robustness tests, I put through SQ walk forward. Sometimes they pass walk forward sometimes not. If they do pass walk forward, I put them on incubating, so paper trading on NT.

It depends on the building blocks, how many instances running, but every Friday, I’m collecting anywhere from ~5k to ~10k strategies that have passed my initial data bank filters.

By the time I have finished the additional OOS testing, I might have ~500 to 1k left.

After the robustness testing, I might have ~10 or so, sometimes less. Sometimes nothing passes this. Sometimes, they all fail at the first OOS tests.

By the time I walk forward the last couple of strategies, sometimes I’m left with 1, 2 or nothing…but by now, the initial 5-10k is very, very, very few.

So, sometimes I might get 1 strategy a week, starting with anywhere from 5-10k from the databanks…sometimes nothing…..it’s not press a button, and then you get thousands of great strategies, there’s a process to follow, and work involved. Once it’s live on a platform, then it’s a whole new assessment also regular re-optimization/WF, if you did that, so…yeah, work, but it’s fun and interesting (at least imo, otherwise, why do it).

For the ebook settings, it’s really a starting point…I stared with all the ebook settings, but have moved on from them, as my knowledge improves, and I see ways to tweak on the program, for example a good thing I found is setting a ratio between IS and OOS DD %, to improve stability and filter out strategies that have very different IS and OO performance (ie bad stability and consistency).

@mabi, mine also complains about noise, funny! I’m running most on a remote VPS, so I actually only have 1 in the house as well. 🙂

Thanks for the responses guys.  I will hang in there 😀

My experience with SQ eBook is very similar to _Cujo

I found eBook settings great for finding EURUSD strategies, usually finding at least 1x strategy per week, out of a weekly Databank pool of ~3-5k. Using same eBook settings, I did not initially have such success with USDJPY & EURGBP: databank pool reduced; retest processing less successful. So I started to explore other building block settings, which has significantly improved the weekly Ductbank pool size and quality.

In addition, a few process changes I have made are:

1. Run ‘Randomize strategy parameters, …’ robustness test 2nd (before ‘Randomly skip trades’). This helps reduce process time as the ‘Randomize strategy parameters, …’ robust test generally excludes far more strategies than the ‘Randomly skip trades’ robust test.
2. Before running final WFM tests, I run all short-list strategies with ‘Real Tick’ precision, to verify the ‘M1 precision results. Usually results are similar, but occasionally a strategy fails with Real Tick data. At this stage I also run all short-list EA’â„¢s in MT4 using Birt’s TDS. The results are usually similar to SQ Real Tick, but MT4-TDS consistently yields more conservative Drawdown results which I feel need to be verified as acceptable. (Though make sure both SQ and MT4-TDS price data sources are setup to your broker’s server time-zone, not your local time-zone, if you want to avoid a headache.)
3. When using WFM I recommend you do not only use ‘Net Profit Stability’ as ‘Robustness Score Component’, as recommended in eBook, as this will not prevent non-profitable strategies from passing the WFM test. To help avoid this I have added ‘Net Profit in % of Original Strategy’ to the ‘Robustness Score Components’ check list, to ensure only consistently profitable strategies pass the WMF tests

Overall I think the SQ eBook a great primer, saving a newcomer to SQ much setting research time. Compared to my previous varied attempts at trading, over the last 4 years, SQ is saving me much time while also providing me with far greater confidence in the strategies and the backtest process. Much thanks to the SQ team!!! Keep up the good work!

Thanks for the reply AC1962!

Hi 

I was wondering, if a couple of strategies that perform well on backtest specially out of sample and in live trading start to perform poorly,

what do you do? do you shut them down? do you keep trading them at small sizes? put them in a demo and wait for them do show better results?

what criteria do you use to assess if a strategy is broken or not?

Hi mkjones320,

I suggest to generate 20000-50000 Strategies

You can be happy If one strategy will be found.

thomas

That’s what I normally get, one or two passes  all the way to WFM. Occasionally I ended up with nothing and had to redo the whole exercise all over again. I normally generates 1 to 2 good EAs in one a month….  🙁

 

The time consuming aspect is during the Robustness test where I have to screen the strategies one by one, i.e. the 95% should not drop below half of the original values, I wish someone can tell me a better way to screen it …

 

Tomas & Marc, can you make a filter selection for us to shorten the process in this stage…

 

 

Ok, Ok, I now the wish list for SQ4 is growing by the day. Lets get SQ 4 out first and then tackle our wish list later..  Keep up the good work guys, greatly appreciate 😀  

You may create a new View for the “Test Strategies” tab, where you add the column Ret/DD Ratio (RT), which, by default, is related to the 95% probability (also configurable).

Hi rafaeldelrey

Yes, Ret/DD Ratio (RT) but one still needs to shift through the strategies to filter out the ratio between Original vs RT and filter out those fall below 50% (ebook). If there are 300 strategies to filter through and only select the ratio that meets the criteria and then it time-consuming  

Hi CMKCMK

If you amend your Databank View setup to include both ‘Ret/DD Ratio (IS)’ and ‘Ret/DD Ratio (RT)’ you can then export complete Databank list as CSV file to Excel, add a new column to calculate ratio of Ret/DD Ratio (RT) / Ret/DD Ratio (IS) and use Excel Conditional Formatting to highlight all strategies with a ratio <50%. Place SQ and Excel windows side-by-side and select all SQ strategy highlighted in Excel and delete them. When doing this I suggest you initially sort your Databank by ‘Ret/DD Ratio (IS)’ as this generally helps to group most (not all) failed runs to one end of the list, making subsequent selection to delete easier.

I’ve also given up viewing ‘each’ Equity Curve after recommended ‘Slippage’ and ‘Multiple Market’ test, I nowadays simply delete all of these tests where Profit Factor is <1.15 (half original strategy pass criteria).

I hope this helps.

AC1962

What is the reason for the OOS 3 Test, since there is no improvement or optimization during the ebook workflow?  If you apply OOS3 test and filtering right after the OOS2 (or make OOS2 = OOS2 + OOS3), it would be the same, isnt it?  Am I missing something?

Hi rafaeldelrey

 

 

Yes, Ret/DD Ratio (RT) but one still needs to shift through the strategies to filter out the ratio between Original vs RT and filter out those fall below 50% (ebook). If there are 300 strategies to filter through and only select the ratio that meets the criteria and then it time-consuming  

You are right.  Maybe some “custom column” would be nice in SQ4.