Finding Robust and High-performing strategies

1 – 2 hours of build doesn’t sound a lot of time to me, unless you’ve got some kind of staggering multi-Xeon server.

I’ve found some of my best strategies after a few days, so maybe let it run a bit longer….

Oh and do you really mean Out-of-sample Net Profit <0 ?

If my Databank has no more than 5,000 strategies I will not stop generating….. normally I need one week to generate that amount of strategies.

Eventually it all depends on your Selection Criteria… you can be very relax and Abracadabra you can have 5,000 strategies in your Databank in less than 1 hour and if you are very strict, you can wait till the cows come home  😀 and still have nothing….

As for me, a relatively newbie,  I need about a month to generate a few good strategies that can qualify to go live….

I’m generating strategies on a 32 core server for 1 month(720 hours) and on 30 years of historical data and 99.9999% I will consider not robust. Ranking options don’t have anything to do with robustness.

Running for 1week-1month seems like overfitting because eventually it will find a system that overfits the Robustness criteria and out-of-sample period.  

 – If it takes 1week-1month to Build it, how long does the system perform live before it fades out and you need to find another strategy?  

– Also, what In-sample/Out-of-Sample periods do you use?  I’m using short time periods (enough to have 100-500 trades during In-Sample), so for 1 hour bars. i use ~6 months In-sample and for 5 minute bars, only need 2-3 months.  My other concern is that using longer time periods (e.g. 5 years-10 years) gets us into different market regimes where the strategy won’t show consistent performance.

Also, any thoughts on the settings I’m using (attached above in my first post), that would help SQ engine find robust, performant strategies faster?

 – Building Blocks – i’ve selected most Entry Rules and most Order Types (not sure limiting them would help find faster)

 – Stop Loss and Profit Target – optional (not required)

 – Genetic Options – 125 Population size, 20 generations, Max Tree Depth 4, Mutation Prob 20%, Crossover Prob 80%

thanks

Kiran

It takes 1 month to find a system because of rigorous robustness testing of millions of systems, not looking for the best curve fitted strategy. 75% of that time is spent random generation, not genetic evo optimization. In 1 or 2 hours of generation I doubt you will find highly robust strategy.

I like your genetic settings, I use similar, but I’m running 32 SQs with 200 genetic population size each. Not to find the best curve fitted, but because I’m going to trash 99.99999% of strategies after all my robustness tests.

For building blocks I use almost everything except time (minute hour day) and Heikin Ashi candles.
Stop loss required, take profit optional.
Increase indicator max parameter to 300 from 200.
Increase max range size to 150 from 50.

Custom fitness- max dd%, net profit, stagnation%, stability.
I don’t use IS/OOS because you literally are still cherry picking a strategy that passes the *whole data series anyway*.  I will test the strategy on the last few recent months of the market as a final test but that’s it.I just use 30 years of data and look for over 1500 trades for patterns that have worked through multiple wars, depressions, market crashes, the shift to electronic automated markets and any type of event and keeps on working. Thats real robustness. If you are using short data series you are definitely curve fitting. You should be building a system through multiple crashes especially for ES strategies. If you build it on only bull market data you are going to get murdered by the coming crash because you curve fitted a strategy to a bull market.

After I get strategies… Then I put them through rigorous robustness testing. I want to see them pass 100% of over 100 Walk Forward Matrix variations (not for optimization, only to see how robust). I want to retest them on multiple pairs and timeframes to see if the strategy fails. I test on some recent few months OOS data, I look at the pseudo code rules and make sure the strategy makes some sense. I run it through the traditional robustness tests of variable spread slippage ATR data adjustment etc etc.

This actually takes a few weeks to find just 1 strategy that will do all this to my liking.

I have a small public version of my portfolio linked (when it was started I did not use this method, the early DD was caused by a strategy that was built on short data series and I removed it long ago) so whether you disagree or agree and question how long these strategies will work then just follow the portfolio. I’m pretty confident in my method.

 

I’ve been working on SQ for over a week, and had trouble generating a Futures strategy that is robust and delivers strong performance.  I’ve tried 5min, 15min, 1hour, True Renko (with true wicks) – typically on 4-6 month In-sample (Test + Validation) and Out-of-sample on either end (beginning and end of IS) of another 1-2 months.

Even if i get 1-2 strategies in Databank after 1-2 hours of Build, it doesn’t pass Retest when i test for the other 1-month out-of-sample period at other end.

 

My Metric function – Weighting of Net Profit (2x – also tried Avg Profit/Month), SQN (1x), Return/Drawdown (1x).

 

 My criteria for acceptable performance (filters for Databank) –

 1) Net Profit (Robust) > x where x gives an annualized return of 20% on total capital (I’m using $25K capital for 1 ES contract and $15K for less volatlie futures such as Corn etc, which is quite aggressive to report a strong return)

 2) Max Drawdown < 15%

 3) In-sample Profit/Month / Out-of-sample Profit/month <600%

 4) Out-of-sample Net Profit <0

 5) SQN (In-sample)  >2

 

This is a reasonable expectation of investors because these strategies compete with the stock market hedge funds and need to offer a better return since it’s based on riskier asset classes.

 

Attached are a couple of Settings files i’m using for ES-5min, ES-1hour, ES-5tick-TrueRenko

-> Appreciate if anyone has thoughts on my Settings and advise if i’m making some fundamental mistakes

-> Also, any guidelines on what typically works well (IS/OOS test period, bar-type, instrument, GP settings/metrics etc.) would be helpful.

 

thanks,

Kiran

 

I kiran,

Are you sure ?  renko support these blocks of buildings now? limit order? price value? does backtest matching ? 

thanks,

olivier

(Removed in retrospect because it was an embarrassing beginner question and didn’t add any value, haha)

It is not only about cores ! SQ was not perfectly built to use all cores so 32 cores = 32 or 16 sq instances and so on (probably dedicate 4 hyperthread for 1 instance).
For sq3 i would suggest a i7 high clock (OC) rather than xeon . A 5965x and for sure the new 6965 x (1700$) cpu work much better than a 32 cores xeon with 2.0 clock speed.
A 5965x 8 cores OC to 4.5 -4.6
Or
6965x 10 cores OC to 4.3 would be a monster for sq3 .
1 week on such pc = 1 month on xeons sllw clock speed.
That is usually not the case with all softwares but with sq3 it is. Get rams from corsair like corsair platinum 4000mhz .
A m.2 samsung 950 (2500 write speed) and for sure water xooling for the cpu OC for that ull pay around 4000$ computer but its gonna be huge for sq3 or sq4. Runaway from xeons (the only advantage they have is the ecc memory) but 10 mistakes out of million is not gonnabreak the bank :p your bank.

Finally what i am not sure off !! Is running SQ with a vmware (parallels desktop) ! Weird but i felt on a vmware not dedicated operating system the sq instance which is one only running eats up 100% of cpu 80-100 all the time.
Magicly multi cores works on parallels desktop(vmware) of sq .

At GACKT: see data sources thread.

@ gentmat: I am getting getting 85% usage via parellelization. There is an entire thread about this. I don’t have a 32 core Xeon. I have a server with 4 8 core xeons. I can 100% usage by running more SQs.
Speed is 2.4ghz and it has turbo throttle to 2.6ghz.

You will want more cores and threading for SQ 4.

Cant agree more, i have 32 (64 threads) cores running sq each core 2.4, running SQ 10 times at the same time,

working great, theoretically the more threads the better because you scanning 64 times at once..

so…, i dont know if i7 will deliver the same results..

No Karish not totally true and that was i have explained up. A i7 will outperform your 32 cores ill re tell you more details:
Data goes from SSD to Rams To CPU Cache to core to 2 virtual threads …
32 cores pulling data will be slow as your cache is 30 at most i guess a 32 cores will have 15 mb not even close to 30 l3 cache
So the bottleneck will be your ssd then xeon rams have around 1600MHZ then cpu with 2.4 who can boost for 3-15 sexs to 3.6 depand on voltage.

A i7 4.3 , 10 cores , 20 threads .. Cache 25MB OC with 3600 mhz rams oc to 4000 (new x99 rampage v v 10 can handle it with new bios update).
With m.2 hard disk with 2500 rather then 450 mb speed .
The cores threads wont wait each other to finish task . They will finish in no time. Splitting the work and waiting adds so much latency. So if u can grab a i7 not a xeon i would highly suggest u try the difference, u ll
Be surprised.

Now that you explained that to me it might be interesting with a motherboard for 4 CPUs when each cpu is an i7 from the new series of skylake, heard they are better but im not that smart to understand it all :/

would be great to hear what do you think about skylake in compare to the i7x though.

cool thanks.

Hey…

I’m also using SQ on futures (I’ll make a big guess and assume your on eminis??), some advice, but I’m by no means an expert, and still on my SQ learning journey….

I use 2 different time frames. 1 hour and 1 day. I have found that anything less is far to noisy, to say nothing of abusing my processors, because…10 years (or whatever) of 15 minute data, ugh, like 2 minutes a strategy on my machines..

4-6 months of IS and 1-2 months OOS is (imho) NOWHERE near enough. I use 15 years IS (also 3 part – generation, training, validation OOS, so that’s good :), and then run multiple manual OOS tests, with varying time frames. Yeah, yeah, I get the argument that on the 2002/3 change in eminis (electronic), but it works for me. For example, the very first OOS I run on strategies I get from my data bank is last 6 months, to see if I’d be happy right now. That kills at least 80% of my strategies. For length of testing, I found this – if I was generating on a short period (I defined short as 3 years, so a few months, no way…), then I found my PCs could generate THOUSANDS of strategies very, very fast, but , most of them failed in my OOS/robustness very, very fast. By switching to much, much longer time frames for generating, the number of strategies dropped dramatically, but the fewer I generated become much, much better at passing my manual OSS/robustness tests. Sounds like you need to buy some data to expand your testing periods (a lot).

For your databank settings…sure, I mean, whatever works for you. I typically find the more simple the better…things like SQN..yeah..I mean, I have a much more simple view – did it make me money, did it not freak me out with big DD % (or ret/DD, or whatever you want ot use), and did it not panic me with a low win %. Also, I like a nice payout ration, so it’s not to fragile, for example, if NT, or my VPS goes down, and it misses that 1 trade it was depending on, it won’t break…things like SQN, R, expectancy, I get it, but…I’m just a simple guy. 🙂

Also, will point out, that while I might want a great sharpe ratio of 3.0 or more, but I don’t actually care if it had a high sharpe, or SQN, or whatever, on data a few years ago, what I want is it to make me money today.

I found I generate about 10k strategies for every 1 strategy that I feel ok enough about to incubate on my platform…with my data bank settings as above…the length of time to generate those strategies depends on your machine, your settings, etc…but I collect my strategies every Friday, so only once a week (sometimes less…). If you find you can’t leave it alone, get a VPS, and put it out of sight (and out of mind) and just let it run….

***sorry, minor addition – also, important is frequency of trade. You need enough trades to be statically significant, and not make you so impatient when you incubate that you rush it to live. For example, if it trades 1 time a year, but wins 100% great…but are you really going to incubate it for 20-30 trades (20-30 years) so make sure it’s ok?? Probably not. I aim for a system to make a trade a week..some do, many don’t, but that’s my goal, at least, so I can watch it make enough trades in my lifetime and put it live.

Hey…I’m also using SQ on futures (I’ll make a big guess and assume your on eminis??), some advice, but I’m by no means an expert, and still on my SQ learning journey….

I use 2 different time frames. 1 hour and 1 day. I have found that anything less is far to noisy, to say nothing of abusing my processors, because…10 years (or whatever) of 15 minute data, ugh, like 2 minutes a strategy on my machines..

4-6 months of IS and 1-2 months OOS is (imho) NOWHERE near enough. I use 15 years IS (also 3 part – generation, training, validation OOS, so that’s good :), and then run multiple manual OOS tests, with varying time frames. Yeah, yeah, I get the argument that on the 2002/3 change in eminis (electronic), but it works for me. For example, the very first OOS I run on strategies I get from my data bank is last 6 months, to see if I’d be happy right now. That kills at least 80% of my strategies. For length of testing, I found this – if I was generating on a short period (I defined short as 3 years, so a few months, no way…), then I found my PCs could generate THOUSANDS of strategies very, very fast, but , most of them failed in my OOS/robustness very, very fast. By switching to much, much longer time frames for generating, the number of strategies dropped dramatically, but the fewer I generated become much, much better at passing my manual OSS/robustness tests. Sounds like you need to buy some data, to expand your testing periods (a lot).

For your databank settings…sure, I mean, whatever works for you. I typically find the more simple the better…things like SQN..yeah..I mean, I have a much more simple view – did it make me money, did it not freak me out with big DD % (or ret/DD, or whatever you want ot use), and did it not panic me with a low win %. Also, I like a nice payout ration, so it’s not to fragile, for example, if NT, or my VPS goes down, and it misses that 1 trade it was depending on, it won’t break…things like SQN, R, expectancy, I get it, but…I’m just a simple guy. 🙂

Also, will point out, that while I might want a great sharpe ratio of 3.0 or more, but I don’t actually care if it had a high sharpe, or SQN, or whatever, on data a few years ago, what I want is it to make me money today.

I found I generate about 10k strategies for every 1 strategy that I feel ok enough about to incubate on my platform…with my data bank settings as above…the length of time to generate those strategies depends on your machine, your settings, etc…but I collect my strategies every Friday, so only once a week (sometimes less…). If you find you can’t leave it alone, get a VPS, and put it out of sight (and out of mind) and just let it run….

***sorry, minor addition – also, important is frequency of trade. You need enough trades to be statically significant, and not make you so impatient when you incubate that you rush it to live. For example, if it trades 1 time a year, but wins 100% great…but are you really going to incubate it for 20-30 trades (20-30 years) so make sure it’s ok?? Probably not. I aim for a system to make a trade a week..some do, many don’t, but that’s my goal, at least, so I can watch it make enough trades in my lifetime and put it live.

I am Good at computers . I suck with SQ

@Threshold Will help you better than me as he is good and his portfolio is a little proof of success or at least call it achievement

I ll try to help you as much as i can

“I found I generate about 10k strategies for every 1 strategy that I feel ok”

Yes this is good enough ! I find 1 out of 4 million strategy to be okay . Thats why most of the guys here leave their servers (Dedicated) Not vps . for 1 month than they come and check strategies.

To make it faster to search for strategies put some limitation for the bank

dont add stratgies in data bank  :

is

net worth <=0

DD >=15%

Stagination >=15%

profit factor < 1.15

fitness <0.65

number of trade for 10 years you must have 480 trades

dont bother with sqn , use dd

” I might want a great sharpe ratio of 3.0″

Thats a lot i guess but also might some ppl help you with that . i think you must exclude < 1.3 not <3.0 !! you dont want strategies to be perfect as they will be curve fit ! there is nothing perfect and no script can be magic

trading = human sentimentals and no computer can predict the idiot human minds ! sometimes they read the news and the news say good stuff about dollars BOOOM dollar go down ! no one knows why !

Now that you explained that to me it might be interesting with a motherboard for 4 CPUs when each cpu is an i7 from the new series of skylake, heard they are better but im not that smart to understand it all :/

would be great to hear what do you think about skylake in compare to the i7x though.

cool thanks.

I can not compare skylake to i7 because skylake is an i7 technology 6700 , 6800 … ,6900, .. 6960 x

i7’s are not like Xeons So you can not use them in parrallel  as 4 cpus ( Not possible ) . 

There is somehow some boards for xeons that accept 1 i7 and 1 xeon ! but that what u max can have ! and i dont suggest it ! its better to go with 2 xeons than 1 i7 and 1 xeon.

1 cpu (6960 x ) is already an overkill for SQ because SQ is not something that you want now now now to find a strategy , u can wait a day and such cpu on h1 timeframe can find app ( and im just giving a guess ) it can generate around 10+ million strategies ( i bet its much more but just gave u the minimum of the minimum)

6960x comes as 3.5 go to siliconlottery.com and get a silicon cpu that can be OC to 4.4 ! now you have 10 cores (20 threads) at 4.4 ! that is insane . and leave your computer open for 1 week or more .

Note: do not go for motherboard with double cpu’s like 2 xeons or 3 or 10 ! useless !

1 xeon computer 

1 xeon computer 

is not equal to 1 computer with 2 xeons ! No the 2 xeons will share the hdd and the ram and all resources . so its better to get 2 pc and not a server for SQ.

—– Anyway there is a post for this subject please search for it where everyone contributed the theory u ll learn more . as i intend to keep the post here as its original purpous 

finding-robust-and-high-performing-strategies 

or send me pm and i ll help you out with your pc components and where to buy each one for cheaper prices ( i dont do affliate :p dont worry )