TRY TO UNDERSTAND CALCULATION OF BUILD MODE

Hi,

I am new here and try to understand the calculation and working methode of SQ.

To start with te builder:
Let’s say I have 500 strategies in my databank from genetics searching. Then I restart the random build mode.
Will the new startup with random build mode use my databank of 500 genetics one or will it work separately.
So does the databank have anything to do with the generating?

If that is the case, I can run the genetics for a while and if I have enough strategies I can run the random for even better results?

I am looking for a method to find even faster strategies
Now I do +/- 140.000 strategies per hour (with my (maybe wrong) settings).

NOW, I have the following settings:

For the genetic options:
– Max of generations = 50
– Population size = 250
– Crossover probability = 80%
– Mutation probability = 40%

– Islands = 4
– Migrate every Xth generation, X = 15
– Population migration rate = 5%

– Filters are empty

– Initial population generation = not in use

– Fresh blood = replace 20% / every 4 generations

– Evolution management = Start again / Restart in sample (whole)
stagnates for 4 generations

—————————

For the ‘ranking’ filters (minimum to get as many strategies as possible):
– Profit factor (IS)> 1
– Profit factor (OOS)> 1
– Ret / DD ratio (IS)> 0.7
– Ret / DD ratio (OOS)> 0.7
– of trades = 50
– Net profit = 50

– Max. strategies in database = 3000 (never stops, replace the weakest strategies)

—————————

For the data I use the GBPUSD / H1 timeframe
– 2006.01.02 -> 2019.01.01
(2019.01.01 -> 2020.06.16 is for the retester if my strategies will work for the future)
– Precision = Fastest
– Data range parts = 10x 4% IS / 10x 4% OOS

—————————

For the Building blocks I select almost everything except Volume – bar and time – and some I dont like it.
I have 281 entry blocks, 5 exit types.

– Order types = everything except ‘MKT Enter/reverse’
– Exit types = everything except ‘exit after bars’
– Calibrate indicators = Max. 30 steps (auto calibrate)

—————————

– Trading directions = Both Long & Short (symmetry ON)
– Conditions = min 1 and Max 3
– Stop Loss same as Profit Target = Fixed: 40-250pips  and  ATR: multiple 1.4 – 10  / Period  20

Thanks,

welcome to the forum.

Random generation doesn’t use anything in the databank as an input. It generates strategies randomly, it doesn’t need any entry strategies.

It can work another way around – you can build for example 500 strategies that pass some basic filters using random generation or genetic evolution, and then use them as Initial population for a new genetic evolution.

And I think you shouldn’t focus on generation speed that much, more to the speed of accepted strategies per hour. I can make a setting that generates 1 million strategies per hour, but none will be accepted.

Ovewall, also be careful about curve fitting – genetic evolution and too much “playing” and reoptimizing existing strategies can lead to overoptimization, which is not what you want.

Hi,

Thanks for your reply.
Every day I learn more and more.

For the curve fitting:
If I reduce the changing steps for the calibrate indicators in building blocks? e.g. on 25 steps.
+ also for the order types and exit types, eg ‘Enter at stop’ min. 5 / max. 200 and step in 5 or 10 or even 15?
Will this reduce curve fitting?

I am also working to understand the genetics options very precisely:
Reduce Max generations and increase population size,
Migrate X generation size on eg 15 and population migration size on 1% => Will I get more diverse strategies? Or do I lose the better strategies too?

Because if I don’t choose the perfect settings in the genetics options and set too low filters in the ranking I get, as you say, too many strategies none of which will work. With too strict filters do I get every 1.000.000 one strategy through the filter?

Yesterday I also had 3000 strategies in the database with all positive in IS and OOS but as if they were all the same? None came through the retester?
Does this have to do with the migration size?

That is why I want to focus first in genetic options and then ranking. When I have this under control I want to manually set my building blocks.

Thanks

Thanks for the reply,

I think I have figured out the calculation behind the genetics.

Like my settings attached:
Max. generations: 70
Population Size: 100
Islands: 15

So means:
100 strategies x 15 times in 1 generation = 1500 strategies
So I will wait 70 generations for it to be renewed.
So 105,000 strategies for 70 generations
My computer does +/- 140,000 strategies per hour -> just over an hour en few minutes and the genetics are restarted??

In the meantime I have also adjusted the entire settings so that I get +/- 20 strategies per hour in my database.

Thanks!!!

Hello,

no need to limit number of steps or disable certain types of order in my opinion. The best ways to avoid curve fitting are: use enough data for the out-of-sample test, use various cross-check test to check strategy sensitivity, use walk-forward testing and ideally let the strategy trade in demo mode for certain amount of time so it proves its performance character

As for genetic options there is no one best. If you have more RAM you can work with larger genetic populations and less generations but you will also achieve success with smaller populations and higher number of generations. The genetic algorithm is flexible enough here. I quite like the approach of generating larger population and smaller number of generations. I try to generate as many member as possible. If there are some good strategies you do not need that much generations to “train” them for the better in my opinion. Less members in a population lower the change is there won’t be too much great strategy. Just let the numbers work in your favor here.

Also I would be careful about all the filters you apply. It is quite easy to filter a good strategy if you set the filters too strictly. You can apply filters gradually to better understands what filters cause the test to fail. You can then review the test setting to make sure it is not too strict

Here a screenshot of my 2 old servers at building.

Left old server with XEON E5440 2cpu’s 8 cores / 64gb RAM (I think from 2010)

Right newer one with XEON E5-2630 2cpu’s 12 cores / 80gb RAM (from 2014 I think)

I have enough ram (80gb) and I want as many diverse strategies in my databank as possible.
Because of this I try to keep the number of generations low and population size + island higher. So that I get many more diverse strategies in my databank (limit at 3000).

I hope to get less curve fitting with this ??

Its gonna my first building.
After this I will see what the retesting will give between 2019 and 2020 (this last year is not included in the builder, it is the future)

first thing first – i see, that you are generatin on EURUSD data, are those the original one from dukascopy? they are UTC0, what UTC using your broker?

do you know that you need to clone those data to a timezone of your broker?

thats a crucial basics

15 islands are nonsense – islands are nothing more that new independent random strategies and you have set to migrate after 65 generations, this will not happen to often…and i dont like migrating at all – its leading to duplicities. I am using 8 islands only for that purpose to run my CPU at 100% load

exiting friday at 23:00 is late and UTC0 data (if you are using not cloned dukascopy data)  dont have friday candles at 23 hours. UTC0 data ends at 20:59, so you will not be closing your strats on Friday

why dont you use any indicators in building, for example the OHLC?

generating STOP and LIMIT strategies together doesnt make sense, for bot you need different settings i think…and LIMIT strategies who know, does anyone trading them? I was trying to find for many years but without any luck

trailing stop min 40 max 500 doesnt make sense – your takeprofit has max 200, so trailing profit even by 40 pips is very big value

your ranking criteria – i dont like them, but it doesnt matter 🙂

but number of trades 25 for IS and OOS for 5.5 years of data is very low, i am using something like 20 trades per year as minimum to get statistically significant strategies with number of trades 300 or more

profit factor and RDD set as only 1 is a very low value and this kind of strategies will not pay you the risks

Thanks Hankeys!
Your information is a great gain in time for me!

I’ll make an adjustment in my settings and do a test. I’ll let you know.

I will also adjust my data to GMT +3 (icMarkets / raw spread account -> eurousd +/- 2spread).
I also have accounts with other brokers but the spreads are around 10 for eurusd.

Thanks!

for ICM use EST07 timezone – its UTC2 with US DST

Hi,

I’ve clone my data to UTC +2 for ICMARKETS,  but…
– From their website they have GMT+3 in summer time, i think thats UTC +3 from the official calculation? no??

For my other brokers they are UTC0, so I can use the standard data.

Thanks,

IC Markets seems to be Eastern EU which is UTC+2 winter / +3 summer