Data Window size

Hi guys, i’m here to open a productive topic.

I’m wondering why Mark and the developer team haven’t put a main feature yet.
I’m talking about the automatic data windows size.
Like you know you can’t choice a random windows size, or at least yes you can but this can be unproductive. I’m explain better myself, when you start a process of building, we are searching for productive/robust strategies in the nearest future, and of course for example, if you are searching H1 strategies that are working in the next month, is totally useless use 12 years data old, don’t you think? Perhaps maybe not everyone know, there is a scientific way to spot the right windows size based on your purpose.
So here it is, before to write it down, i wanna hear some community thought about it, how do you choose the windows size? Do you know the scientific method?
This is for you Mark, do you know it? Do you and your team know how to calculate the right windows size of the data?

After a little bit of comments i’ll explain how to do it.

I wanted to know more about it.

Seems to me that the more data a strategy worked on in the passed the more likely it is to work on data going forward. Since you have the possebility to make plenty such strategies it would be better to just retest them on recent data to see which ones is working now and use those. I think you will find that the % that works will then be much higher then just strategies made on a recent window.

Not totally correct, yes of course, more a strategy is working during the History and more is reliable through different kind of markets, but we aren’t searching for one “holy grail” strategy working well always and working forever.
A strategy created on 12 years or more, will be a very slow strategy, and will miss a lot of gain opportunities. And consequence of the big data window size is an overfitting  of the builded strategies, to aim a good return/DD during all this time SQ will create a very complex rule/rules of entry exit, managing etc.

IMHO I prefer create strategies more frequently, and optimising/deleting and swapping it when they lose the reliability by monitoring all the stats.

It’s weird that this topic have so low success, i was thinking a more light on by the community…

This topic is difficult. For example make strategies on last years data but then exclude this recent may. You find that 95 % is looser in may well depending on how you rank them . I did another test as well yesterday, Tested 3800 EU that was really good during 2018 rdd >5 ( they were from a batch of 25000).  700 of these 3800 worked great month Jan-march the rest was losers  . 525 of these worked great in April 2019  80%.  But only 3 of these 525 was profitable in may the rest really big losers. So then I tested all 25000 on may 2019 they are made and tested on data from 1986-2018-09. Now I had about 2700 that was profitable in may 2019 .  Now I went further and test them month by month for no loosing months ( but flat is okey) going back 12 months.  I have 37 left they do not have an RDD above 5 on 2018 so they cant be from my original test of the 3800.

So maybe that a good workflow. I know have 37 strategies that worked great the last 12 months but also ok for the last 30 + years.

Attached is how they look on Duka.

“Do you know the scientific method?” Tell us

Clonex i’ll tell you, but before i wanna listen some different point of view about it.

Dead end for this topic?  For example on EURUSD H1 in this moment there is a valid window

From: “10 October 2018”
To:       “26    April    2019”

Tradable in two next windows:
Window 1: From: 27 April 2019 to 21 May 2019 with accuracy of 90%

Window 2: From: 22 May 2019 to 27 June 2019 with accuracy of 80%

No one guess how i know this?

C’mon guys be active!

I have no clue. Only thing i know is that You are referring to History.  I suppose it is easy to prove later by testing  performance on a couple of hundred live strategies on Your suggested history data

Hopefully You are right thought  🙂

hi …Gianfranco    hurst exponent….for scan   symbols  and   data Windows….up 0.5    ?

is my little idea….automatic scan   with  min and max  days…like  WFM

thanks

Gianfranco

Someone say it, finally!!!
Yes is the Hurst Exponent, we have a lot of statistic tools to be sure of working properly.
Now i explain my problem, i have a lot of python scripts: to clear the data, find the data window,  correlation between data, manage the building, find the uncorrelated strategies, monitoring the equity and chose for reoptimization or deleting.

There are a lot of trick to speed up the entire process, for example, correlation of strategies with a very big pool, 10k strategies, instead of testing entirely with all the data window, i’m shrinking the data window to 1/3 (IS-OOS) of the original, and using those equities, less computation power, split the pool in two parts (good correlations between -0.5 and 0.5, and bad correlations between -1 and -0.51, 0.51 to 1 retesting the correlations of goods with the original data size to be sure of the partial results, and that’s it, a lot of energy and time saved.

I’m asking myself why they didn’t include yet all this tools inside the suite?
However if someone need those scripts i’ll share it, but i prefer share it with the development team to join it inside the software.

yes, if you are or have a good programmer many things could be much more easier….we have for example python script for old SQ to control MC tests, etc.

i have read some article for these hypotesis – https://www.mql5.com/en/articles/2930

but i dont know how to use them in our “bussiness”

I do not belive this would work using timeseries trying to predict short future periods in FX which is totally randome. If it works one period it was only luck. I rather Optimize an entry over 100 years of data to get an good average performance and then by walfkforward try to adapt the exits instead to current market. This way you can also  get strategies with >90% winning 6 months periods looking back 30 years and automate the workflow  with current tools.