Fake backtest IS OOS, overfitting

SQ generates a “good” strategy with good IS and OOS

but it turns out SQ peeks into the OOS and fits curve to it

why is OOS so fake then?

How to disable SQ from peeking into OOS?

try just building a strategy on  IS and have no OOS. then test out of sample after the build

You never avoid some kind of curve fitting in my opinion. That’s all we have as traders. We build models and we verify them on history. Even if you later use some extra data to verify that strategy (data never used in SQ) you also fit the strategy to that data. If the strategy does not work on that data you probably won’t use it but you will if the strategy ‘fits’ to that data 🙂

this is a bad idea

manual inefficient work

just stop SQ to peek into the future OOS and be honest with backtests and if OOS is no good then drop it and move on

I’ve always asked to myself the same question: If you select an OOS range data, will SQ actively blend strategies on the IS results to get a better fit on OOS? Or will just discard those that get the best fit on IS and don’t achieve the minimum criteria selected on OOS?

I suspect that SQ works on the first way. So if you choose OOS, the Building process is influenced by this OOS because it forces the strategies to the best OOS instead of building on base only to the IS period. That’s my guess. So, if I’m right it would be better to build only with IS and make a second run on the data range that would be your OOS

if you do IS only

Then you will have to do a  manual work by testing OOS and all these strategies will fail 99.99%

What is the point in automation and this software then?

Well, you can always create a Custom Project to automate it. That’s what I did.

I mean ‘did’ because from some weeks I started to Build without OOS. All the data I have goes to IS. Not bad my findings so far

using only IS without some robustnests tests and using genetics is a first way how to start loosing

i tested many and many approaches and yes, i can tolerate opinions that you dont need some OOS testing, but without this check on never seen data of your model (workflow) you have no idea how your model is robust to the future

Tested many approaches as well. Got the conclusion after years that OOS doesn’t bring value to the process.

It doesn’t hurt either so if you want to use it just in case, should be Ok I suppose

same problem for me… when i use OOS in the builder, genetic algorithm seems to look at the IS+OOS fitness to generate new strategies…

For exemple, if  i use OOS in the builder with criterias like ret/DD(IS) > 2 and no criteria for OOS and let it generate 100 strategies, they all have good OOS.

But if i use only IS with the same criteria and after, retest my 100 strategies on OOS, 99% are overfit and crash in the OOS area.

So, for me, OOS is absolutly useless in the builder :/

hankeys wrote:

using only IS without some robustnests tests and using genetics is a first way how to start loosing i tested many and many approaches and yes, i can tolerate opinions that you dont need some OOS testing, but without this check on never seen data of your model (workflow) you have no idea how your model is robust to the future

why don’t you like genetics?

you don’t believe in evolution and natural selection?

only random?

oh shit… sorry… i’ve made a big mistake…

after verification, SQX builder looks only IS fitness. the results are exactly the same if i use OOS during the building process or after. sorry.

on the other hand there is a big problem.

If you generate strategies with the “backtest on additional market” activated, the builder erase the IS fitness and OOS fitness for look only at the IS+OOS fitness.

So, if we use additional markets in the builder, the OOS become fake OOS and the builder overfit the strategies.

If, like me, you use precedent generation as initial population, that break all the process and cancel your OOS area.

Hi Charles,

I think you are wrong. After reading you I was concerned because I use several Crosschecks on my Buidling process and by any means I wouldn’t want that the optimization process would include the Additional Markets. So I took the SQ manual. According to the manual, ” the strategy is evolved only on the In Sample part of data”. So the Genetic Evolution; that is the optimization process. It would be working just on IS – at least I hope so!

I’ve built and am testing quite a number of portfolios for M5 H1 and H4 in MT4.

The strategies in the portfolios range from heavily robustness tested strategies with WFM, MC etc. to no OOS at all.

By far my best performing portfolio is my H4 portfolio consisting of 96 strategies accross 12 FX pairs and gold (13 pairs all up).

This, ironically, is the portfolio I built consisting of strategies built using only the builder task using genetic evolution with no OOS. None of these strategies are robustness tested either. All I did was build them using ridiculously high spreads and the maximum data available.

I wanted to run this experiment because I figured if I can get a really good equity curve for each strategy, with spreads I’ll most likely never see in real trading, over a very long period of time, on a high time frame where there is likely to be far less noise, that even though this is “curve fitted”, it will need to continue to have performed in a large array of varying market conditions for a pair.

Seems to be working very well 🙂

Honestly, a pretty interesting approach. Like, we are curve fitting to some extent – why not embrace that & just make it tougher? Also, how long did it take for you to reach that many strategies? Just from a curiosity / comparison standpoint(?)