Not logged in
Viewing 15 posts - 1 through 15 (of 17 total)

Forums>StrategyQuant>General Discussion>Fake backtest IS OOS, overfitting

  • #259801 |
    86 Posts

    SQ generates a “good” strategy with good IS and OOS

    but it turns out SQ peeks into the OOS and fits curve to it

    why is OOS so fake then?

    How to disable SQ from peeking into OOS?



    117 Posts

    try just building a strategy on  IS and have no OOS. then test out of sample after the build

    1827 Posts

    You never avoid some kind of curve fitting in my opinion. That’s all we have as traders. We build models and we verify them on history. Even if you later use some extra data to verify that strategy (data never used in SQ) you also fit the strategy to that data. If the strategy does not work on that data you probably won’t use it but you will if the strategy ‘fits’ to that data :)

    86 Posts

    this is a bad idea

    manual inefficient work

    just stop SQ to peek into the future OOS and be honest with backtests and if OOS is no good then drop it and move on



    • This reply was modified 11 months, 2 weeks ago by gin.
    110 Posts

    I’ve always asked to myself the same question: If you select an OOS range data, will SQ actively blend strategies on the IS results to get a better fit on OOS? Or will just discard those that get the best fit on IS and don’t achieve the minimum criteria selected on OOS?

    I suspect that SQ works on the first way. So if you choose OOS, the Building process is influenced by this OOS because it forces the strategies to the best OOS instead of building on base only to the IS period. That’s my guess. So, if I’m right it would be better to build only with IS and make a second run on the data range that would be your OOS

    86 Posts

    if you do IS only

    Then you will have to do a  manual work by testing OOS and all these strategies will fail 99.99%

    What is the point in automation and this software then?



    110 Posts

    Well, you can always create a Custom Project to automate it. That’s what I did.

    I mean ‘did’ because from some weeks I started to Build without OOS. All the data I have goes to IS. Not bad my findings so far


    797 Posts

    using only IS without some robustnests tests and using genetics is a first way how to start loosing

    i tested many and many approaches and yes, i can tolerate opinions that you dont need some OOS testing, but without this check on never seen data of your model (workflow) you have no idea how your model is robust to the future

    You want to be a profitable algotrader? We started using StrateQuant software in early 2016. For now we have a very big knowhow for building EAs for every possible types of markets. We share this knowhow, apps, tools and also all final strategies with real traders. If you want to join us, fill in the FORM. 1500+ final SQX strategies for members running on demo accounts to verify the edge DEMO ACCS. We provide also strategies for indices - DAX, DOW JONES, NASDAQ, SP, UK, because we have realtick data from our brokers.

    110 Posts

    Tested many approaches as well. Got the conclusion after years that OOS doesn’t bring value to the process.

    It doesn’t hurt either so if you want to use it just in case, should be Ok I suppose

    32 Posts

    same problem for me… when i use OOS in the builder, genetic algorithm seems to look at the IS+OOS fitness to generate new strategies…

    For exemple, if  i use OOS in the builder with criterias like ret/DD(IS) > 2 and no criteria for OOS and let it generate 100 strategies, they all have good OOS.

    But if i use only IS with the same criteria and after, retest my 100 strategies on OOS, 99% are overfit and crash in the OOS area.


    So, for me, OOS is absolutly useless in the builder :/

    86 Posts

    using only IS without some robustnests tests and using genetics is a first way how to start loosing i tested many and many approaches and yes, i can tolerate opinions that you dont need some OOS testing, but without this check on never seen data of your model (workflow) you have no idea how your model is robust to the future


    why don’t you like genetics?

    you don’t believe in evolution and natural selection?

    only random?


    32 Posts

    oh shit… sorry… i’ve made a big mistake…

    after verification, SQX builder looks only IS fitness. the results are exactly the same if i use OOS during the building process or after. sorry.

    32 Posts

    on the other hand there is a big problem.

    If you generate strategies with the “backtest on additional market” activated, the builder erase the IS fitness and OOS fitness for look only at the IS+OOS fitness.

    So, if we use additional markets in the builder, the OOS become fake OOS and the builder overfit the strategies.

    If, like me, you use precedent generation as initial population, that break all the process and cancel your OOS area.

    110 Posts

    Hi Charles,

    I think you are wrong. After reading you I was concerned because I use several Crosschecks on my Buidling process and by any means I wouldn’t want that the optimization process would include the Additional Markets. So I took the SQ manual. According to the manual, ” the strategy is evolved only on the In Sample part of data”. So the Genetic Evolution; that is the optimization process. It would be working just on IS – at least I hope so!

    28 Posts

    I’ve built and am testing quite a number of portfolios for M5 H1 and H4 in MT4.

    The strategies in the portfolios range from heavily robustness tested strategies with WFM, MC etc. to no OOS at all.

    By far my best performing portfolio is my H4 portfolio consisting of 96 strategies accross 12 FX pairs and gold (13 pairs all up).

    This, ironically, is the portfolio I built consisting of strategies built using only the builder task using genetic evolution with no OOS. None of these strategies are robustness tested either. All I did was build them using ridiculously high spreads and the maximum data available.

    I wanted to run this experiment because I figured if I can get a really good equity curve for each strategy, with spreads I’ll most likely never see in real trading, over a very long period of time, on a high time frame where there is likely to be far less noise, that even though this is “curve fitted”, it will need to continue to have performed in a large array of varying market conditions for a pair.

    Seems to be working very well :)

    • This reply was modified 10 months, 2 weeks ago by Insanity82007Insanity82007.
Viewing 15 posts - 1 through 15 (of 17 total)

You must be logged in to reply to this topic.