Not logged in
Viewing 15 posts - 1 through 15 (of 76 total)

Forums>StrategyQuant (formerly named Genetic Builder)>General Discussion>Monte Carlo simulations seem to conclude nothing about future performance of a strategy

  • #115162|
    Customer
    645 Posts

    Just found this interesting article from Daniel today: http://mechanicalforex.com/2016/05/do-monte-carlo-simulations-say-anything-about-system-robustness.html

     

    Very interesting find and in conclusion with my findings so far…. I´ve been running systems live for about 8 years in total (not just from SQ) that have been Monte Carlo simulated before and yet have found no conclusion so far about that strategies that had bad Monte Carlo simulation results before going live did worse than the ones which had great Monte Carlo simulation results. Daniel describes it very well, Monte Carlo simulations tend to prefer strategies that work well on smoothed data only and can make you bin profitable live strategies that work on precise price-action only and still would do great going forward (like Daniel describes it with the company that is buying forever after 2 new highs, etc). So in fact, Monte Carlo simulations can make you discard really good strategies that would have done nice going forward and hence work counter-productive for us.

     

    Does anyone else with a solid base of live trading have any other conclusions about Monte Carlo stability versus live trading so far? Would be great to hear…

    #137219
    Customer
    941 Posts

    For me the MC tests are about varying the spread, the slippage and the data.  I never use MC to vary the parameters.

     

    In this case MC seems to be useful, as it highlights which strategies are sensitive to data feed and execution changes.  I don’t want to vary the parameters in a MC simulation.

     

    Does it mean I bin possibly good strategies?  Maybe. 

    #137220
    Customer
    734 Posts

    MC is for measuring RISK.

    Your backtest might say 4% DD.
    Monte Carlo sim will show a 7% DD is possible. This is what you adjust your trade size to.

    If you want to use monte carlo to see if a system can blow up you need to do millions of iterations to eventually find that 0.001% chance that it had 100 consecutive losers which banks and HFs will do but the real purpose of Monte Carlo is for finding likely drawdowns.

    Real robustness testing is seeing how it performs on alternate time frames ( I make sure my H1 systems work on M15, M30 AND H4), OOS data(earlier years and later years), alternate assets/pairs(I make sure a EURUSD will work on GBPUSD/USDCHF), wider spreads, wider slippage, longer historical data, etc etc etc.

    #137222
    Customer
    522 Posts

    Forgive me: I ran this type of experiment ages ago using close to 100000 models.  I’ve recreated a simple segment of the experiment using a smaller data set.

     

    Attachments in this forum are visible only for registered users.
     
     
    I created a Support Vector Machine to determine the efficacy of robustness test results for predicting the future profitability of models.  I used the in sample robustness data as the input to the SVM.  The models were built using up to 15 years of data. However, the size of the training set was varied depending on the experiment batch from which the models were drawn.  Each model had 20% of data reserved for testing.  The desired data used for the research is the annual return of the test period.
     
     Thus, the inputs as seen in the image above are:
     
    Sharpe Ratio (RT), Ret/DD Ratio (RT), R Expectancy (RT), Profit factor (RT), Net Profit (RT), Max DD % (RT), Annual % Return (RT)
     
     
    The annual return was split into two categories, 1. an annual return of at least 50% and 2. an annual return of less than 50%.
     
    The data fed into the SVM was split into training – 75%, cross validation – 15% and testing – 10%
     
     
    Attachments in this forum are visible only for registered users.
     
     
    The stats in the confusion matrix above depict the performance of the SVM predicting the future performance of the models.
     
     
     
    Sum of return obtained by classification No of 1 models Actual annual return per model 5604 109 51
     
     
    The above is taken from the spreadsheet.  The sum of the out of sample return correcting predicted by the SVM is 5604%, the number of models selected by the net is 109 so the average return per model is 51%.  
     

     

    I performed sensitivity of the RT inputs to determine which made a significant contribution to prediction accuracy:

     

     

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

    Attachments in this forum are visible only for registered users.

     

     

    The original SVM (larger data-set) achieved a 75% prediction accuracy and this is reflected in live model performance.  For me, sophisticated Monte Carlo robustness testing is invaluable.

    #137223
    Customer
    734 Posts

     

    Forgive me: I ran this type of experiment ages ago using close to 100000 models.  I’ve recreated a simple segment of the experiment using a smaller data set.

     

     
    Summary
    I create a Support Vector Machine to determine the efficacy of robustness test results for predicting the future profitability of models.  I used the in sample robustness data as the input to the SVM.
     

     

     

    QA 4 basically has this in equity chart form.

    #137224
    Customer
    522 Posts

    QA 4 basically has this in equity chart form.

     

    I haven’t finished the post yet.

    #137227
    Customer
    645 Posts

    I have invited Daniel over here, maybe he wants to reply to the thread directly.

     

    I absolute agree with spread, slippage sensitivity etc, but that isn´t a MC simulation at all. I´ve been doing this without any MC since I started out. Just do various backtest runs with different spreads and slippage values to see if the strategy is spread sensitive in NORMAL backtests and you get that idea as well. However, apart from this, REAL MC never told me anything about if the strategy works better than others live, at least that´s my conclusion after 8 years. Funnily almost 80% of systems that failed MC (using distorted data-sets + parameter variation, so “real” MC), did well going forward, while the ones that survived these MC simulations did just as well with a 75% success rate in live trading. So no real difference.

     

    However, what becomes clear is that the ones that failed MC simulations are the ones that use specific PA relations most of the time, e.g. that refer to “Open[85]”, while the ones that do well with the MC simulations are the ones that use for example MA´s with bigger values (300 to 500 bars), which underlines the findings of Daniel that MC prefers “smoothed” systems since they work better on distorted data, yet seem to say nothing about their future stability at least for him and for me since my “Open[85]” systems did just as well in live trading.

     

    Using different TF´s to simulate stability btw is nothing else than using distorted data of the main TF, MA strategies (smoothed strategies) just work better there as well in the MC simulations, yet seem to conclude nothing going forward.

     

    My findings hence are: of course to make sure that strats are not spread and slippage sensitive, but apart from this it didn´t make a difference for me in the live trading success rate if they survived the other MC tests or not.

    #137228
    Customer
    522 Posts

    I have invited Daniel over here, maybe he wants to reply to the thread directly.

    I can imagine the invitation:

    “Please Daniel come to protect me at the SQ forum. I have told them you are my very good friend but I won’t tell them you’re also my secret internet protector. I will always be your little sugar and you are my Dannipoops.”

    #137229
    Customer
    734 Posts

    I’m sort of agreeing that’s its useless for robustness testing, because in my view its not for robustness testing. Its for position sizing, and for projecting a future equity-band.
    It is especially useless for proving a strategy is over-fitted. The more a strategy is curve fitted the better it looks in Monte Carlo(randomized trade order). However, the more a strategy is curve fitted, the worse it performs on other pairs and other timeframes. There are always some exceptions though in trading. I disagree that moving averages/bands/candlesticks/and ‘bars since exits’ cause strategies to fail on alternate timeframes/pairs. Its sometimes true but not always. I’ll post an example.

    I appreciate his contrarian articles and yours.

    #137230
    Customer
    645 Posts

    At notch: wow, how old are you and how few arguments do you have to come down to that level? I have definitely no time to waste with persons that have such a low IQ, you went straight to my ignore list.

     

    Threshold: yes, I agree with that indeed, I use MC for WC scenario tests to see when a strategy needs to be stopped trading. For the rest, it´s not because I don´t believe in this, but because it has shown in 8 years of live trading to really tell me nothing about future performance and system stability (overfitted or not?), apart from the usual spread / slippage test that are useful indeed of course to spot scalpers that will never work live. I am open minded though if someone has other live stats, apart from theoretical ones why it should work.

    #137231
    Customer
    941 Posts

    I can imagine the invitation:

    “Please Daniel come to protect me at the SQ forum. I have told them you are my very good friend but I won’t tell them you’re also my secret internet protector. I will always be your little sugar and you are my Dannipoops.”

    :D Made me chuckle

    #137232
    Customer
    734 Posts

    Pending live stats….
    I try to stick close to what I’ve read from traders who been successful for 20+ years. Is Daniel successful in building systems? It seems he writes articles about what’s not successful (and only data mines them?). Seems like hes still on the journey like many of us.

    #137235
    Customer
    645 Posts

    :D Made me chuckle

     

    sqf.jpg
     
    Makes me chuckle too  :lol:
    #137236
    Customer
    645 Posts

    Pending live stats….
    I try to stick close to what I’ve read from traders who been successful for 20+ years. Is Daniel successful in building systems? It seems he writes articles about what’s not successful (and only data mines them?). Seems like hes still on the journey like many of us.

     

    Yes indeed, but most of the time these are manual traders…. Daniel´s systems are picking up lately it seems, his approach is different, they data-mine on GPUs in the community, hence can test millions of combinations within a few minutes / hours. They explore the whole possible space of combinations by brute forcing it on this network of GPUs. Once a profitable system is found, they right away boot-strap the underlying data for a few million times and the system has to do well on all those data-sets as well, which eliminates mining bias by ~98%. Only then a system is being traded live, so far they have a few hundred fullfilling these and are making money if looking at the MyFxBooks of a few of the users that publish their portfolios (which you can freely compile with different portfolio theory algos).

    #137238
    Customer
    436 Posts

    Monte Carlo is part of puzzle, together with other steps its useful tool. It is not perfect and some good strategies will not pass trough but most of bad fail and thats important. I think Daniel wanted to mention that not all strategies that fail MC are bad. The process benefits from this test more than looses.

     

    @Notch Funny.

    @others Dont be cool as cucumbers, it was for laughs. ( i laught at least)

Viewing 15 posts - 1 through 15 (of 76 total)

You must be logged in to reply to this topic.