Monte Carlo simulations seem to conclude nothing about future performance of a strategy

Just found this interesting article from Daniel today: http://mechanicalforex.com/2016/05/do-monte-carlo-simulations-say-anything-about-system-robustness.html

 

Very interesting find and in conclusion with my findings so far…. I´ve been running systems live for about 8 years in total (not just from SQ) that have been Monte Carlo simulated before and yet have found no conclusion so far about that strategies that had bad Monte Carlo simulation results before going live did worse than the ones which had great Monte Carlo simulation results. Daniel describes it very well, Monte Carlo simulations tend to prefer strategies that work well on smoothed data only and can make you bin profitable live strategies that work on precise price-action only and still would do great going forward (like Daniel describes it with the company that is buying forever after 2 new highs, etc). So in fact, Monte Carlo simulations can make you discard really good strategies that would have done nice going forward and hence work counter-productive for us.

 

Does anyone else with a solid base of live trading have any other conclusions about Monte Carlo stability versus live trading so far? Would be great to hear…

@Treshold: that strategy you mention there, I run it exactly like that for 1,5 years already 🙂 Think everyone of us has found it exactly like that, it´s one of the most common ones (break out). Anyhow, this one is liked by MC for the reason that it´s based on break outs. It also passes all tests here, even with distorted data (70% chance, 30% ATR variation, 400 tests). The reason though is this: it´s a breakout strategy based on rather big breakouts and big trailing values, it also refers to the last 300 highs or lows, so some distortion in the data, even heavy ones or variation of the parameters still lets it pass the MC fine because the breakouts anyway occur even in heavily distorted data. These type of strategies are favored by MC too.

 

However, I have a strategy which I´ve created about 6 years ago (not in SQ of course) which indeed refers to Open[85] and other very specific values in time. It also uses nice amount of parameters, quite a lot actually that we all would call curve-fitted. Yet this strategy performs well since 6 years! I did code it in SQ a while ago and run MC, any kind of test (apart from spread and slippage) made it look TERRIBLE – that kind of strategy that you would right away put to the bin. Yet it did better than most strategies I´ve ever created and that had a great MC. So yes, as I already mentioned in the initial post, strategies that don´t pass MC can as well be very profitable going forward, hence MC doesn´t say anything about future performance or stability, otherwise this 6 years old strategy should have failed already. That´s also what Daniel concludes and why I´ve posted this because I wanted to hear what others have found about this.

 

Actually it is also not to hard to simulate this. We can do it this way:

 

1) Create a strategy on data from 1989 to 2000

2) Run MC on that strategy also from 1989 to 2000 and note the results (good or bad MC?)

3) Now run the strategy on data from 2000 to 2016, note the results too (performed well or not?)

4) Note these results in Excel and do it for at least 100 strategies to get a meaningful statistical relevance.

5) See if any relation can be drawn between a good (pseudo) OOS performance from 2000 to 2016 and the MC simulation results on data from 1989 to 2000 (good MC = good pseudo OOS results in >50% of the cases?)

 

I think it´s indeed best to not start such topics here anymore. And actually I´ve kept most of the stuff I´ve found lately about creating more successful strategies and how to speed up SQ even further out of here already  since it´s not the first time that there is flaming instead of a good discussion evolving around such posts. It seems it´s just a minority here that wants to really discuss important topics about how to get profitable in automated trading apart from the usual “it should work like that but I have no evidence nor idea if it really does, I put it live and fail again, damn…” stuff and instead go in circles forever without making any money. Now I totally understand the value of a closed community where everyone paid to be part of it – you get great quality discussions instead of stupid flaming because everyone did put money on the table for it and really wants to succeed and learn something there. It´s a community free of time-wasters like we have many here unfortunately.

 

P.S.: Just got Daniels (unfortunately understandable) reply. Thanks to some of the users here that are not interested in a meaningful discussion but prefer to insult and flame in this thread:

 

“Hi Geektrader,

Thanks for letting me know, I’â„¢ll keep an eye on it! However I am not a big fan of flaming so I will refrain from getting into the aforementioned discussion. Thanks again for commenting,

Best Regards,

Daniel”

Geek. Just Chillout. We have here good discussion including yours and others posts. This thread is excellent. So unhide and letsgo start together theese tests .  At this point i can only agree from my own research that for  me  some strategy settings (1day , 4 hour TF strategies) cannot pass via my MC tests. Daniels findings are good starting point for research. And without any discussion he is running one of the best blogs 😉 🙂

I’ve worked in a real trading company, you need a thick skin to survive in those environments.  :ph34r: 

Geek. Just Chillout. We have here good discussion including yours and others posts. This thread is excellent. So unhide and letsgo start together theese tests .  At this point i can only agree from my own research that for  me  some strategy settings (1day , 4 hour TF strategies) cannot pass via my MC tests. Daniels findings are good starting point for research. And without any discussion he is running one of the best blogs 😉 🙂

 

Updated my previous post about this with Daniels reply who decided to not opt in here thanks to the flamers too. It´s not about chilling out, I chill out very well, I just don´t have any time to waste on flamers like notch or mikeyc, simply not worth a single minute of my time. Either we have a fundamental and meaningful discussion about how we can test things instead to assume anything and without flaming, or I simply keep on doing the research on my own like I´ve anyway been doing lately instead to post it here since it always ended in wasting time with flamers and nay-sayers that don´t have any kind of proof nor tested it at all, but have time to post childish comments. Sorry, but I am not 19 or 20 anymore like these guys seem to be and I know that life-time is our biggest value and gift. People that don´t understand this are “off my list” within the blink of an eye.

these days bought asirkuy mmbrshp i have to prepare data and then  ill do theese tests with my current strategies and results will post here . i agree . dont waste time

 

Great to hear clonex. I guess your current strats are created on 15 years of data 2000 to 2016 or something in that range? If so, it´s even easier for you, you can do it the other way around then:

 

1) Run MC simulations for those strats on your current data set 2000 to 2016 and note the results if good or bad MC

2) Run the strats on the new Asirikuy data from 1989 to 2000 and see if they have a good pseudo OOS performance there and write that down too.

3) See if any correlation between good MC on 2000 to 2016 and good pseudo OOS from 1989 to 2000.

 

Really looking forward to it!

Great to hear clonex. I guess your current strats are created on 15 years of data 2000 to 2016 or something in that range? If so, it´s even easier for you, you can do it the other way around then:

 

1) Run MC simulations for those strats on your current data set 2000 to 2016 and note the results if good or bad MC

2) Run the strats on the new Asirikuy data from 1989 to 2000 and see if they have a good pseudo OOS performance there and write that down too.

3) See if any correlation between good MC on 2000 to 2016 and good pseudo OOS from 1989 to 2000.

 

Really looking forward to it!

Ill doit and let you know 😉

1) Create a strategy on data from 1989 to 2000

2) Run MC on that strategy also from 1989 to 2000 and note the results (good or bad MC?)

3) Now run the strategy on data from 2000 to 2016, note the results too (performed well or not?)

4) Note these results in Excel and do it for at least 100 strategies to get a meaningful statistical relevance.

5) See if any relation can be drawn between a good (pseudo) OOS performance from 2000 to 2016 and the MC simulation results on data from 1989 to 2000 (good MC = good pseudo OOS results in >50% of the cases?)

Again

 

MC is for measuring RISK.

Your backtest might say 4% DD.
Monte Carlo sim will show a 7% DD is possible. This is what you adjust your trade size to.
 but the real purpose of Monte Carlo is for finding likely drawdowns.
 

 

I’m sort of agreeing that’s its useless for robustness testing, because in my view its not for robustness testing. Its for position sizing, and for projecting a future equity-band.

I agree, MC is for adjusting your risk, not proving robustness.^ I think other tests are far better for robustness than MC.

MC is for this-
You build a sys that backtests 10% DD wit 1% risk per trade.
You monte carlo it 10,000 times. You find 95% confidence 20% DD.
You really can only tolerate 10%DDs and that’s why you chose this system.
What the monte carlo sim is showing is that you should reduce you risk to 0.5% per trade to keep a 95% confidence level you’ll stay within a 10% DD.

Thats what most professionals use MC for. Not robustness testing but it seems to be conventional wisdom that its a good robustness test… its not. I tend to agree with this.

these days bought asirkuy mmbrshp i have to prepare data and then  ill do theese tests with my current strategies and results will post here . i agree . dont waste time

I think this was the first gif posted on SQ forums.
Many great things to come.

this is realy interesting topic, but also sometimes to personal.

 

we should discuss the merit, not the persons involved. 

 

Geektrader, you posted a few interesting articles lately that bring another opinion and contradict my own.

 

Despite Daniels test I still think it is important to test the strategy on different symbols/timeframes and it gives some indication whether the strategy is curve fitted to given data.

I also still believe Monte Carlo is a good tool to estimate risk and the range of performance we can expect.

 

We plan to use bootstrapping in SQ4 as additional robustness test, I just have to implement it.

 

By the way, I was also a member of Asirikuy some time ago, and I found Daniels articles very interesting.

Great discussion People!

 

For me MC is for my risk determination as others have said in this thread, infact I don’t use the MC in SQ other than to see 1) what the 95% WC is in terms of original strategy and 2) just a visual indication of MC runs that they aren’t “too all over the place”  The MC Analysis in SQAnalysis its much better and is what I use most of all, I would really like to see this in SQ, also I use the MC in SQA for strategy failure determination, i.e. my OOS on demo/live accounts are they inline with my Worst case DD.  

 

As @threshold has said I also agree with him and use it myself for determining position size and risk,

 

Since first using SQ I’ve always thought that the MC in SQ has been misnamed as robustness tests and should more along the lines of “WC/DD and Risk Analysis”

 

I’ve stated many times in this forum that I’m also a member of Daniels, Asirikuy, for a number of years now and value the information he has to offer and have not yet read that blog post (haven’t caught up on my news feeds this week) so will have a gander over the weekend.

 

Robustness tests for me is ability to trade profitably under many different circumstances, i.e. different pairs, market conditions, different broker data.  As of late K.I.S.S. has been my motto in strategy generation.   I just run my mining with only the simplest indicators used, time will tell if they turn out to be more robust.

 

This then sort of leads us back to the discussion we had on a previous thread where I tended to agree with you @geektrader that what we are seeing as of late few years with the upsurge of intelligent system generation, GA, NN and so forth.   I think for us as retail traders the forex market dynamics may have changed and I question the validity of historical data on certain timeframes, because tools like SQ totally rely on history to produce and test robust strategies, so is this approach valid including MC analysis as a whole if the dynamics have changed?

 

I do however believe they will be trends as markets are moved by fundamentals and govt policy, i.e. a country will have a target interest rate that they want to achieve to control other economical fundamentals like spending, exports etc, so flow of cash in the forex market will flow in or out of the country depending on these policies.  So are applications like SQ and Asirikuys Kantu going to work in the future, again time will tell, but again I still tend to be positive on this front as every second of the day new market data is being generated so will these systems will eventually adapt?  who can tell? only time, but in the mean time I’ll be plugging away on the grind of my strategy generation using this great tool.

 

Also in terms of robustness I tend to believe that in this market (forex) where the exchange is decentralised the broker and their antics has a lot to do with things, especially for the majority of us that are retail traders.   What I’ve seen on two different brokers on their premium retail accounts (pepperstone and Go Markets Australia) is that strategy performance seems to be impacted proportionally to the amount of capital I have in the account, i.e. the more I have in there, the worse my strategies perform, while the account with the least seems to perform very well.  I’ve also check by transferring bulk of $ from the high account to my smaller account on the other broker so it became my larger account, lo and behold this account started to perform poorly while my now small account started to perform as expected, both these accounts run same strategies at same risk levels. only difference is brokers and account capital.   My thoughts are the golden level is 5K but have in no way validated this, I am now looking at opening more accounts and spreading my capital across them in smaller accounts and making sure I keep the $ under 5K to see if overall my portfolio performance picks up.

 

Lastly I do think that to improve performance / robustness that strategies need to trade under control of a trade manager that will shadow trade and determine final contract sizing dynamical based on the performance of the strategy under live conditions, part of this is to do a MC within the management loop on each strategy it manages to determine its current risk, so that it will effectively take a poorly performing strategy offline automatically as it starts failing, but inverse to that it should bring it back online as market conditions come back inline with it underlying fundamentals and even increase contract sizing if strategy performance starts increasing, so effectively the strategy still trades in the background.  I think I posed a feature request for this on the relevant forum here.

 

I do sincerely hope that no-one stops their input into this community forum I’m sure we can all be adults about the discussions we have so we can positively input to help each other, I know we can get excited at times 😀  Me personally I value everyones contributions, so thank you one and all for all of your input.

 

For me the journey is still going!

 

Stephen

When i see shit like this:

I just know trading and finding simple robust strategies is not this f’ing complicated. Quants like to get fancy, especially ones that worked for banks… They all go back to simplicity in the end. Ernie Chan wrote a few books on this where he does get very algorithmic-heavy. He programmed AI HFTs for banks (Goldman Sachs I think) and when he started trading his own money he went back to simplistic models and robustness tests. Some of these pdf papers really go above and beyond what is actually requires to finding a robust strategy that makes money. Its not that complicated.

As a generalization, very few things in trading are true all the time, but they can be mostly true.

My self-made ADX systems that take advantage of a real pattern that happens in ANY market and assets fail Monte Carlo because they only have about 80 trades of historical data per pair. Its a rare event that only happens in each pair maybe 3-5 times per year and yet its incredibly reliable in ANY market. They’re the most successful thing mainly holding up my portfolio. So from my direct experience this taught me to trust MC less.
Some perfect equity curve generated strategies that take advantage of a non-existent pattern (just buy stop X distance away curve fitted to the historical deviations of the currency pair) pass MC with flying colors and break down as soon as they go live. From my direct experience this taught me to trust MC less.

I’m not saying its completely useless and I’m also not 100% trusting of MCA. Its not that easy as Monte Carloing a strategy and its a winner. No body on this forum would be bitching anymore and they’d all we wildly successful, even you if that were true.
Nothing is ever 100% reliable in trading. Sure maybe it can be another filter, but first and foremost its for risk sizing.
I just use WFM for exactly what your using MC for (variable parameters/data sets) but WFM actually has criteria that i can define for pass/fail. MC in SQ doesn’t.

Good talk,

From my experience, one of the very best robustness tests are:

¢ 1 ~ 2 Timeframes Above/Bellow the current Timeframe,

¢ The parameters of the indicators must work in range of the 10~25% change the same as the current original parameters,

 

kind of MC…,

in an alternative software to SQ i found that they have different timeframes built-in into the MC tests.., that might improve the work flow with SQ4 🙂

 

have a great weekend!

 

Also in terms of robustness I tend to believe that in this market (forex) where the exchange is decentralised the broker and their antics has a lot to do with things, especially for the majority of us that are retail traders.   What I’ve seen on two different brokers on their premium retail accounts (pepperstone and Go Markets Australia) is that strategy performance seems to be impacted proportionally to the amount of capital I have in the account, i.e. the more I have in there, the worse my strategies perform, while the account with the least seems to perform very well.  I’ve also check by transferring bulk of $ from the high account to my smaller account on the other broker so it became my larger account, lo and behold this account started to perform poorly while my now small account started to perform as expected, both these accounts run same strategies at same risk levels. only difference is brokers and account capital.   My thoughts are the golden level is 5K but have in no way validated this, I am now looking at opening more accounts and spreading my capital across them in smaller accounts and making sure I keep the $ under 5K to see if overall my portfolio performance picks up.

 

 

Hi Steven,

 

why is the perfomance worse on bigger accounts? Is it because of slippage? This is the only think which should make difference in profit between two different brokers, isn’t it? (spread is different-off course, but this should had not the impact you wrote about)

 

Patrick