Reply

Monte Carlo simulations seem to conclude nothing about future performance of a strategy

63 replies

geektrader

Customer, bbp_participant, community, 522 replies.

Visit profile

7 years ago #115162

Just found this interesting article from Daniel today: http://mechanicalforex.com/2016/05/do-monte-carlo-simulations-say-anything-about-system-robustness.html

 

Very interesting find and in conclusion with my findings so far…. I´ve been running systems live for about 8 years in total (not just from SQ) that have been Monte Carlo simulated before and yet have found no conclusion so far about that strategies that had bad Monte Carlo simulation results before going live did worse than the ones which had great Monte Carlo simulation results. Daniel describes it very well, Monte Carlo simulations tend to prefer strategies that work well on smoothed data only and can make you bin profitable live strategies that work on precise price-action only and still would do great going forward (like Daniel describes it with the company that is buying forever after 2 new highs, etc). So in fact, Monte Carlo simulations can make you discard really good strategies that would have done nice going forward and hence work counter-productive for us.

 

Does anyone else with a solid base of live trading have any other conclusions about Monte Carlo stability versus live trading so far? Would be great to hear…


🚀 Unlock Your Edge in Automated Forex Strategy Development 🚀

Historical Forex Data Starting From 1987, 28 Pairs, M1, 99% Error-Free, Lifetime Free Updates

0

mikeyc

Customer, bbp_participant, community, 877 replies.

Visit profile

7 years ago #137219

For me the MC tests are about varying the spread, the slippage and the data.  I never use MC to vary the parameters.

 

In this case MC seems to be useful, as it highlights which strategies are sensitive to data feed and execution changes.  I don’t want to vary the parameters in a MC simulation.

 

Does it mean I bin possibly good strategies?  Maybe. 

0

Threshold

Customer, bbp_participant, community, 723 replies.

Visit profile

7 years ago #137220

MC is for measuring RISK.

Your backtest might say 4% DD.
Monte Carlo sim will show a 7% DD is possible. This is what you adjust your trade size to.

If you want to use monte carlo to see if a system can blow up you need to do millions of iterations to eventually find that 0.001% chance that it had 100 consecutive losers which banks and HFs will do but the real purpose of Monte Carlo is for finding likely drawdowns.

Real robustness testing is seeing how it performs on alternate time frames ( I make sure my H1 systems work on M15, M30 AND H4), OOS data(earlier years and later years), alternate assets/pairs(I make sure a EURUSD will work on GBPUSD/USDCHF), wider spreads, wider slippage, longer historical data, etc etc etc.

0

Threshold

Customer, bbp_participant, community, 723 replies.

Visit profile

7 years ago #137223

Forgive me: I ran this type of experiment ages ago using close to 100000 models.  I’ve recreated a simple segment of the experiment using a smaller data set.

Summary
I create a Support Vector Machine to determine the efficacy of robustness test results for predicting the future profitability of models.  I used the in sample robustness data as the input to the SVM.

QA 4 basically has this in equity chart form.

0

geektrader

Customer, bbp_participant, community, 522 replies.

Visit profile

7 years ago #137227

I have invited Daniel over here, maybe he wants to reply to the thread directly.

 

I absolute agree with spread, slippage sensitivity etc, but that isn´t a MC simulation at all. I´ve been doing this without any MC since I started out. Just do various backtest runs with different spreads and slippage values to see if the strategy is spread sensitive in NORMAL backtests and you get that idea as well. However, apart from this, REAL MC never told me anything about if the strategy works better than others live, at least that´s my conclusion after 8 years. Funnily almost 80% of systems that failed MC (using distorted data-sets + parameter variation, so “real” MC), did well going forward, while the ones that survived these MC simulations did just as well with a 75% success rate in live trading. So no real difference.

 

However, what becomes clear is that the ones that failed MC simulations are the ones that use specific PA relations most of the time, e.g. that refer to “Open[85]”, while the ones that do well with the MC simulations are the ones that use for example MA´s with bigger values (300 to 500 bars), which underlines the findings of Daniel that MC prefers “smoothed” systems since they work better on distorted data, yet seem to say nothing about their future stability at least for him and for me since my “Open[85]” systems did just as well in live trading.

 

Using different TF´s to simulate stability btw is nothing else than using distorted data of the main TF, MA strategies (smoothed strategies) just work better there as well in the MC simulations, yet seem to conclude nothing going forward.

 

My findings hence are: of course to make sure that strats are not spread and slippage sensitive, but apart from this it didn´t make a difference for me in the live trading success rate if they survived the other MC tests or not.


🚀 Unlock Your Edge in Automated Forex Strategy Development 🚀

Historical Forex Data Starting From 1987, 28 Pairs, M1, 99% Error-Free, Lifetime Free Updates

0

Threshold

Customer, bbp_participant, community, 723 replies.

Visit profile

7 years ago #137229

I’m sort of agreeing that’s its useless for robustness testing, because in my view its not for robustness testing. Its for position sizing, and for projecting a future equity-band.
It is especially useless for proving a strategy is over-fitted. The more a strategy is curve fitted the better it looks in Monte Carlo(randomized trade order). However, the more a strategy is curve fitted, the worse it performs on other pairs and other timeframes. There are always some exceptions though in trading. I disagree that moving averages/bands/candlesticks/and ‘bars since exits’ cause strategies to fail on alternate timeframes/pairs. Its sometimes true but not always. I’ll post an example.

I appreciate his contrarian articles and yours.

0

geektrader

Customer, bbp_participant, community, 522 replies.

Visit profile

7 years ago #137230

At notch: wow, how old are you and how few arguments do you have to come down to that level? I have definitely no time to waste with persons that have such a low IQ, you went straight to my ignore list.

 

Threshold: yes, I agree with that indeed, I use MC for WC scenario tests to see when a strategy needs to be stopped trading. For the rest, it´s not because I don´t believe in this, but because it has shown in 8 years of live trading to really tell me nothing about future performance and system stability (overfitted or not?), apart from the usual spread / slippage test that are useful indeed of course to spot scalpers that will never work live. I am open minded though if someone has other live stats, apart from theoretical ones why it should work.


🚀 Unlock Your Edge in Automated Forex Strategy Development 🚀

Historical Forex Data Starting From 1987, 28 Pairs, M1, 99% Error-Free, Lifetime Free Updates

0

mikeyc

Customer, bbp_participant, community, 877 replies.

Visit profile

7 years ago #137231

I can imagine the invitation:

“Please Daniel come to protect me at the SQ forum. I have told them you are my very good friend but I won’t tell them you’re also my secret internet protector. I will always be your little sugar and you are my Dannipoops.”

😀 Made me chuckle

0

Threshold

Customer, bbp_participant, community, 723 replies.

Visit profile

7 years ago #137232

Pending live stats….
I try to stick close to what I’ve read from traders who been successful for 20+ years. Is Daniel successful in building systems? It seems he writes articles about what’s not successful (and only data mines them?). Seems like hes still on the journey like many of us.

0

geektrader

Customer, bbp_participant, community, 522 replies.

Visit profile

7 years ago #137235

😀 Made me chuckle
Makes me chuckle too  😆


🚀 Unlock Your Edge in Automated Forex Strategy Development 🚀

Historical Forex Data Starting From 1987, 28 Pairs, M1, 99% Error-Free, Lifetime Free Updates

0

geektrader

Customer, bbp_participant, community, 522 replies.

Visit profile

7 years ago #137236

Pending live stats….
I try to stick close to what I’ve read from traders who been successful for 20+ years. Is Daniel successful in building systems? It seems he writes articles about what’s not successful (and only data mines them?). Seems like hes still on the journey like many of us.

 

Yes indeed, but most of the time these are manual traders…. Daniel´s systems are picking up lately it seems, his approach is different, they data-mine on GPUs in the community, hence can test millions of combinations within a few minutes / hours. They explore the whole possible space of combinations by brute forcing it on this network of GPUs. Once a profitable system is found, they right away boot-strap the underlying data for a few million times and the system has to do well on all those data-sets as well, which eliminates mining bias by ~98%. Only then a system is being traded live, so far they have a few hundred fullfilling these and are making money if looking at the MyFxBooks of a few of the users that publish their portfolios (which you can freely compile with different portfolio theory algos).


🚀 Unlock Your Edge in Automated Forex Strategy Development 🚀

Historical Forex Data Starting From 1987, 28 Pairs, M1, 99% Error-Free, Lifetime Free Updates

0

Patrick

Customer, bbp_participant, community, 424 replies.

Visit profile

7 years ago #137238

Monte Carlo is part of puzzle, together with other steps its useful tool. It is not perfect and some good strategies will not pass trough but most of bad fail and thats important. I think Daniel wanted to mention that not all strategies that fail MC are bad. The process benefits from this test more than looses.

 

@Notch Funny.

@others Dont be cool as cucumbers, it was for laughs. ( i laught at least)

0

Threshold

Customer, bbp_participant, community, 723 replies.

Visit profile

7 years ago #137240

I love this forum.

Just finishing a reply-

This isn’t a live strategy, I was just recently generating new EURUSD h1 strategies  but then my new server arrived so I’m just gona throw the results away. I didn’t like the equity curve and number of trades anyway.

For example’s sake I’m just using this, there isn’t anything special about it or “final” but I would definitely consider it a pass. I know this doesn’t have MAs or candle patterns, etc its not perfect example but its what I found on short browse of my currently open databank.

Main EURUSD H1

File: main.pngmain.png

EURUSD M15 ‘robustness test’

File: main2.pngmain2.png

EURUSD M30 ‘robustness test’

File: main3.pngmain3.png

EURUSD H4 ‘robustness test’

File: main4.pngmain4.png

GBPUSD H1 ‘robustness test’

File: main5.pngmain5.png

Pseudo code

File: main6.pngmain6.png

Walk Forward Matrix pass (5 to 30 OOS with 5 Step, 3 to 20 iterations with 1 step, 20% parameter preset)

File: main7.pngmain7.png

The strategy has only 3 rules and 3 optimizable parameters.
This is what I consider real robustness besides using extensively long historical data… I just didn’t like the strategy.
I don’t have much else to add to the topic.

 

0

clonex / Ivan Hudec

Customer, bbp_participant, community, sq-ultimate, contributor, author, editor, 271 replies.

Visit profile

7 years ago #137241

Yes i think so trshld. Combination multi tf and multi mrkt robustness, lower complexity, some brutal mc tests and performance monitor system is the way. But it is a fact some strategies cant pass viac mc tests

0

Threshold

Customer, bbp_participant, community, 723 replies.

Visit profile

7 years ago #137242

 

 Its sometimes true but not always.

0

Threshold

Customer, bbp_participant, community, 723 replies.

Visit profile

7 years ago #137243

BTW I’ll actually just add that GPUs should be mining Ethereum, not running Asirkuy. 😀
Just having fun.

0

Viewing 15 replies - 1 through 15 (of 63 total)

1 2 3 4 5