How Robust is Robust Enough?
31 replies
_Cujo
10 years ago #115074
What settings do you use on Monte Carlo Robustness testing? I appreciate that you can never be to careful, and it’s money, and need to be certain, so not wanting to cut down the robustness tests to much, but…
I started out at the top image. Every strategy tested was not just failing, but was very, very, very bad. It got me thinking, maybe my settings (top image) are to severe. It would turn moderately successful strategies, using the same IS data for the Monte Carlo to not just account destruction, but total obliteration.
Adding it all, up, for example, there’s a 60% chance something happens like 20% parameter change, or 20% price change, etc…logically, that feels very heavy.
Where I started

I did some experimentation, initially thinking – if I take a bunch of strategies, on the same IS, run them through the Monte Carlo Simulation, and tweak it so the robustest net profit was ~10% either side, possibly about right. But, that felt a bit over optimized of risk by seeking a specific result with my robustness settings, which felt wrong (and was really hard anyway).
Then I thought – I know, I’ll just half everything, which is still leaving a 30% chance something happens, like a trade is skipped or parameter change, etc..which left me with this below…
Where I am now

Then I started second guessing that, so thought I’d ask for advice. What settings do you use to test for robustness in the Monte Carlo Simulation? Maybe even image 2 is to heavy?
How Many Monte Carlo Simulations to Run
This really got me thinking on how many simulations to run:
- 5000 is sort of enough, 7500 is better and 10,000 is over kill, from a study that studied studies on Monte Carlo simulations.
- 500-1000 is enough if you’re testing weapons, but results stabilising after ~500. I guess the difference between 98% dead and 95% dead is moot.
- At least 440 times, if you’re using dedicated, professional risk analysis software (note – professional risk analyser not professional trader).
- Much reading cited that the number of times you need to run it depending on the standard deviation of the data you were running it on.
For how many times to run it for trading system assessment, ambi broker advises 1000 (here and here), Vanguard do 5000 runs, the lowest recommendation I found was 500, but the best advice seemed to be scale it around 4000, depending on how many parameters in your system…how many are you running?
I changed my confidence level from 95% to 98%, which made me feel more confident, but still wondering about what settings to use…??
0
mabi
10 years ago #136717
We are just using Robustness tests to se if the strategy generated is curve fitted to the noise in the market., If it is you will not need even 500 runs since it will show immediately if it is curve fitted with drops in equity curves .. Anyway thats my understanding of Robusness tests.
However u lower time frame u more noise and higher risk for a curve fitted strategy..
0
mabi
10 years ago #136718
So in short the Curve fit test ” Robustness” is just a quick way to remove bad strategies . The real test is the walkforward test if it does not pass that delete or improve the strategy.
0
_Cujo
10 years ago #136722
So in short the Curve fit test ” Robustness” is just a quick way to remove bad strategies . The real test is the walkforward test if it does not pass that delete or improve the strategy.
They’re for different purposes, ofc. Monte Carlo is more about risk, confidence in results and testing robustness via noise, randomisation, etc.
Walk forward is more about telling if parameters should be optimised (but not what the optimisation is), then testing on OOS (which does tests robustness).
Here is a set of good articles about the 2, here and here (not necessarily the differences, but system trader is a good web site, guess most ppl here know of it).
But, setting aside the differences between Walk forward vs Monte Carlos Simulation, focussing on the Monte Carlo Simulation…what settings are you using in Monte Carlo compared to the images in the OP? Sounds like you gave a lot of advice to give… 🙂
0
mabi
10 years ago #136727
Thanks ! Okey i actually have Kevin Davey free tools and his book. I guess i have to read it again ! ( He actually told me that 2 weeks ago ) However when building a strategy we are looking for the true signal without adapting the strategy to the market noise and that whats we use Montecarlo for. If we have to many indicators the strategy will for sure be adapted so i am only looking for really simple ones . One way i found to do this with SQ is to generate only on 1-4 years of data and let 15 years be out of sample. I then find that the only ones that actually have a okey curve is very simple and actually robust.
(I do not know Kevin Davey but he hangs out at another forum and are very active)
https://www.youtube.com/watch?v=kBW7-p2DMVU
0
eastpeace
10 years ago #136763
500 simulations would let SQ halt. (831 strategies in test pool)
i7-4790, 16G ram.
0
geektrader
10 years ago #136858
Nope, it doesn´t, I use 1000 always, even at many strategies at once in the Re-Tester. Doesn´t hold anything here, barely uses much RAM either, just takes a long time:) But never did SQ hang, just took several days sometimes;)
0
mabi
10 years ago #136859
1000 tests simply means you are looking for 950 of 1000 test’s has to fall within the acceptable range of deviation. I am finding it some times difficult to determine which tests and how large change to apply to have a reasonable impact on the strategies performance without totally destroying them depending on their complexity. This brings up another issue and that is to how to set up SQ to find a certain type of strategies so that you know that the settings applied to Monecarlo test are having desired effect. Just choosing evrything in builder and let it run will produce a wide array of different type of strategies and they can not be Montecarlo tested with the same settings and should then be splitted up before tested. Or am i wrong ?
0
_Cujo
10 years ago #136919
I was thinking about robustness and re-listening to a Kevin Davey podcast from System Trader Success, I guess everyone here knows both of them, of course. 🙂
He mentioned he did 5000 Monte Carlo simulations. I am doing more like 50, sometimes 550, right after generation as part of the weeding out process. I think I only get close to that number on strategies that have passed a few basic robustness tests, and a walk forward, and I’m getting more interested in. Certainly not when I’m clearing out strategies from the data bank by the thousand…
[media]https://www.youtube.com/watch?v=TpmDzRFIu0k[/media]
0
_Cujo
10 years ago #136920
Thanks ! Okey i actually have Kevin Davey free tools and his book. I guess i have to read it again ! ( He actually told me that 2 weeks ago ) However when building a strategy we are looking for the true signal without adapting the strategy to the market noise and that whats we use Montecarlo for. If we have to many indicators the strategy will for sure be adapted so i am only looking for really simple ones . One way i found to do this with SQ is to generate only on 1-4 years of data and let 15 years be out of sample. I then find that the only ones that actually have a okey curve is very simple and actually robust.
(I do not know Kevin Davey but he hangs out at another forum and are very active)
https://www.youtube.com/watch?v=kBW7-p2DMVU
YES, Kevin is really a good model. I read his book and re-read it any times. He is really active on forums and …well, a good model, teaches a lot. 🙂 I don’t know him either, but he comes across as very approachable. I guess he sold many, many of his book too.
0
statistic
10 years ago #137119
How reliable the Robust test?
The original strategy fails but Robust test shows very good results.
See the snapshot of the configuration.
http://s33.postimg.org/9gc31fw5r/Robust_test_failure_settings.png
Any one can comment?
Thanks.
0
mabi
10 years ago #137120
@ Statistic,
Robustness test tries to simulate different conditions in the market by changing its data. Seems your strategy would do better on any other market conditions then on the one you created it on.
0
statistic
10 years ago #137121
@ Statistic,Robustness test tries to simulate different conditions in the market by changing its data. Seems your strategy would do better on any other market conditions then on the one you created it on.
Thank you @Mabi, for rocket fast answer. I generated a bunch of strategies and they work in OTHER condition better then on original data.
Never seen it before I switched to Dukas Copy data.
Strategy are generated on M15 , Trade On Bar Open Fast but when do Retest use Real Tick.
It doesn’t make a lot of sense how it happens.
I was usually using M1 commercial data, and reviewed manually over a million of strategies, but never happen seeing such discrepancy before moving on Dukas Copy Tick data.
Do you guys see that happening with Dukas Copy tick data?
Thank you.
0
statistic
10 years ago #137125
500 simulations would let SQ halt. (831 strategies in test pool)i7-4790, 16G ram.
Hi mate, just tested 1000 tests on 100 strategies , got a 35% CPU see the screenshot.
So nothing to do with you SQ, just need to use higher spec Severs.
Settings:
0
_Cujo
10 years ago #137126
I also have no trouble running 1,000 or 5,000 Monte Carlo Simulations on strategies.
0
statistic
10 years ago #137127
I also have no trouble running 1,000 or 5,000 Monte Carlo Simulations on strategies.
Just run 1000 tests on 100 strategies, too a long time over 30 minutes! But it is Ok, for the sake of testing.
it is better to rule out the failing strategies on 20/100/300 tests first this
way you do not need to run all the bunch failing strategies for 5 minutes each.
0