READ-ONLY

The forum is now a read-only archive.

For bug reports & platform questions → [email protected]

Our community lives on Discord and YouTube — come join us!

How Robust is Robust Enough?

31 replies

_Cujo

Customer, bbp_participant, community, 97 replies.

Visit profile

10 years ago #115074

What settings do you use on Monte Carlo Robustness testing? I appreciate that you can never be to careful, and it’s money, and need to be certain, so not wanting to cut down the robustness tests to much, but…

 

I started out at the top image. Every strategy tested was not just failing, but was very, very, very bad. It got me thinking, maybe my settings (top image) are to severe. It would turn moderately successful strategies, using the same IS data for the Monte Carlo to not just account destruction, but total obliteration.

 

Adding it all, up, for example, there’s a 60% chance something happens like 20% parameter change, or 20% price change, etc…logically, that feels very heavy.

 

Where I started

6jtN7Fe.png

 

I did some experimentation, initially thinking – if I take a bunch of strategies, on the same IS, run them through the Monte Carlo Simulation, and tweak it so the robustest net profit was ~10% either side, possibly about right. But, that felt a bit over optimized of risk by seeking a specific result with my robustness settings, which felt wrong (and was really hard anyway).

 

Then I thought – I know, I’ll just half everything, which is still leaving a 30% chance something happens, like a trade is skipped or parameter change, etc..which left me with this below…

 

Where I am now

E7kT8xy.png

 

 

Then I started second guessing that, so thought I’d ask for advice. What settings do you use to test for robustness in the Monte Carlo Simulation? Maybe even image 2 is to heavy?

 

 

How Many Monte Carlo Simulations to Run

This really got me thinking on how many simulations to run:

  • 5000 is sort of enough, 7500 is better and 10,000 is over kill, from a study that studied studies on Monte Carlo simulations.
  • 500-1000 is enough if you’re testing weapons, but results stabilising after ~500. I guess the difference between 98% dead and 95% dead is moot.
  • At least 440 times, if you’re using dedicated, professional risk analysis software (note – professional risk analyser not professional trader).
  • Much reading cited that the number of times you need to run it depending on the standard deviation of the data you were running it on.

For how many times to run it for trading system assessment, ambi broker advises 1000 (here and here), Vanguard do 5000 runs, the lowest recommendation I found was 500, but the best advice seemed to be scale it around 4000, depending on how many parameters in your system…how many are you running?

 

I changed my confidence level from 95% to 98%, which made me feel more confident, but still wondering about what settings to use…??

0

_Cujo

Customer, bbp_participant, community, 97 replies.

Visit profile

10 years ago #137131

Is Monte Carlo over the full selected data series, or is it run over just the IS or OOS?

 

For example, if I am testing on 2004 and 2005, with 50% IS/OOS, is the Monte Carlo run over both 2004 and 2005…? Do you know?

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137132

Is Monte Carlo over the full selected data series, or is it run over just the IS or OOS?

 

For example, if I am testing on 2004 and 2005, with 50% IS/OOS, is the Monte Carlo run over both 2004 and 2005…? Do you know?

 

SQ manual is your friend! 🙂

If you do not read manual you can read Robustness page see screenshots : Results in the table are for complete data (In Samlple + Out of Sample)

 

I have another issue will post in next post.

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137133

Check this out , Monte Carlo based on chance so if you press a few time RUN you will get absolutely diffirent results in MC section . Whis is correct but defits the purpose of the test.

it is like lottery for the stratagies. one can just Win on random chance 🙂

So my simple question how many times should I press RUN . lol ?

I can make all stratagies pass or failed just using RUN buttom. hehe.

See screenshot.

 

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137139

So how can I trust someting which totaly depends on LUCK?

it works as expected, no questions are asked about it as it is clearly states in descriptions : Randomize this and this .

 

Randomize means Luck 🙂

But we put all our trust on this LUCK to select a working stratagies. how Ironic it is 🙁

 

so any one with siencific degree can comment please ?

 

I should have never pressed “START” twice, I have been supporter of MC for over decade.

Now, I totaly lost, as i run a small test with 5 stratagies , making them profitable with 100% confidence and then next few runs , completely failed !

 

But it is better late than never, right.

 

Thank you in advance for any comments.

0

Patrick

Customer, bbp_participant, community, 422 replies.

Visit profile

10 years ago #137140

RT test do not relay on luck.

 

1000 tests in statisticly “100% sure”, but ussualy for raw filtering 100 runs is enough. 

 

so run more runs and everything will be ok. sure when you run 10 runs it is not enough to see all possiblities-thats why its good and after bad

 

when you make 100 runs first and after 1000 runs, what goes through 100 most of will come trough thousand. but 10 test its realy ridiculous.

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137142

Thanks Patrick , 1000 tests it is the same as running 100 times with 10test 🙂

So this topic proves that power in Numbers any one wants to run 100test need to read it .

I ran 10 test just to prove the point, let me do 1000 first as it might happen too 🙂  as even 1000 will not cover endless posobilites but gives us more confidence in the stratagy.

 

i will come back with more testing as I want to close this off today.

0

Patrick

Customer, bbp_participant, community, 422 replies.

Visit profile

10 years ago #137143

no its different. 1000 tests is not 10×100  tests,       in 10×100 tests may some test repeat.       in 1000 tests is every test different to others. no test is repeated unlike 10×100 tests.

 

you dont have to do 1000, 100 is enough to see the most probably results of simulation. slight deviations may occur-important is that most of the equitys ate rising equities. it is not important to have all equities looking great.

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137144

10x tests

  Running robustness tests on strategy USDJPY-M15-14.40
Testing USDJPY-M15-14.40  …..   6.219 s.
DONE in 7.641 s.

 

100x tests

  Running robustness tests on strategy USDJPY-M15-14.40
Testing USDJPY-M15-14.40  …..   55.067 s.
DONE in 58.442 s.

 

1000x tests

  Running robustness tests on strategy USDJPY-M15-14.40
Testing USDJPY-M15-14.40  …..   542.407 s.
DONE in 566.611 s.

 

Thank you again Patrick,

Now the questions about the Speed of the test.

I’m testing on a server and using only 32 cores for now with 32GB RAM (I will increase up to 128Core/128GB). During the this test CPU is not used (6%) ram about  6GB (plenty spare one left) , Disk is not used.

SQ configured to use all 32 cores, but it won’t use them. However in WFO all 32 cores are hummered 80-90% .

Can you please explain as it takes a long time to process those 100 tests , 10 tests takes 58 seconds, so 100 will take about 10 minutes , I guess it is better to do correlation first and remove repeated Strategies then do Robust Test simulation. 

 

Please advise.

Thanks.

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137145

I took a stratagy which has no significat Equty curve just for the sake of testing.

And run all tests 10x / 100x / 1000x ,

In this case diffirence is very small , and there is almost no diffirence at all between 100x and 1000x tests

see my tests below:

I will try to find something better to see if it changes much with the amount of tests but I use Robust settings as per my early post, I guess it is overkill , but this is what Mark / Partic put in default 🙂 lol

Thanks for any comments.

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137146

All previous tests were run on Dukas Copy M15 with resolution M1

Now time to run on Real tick data

Running 100x tests all passed! but the Original strategy failed miserably.

 

The stratagy was generated on M15 (Selected Timeframe only ) not on M1/Tick , Robustness was turn off during generation.

Any comments?

Thanks.

0

_Cujo

Customer, bbp_participant, community, 97 replies.

Visit profile

10 years ago #137162

Yeah, looking at this..I think between 100 and 1000 tests, little difference, they look similar. 

 

Right now, I’m running 550 tests just because (based on some of the reading I highlighted in OP).

 

Looking at your study, I think I change that to 100.

0

Tomas Matejka

Administrator, sq-ultimate, 2 replies.

Visit profile

10 years ago #137169

All previous tests were run on Dukas Copy M15 with resolution M1

Now time to run on Real tick data

Running 100x tests all passed! but the Original strategy failed miserably.

 

This sounds odd. We will check it whether there is any possible issue

Could you also attach USDJPY 15 min 14.40 strategy?

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137174

Yeah, looking at this..I think between 100 and 1000 tests, little difference, they look similar.Right now, I’m running 550 tests just because (based on some of the reading I highlighted in OP).

Looking at your study, I think I change that to 100.

This is correct , looking at this 1000 tests will you deselect the strategy just because ONE goes a bit south for 40 trades, well it just means the strategy has 0.1% of chances to go that way . In reality it is less, as I set very tough conditions.

 

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137175

This sounds odd. We will check it whether there is any possible issueCould you also attach USDJPY 15 min 14.40 strategy?

Thanks Tomas,

This is absolutely ODD, I cannot reproduce the same results , Nothing changed , the same data the same settings.

I think I used the same PC. but it was late so I might make a mistake . Today all working as expected.

But from screenshot you can see that all the same , dates, pairs .

Running 100 tests on the same strategy.

This is tested with M1 resolution 100 tests.

 

0

statistic

Subscriber, bbp_participant, community, 31 replies.

Visit profile

10 years ago #137178

sorry I should have taken better trading system, but it was not the point I wanted to see how MC affects , and it was ODD that some trading systems  had RT better then Equity Curve , this was very odd indeed.

So the Euity curve doesn’t matter but if you want to see it , it is not very good stratagy,

I’m not sure what happened yesturday, I think I tested it on Windows 2012 server and this time I’m testing on Windows 2008, let me test on the same platform and this way we will know what happened.

 

0

Viewing 15 replies - 16 through 30 (of 31 total)

← 1 2 3 →