読み取り専用

フォーラムは現在、読み取り専用のアーカイブとなっています。.

バグ報告やプラットフォームに関する質問はこちら → [email protected]

私たちのコミュニティはDiscordとYouTubeで活動しています。ぜひ参加してください!

ご参加ください Discord YouTube

How Robust is Robust Enough?

31件の返信

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #115074

What settings do you use on Monte Carlo Robustness testing? I appreciate that you can never be to careful, and it’s money, and need to be certain, so not wanting to cut down the robustness tests to much, but…

 

I started out at the top image. Every strategy tested was not just failing, but was very, very, very bad. It got me thinking, maybe my settings (top image) are to severe. It would turn moderately successful strategies, using the same IS data for the Monte Carlo to not just account destruction, but total obliteration.

 

Adding it all, up, for example, there’s a 60% chance something happens like 20% parameter change, or 20% price change, etc…logically, that feels very heavy.

 

Where I started

6jtN7Fe.png

 

I did some experimentation, initially thinking – if I take a bunch of strategies, on the same IS, run them through the Monte Carlo Simulation, and tweak it so the robustest net profit was ~10% either side, possibly about right. But, that felt a bit over optimized of risk by seeking a specific result with my robustness settings, which felt wrong (and was really hard anyway).

 

Then I thought – I know, I’ll just half everything, which is still leaving a 30% chance something happens, like a trade is skipped or parameter change, etc..which left me with this below…

 

Where I am now

E7kT8xy.png

 

 

Then I started second guessing that, so thought I’d ask for advice. What settings do you use to test for robustness in the Monte Carlo Simulation? Maybe even image 2 is to heavy?

 

 

How Many Monte Carlo Simulations to Run

This really got me thinking on how many simulations to run:

  • 5000 is sort of enough, 7500 is better and 10,000 is over kill, from a study that studied studies on Monte Carlo simulations.
  • 500-1000 is enough if you’re testing weapons, but results stabilising after ~500. I guess the difference between 98% dead and 95% dead is moot.
  • At least 440 times, if you’re using dedicated, professional risk analysis software (note – professional risk analyser not professional trader).
  • Much reading cited that the number of times you need to run it depending on the standard deviation of the data you were running it on.

For how many times to run it for trading system assessment, ambi broker advises 1000 (ここ そして ここ), Vanguard do 5000 runs, the lowest recommendation I found was 500, but the best advice seemed to be scale it around 4000, depending on how many parameters in your system…how many are you running?

 

I changed my confidence level from 95% to 98%, which made me feel more confident, but still wondering about what settings to use…??

0

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #137131

Is Monte Carlo over the full selected data series, or is it run over just the IS or OOS?

 

For example, if I am testing on 2004 and 2005, with 50% IS/OOS, is the Monte Carlo run over both 2004 and 2005…? Do you know?

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137132

Is Monte Carlo over the full selected data series, or is it run over just the IS or OOS?

 

For example, if I am testing on 2004 and 2005, with 50% IS/OOS, is the Monte Carlo run over both 2004 and 2005…? Do you know?

 

SQ manual is your friend! 🙂

If you do not read manual you can read Robustness page see screenshots : Results in the table are for complete data (In Samlple + Out of Sample)

 

I have another issue will post in next post.

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137133

Check this out , Monte Carlo based on chance so if you press a few time RUN you will get absolutely diffirent results in MC section . Whis is correct but defits the purpose of the test.

it is like lottery for the stratagies. one can just Win on random chance 🙂

So my simple question how many times should I press RUN . lol ?

I can make all stratagies pass or failed just using RUN buttom. hehe.

See screenshot.

 

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137139

So how can I trust someting which totaly depends on LUCK?

it works as expected, no questions are asked about it as it is clearly states in descriptions : Randomize this and this .

 

Randomize means Luck 🙂

But we put all our trust on this LUCK to select a working stratagies. how Ironic it is 🙁

 

so any one with siencific degree can comment please ?

 

I should have never pressed “START” twice, I have been supporter of MC for over decade.

Now, I totaly lost, as i run a small test with 5 stratagies , making them profitable with 100% confidence and then next few runs , completely failed !

 

But it is better late than never, right.

 

Thank you in advance for any comments.

0

パトリック

顧客、bbp_participant、コミュニティ、422件の返信。.

プロフィールを表示

10年前 #137140

RT test do not relay on luck.

 

1000 tests in statisticly “100% sure”, but ussualy for raw filtering 100 runs is enough. 

 

so run more runs and everything will be ok. sure when you run 10 runs it is not enough to see all possiblities-thats why its good and after bad

 

when you make 100 runs first and after 1000 runs, what goes through 100 most of will come trough thousand. but 10 test its realy ridiculous.

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137142

Thanks Patrick , 1000 tests it is the same as running 100 times with 10test 🙂

So this topic proves that power in Numbers any one wants to run 100test need to read it .

I ran 10 test just to prove the point, let me do 1000 first as it might happen too 🙂  as even 1000 will not cover endless posobilites but gives us more confidence in the stratagy.

 

i will come back with more testing as I want to close this off today.

0

パトリック

顧客、bbp_participant、コミュニティ、422件の返信。.

プロフィールを表示

10年前 #137143

no its different. 1000 tests is not 10×100  tests,       in 10×100 tests may some test repeat.       in 1000 tests is every test different to others. no test is repeated unlike 10×100 tests.

 

you dont have to do 1000, 100 is enough to see the most probably results of simulation. slight deviations may occur-important is that most of the equitys ate rising equities. it is not important to have all equities looking great.

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137144

10x tests

  Running robustness tests on strategy USDJPY-M15-14.40
Testing USDJPY-M15-14.40  …..   6.219 s.
DONE in 7.641 s.

 

100x tests

  Running robustness tests on strategy USDJPY-M15-14.40
Testing USDJPY-M15-14.40  …..   55.067 s.
DONE in 58.442 s.

 

1000x tests

  Running robustness tests on strategy USDJPY-M15-14.40
Testing USDJPY-M15-14.40  …..   542.407 s.
DONE in 566.611 s.

 

Thank you again Patrick,

Now the questions about the Speed of the test.

I’m testing on a server and using only 32 cores for now with 32GB RAM (I will increase up to 128Core/128GB). During the this test CPU is not used (6%) ram about  6GB (plenty spare one left) , Disk is not used.

SQ configured to use all 32 cores, but it won’t use them. However in WFO all 32 cores are hummered 80-90% .

Can you please explain as it takes a long time to process those 100 tests , 10 tests takes 58 seconds, so 100 will take about 10 minutes , I guess it is better to do correlation first and remove repeated Strategies then do Robust Test simulation. 

 

ご回答ください。.

ありがとう。.

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137145

I took a stratagy which has no significat Equty curve just for the sake of testing.

And run all tests 10x / 100x / 1000x ,

In this case diffirence is very small , and there is almost no diffirence at all between 100x and 1000x tests

see my tests below:

I will try to find something better to see if it changes much with the amount of tests but I use Robust settings as per my early post, I guess it is overkill , but this is what Mark / Partic put in default 🙂 lol

Thanks for any comments.

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137146

All previous tests were run on Dukas Copy M15 with resolution M1

Now time to run on Real tick data

Running 100x tests all passed! but the Original strategy failed miserably.

 

The stratagy was generated on M15 (Selected Timeframe only ) not on M1/Tick , Robustness was turn off during generation.

Any comments?

ありがとう。.

0

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #137162

Yeah, looking at this..I think between 100 and 1000 tests, little difference, they look similar. 

 

Right now, I’m running 550 tests just because (based on some of the reading I highlighted in OP).

 

Looking at your study, I think I change that to 100.

0

Tomas Matejka

管理者、sq-ultimate、2件の返信.

プロフィールを表示

10年前 #137169

All previous tests were run on Dukas Copy M15 with resolution M1

Now time to run on Real tick data

Running 100x tests all passed! but the Original strategy failed miserably.

 

This sounds odd. We will check it whether there is any possible issue

Could you also attach USDJPY 15 min 14.40 strategy?

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137174

Yeah, looking at this..I think between 100 and 1000 tests, little difference, they look similar.Right now, I’m running 550 tests just because (based on some of the reading I highlighted in OP).

Looking at your study, I think I change that to 100.

This is correct , looking at this 1000 tests will you deselect the strategy just because ONE goes a bit south for 40 trades, well it just means the strategy has 0.1% of chances to go that way . In reality it is less, as I set very tough conditions.

 

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137175

This sounds odd. We will check it whether there is any possible issueCould you also attach USDJPY 15 min 14.40 strategy?

ありがとう、トマス。,

This is absolutely ODD, I cannot reproduce the same results , Nothing changed , the same data the same settings.

I think I used the same PC. but it was late so I might make a mistake . Today all working as expected.

But from screenshot you can see that all the same , dates, pairs .

Running 100 tests on the same strategy.

This is tested with M1 resolution 100 tests.

 

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137178

sorry I should have taken better trading system, but it was not the point I wanted to see how MC affects , and it was ODD that some trading systems  had RT better then Equity Curve , this was very odd indeed.

So the Euity curve doesn’t matter but if you want to see it , it is not very good stratagy,

I’m not sure what happened yesturday, I think I tested it on Windows 2012 server and this time I’m testing on Windows 2008, let me test on the same platform and this way we will know what happened.

 

0

15件の返信を表示中 - 16 - 30件目 (全31件中)

← 1 2 3 →