読み取り専用

フォーラムは現在、読み取り専用のアーカイブとなっています。.

バグ報告やプラットフォームに関する質問はこちら → [email protected]

私たちのコミュニティはDiscordとYouTubeで活動しています。ぜひ参加してください!

ご参加ください Discord YouTube

How Robust is Robust Enough?

31件の返信

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #115074

What settings do you use on Monte Carlo Robustness testing? I appreciate that you can never be to careful, and it’s money, and need to be certain, so not wanting to cut down the robustness tests to much, but…

 

I started out at the top image. Every strategy tested was not just failing, but was very, very, very bad. It got me thinking, maybe my settings (top image) are to severe. It would turn moderately successful strategies, using the same IS data for the Monte Carlo to not just account destruction, but total obliteration.

 

Adding it all, up, for example, there’s a 60% chance something happens like 20% parameter change, or 20% price change, etc…logically, that feels very heavy.

 

Where I started

6jtN7Fe.png

 

I did some experimentation, initially thinking – if I take a bunch of strategies, on the same IS, run them through the Monte Carlo Simulation, and tweak it so the robustest net profit was ~10% either side, possibly about right. But, that felt a bit over optimized of risk by seeking a specific result with my robustness settings, which felt wrong (and was really hard anyway).

 

Then I thought – I know, I’ll just half everything, which is still leaving a 30% chance something happens, like a trade is skipped or parameter change, etc..which left me with this below…

 

Where I am now

E7kT8xy.png

 

 

Then I started second guessing that, so thought I’d ask for advice. What settings do you use to test for robustness in the Monte Carlo Simulation? Maybe even image 2 is to heavy?

 

 

How Many Monte Carlo Simulations to Run

This really got me thinking on how many simulations to run:

  • 5000 is sort of enough, 7500 is better and 10,000 is over kill, from a study that studied studies on Monte Carlo simulations.
  • 500-1000 is enough if you’re testing weapons, but results stabilising after ~500. I guess the difference between 98% dead and 95% dead is moot.
  • At least 440 times, if you’re using dedicated, professional risk analysis software (note – professional risk analyser not professional trader).
  • Much reading cited that the number of times you need to run it depending on the standard deviation of the data you were running it on.

For how many times to run it for trading system assessment, ambi broker advises 1000 (ここ そして ここ), Vanguard do 5000 runs, the lowest recommendation I found was 500, but the best advice seemed to be scale it around 4000, depending on how many parameters in your system…how many are you running?

 

I changed my confidence level from 95% to 98%, which made me feel more confident, but still wondering about what settings to use…??

0

マビ

Customer、bbp_participant、community、sq-ultimate、252件の返信。.

プロフィールを表示

10年前 #136717

We are just using Robustness tests to se if the strategy generated is curve fitted to the noise in  the market., If it is you will not need even 500 runs since it will show immediately if it is curve fitted with drops in equity curves .. Anyway thats my understanding of Robusness tests.

 

 

However u lower time frame u more noise and higher risk for a curve fitted strategy..

0

マビ

Customer、bbp_participant、community、sq-ultimate、252件の返信。.

プロフィールを表示

10年前 #136718

So in short the Curve fit test ” Robustness” is just a quick way to remove bad strategies . The real test is the walkforward test if it does not pass that delete or improve the strategy.

0

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #136722

So in short the Curve fit test ” Robustness” is just a quick way to remove bad strategies . The real test is the walkforward test if it does not pass that delete or improve the strategy.

 

They’re for different purposes, ofc. Monte Carlo is more about risk, confidence in results and testing robustness via noise, randomisation, etc.

 

Walk forward is more about telling if parameters should be optimised (but not what the optimisation is), then testing on OOS (which does tests robustness).

 

Here is a set of good articles about the 2, ここ そして ここ (not necessarily the differences, but system trader is a good web site, guess most ppl here know of it).

 

But, setting aside the differences between Walk forward vs Monte Carlos Simulation, focussing on the Monte Carlo Simulation…what settings are you using in Monte Carlo compared to the images in the OP? Sounds like you gave a lot of advice to give… 🙂

0

マビ

Customer、bbp_participant、community、sq-ultimate、252件の返信。.

プロフィールを表示

10年前 #136727

Thanks !  Okey i actually have Kevin Davey free tools and his book. I guess i have to read it again ! ( He actually told me that 2 weeks ago )  However when building a strategy we are looking for the true signal without adapting the strategy to the market noise and that whats we use Montecarlo for. If we have to many indicators the strategy will for sure be adapted so i am only looking for really simple ones . One way i found to do this with SQ is to generate only on 1-4 years of data and let 15 years be out of sample. I then find that the only ones that actually have a okey  curve is very simple and actually robust.

 

 

(I do not know Kevin Davey but he hangs out at another forum and are very active)

 

 https://futures.io/vendors-product-reviews/26335-kj-trading-systems-kevin-davey-ask-me-anything-ask-me-anything.html

 https://www.youtube.com/watch?v=kBW7-p2DMVU

0

イーストピース

Customer、bbp_participant、community、sq-ultimate、298件の返信。.

プロフィールを表示

10年前 #136763

500 simulations would let SQ halt.  (831 strategies in test pool)

 

i7-4790, 16G ram.

0

geektrader

顧客、bpb_participant、コミュニティ、503件の回答。.

プロフィールを表示

10年前 #136858

Nope, it doesn´t, I use 1000 always, even at many strategies at once in the Re-Tester. Doesn´t hold anything here, barely uses much RAM either, just takes a long time:) But never did SQ hang, just took several days sometimes;)


🚀 自動化された外国為替戦略開発であなたの優位性を引き出せます 🚀

1987年から開始された歴史的な外国為替データ。28ペア、M1、99%エラーなし、生涯無料アップデート

0

マビ

Customer、bbp_participant、community、sq-ultimate、252件の返信。.

プロフィールを表示

10年前 #136859

1000 tests simply means you are looking for  950 of 1000 test’s has to fall within the acceptable range of deviation. I am finding it some times difficult to determine which tests and how large change to apply to have a reasonable impact on the strategies performance without totally destroying them depending on their complexity. This brings up another issue and that is to how to set up SQ to find a certain type of strategies so that you know that the settings applied to Monecarlo test are having desired effect.  Just choosing evrything in builder and let it run will produce a wide array of different type of strategies and they can not be Montecarlo tested with the same settings and should then be splitted up before tested. Or am i wrong ?

0

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #136919

I was thinking about robustness and re-listening to a Kevin Davey podcast from System Trader Success, I guess everyone here knows both of them, of course.  🙂

 

He mentioned he did 5000 Monte Carlo simulations. I am doing more like 50, sometimes 550, right after generation as part of the weeding out process. I think I only get close to that number on strategies that have passed a few basic robustness tests, and a walk forward, and I’m getting more interested in. Certainly not when I’m clearing out strategies from the data bank by the thousand…

 

 

[media]https://www.youtube.com/watch?v=TpmDzRFIu0k[/media]

 

 

0

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #136920

Thanks !  Okey i actually have Kevin Davey free tools and his book. I guess i have to read it again ! ( He actually told me that 2 weeks ago )  However when building a strategy we are looking for the true signal without adapting the strategy to the market noise and that whats we use Montecarlo for. If we have to many indicators the strategy will for sure be adapted so i am only looking for really simple ones . One way i found to do this with SQ is to generate only on 1-4 years of data and let 15 years be out of sample. I then find that the only ones that actually have a okey  curve is very simple and actually robust.

 

 

(I do not know Kevin Davey but he hangs out at another forum and are very active)

 

 https://futures.io/vendors-product-reviews/26335-kj-trading-systems-kevin-davey-ask-me-anything-ask-me-anything.html

 https://www.youtube.com/watch?v=kBW7-p2DMVU

 

YES, Kevin is really a good model. I read his book and re-read it any times. He is really active on forums and …well, a good model, teaches a lot. 🙂 I don’t know him either, but he comes across as very approachable. I guess he sold many, many of his book too.

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137119

How reliable the Robust test?

The original strategy fails but Robust test shows very good results.

See the snapshot of the configuration.

 

http://s33.postimg.org/9gc31fw5r/Robust_test_failure_settings.png

Any one can comment?

ありがとう。.

0

マビ

Customer、bbp_participant、community、sq-ultimate、252件の返信。.

プロフィールを表示

10年前 #137120

@ Statistic,

 

Robustness test tries to simulate different conditions in the market by changing its data. Seems your strategy would do better on any other market conditions then on the one you created it on.

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137121

@ Statistic,Robustness test tries to simulate different conditions in the market by changing its data. Seems your strategy would do better on any other market conditions then on the one you created it on.

Thank you @Mabi, for rocket fast answer.  I generated a bunch of strategies and they work in OTHER condition better then on original data.

Never seen it before I switched to Dukas Copy data.

Strategy are generated on M15 , Trade On Bar Open Fast but when do Retest use Real Tick.

It doesn’t make a lot of sense how it happens.

I was usually using M1 commercial data, and reviewed manually over a million of strategies, but never happen seeing such discrepancy before moving on Dukas Copy Tick data.

Do you guys see that happening with Dukas Copy tick data?

ありがとうございます。.

 

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137125

500 simulations would let SQ halt.  (831 strategies in test pool)i7-4790, 16G ram.

Hi mate, just tested 1000 tests on 100 strategies ,   got a 35% CPU see the screenshot.

So nothing to do with you SQ, just need to use higher spec Severs.

Settings:

 

 

0

_Cujo

顧客、bpb_participant、コミュニティ、97件の回答。.

プロフィールを表示

10年前 #137126

I also have no trouble running 1,000 or 5,000 Monte Carlo Simulations on strategies. 

0

statistic

登録者、bbp_participant、コミュニティ、31件の返信.

プロフィールを表示

10年前 #137127

I also have no trouble running 1,000 or 5,000 Monte Carlo Simulations on strategies.

Just run 1000 tests on 100 strategies, too a long time over 30 minutes! But it is Ok, for the sake of testing.

it is better to rule out the failing strategies on 20/100/300 tests first this
way you do not need to run all the bunch failing strategies for 5 minutes each.

 

0

15件の返信を表示中 - 1 - 15件目 (全31件中)

1 2 3 →