Is my setting for robustness test too harsh?

Hi everyone,

 

I used to run robustness tests this way, following the same method mentioned in Ebook:

1. I set the number of experiments to 200.

2. I run only one type of test each time. For example, randomize trades first, and then randomize parameters second.

3. decision rule: ret/DD > half of original level. Shape must be good.

 

With this method I got some 20+ strategies that pass the tests.

 

Now I am trying a different approach:

1. I set the number of tests to 1000.

2. I run the first 5 types of tests simoutaneously.

3. decision rule is the same.

 

Now all of the 20+ strategies failed the tests.

 

How should I interpret this results? Am I testing too much? Or is it a sign that my strategy is not robust?

 

Thanks for your reply.

Hi ybhx0315,

 

I started being super, super hard on my robustness as well and found no strategies getting through. then I changed what I was looking for. Once I understood that I’m not actually looking just for profit over historical, but looking for things like ability to pick winners, cut losses/DD, so those sort of metrics, not just “how much money did it make 10 years ago”, but was the win % good a few years ago and do I think (through robustness) taht it can still predict winners with small DD today/future, it made more sense. I was doing multiple OOS robustness tests on data going back to the mid 1990s, and nothing was passing when looking just for profit. But now I look for different things, also with walk forward, ofc.

 

There’s also some discussion on robustness in this thread.

Hi ybhx0315,

 

I started being super, super hard on my robustness as well and found no strategies getting through. then I changed what I was looking for. Once I understood that I’m not actually looking just for profit over historical, but looking for things like ability to pick winners, cut losses/DD, so those sort of metrics, not just “how much money did it make 10 years ago”, but was the win % good a few years ago and do I think (through robustness) taht it can still predict winners with small DD today/future, it made more sense. I was doing multiple OOS robustness tests on data going back to the mid 1990s, and nothing was passing when looking just for profit. But now I look for different things, also with walk forward, ofc.

 

There’s also some discussion on robustness in this thread.

Hi Cujo,

 

Thanks for your reply. I followed your thread and let me share 2 of my findings:

1. robustness test results usually stablize after 100 runs. So I see similar results in 100 runs, 500 runs and even 1000 runs.

2. Some robustness tests results have very good shape, but very bad numbers. And the opposite is true. For example, the original ret/dd is 20 and at 95% confidence level the ret/dd is around 6. DD% is almost doubled. But the robustness lines look quite densed to the center and looks pretty. Another example is original ret/dd is 4 and at 95% level the ret/dd is around 3. The robustness lines are very wide spread. Did you notice the same thing and how you interpret this?

 

Thanks.

Hi Cujo,

 

Thanks for your reply. I followed your thread and let me share 2 of my findings:

1. robustness test results usually stablize after 100 runs. So I see similar results in 100 runs, 500 runs and even 1000 runs.

2. Some robustness tests results have very good shape, but very bad numbers. And the opposite is true. For example, the original ret/dd is 20 and at 95% confidence level the ret/dd is around 6. DD% is almost doubled. But the robustness lines look quite densed to the center and looks pretty. Another example is original ret/dd is 4 and at 95% level the ret/dd is around 3. The robustness lines are very wide spread. Did you notice the same thing and how you interpret this?

 

Thanks.

 

…a beautiful equity curve is no good if it ruins your account as you won’t recover after a big DD until you put more $ in.