Stunning article about MONTE CARLO analysis

What are your opinion? I am very surprised… Because we hear, that we should use symmetrical approach at the building + we of course we use data-mining…

https://www.priceactionlab.com/Blog/2015/05/fooled-by-monte-carlo-analysis/

Just 2 things in advance:

Forced long/short symmetry

Many trading strategies employ forced long/short symmetry. Long/short symmetric strategies have several advantages but also some disadvantages but an analysis is beyond the scope of the article. The important point here is that most Monte Carlo algorithms break the symmetry that is present by design. As a result, any inferences made from them are false.

AND

When a strategy is developed via data-mining, Monte Carlo analysis is not useful in determining whether it is robust or random. However, this is exactly what many users and developers of data-mining programs do in many cases. The reason for the non applicability of this method is that when Monte Carlo analysis becomes part of the data-mining process, it loses it effectiveness as a validation tool due to data-snooping and selection bias. Although Monte Carlo analysis can be used to estimate probabilities of future drawdown levels, assuming it is applicable, it cannot be used as a validation tool. Amateur quant traders use Monte Carlo analysis for the purpose of validating strategies discovered via data-mining bias.

I follow this topic

Thanks for the link, the first thing I notice is that the article is talking specifically about tests which  only randomize the order of trades. SQ MC can also skip, randomize the market data or strategy perimeters which should be more useful.

Case 1 is obvious. Of course a randomize order MC test based from a curve fitted strategy is not useful.  Author could’ve included too small of a data-set here too.

<b>Randomizing order of trades breaks all patterns and sequences by design, not just longs shorts symmetry or probability of an extended downtrend.</b> MC can never show the true probabilities of actual events taking place. The limitations of MC are already well documented elsewhere. The author has outlined some specific cases but there literally are no specific cases where we can say MC is producing realistic results in the context of trading. It’s not meant to be realistic. It’s just an estimate or even (hopefully) a safely pessimistic risk analyses since humans tend to underestimate risk. There certainly are cases where (randomize order) MC is less useful than “normal” but I don’t think the author picked many great examples of that.

There’s another flaw in the case 3 report. I think here we have a good example of a human underestimating risk. It’s not impossible that the SPY can chop in a slow downtrend or move in some other way (other than an impossible downtrend in SPY) to cause this system to DD in a huge way the author is talking about and claiming it to be “unrealistic.”

Case 8 makes sense of course if you include MC as part of the data-mining process and run billions of strategies through MC or any other kind of test for that matter, we are data-mining to some degree. Although, just because a strategy was found by data mining does not mean the MC was necessarily part of the data mining process.