Can QuantAnalyzer show helpfull results? Think again…

While the post’s nature will be mainly technical it’s main purpose is giving feedback to you, so I put it here. I can’t improve your products, only you can.

I’m disappointed, I confess. Until now I liked what I’v seen on your website a lot and was ready to spend money for two full life-time licenses. After my experience and the post now you may say I’m just ranting. So I will let the facts speak for themself and not post any more opinions. So be it.

I will run a basic BollingerBand mean reversion stratgey in MetaTrader, import the results into QuantAnalyzer and compare the results with the MetaTrader results and my own statistics.

1) The test. The strategy is very simple and has an obvious flaw (no stoploss). The flaw will help us later to spot discrepancies and irregularities more easily. The rules are simple and not important (in fact any strategy might be used):
– Long: Buy if price breaks out of the BBand and returns into the channel. Open max. two more positions on two more consecutive buy signals. Close all positions if price crosses the BBand mean line (the moving average) or at least one position is in profit.
– Short: opposite to long
– As there is no stoploss we will experience times where price quickly trends away and positions are not closed for a considerable amount of time. Sooner or later the opposite position side will kick in and limit or lock the losses.

Taken trades are visualized with a custom script in a regular chart. As can be seen there are two periods where the missing stoploss causes major havoc. Positions were opened but never closed. The test ends with both 3 long and 3 short positions open and locking each other, thus causing a seemingly sudden drawdown at the end. Seemingly because in fact there never was a profit a drawdown could have started from.

@see attachement: 1-Test.png

2) The MetaTrader results as we all know it. Nothing special, so far so useless. A novice might even think the strategy is good. Full report in the ZIP file.

@see attachement: 2-Test.png
full report: MetaTrader-Report.zip

3) The real results as recorded by my own statistic tools during the test. Clearly visible the long stagnation periods (long and short sides locked) and the drawdowns in between. Now even the novice trader realizes the strategy has a problem.

@see attachement 3: 3-Real-Results.png

4) Finally the analysis made by QuantAnalyzer. Indeed the main numbers are calculated correctly but hardly anything could be more misleading and wrong than the given visual presentation. Everything from equity (wrong, in reality it’s hypothetical balance) to drawdown (completely wrong) to stagnation period (completely wrong) is so useless that I still don’t know what to say. What makes it worse to a novice trader the QuantAnalyzer results look even better than the MetaTrader results because in MetaTrader the drawdown is at least kind of visible. However in QuantAnalyzer drawdown and stagnation are almost completely suppressed.

@see attachement 4: 4-QuantAnalyzer-Results.png

For now I don’t dare anymore to ask for distribution analysis, for Sortino, for Calmar, for linear regressions. Just to start with. Still wondering…

ps: Did I miss the web page where all this is mentioned and explained? Please show me I missed it.

Hello,

thank you for long article and interesting topic.

I understand your point, but you have to understand that QuantAnalyzer can work only with data that are imported to it. It canot display accurate equity or drawdown, when this information simply is not in the report produced by MT4, and there is no way to somehow compute it only from the list of executed trades.

But, if you’ll switch your equity chart from Trades to Time, you’ll see the big periods of stagnation – :
@see attachement: equity_time.png

There is also a possibility to use minute historical data to compute some of the missing values – by using Open balances computer. I used it with AUDUSD data from Dukascopy, and this is the result on equity chart. There are some strange spikes (probably some issue with simulation engine), but overall the red line shows open equity:

@see attachement: equity_time_openbalances.png

The power and functionality of QA is in other things – you have advanced tools like What If scenatios, Money management or Portfolio simulation, Monte Carlo analysis, etc.

As for the documentation, I know it is not always ideal, but you can just “play” with the program to see how things work.

We have to realize that tools like QuantAnalyzer, which analyze the trade history records, will never be able to accurately display all the things that go on during live trading – simply because there is no data for this in the results file. There is only something that can be computed from history results.

Our vision is to make StrategyQuant a platform where you can generate strategies, trade using advanced features (equity control, programatical switching strategies on/off, etc.) and performing such analysis on its running and historical results. We are moving towards this goal with SQ4, although there is still some way to go.