Evaluating the trading performance of strategies

A common problem for a trader is to know when his strategy has lost the advantage, or in short, when there is such a situation that does not fit today’s strategies. Strategies generated by data mining can experience performance degradation quite quickly, and these snippets can help analyze such situations. This episode is the introduction to a series of articles in which I will introduce new WHAT-IF snippets to help us deal with this problem. In the following episodes, we will also introduce snippets based on statistical quality control logic, and finally, we will conduct a study comparing whether these snippets statistically reduce the risk profile of strategies.

In today’s part, I have prepared a snippet that allows the strategy developer to measure the maximum drawdown in the in-sample period (the training period) and then terminate the strategy when it reaches the percentage value of this drawdown in the out-of-sample period.

.

I’ve prepared two versions of the WHAT-IF Analysis

 

1.Training/Test logic

e WHAT-IF snippet, you can set the percentage threshold of the percentage drawdown in the sample. For example, you can choose 50% of IS DD or 150% of IS DD. Finally, the strategy turns off during the out-of-sample period when it reaches the percentage drawdown measured in the DD sample.

2. Training/Validation/Test logic

This WHAT-IF snippet works with training/validation test logic. This way of analysis is based on the hold-out method. In the hold-out method for model selection, the dataset is split into three different sets – training, validation, and test dataset.

Source: https://vitalflux.com/hold-out-method-for-training-machine-learning-model/

 

The What If snippet has two mods.

 

  1. The maximum drawdown is measured in the training part of the data, and the strategy stops trading in the validation and testing part of the data after reaching the maximum training drawdown.
  2. The WHAT-IF snippet determines the maximum drawdown in the validation part of the data and stops trading in the testing part of the data after reaching the percentage drawdown measured in the validation data part.

 

In Strategy Quant X you can set the training/validation/testing in the Data section.

 

More information about how to use and set training/validation/testing you can find here.

 

Let’s take a look at the use of these snippets in practice:

 

Training/Testing What-IF snippet

 

In the figure above, we see a simple RSI strategy backtest divided into an in-sample period and an out-of-sample period. We can see that the strategy reaches the maximum drawdown in the out-of-sample period.

 

After using the WHAT-IF snippet, we can see what happens when the strategy exits the trade after reaching 50% of the maximum drawdown from the sample part of the data. The strategy in Out of sample will end the trade early. At the bottom of the database, we can see how the statistics of each strategy have changed. The drawdown in What If Out Of Sample is lower.

 

Training/Validation/Testing WHAT-IF snippet

 

Snippet works in two modes:

Mode =  1

Operates under the assumption that the stratégy trading is turned off when the percentage value of the maximum drawdown from the training period is reached. This logic is applied both in the validation and testing period.

Mode =  2  

Works with the assumption that the stratégy trading is turned off when the percentage value of the maximum drawdown is measured in the validation period. This logic is applied in the test period.

 

 

Basic backtest

In the following figure, we see the initial backtest of the strategy divided into Training/Validation/Testing.

 

 

Mode = 1

After using the WHAT-IF snippet, we see that the strategy failed to achieve 50% of its drawdown measured in the Training period. 50% of the drawdown was reached only in the test part of the data, where the trading was stopped.

 

Mode =  2

In this mode, I set a drawdown of 100% in the validation part of the data. In the testing part, the strategy reached a drawdown of 100% quite quickly and stopped trading.

 

These “WHAT-IF” snippets are an introduction to the topic. The goal is to work out the entire workflow. As for implementation in platforms, → the above solution can be implemented in MT4/MT5 and also in Tradestion/Multicharts. Anyway, in terms of modularity, the best platform is Metatrader 5.

At the end of the blog, I would like to open a discussion about ideas that could be integrated into the snippets. If you have ideas, please post them in the discussion below the article, or directly to [email protected]

 

Both What if snippet you can download here.

 

Subscribe
Notify of
11 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Bee trader
2. 3. 2022 10:01 am

Hi Clonex,

Thanks for the writeup and snippets. Have you considered using a method of fitting a distribution like the Beta and then use the statistical properties to stop trading if there is a deviation beyond the allowable/chosen threshold? Tim Masters talks about this in his books.

Thanks
Bee Trader

Bee trader
Reply to  clonex / Ivan Hudec
2. 3. 2022 4:51 pm

Which SQC methods are you considering for your next research?

Emmanuel
11. 4. 2022 6:03 pm

Excellent !!!!!! Thank you very much Clonex for this awesome work !!!!!

hannahis
14. 6. 2022 7:52 pm

“A common problem for a trader is to know when his strategy has lost the advantage, or in short, when there is such a situation that does not fit today’s strategies. Strategies generated by data mining can experience performance degradation quite quickly…” Merry go round and not progressing any further..endless cycle of trying and not quite succeeding… Have anyone ever sat down and consider why strategies often loses their edge (especially so for data mined strategy that are not based on sound trading theory)? Unless one sits down and thinks really hard, we will be ever chasing the wind and… Read more »

Last edited 1 year ago by hannahis
hannahis
Reply to  hannahis
14. 6. 2022 8:11 pm

You will no longer be mining or finding robust strategies. You will know the ingredients for robust strategies and start building them (without the need to search anymore because you know how to play along with the market makers, the key to develop robust strategies). Softwares won’t know about market manipulation, they only look at data but not the intention behind the data (market manipulation). So by purely looking at (historical) data without understanding market manipulation, the software won’t know what to look out for and what features are important to have etc.

Adam Kundrát
Adam Kundrát
Reply to  hannahis
8. 8. 2022 8:06 am

Interesting line of thinking and most likely right in terms of the fact that the market makers essentially have different intentions to trade compared to usual traders. Not sure whether you put equal sign between activities of market makers and market manipulation but I believe yes. Then the question is whether the manipulation is an intentional activity, or rather activity coming from an existence of the market makers? One or the other and considering your thoughts the questions would be: (1) can profitable strategies be found at all? based on your thoughts, I believe the answer is YES but your… Read more »

Adam Kundrát
Adam Kundrát
8. 8. 2022 8:20 am

Hi Clonex, thanks for your article. I am not sure whether I can still catch a full significance of your add but do I understand well that the outcome of your test essentially compares the highest DD in in-sample dataset to the current defined value of DD of a trading strategy on live account and stops trading the strategy when the defined value is exceeded? E.g. in-sample DD = $1.000, predefined value = 130% of the in-sample DD = $1.300. When the strategy exceeds the predefined value of the DD ($1.300) on live account then, MT4/MT5 closes all the trades… Read more »

SteveChou
9. 3. 2023 4:06 am

This is a good idea let trader to know how long will be failed with their strategy after they create.
But, I have a suggestion maybe it can be improve this evaluate.
It can change the reference DD from original strategy result to Monte Carlo 95%(or else %) result.
Because we know that MC result can predictor strategy possibility result in the future.
So, if reference result can be modify to MC result, maybe it will be better.
It just a thinking idea.

Continue reading