What we have learned from analyzing 1.2 million FX strategies – Part 3

Part 3:Profit Factor, Ret/DD Ratio/ Stability SQ3/ Annual % Return,Winning Percent, Win/LossRatio/ Sharpe Ratio

 

Before reading this part, I recommend that you read the two parts that cover Part 1 and Part 2

In today’s episode, we will analyze the commonly used strategy metrics that many algo developers use when evaluating strategies.

As always, let us get back to how we plotted 1.2 million strategies.

So we have generated about 1.2 million strategies using only basic filters like a minimum number of trades and Stability SQ3 > 0.6. We used the largest possible data sample for the period 1987-2017 IS + OS and for the period 2017-2020. (WFOS)  As True Out-of-sample (WFOS ) we used the period 2019-2020 – two years. During this period we observed the effects of selection, robustness tests, and other analyses. We have developed strategies for a 1-hour time frame and a 4-hour time frame, we used the Darwinex fee structure and had 1 pip slippage in each market. We modify the dataset by limiting the average number of trades per year to 100. Too high number of trades is unacceptable from the point of view of cost reduction and slippage, in trading reality.

 

I tried to adjust each strategy performance metrics for outliers. That did not always work, and it was not always perfect. Most of the time, I adjusted the strategy performance metrics to show 0.5 to 99.5 percentile values. Then I did a quantile normalisation of the data by splitting it into 10 deciles using qcut.

 

We will analyze these strategy Out of Sample metrics

  • Profit Factor
  • Ret/DD Ratio
  • Stability SQ3
  • Annual % Return
  • Winning Percent
  • Win/Loss Ratio
  • Sharpe Ratio

 

More information on the strategy metrics can be found on this page:

https://strategyquant.com/doc/strategyquant/results-overview/strategy-analysis-metrics/

 

Profit factor  ( OOS )

As we can see in the table, higher values of the Profit factor ( OOS ) do not lead to higher values of the WFOS Net profit and the WFOS Profit factor.

 

Winning Percent (OOS)

In Winning Percentage (OOS) we see better values for WFOS Net profit and WFOS Profit factor. We get better values in WFOS Period at lower Winning Percent (OOS) values.

 

Annual % Return  ( OOS)

 

We also see no significant differences in Annual % Return (OOS) between the same periods. In this case, we see the weakness of quantile normalization when the last decile represents strategies with a value of  4.62 -9.21%. Interval si simply too big.  Therefore, I tried to divide the annual percentage return metric into simple intervals. You can see the result in the table below:

 

 

In this case, we see that higher values of the Annual % Return (OOS) can lead to higher values of the WFOS Net profit and the WFOS Profit factor, but with less statistical significance, apart from the fact that there are significantly less  strategies in the last two intervals.

 

Stability SQ3 (OOS)

 

For stability Stability Q3 (OOS) , we also see no significant differences between the same intervals. In this case, I tried to split the Stability Q3 (OOS)  into simple intervals. You can see the result in the table below:

 

 

In this case, we see that higher values of Stability SQ3 (OOS) can lead to higher values of WFOS Net profit and WFOS Profit factor, but with lower statistical significance, apart from the fact that there are significantly less strategies in the last two intervals.

 

Win/Loss Ratio (OOS)

As we can see in the table above, higher values for the Win/Loss ratio (OOS) do not lead to higher values for the WFOS Net profit and the WFOS Profit factor. Rather, the results show that better WFOS values are obtained by lower values for the Win/Loss ratio (OOS).

 

Return/DD Ratio  (OOS)

 

As we can see in the above table, higher values of Return/DD ratio (OOS) do not lead to higher values of WFOS Net profit and WFOS Profit factor. The last decile was 5.5-13.43, so I opted for a simpler form of interpolation than in the previous cases. (I took the interval of all the values and divided them by ten).

 

 

In this case, I do not see us getting significantly better results at high intervals in WFOS.

 

Sharpe Ratio  (OOS)

 

As we can see in the table above, higher Sharpe Ratio (OOS) values do not lead to higher WFOS Net profit and WFOS Profit factor values.

 

Conclusion

The purpose of today’s article was to point out that many of the metrics I use in rankings or as fitness metrics in genetic starvation have relatively little predictive power. It’s a good idea to do more research before creating a strategy to find out which factors have predictive value.

 

In the next two episodes, we will look at the same metrics, but with the insights we gained in parts one and two.

 

clonex

clonex

7 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Emmanuel
Emmanuel(@emmanuel1)
November 27, 2021 11:51 pm

Thank you for excellent study !!!!!! This is impressive !!!!!

tnickel
tnickel(@170-tnickel)
November 28, 2021 10:29 am

Hi, very good work.
This means, the most matrics are not able to filter profitable strategies in the analysis.
In my opinion the curvefitting is too high. If the generated pool of strategies have more curvefitted than non curvefitted strategies => the filter are not working good.

I think: If we use a small building-block- set which produces less curvefitted strategies the filtering with metrics will be better.

binhsir
binhsir(@binhsir126-com)
Reply to  clonex
January 8, 2022 6:42 am

I am looking forward to the research on the setting of building blocks. Sometimes I am hesitant about what range of building blocks should I choose.

Last edited 15 days ago by binhsir
Oliver
Oliver(@oli_georgehotmail-com)
November 28, 2021 3:33 pm

With so many strategies id be interested in knowing…….if you backtest them all on all 28 currency pairs(insample)…..then select strategies that as a portfolio produce a return to dd ratio of say 1 or 1.5…… then see if that produces more predictive powers on the OOS. thanks

binhsir
binhsir(@binhsir126-com)
January 8, 2022 6:38 am

These 3 blogs is excellent, thanks very much.
But the pictures is very unclear, could you please reload these pictures?

Continue reading

New What If Snippets

clonexDecember 28, 2021

Two new What If snippets, which allow us to test how strategy can be managed, depending on its current performance, have been added to the sharing server. The first is …

Coding chat transcript from 18/11/2021

Mark FricNovember 24, 2021

Thanks again to all the participants of the last coding chat. The next coding session will be on 25th November at 14:00 CET time, which is 13:00 London time, 8am …