What we have learned from analyzing 1.2 million FX strategies

Part 1: Complexity and Number of Trades

About a year and a half ago, I came to the conclusion that the edge of my strategy development workflow was shrinking. . In 2014-15, making 40-55% per year on FX was a no-brainer. In 2016-2017, it became increasingly difficult to make profits with the standard workflow. Therefore, I decided to examine and analyze some paradigms of some procedures in statistical detail.

So we generated about 1.2 million strategies using only basic filters like a minimum number of trades and Stability SQ3 > 0.6. We used the largest possible data sample for the period 1987-2017 IS + OS and for the period 2017-2020. ( WFOS )
As True Out-of-sample (WFOS ) we used the period 2019-2020 – two years. During this period we observed the effects of selection, robustness tests, and other analyses.

We developed strategies for a 1-hour time frame and a 4-hour time frame, we used the Darwinex fee structure and had 1 pip slippage in each market.

The goal was to explore:

which factors in strategy selection influence True-Out-of-sample performance  ( WFOS )

How basic robustness tests work

• complexity vs True Out-of-sample ( WFOS)  performance
• the effect of the number of trades on strategy performance
• spread analysis vs True Out-of-sample (WFOS)  performance

How it works

• Monte Carlo filtering
• SPP
• Multimarket testing

In designing the analysis, I drew inspiration from interviews with Better System traders and also from several other interviews with various hedge fund managers, and of course from the robustness testing tools available in StrategyQuant X.

I used Python tool, at least the Pandas library, and worked with the Jupyter Notebook IDE.
To generate such a large set of strategies required 3 Strong PCs with about 150 cores in total. The generation itself took a month and another month and a half for preparation and testing. We had two people working on it. The initial form of analysis was manual testing of factors in Python, later I used machine learning methods to validate my results.

We have generated the following honors in these markets:

Edit: Added Additional informtions:

• In builder whe enabled all possible exit types except Exit Rule. We have also enabled Exit On Friday to avoid weekend gaps.
• Only used Entry type of our strategies was Enter at stop

Complexity

I was initially interested in complexity. If we consider the strategy as a model obtained by data mining, we can use the machine learning paradigm and various other sources about how the complexity of the model affects out-of-sample performance. In our case, we can assume that the complexity of the strategy obtained through data mining has an impact on the performance of the strategy in its True Out-of-sample Performance (WFOS – a portion of the data that was never used in the development or selection of the strategy).

In the figure above, we see all the strategies bucketed by complexity and their effect on the median strategy in a given complexity interval ( Interval 2-3 means all strategies with complexity 2, etc. ).

• WFOS Median  Net Profit
• WFOS Median Profit Factor

In the orange frame, we see two columns

• Rel. freq:  relative frequency of strategies in a given interval ( bucket)
• Count: number of strategies in a given interval ( bucket)

Note that strategies with high complexity had lower median values for net profit and profit factor in WFOS (True Out-of-sample ). Conversely, strategies with higher complexity had worse median values.

Now let us look at the graphical representation of the above table

Complexity vs. WFOS Median Net profit

Note: The red line represents the average complexity value across the dataset

In the graph above, we see that strategies with complexity 2 and 3 achieved very poor True Out-of-sample median results. Unusually poor results were also achieved by the strategies with the highest complexity of 19. However, it is important to note that these are outliers – strategies that are very few in numbers.

Complexity vs. WFOS Profit Factor

If we clean the dataset of outliers (I select only the values of complexity where the highest number of strategies is found), we see the above observations from the previous graphs even more clearly.

I only select strategies whose complexity is higher than 3 and lower than 13.

In the figure above, we see that the lower the complexity, the better the median values of net profit and profit factor (complexity 4, 5, 6 and median values in the green frame).

Let us take a look at the visual representation of the above table

The lower the complexity of the strategy, the better the result of the WFOS Median Net and the WFOS Median Profit Factor.

In the analysis, I also use inferential parametric and nonparametric Tests. The null hypothesis was formulated that there is no statistically significant difference between WFOS NP and PF with complexity 4,5,6 compared to strategies with complexity 7-13. The test results show that we can reject the null hypothesis and accept the alternative hypothesis. A large number of strategies significantly reduced the standard deviation and boosts the significance of the test.

Number of trades

In analyzing the number of trades, I assumed that the higher the number of trades ( the larger the sample of trades ), the more meaningful insights we get into the ultimate quality of the model. ( Strategies )

Which outcome of the strategy generated in SQX is more credible? The one that generates 10 trades in 10 years or the one that makes 1000 trades in 10 years?

Let us look at a basic analysis of our data set:

Distribution of strategies according to the number of trades in the whole backtest

Distribution of strategies by several trades binned into deciles

We can see the basic characteristics of the Number Of Trades distribution. Note the 1st decile and the 10th decile. They represent the intervals 97-470 trades ( 1 decile ) and 1598 to 9575 trades. ( 10 decile)
I will try to solve this problem by removing the outliers.

In the above figure, I see that even after cleaning the dataset, the increasing number of trades and hence statistical significance is not conducive to the performance of WFOS Median Net Profit a WFOS Profit Factor.

In the above graph, we can see that a higher number of trades does not lead to better WFOS Median Net Profit and WFOS Profit Factor results. In particular, please keep in mind that I am only making a selection. We do not select strategies based on other parameters.
On the contrary, the results show that fewer trades can mean better results in WFOS.

Complexity + NoT

Let us try to combine the findings from the previous complexity analysis and select strategies with a complexity of 4 to 6 and several trades between 480 and 2600.

The change is noticeable. The combination of strategies with lower complexity and number of trades in the intervals 1326-1708 and 1708-2599 leads to significantly better results in WFOS Median Net Profit and WFOS Profit Factor.

Conclusion of the first part

It can be seen that the complexity in our dataset can affect the True Out-of-sample 1,2 strategies. The number of trades without combinations has no influence, but in combination with complexity, it is a strong factor. We remind you that this is the first part of a larger series.

The number of trades without combinations has no influence, but in combination with complexity it is a strong factor.

What does it mean exactly? That you should aim to build strategies that are as simple as possible while producing as many trades as possible..
If you want to affect the complexity of your strategies in SQX you should keep the number of entry and exit rules low. I personally prefer Maximum Entry conditions set to 1-2.

Here you can specify the number of entry conditions that directly affect the complexity:

In the next episode next week, we will look at the robustness of the strategy to a higher spread. In the next episodes, we will go over the standard recommended metrics such as Profit Factor, Rer/ DD Ratio, and others. In the following episodes, we will take a look at the effectiveness of Monte Carlo tests and SPP.