Darwinex and Dukascopy: The Difficulties with Changes of Historical Forex Data

Have you experienced changes in historical data? Or have you ever wondered why different providers have different data? Let’s discuss in detail the reasons why these situations occur. 

Forex trading can be complex and challenging. One of the most critical components of successful forex trading is data consistency. Keep reading why. On the other hand, achieving 100% accuracy is not possible for every forex broker. This can have significant implications for traders who use algorithmic trading strategies.

Why is it so important? Data consistency is related to the accuracy and reliability of the data that are used for decisions in trading. Such data include historical prices, economic indicators, news releases and other factors that impact currency exchange rates (prices). Inaccurate or inconsistent data may lead to incorrect trading decisions, resulting in losses for traders.

One of the challenges of achieving data consistency in forex trading is the fact that different brokers have different data sets –  there is no unified global data set (no centralized exchange). Every forex exchange and broker has its own data. There might be a variety of factors, such as differences in data collection methods or differences in the way data are reported. As a result, it’s common to see differences in price data between different brokers.

Another trap is the situation with data changing over time. For example, historical price data may be revised, economic indicators can be updated, and news releases can be corrected or retracted. These changes may have a significant impact on algo-strategies and their results, particularly in case a strategy has been developed and implemented.

It’s important to note that data changes may occur even with well-respected forex data providers such as Darwinex and Dukascopy. These providers have historically changed their data, and this typically means adding more ticks to original data. It could change Open, High, Low, Close price or Volume.

This isn’t necessarily a bad thing – brokers update their historical data to make them more precise (by adding ticks that have not been available before – at the time the data are created).

We just need to know it happens and to count with it.

Examples of data changes:

In the screen below, you can see that tick files for USDCAD have been re-uploaded one year later.

In the following case – all the data were uploaded on the same day except the marked one which looks like it was uploaded 3 days later, so the data doesn’t match in any source and the only single source of truth is Darwinex FTP.

Another example of data error -nonsense timestamp:

In conclusion, data consistency is a critical component of successful forex trading, particularly for algo-strategies. However, achieving 100% accuracy is not possible for every forex broker. Traders should be aware of the possibility of data changes that might impact their strategies. By being proactive and monitoring data changes, traders can help to mitigate the impact on their trading strategies and to improve their chances of success.

Our recommendations:

→ In case you want to preserve the backtest history strategy I would recommend to keep the older data (those with more records) and only to update the new data. Data can be backed up and moved between workstations by copying the ‘SQ X installation folder’/user/data.

→ In case you want to have 1:1 backtests with Darwinex FTP data – we recommend downloading clean data, but in this case you have to take into account that the results of your strategies might change with the small updates in historical data.

→ Our recommendation is rather to accept the change in the data, but to have 1:1 data with Darwinex. We also recommend testing the strategies for sensitivity to OHLC prices so that they are not sensitive to such changes.

→ This shows the importance of proper robustness testing – to ensure that small changes in data don’t have significant impact on your strategy performance. If this happens it indicates the strategy was overfitted to the historical data.
More about robustness here: https://strategyquant.com/doc/strategyquant/cross-checks-automated-strategy-robustness-tests/

Disclaimer:

StrategyQuant’s free data service is a tool designed to assist users in downloading historical forex data from third-party data providers. StrategyQuant is not a data provider and does not guarantee the accuracy, completeness or reliability of the data obtained through this service.

The data downloaded through StrategyQuant’s free data service should not be considered as investment advice or a recommendation to trade on a forex market. Users should exercise their own judgment when using the data and assume all risks associated with the use of the data.

StrategyQuant disclaims all liability for any loss or damage arising from the use of the data downloaded through its free data service. Users are responsible for ensuring the suitability and legality of their use of the data.

By using StrategyQuant’s free data service, users acknowledge and agree to the terms of this disclaimer.