multiple testing

Hi,

It’s well known that the methodology used in strategy quant introduce false discovery of the strategies.

http://link.brightcove.com/services/player/bcpid2207682327001?bckey=AQ~~,AAAAt6-8xFk~,r8nyyutNWdB9cOzqmiiiQxC0oXMNGP7d&bclid=2204845638001&bctid=3878525144001

http://eranraviv.com/multiple-testing/
 

http://www.financial-math.org/

see also David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado and Qiji Jim Zhu, “Pseudo-mathematics and financial charlatanism: The effects of backtest over fitting on out-of-sample performance,” Notices of the American Mathematical Society, May 2014, pg. 458-471

Can you introduce the measure probability of backtest overfitting ??

Krzysztof

I  have found in general, SQ backtest results and MT4 results align.  If they are not, I suspect you are not backtesting in MT4 correctly.  Getting MT4 to backtest propertly needs third party support such as Birt’s Tick Data Suite.

 

Sounds like you’ve made your mind up, SQ and EA analyzer are not the products for you at present.  Maybe some future versions will align with what you are looking for.

 

Good luck in your trading.

I forgot one thing. Of course not taking into account bias due to multiple testing contaminate whole robustness analysis as input data (strategy chosen) has a bias so it is also a show stopper…..

MT4 backtester is OK, what is not OK are the internal build in functions of MQL like iMAOnArray, iRSI and others. I had the same problem with mismatch between MT4 and MATLAB but when i replaced those functions with custom MQL functions it was OK.

Personally i think SQ and EA are great tools as even with those what i wrote they are ahead of competition due to e.g. possibility of custom extend but for sure they need more work.

Krzysztof

Krysztof, I read these articles some time ago already, and I want to incorporate some of the things mentionesd there to the new SQ 4.

But most importantly, we are developing new SQ 4 as more open and extendable system, so it will be possible for you as user to add functions like these to the programm with more or less programing.

As for the difernces between SQ and MT4 – this is a neverending theme, and usualy the problem is in either diferent data or diferent test settings. Small diferences in data or settings could have bug impact on results – just 1-2 missed trades could make very big diference.

When run on the same data and settings, the test in SQ is virtualy equivalent to NT4, except small rounding errors – which can sometimes create a big difference in some strategies.

But what I think is important to realize is that if strategy has diferent backtests in SQ and mT4 it could also mean that it is not god enough – because every test is only an aproximation.
Your real trading will be again diferent than any backtest and god strategy should be able to perform also in changed environment.

Just throw such strategy away and look for new ones.

Krysztof, I read these articles some time ago already, and I want to incorporate some of the things mentionesd there to the new SQ 4.

 

But most importantly, we are developing new SQ 4 as more open and extendable system, so it will be possible for you as user to add functions like these to the programm with more or less programing.

 

As for the difernces between SQ and MT4 – this is a neverending theme, and usualy the problem is in either diferent data or diferent test settings. Small diferences in data or settings could have bug impact on results – just 1-2 missed trades could make very big diference.

When run on the same data and settings, the test in SQ is virtualy equivalent to NT4, except small rounding errors – which can sometimes create a big difference in some strategies.

 

But what I think is important to realize is that if strategy has diferent backtests in SQ and mT4 it could also mean that it is not god enough – because every test is only an aproximation.
Your real trading will be again diferent than any backtest and god strategy should be able to perform also in changed environment.

Just throw such strategy away and look for new ones.

 

 

 

 

From where those rounding errors come from ??? If you use the same data and the same variable types (double, float) and the same precision in rounding there will be no errors. I have been working with software test like 25 years so i’ts hard to believe for me……Do you know exactly or you just suspect ??

 

Krzysztof

From where those rounding errors come from ??? If you use the same data and the same variable types (double, float) and the same precision in rounding there will be no errors. I have been working with software test like 25 years so i’ts hard to believe for me……Do you know exactly or you just suspect ??

I know exactly, there are two differences:

1. SQ is in Java, while MT4 is probably in C. There are slight differences in rounding functions (or settings they use) between functions in Java and C

2. you don’t consider that MT4 could work with all the trades unrounded, and round only the resulting sum. But In the report that QA loads the trades are already rounded, so thus a small difference in the final sum.

I know exactly, there are two differences:

 

1. SQ is in Java, while MT4 is probably in C. There are slight differences in rounding functions (or settings they use) between functions in Java and C

 

2. you don’t consider that MT4 could work with all the trades unrounded, and round only the resulting sum. But In the report that QA loads the trades are already rounded, so thus a small difference in the final sum.

Ticket	 Open Time	Type	Size	Item 	Price	S / L	T / P	Close Time 	   Price	Commission	Taxes	Swap	Profit
461143512015.08.24 01:28:12sell	0.01	eurusd	1.13764	1.15767	1.11767	2015.08.24 02:36:001.14471	-0.08	         0.00	 0.00	-7.07

.

For curiosity I checked. So here is the example. Open price 1.13764, Close price 1.14471 lot size 0.01 so profit = (1.13764-1.14471)*10000*0.01 = -7.07 so its exactly what is written in report. If u use the same data and technique u will get exactly the same results

If the final sum is rounded u can get only small differences but differences between MT4 and SQ are huge sometimes. I think Richard sent to Tomas like 2 weeks ago such example but no reply so far. If the differences are huge it means that SQ backtester is unreliable.

Krzysztof

I can not post a new topic in QA section, getting ‘page not found’ error. Can u fix it ??

Krzysztof

If the final sum is rounded u can get only small differences but differences between MT4 and SQ are huge sometimes. I think Richard sent to Tomas like 2 weeks ago such example but no reply so far. If the differences are huge it means that SQ backtester is unreliable.

 

you are mixing two different things. Big differences between SQ and MT4 could be caused by differences in data or settings, not by rounding. 

Rounding could cause differences in cents only.

If the differences are huge it means that SQ backtester is unreliable.

this is a misunderstanding of how backtesting works. Backtesting in both MT4 and SQ is only an approximation. You cannot say that backtesting in MT4 is better than in SQ, just because it produces different results for some strategy.

What you can say is that this exact strategy probably has some inherent properties that make it difficult to backtest, so no backtest will be reliable.

If you’ll take the same strategy and backtest it in yet another program you’ll get again different results – this is a problem of a strategy, not a backtester.

It is really a waste of time to work with strategies that have too different results between MT4 and SQ – take them as they are – strategies that cannot be backtested reliably and should be avoided.

Wow

Backtesting in both MT4 and SQ is only an approximation

 

I can’t believe it. I have an AI system with MATLAB and MT4 and MATLAB backtesting matching almost to the pip so is not aproximation at all !!!!

The big mismatch can happen if your internal implementation of indicators is different than MT4 indicators and after in generated code u map them to MT4 indicators and functions. Example of it is iRSI which is implemented different than Wilder RSI. So does the MT4 code generator maps indicators to MT4 build in indicators or custom functions ??

Krzysztof

EA generated by SQ uses standard MT4 indicators, and the computation of indicators in SQ should match the ones from MT4, we reimplemented them all in Java.

But you are right that there could be a bug (although unlikely) or just a rounding error in some of the indicators implementation in SQ that could cause slightly different indicator values -> different entry/exit signals -> different strategy results.

But this is why I say that backtesting is always an approximation.

If you have strategy that performs well in one backtest, and by using slightly different data or indicators implementation (not different logic, only rounding errors) causes the whole strategy to fail then it is not reliable strategy.

It means the strategy is curve fitted to the specific data / indicators data combination.

You should use robustness tests to recognize such strategies vulnerable to change in data or indicators.

Only after your strategy works well also in robustness test you can be more sure it is reliable and not curve fitted.

I don’t want to advocate the differences between MT4 and SQ, the results in both programs should be the same or similar for most cases. But when they aren’t it usually means something else than bug in SQ.

Marc,

There is no approximation in software design – there are just bugs !!!!

The reason of those mismatches is different implementation of basic indicators in MT4 and SQ. Just follow my posts (fajst_k) in this forum

http://www.forex-tsd.com/metatrader-4/8221-rsi-moving-average-30.html#post671448

until post 325

I know this as I had mismatch between MATLAB and MT4 and when i replaced certain build in MT4 functions by custom ones i got a match.

Robustness analysis does not guarantee the screening of strategy which backtest don’t match with MT4. If ‘good’ strategy will be presented to new data during trading, it can take different branch and ‘bad’ MT4 functions can be called. Beside this in such case u trow away the strategy only because of bugs and this strategy can be valuable. Just change your MT4 code generator and replace ‘bad’ build in functions of MT4 (iRSI, IMA, IMAoNArray and who knows what else) with custom ones.

I still cant post a new topic in QA section of the forum, getting ‘page not found’. Can you fix it ??

Krzysztof

my goal is that the indicators in SQ and MT4 match and we have tests for all the implemented indicators.

They match with occasional very small differences at third-forth decimal place caused by rounding errors and slightly different implementation of Math functions in Java.

I don’t know of any bugs in SQ where indicator would have very different values from the values computed by MT4.

I still cant post a new topic in QA section of the forum, getting ‘page not found’. Can you fix it ??

I cannot find what’s wrong, could you send me the exact URL you are trying to open?

after attempt of create a new topic in Quant Analyzer section i’m getting ‘page not found’ error. See screenshot.

Krzysztof

File: scr.jpg

my goal is that the indicators in SQ and MT4 match and we have tests for all the implemented indicators.

They match with occasional very small differences at third-forth decimal place caused by rounding errors and slightly different implementation of Math functions in Java.

 

Do you have a logfile of those tests ?? Can you present it on the forum ??

Krzysztof

Do you have a logfile of those tests ?? Can you present it on the forum ??

what exactly do you want? Do you  want to verify that it is as I say or what?

These are standard JUnit tests, it returns OK when all tests passed. I don’t have log file for it. 

what exactly do you want? Do you  want to verify that it is as I say or what?

These are standard JUnit tests, it returns OK when all tests passed. I don’t have log file for it. 

You should feed the test signal to your indicators than the same signal to e.g. MT4 build in indicators. Than you should compare output results. Only allowed discrepancy should be very small rounding error. This test should be repeated for different combinations and values of input parameters as you don’t know implementation of build in MT4 functions and you can not guarantee that they behave properly. And of course this test should be done against indicators from all trading platforms for which SQ generate strategy code. Have you done such testing ?? If yes provide a logfile at least for MT4.

Krzysztof