What is your experience with IS/OOS and WF Metrics results?

Lately I have been able to find some good strategies that produce good IS/OOS results with $1000 to $1500 of drawdown.  But when I plug these strategies in Walk Forward Matrix and change just one or two parameters the perfomance decreases drastically.  And by drastically I mean the strategy that prodcued $80K of profit over the testing period suddenly loses $20K.

I am just wondering what is the experience of other members here.  Are there any good practices that one should follow while runing walk-forward matrix? 

Thanks!

Vishal

Lately I have been able to find some good strategies that produce good IS/OOS results with $1000 to $1500 of drawdown.  But when I plug these strategies in Walk Forward Matrix and change just one or two parameters the perfomance decreases drastically.  And by drastically I mean the strategy that prodcued $80K of profit over the testing period suddenly loses $20K.

 

I am just wondering what is the experience of other members here.  Are there any good practices that one should follow while runing walk-forward matrix? 

Thanks!

Vishal

Did you import the settings from the previous test first?  I had a similar problem.  You go to RESULTS – STRATEGY SETTINGS and then click the right strategy settings to draw from and click the apply button.  There should be little or no red in the comparison which means you are using the same settings you used in the previous tests.  Otherwise you’re using a different set of parameters than what you just tested.

Thanks Toddone46, I am new to SQ but I love the program so far.  My biggest challenge lies in figuring out why the strategy that shows good results in both IS and OOS testing fails miserably in Walk Forward Matrix.  I can imagine there will be some amount of curve fitting with IS and OOS data while building the strategy but so far the results from Walk Forward matrix baffles me.

What I have also noticed is that once I run Walk Forward matrix, suddenly my original strategy goes in red also.  Now while running the walk forward I do change some of the parameters (two at the most).  I have yet to figure out why my original strategy goes in red suddenly.

And that is why I posted this discussion to understand the experience of other members who have been using this tool for a while.

Thanks!

Hi pandyav,

If the profit decreases drastically the strategy is curvefitted.

You can throw away this strategy.

thomas

Thanks Thomas.  What I’ve also noticed is that even if a strategy passes WF tests and Monte Carlo analysis, when I run the strategy using Market Replay function in Ninja Trader, the strategy gets hammered.  The only reason I could come up with to explain this is that strategies that we create in SQ are based on Open, High, Low, Close prices of an instrument.  This completely ignores the intrabar movement of an instrument which is the reality of trading.  

So far, I haven’t been able to figure out how to deal with this situation?  

it is true, testing in SQ for NinjaTrader doesn’t consider intra bar movement, but again if your strategy is that sensitive on this then it is probably not robust enough.

There should be strategies that use only OHLC prices and work.

Mark,

Thank you for the response.  Is there a way to find strategies that only trades at the close of a bar?  This will help me to avoid any intra bar movement and create more realistic strategies.