WFO consistently crashes SQ 90% of the time

Even if I have more than 8GB free RAM, Walk Forward Optimization cannot be used.  If I invoke it, SQ crashes 90% of the time.  In fact, it only works if I optimization requires about 100 runs.  Memory management is extremely poor.  Clearing last generation strategies or clicking Perform Memory Cleanup does absolutely nothing.

After crash, when I restart SQ, it always offers to restore last strategies (even the ones I deleted under Last Generation).  If I click Yes, it always loads strategies from Last Generation and never the Walk Forward result!  Because the result file is always corrupt.

I’m sorry about this issue, it will be completely rewritten in the new version 4 and we will make it work much better.

Hi my systems work in last time with SQ3.8 very stable.

I have 32GB of Memory.

I increased the memorysize for every SQ to 11 GB.

You can run at least two SQ on one machine.

thomas

Even if I have more than 8GB free RAM, Walk Forward Optimization cannot be used.  If I invoke it, SQ crashes 90% of the time.  In fact, it only works if I optimization requires about 100 runs.  Memory management is extremely poor.  Clearing last generation strategies or clicking Perform Memory Cleanup does absolutely nothing.After crash, when I restart SQ, it always offers to restore last strategies (even the ones I deleted under Last Generation).  If I click Yes, it always loads strategies from Last Generation and never the Walk Forward result!  Because the result file is always corrupt.

I used several machines with 32core /32GB RAM, never failed me but yes, sometimes it is reactive but it is about 30 -60 seconds but only when I stop it, as it need to flash all what is in memory, I think on low spec it will be 5-10 minutes hence you feel it crushes. run 4x SQs for Generation, but they do not use all Cores , but when I run WFO it uses all 32 cores.

” Perform Memory Clean-up” indeed has no affect see the screenshot below:

On picture is 24gb, but I increased it to 32 as I thought Cleanup will work, so now it is with 32GB.

As you see there is no affect of this function. even after I Paused or Stop the Generation.

Mark, you probably already know the answer, please share, if we need t wait for the SQ4 it is fine.

I want to see All dedicated Cores are used (it is Windows Server 2012) . maybe there is another topic on this. in the settings All 32 cores are ticked.

Thanks

I tested on

Windows 10 (8core/16GB) – running 4x SQs – never crashes

Windows 10 (32cores/ 32GB) running 8xSQs – never crashes

Windows 2008 servers ( (8cores/32GB) running 1SQ (for some reason it uses all CPUs 100% ) – never crashes

Windows 2012 servers (32cores/32GB) runs between 8 – 15 of SQs never had any issues and run only on 10% CPU but because it uses all 32GB ram (need to increase it to 48GB)

it is usually.

mx55333,

SQ has no issues, it is fully tested from my end, so Mark, based on my testing SQ works as expected, no crashes what so ever, and I am very heavy user.

Only one issue it doesn’t use all 32 cores. But I think it might be because of disk Queuing , I will look into it, but if Mark already knows the answer please share. So Mark if you need to test SQ4 on any OS, memory/CPU please let me know, I will happy be beta tester. 🙂

Hi Mark,

Good news, I have optimised my ESX servers. server and now machines are not fighting for CPU time.

The machines is Xeion E5-2630 v3 2.4GHz 16 Core (2x sockets with 8Cores each giving 16 Threads per Socket = 32 Threads in total)

However I have better performance if I use only Real cores without Threads.

My questions to you Mark, if SQ is able to use Threads or it uses only Cores?

I have not tested with Hyperthreading Turn Off, but will do that on this weekends and reply accordenly with my finings.

SQ will used only 25% of given Cores, so affectivly 4 intances will be able to load 16 real cores.

I believe that SQ can only use maximum of 4 real cores, can you please confirm that is not optimised for more then 4 cores?

And secondly I need to know whether it uses Hyperthreading or not, I will check it myself but I believe you have the answer already.

This is the snapshot of 4 intances loading Xeon E5-2630

This test is not very accurate as I have other machines running on the same ESX server , and another one with Just two SQ instances, other machines are mostly idel like Domain Controller and other bits.

Even if I have more than 8GB free RAM, Walk Forward Optimization cannot be used.  If I invoke it, SQ crashes 90% of the time.  In fact, it only works if I optimization requires about 100 runs.  Memory management is extremely poor.  Clearing last generation strategies or clicking Perform Memory Cleanup does absolutely nothing.

 

After crash, when I restart SQ, it always offers to restore last strategies (even the ones I deleted under Last Generation).  If I click Yes, it always loads strategies from Last Generation and never the Walk Forward result!  Because the result file is always corrupt.

I have done more testing as I can confirm that SQ does freeze does not matter if I use 16 cores or 4 cores with 96GB or 8GB , running only ONE instance. it doesn’t work well with Retest Strategies. after the test done I cannot delete more then 1000-1500 strategies. Let’s say I generated 20000 strategies and after retest I want to delete 15000 of them, I would take about 30 times to delete them, it WILL NOT delete all 15000 of them, The CPU / Memory usage is the same for a hour so I can confidently say it freezes. So at this moment it cannot work with large arrays of strategies, maximum is 10k which comfortably works.

However if Mark can tell now to strip reporting part and go from 1MB to 78KB, that might help so we can go from GB to MBs. (as in reality we just need strategy without reporting if we want to work with them further)

If I could just script it so it does it from backend then it will be perfect.

Mark,

I suggest a feature:

When we save .STR can be small checkbox (no reports), this way size will be small and I think SQ3 will handle it better and it will not take 100s GBs of storage if you want to take it to further testing.

So I agree with mx55333, I have not get that far as Optimisation, so I cannot comment much on this, but I do confirm that over 10k strategies are not handled very well by SQ3. If Mark wants any testing done, let me know.

Thanks for your suggestions, I will forward the thread to Mark so he can take this into account