Proper settings to avoid intensive SWAP usage

Hi,

I am running StrategyQuant on a quad-core computer with 8G physical RAM. I allocated 3 threads to StrategyQuant and allowed it to use 7G RAM, i.e. I used the following Java VM option when starting the program:

StrategyQuant64.exe -J-Xmx7g

My problem is when the evolution progresses, StrategyQuant will use a lot of SWAP memory (page file on hard drive) to avoid going over physical RAM limit. This significantly slows things done. Though I will upgrade my harddrive to SSD and increase my RAM size, it will still come to a bottleneck at some point.

My question is, approximately how does memory usage scale with the following parameters? And is there a set of recommended/reference settings based on which we can adjust according to hardware spec and the scaling rule? 

* Number of threads

* Population size

* Max number of generations (assuming no stagnation happens) 

* Max number of strategies to keep in databank

* Total number of bars for IS and/or OOS historical data

* Number of building blocks enabled

* Max tree depth for random population generation

* Anything else?

Thanks!!

-Zhong

Hello,

it is not a system memory swap, SQ uses files to store strategy data so that it doesn’t have to keep it all in memory and also as a prevention from crash.

In generally saves every strategy in the databank also as a file, and it saves list of trades (most memory consuming thing) to a temporary files.

The only thing that really affects this is the number of strategies in databanks, both result databank and Last generation.

So you can decrease it by using lower population size and total databank count, but the best solution would be to buy a faster SSD disc.

But even with normal hard disc this shouldn’t be that time consuming, the most time consuming process is running the backtest, which is few times more time expensive than writing/reading files to disc.

Mark,

Thanks for the great product and quick reply!

I installed a 240GB SSD yesterday and migrated my Windows 7 professional 64-bit onto it, together with StrategyQuant. I ran the evolution overnight, this time limiting to only 2 threads (I have quad-core), 250 population size, 2000 for databank with pretty strict rules (so I suspect only a few can make it in), and 6G maximum RAM usage setting (I have 8G total RAM). Unfortunately this morning StrategyQuant window cannot be displayed anymore. Windows is still responding to some commands, but very very sluggish and semi-frozen. StrategyQuant seems to be minimized and restoring or maximizing it does not bring out the window. “System Explorer” shows RAM usage is at 76% and CPU at 55%, StrategyQuant is using 4071MB RAM, and 5026MB SWP (see attachment).

As to the SWAP usage – my guess is Java VM is doing it automatically. If I set the parameter to limit Java VM to using 6G max but the program actually uses 12G, my guess is Java VM will automatically handle the swapping between physical RAM and page file on disk for the extra 6G. Of course this is a simplistic view – even with the RAM limit set higher than actual usage some swapping is possibly still going on.

I will try a smaller population size with 1 thread this time.

Finally a question: when I use 2 threads, does the RAM usage approximately doubles (at least for the portion that stores current population), assuming no limit is coming from the command line parameter?

P.S. Just ended the StrategyQuant process by force, and now RAM usage comes back to 22% and CPU to 2%. The system is responsive again. This desktop is dedicated to StrategyQuant so no other expensive process was running – except TeamViewer for remote access which takes about 500MB RAM and 1 to 3% CPU. Is there any concern of TeamViewer or other remote desktop application not playing well with StrategyQuant?

Thanks,

-Zhong

File: 2015-04-21_07-55-24.png

Just to add more information. 

My overnight run I mentioned above used the following parameters:

File: 2015-04-21_09-55-03.png

I have done extensive tests over weeks and can tell you the following when generation strategies with these parameters:

1 thread only (!)

14 years of data

30M timeframe

Selected Timeframe only mode

All indicators used

250 population size (but decimation factor of 10 for the initial random generation, so 2500 are generated at first)

unlimited number of generations but stop evolution if fitness stagnates for 3 generations

storing max of 50 strategies in the database

You´ll get away with 4GB per instance by using this launch-line and won´t run out of memory in Java nor SQ starting to swap: StrategyQuant64.exe -J-server -J-Xmx4g -J-XX:+UseParallelGC -J-XX:+UseParallelOldGC -J-XX:+DisableExplicitGC -J-XX:+AggressiveOpts

Once you raise population size above 250, RAM-usage goes up extremely, don´t know why, but it´s like that.

When you change to M15 from M30, RAM-usage almost doubles, the same if you then go to M5 and again to M1 (absolutely impossible to generate with that in the long run without assigning at least 32GB RAM to the instance with just 1 thread!)

If you allow more than 1 thread, RAM usage as well doubles from 1 to 2 threads and goes up heavily if allowing even more, while the CPU usage is not really much better. During generating strategies (I´ve already reported this as a “bug”), SQ uses almost always just 1 core on average. So it´s not really worth to assign more than 1 thread when generating strategies, it just wastes RAM (for the Optimizer though, it uses all cores just fine).

So you see, using M1, you´ll have no chance with 6GB of RAM, absolutely not. Try the above suggested things for now and you´ll be fine running SQ for days without problems. In my opinion though, SQ needs to get A LOT MORE EFFICIENT to really be useable. I have high hopes for V4 with that….

Thanks geektrader for such detailed reply! That was really helpful. I am trying your suggestions; also by tomorrow I will have 16G RAM on board to improve the situation.

Two pieces of info I may add:

(1) I made a mistake earlier by including 4 other symbols’ 1min bar data in “Additional data for testing”. I thought that was for test after the evolution, but obviously I was wrong – performance on the additional data was computed along with the main data during the evolution. By unchecking the additional data the evolution is much faster and RAM usage lower.

(2) Since StrategyQuant does a lot of disk reading/writing during execution (multiple GB temporary file would be written to disk during evolution), the use of SSD is highly recommended. But if one has enough RAM to spare, another possible speedup is to run the entire program on a RAM virtual drive (can be created easily by e.g. ImDisk), which will be multiple times faster even than SSD. The downside is of course less RAM available for the program itself, and RAM is volatile so if there is a power loss or unexpected reboot/shutdown all results are lost. May still worth trying at least – I will do so when the 16G arrives and report here if good result is obtained.

-Zhong