Video Tutorial SQ Max Speed & Performance CPU,Ram,Disk & Extra 6k Strategy Quest challenge

https://www.youtube.com/watch?v=l6oRsTNm0yc

// This is the link of the video , Check it out hopefully you will like it . YES i am arabic and the accent follows
me to death so if you have a problem with it “I am verry Sorry but i have to KILL YOU”

A big credit goes to “GeekTrader” , This Video tutorial is to teach you how to max speed of SQ 3 .
Tweaking CPU,rams and hard drives / SSD ‘s

The original post is like 10 pages + which seems bit complicated for beginners here so i explained it
step by step in this video + added more tweaks of my own (Hope it helps out beginners and even pro users of SQ).

End of the video is a quest for people to contribute more by filtering some good strategies and show us the procedure .. I am sure each pro will choose a different strategy(ies) we got to learn from the contributes.

Links of the video:

https://www.youtube.com/watch?v=l6oRsTNm0yc

* Strategies : https://drive.google.com/file/d/0B83k2vKtYK80UWNuWDZsenpQNzQ/view?usp=sharing

* Java Version 9 : http://cdn.azul.com/zulu-pre/bin/zulu…

* The scipt To use for the batch opening of SQ, Create new .bat file and add these line of codes and press
save . Watch the video to learn how to tweak the parameters.

This is a new Script if you want to rename each instance so you can know each instance job . ( if you do not need this option use the above code )
1- NumberOfSQInstances= “to whatever number of instance you want to run” Lets say “X” instances
2- set arrayline[1]=CrossMaStrategyInstance
set arrayline[2]=RSIStrategyInstance
…. Continue declaring more to fit your X instances
lets say we want 3 instances so i ll add one more
arrayline[3]=anotherNewInstance

The code is :

How many strategies did u have in databank

Hi Clonex

1295 strategies in databank

AC1962

Hi gentmat

I have now been running SQ for 2.5 days, using your recommended batch file script set to 1Gb RAM / thread / SQ instance. Each instance is setup with recommended Zulu 9.0.0.2.

While overall strategy creation speed is much improved, I notice that the memory consumption meter for each SQ instance rapidly increases with time, seemingly relative to the amount of strategies saved in databank. See attached screenshot of memory consumption meter from SQ instance with greatest quantity of strategies in databank (~1300). It is almost up to the script setting limit of 1Gb RAM.

In your video you indicate that for this setup you would expect SQ memory consumption meter to stay reasonably flat, not exceeding 500Mb over a week or so period. Are you able to explain to me why, after only 2.5days, my SQ memory consumption meter is already approaching 1Gb, when set to 1 thread and using Zulu 9.0.0.2? My PC setup is: Windows 10 home; 4-core i7 CPU; 16Gb RAM.

When I previously used to run 1x instance of SQ with thread count set to 8, even though strategy creation speed was slower, the memory consumption meter rarely approached 3Gb even after 6 days of running SQ continuously.

I am just curious, because to increase my RAM allocation from 1Gb to 2Gb per SQ instance will require me to upgrade my overall installed RAM from 16Gb to 24Gb or 32Gb.

Any insight you can provide on this issue, would be much appreciated.

Thanks
AC1962

Number of population ? How much is it
What tf are you using ? And market entry !
And are you keeping in indicator the highest lowest ? Are you using amy of these 4 indicators ?

Sent from my iPhone using Tapatalk

Hi gentmat

See below

Number of population ? How much is it

Genetic population size = 100

What tf are you using ?

time-frame = H4

And market entry !

Enter at Stop

And are you keeping in indicator the highest lowest ?

Yes

Are you using amy of these 4 indicators ?

Yes, all 4: HIGHEST; LOSEST; HR; LR

Thanks

AC1962

Hi gentmat,

Your kind sharing of your performance analysis, helpful suggestions, performance tips and that great script for the bat file to create multiple sessions of SQ are greatly appreciated. 

My systems are customized Lenovo D30 workstations each with dual Xeon E5-2670s with 16 (32 virtual) cores and 128GB ram ea.  One is dedicated to searching 24/7 for strategies and the other is used for analyzing the strategies to refine them into a portfolio.  Previously, I was manually updating 32 copies of SQ whenever I wanted to make a change now, I just change the master copy run the batch file and life is so much easier!

The SSDs on my systems are for Windows 10 only and as such are relatively small (240GB).  SQ and the other FX programs run on a 2TB hard drive that easily accommodates the original installations and 32 copies of SQ and still uses less than 25% of the drive space. You mentioned using Supercache and I would like to suggest an alternative. PrimoCache offers a 60 day free trial and then a purchase price of $30.  Set-up is easy and performance is outstanding. (I have no affiliation what so ever with either package.)

I had been wondering for a long time whether it was more efficient to use random generation or genetic evolution so I used your script and created a pseudo scientific study to try to get some data points.  I created two batch files based on your script to generate 10 copies utilizing genetic evolution and 10 copies utilizing random selection.  Both were based on the exact same price data, the same spread and slippage settings, the same date spits between learning & validation and the same ranking conditions.  (Full details of data and ranking are listed at the bottom.)  Both sets of 10 copies each ran concurrently for 24 – 36 hours when they were then stopped and the strategies that were created were counted and saved for further analysis.  On each of the three tests the total of all 10 Genetic Evolution databanks contained at least 200 times the number of strategies that were in the Random Generation databanks. Here are the summary data from the latest 24 hour test:

Genetic Evolution:

Number of instances:  10

Minimum number of strategies in the databank:   33

Maximum number of strategies in the databank: 129

Total number of strategies for all ten instances:  582

Random Selection:

Number of instances:  10

Minimum number of strategies in the databank:   0

Maximum number of strategies in the databank:  2

Total number of strategies for all ten instances:   2

Now, one person performing three tests for different time periods obviously does not make a valid study.  So I would like to ask whether other users are using Random Generation and if so if they are able to generate strategies any faster?  Are there any other advantages to using Random Generation over Genetic Evolution?

________________________________________

Here are the parameters used for this testing:

Data: EURUSD M5 tick data from Dukascopy

Backtest dates:  05.05.2003 through 11.04.2012

In sample 05.05.2003 through 04.01.2010

Out of Sample: 04.01.2010 through 11.04.2012

Spread: 2.0

Slippage 3.0**

Test Precision: M1

Ranking Criteria – reject anything matching:

In sample:

Profit Factor: < 1.3

Return/DD Ratio: < 3.3

% Wins: < 30%

Number of Trades: < 300

Out of sample:

Profit Factor: < 1.3

Return/DD Ratio: < 1.1

% Wins: < 30%

Number of Trades: < 100

** The first robust testing was determining the sensitivity to slippage.  My assumption, which may be incorrect, is that if the strategy already performed with 3 pip slippage it can only improve with no slippage.  Any and all comments or guidance are welcome.

Hi Dave

I initially installed SuperCache on my 2x Windows 10 PC’s. It has worked well on one PC, but caused the 2nd to continually crash on restart. Checking SuperCache support it seems this is a known problem. 3x solutions are offered but first two did not work for me, and third I did not have IT confidence to attempt. Net result was that I had to get by PC supplier involved and ended up resetting Windows 10 to solve problem 🙁 and remove SuperCache. After searching web I also found PrimoCache, which is not only a lot cheaper, but: has better review ratings; offers far better up-to-date on-line support; has an active forum; installs cleanly; more features but still very easy to use. I now have PrimoCache installed on both by PC’s, working great.

After initial testing I now always and only use Genetic Evolution, because the evolution feature automatically provides a proxy of performing multiple strategy improvement once a strategy passes acceptance criteria. Often, when SQ finds a strategy that passes into databank, Genetic Evolution will also create a mass of passable variation strategies based on the 1st. I would much rather see my databank filled with a 100 variations of Genetic Evolved strategies, than just 1x Random strategy. Obviously doesn’t ensure a robust strategy, but clearly improves the chances of hitting on one. Your test seems to support my own finding.

Also, I feel that using gentmat’s excellent batch file improves the chance of hitting on a good strategy, as often 1x of the multiple single thread SQ instances will churn out far more potentially viable strategies than the other instances. Chance are this would not so readily happen if I just used 1x SQ instance set for multiple threads. Thanks again gentmat!

AC1962

This is a special request only use if needed . else use the generic code that i posted in the first page here is the code ( To rename each instance ) .. Each sq3 instance will be named as you wish it to be . Exemple : CrossMaSq3Instance and so on .
Please change NumberOfSQInstances=2 to your need

Lets say i want 10 instances : i ll put NumberOfSQInstances=10
then i have to create 10 of these :
set arrayline[1]=CrossMaStrategyInstance
set arrayline[2]=RSIStrategyInstance
set arrayline[3]..
set arrayline[4]…
set arrayline[5]..

.. until set arrayline[10]..= its name whatever it was .

The code is the following save it in a .bat file

@echo off
setlocal enabledelayedexpansion

set NumberOfSQInstances=2
set arrayline[1]=CrossMaStrategyInstance
set arrayline[2]=RSIStrategyInstance

set MainSQLocation=C:\StrategyQuant
set TempSQLocation=C:\temp
set SQParameters=-J-server -J-Xmx1g -J-XX:+DisableExplicitGC -J-XX:+AggressiveOpts -J-XX:+UseSerialGC

rmdir “%MainSQLocation%\temp” /S /Q
rmdir “%MainSQLocation%\log” /S /Q
rmdir “%TempSQLocation%\strategyquant-temp” /S /Q
mkdir “%TempSQLocation%\strategyquant-temp”

FOR /L %%A IN (1,1,%NumberOfSQInstances%) DO (
mkdir “%TempSQLocation%\strategyquant-temp\!arrayline[%%A]!”
)
FOR /L %%A IN (1,1,%NumberOfSQInstances%) DO (
compact /c /s:”%TempSQLocation%\strategyquant-temp\!arrayline[%%A]!”
)

c:
FOR /L %%A IN (1,1,%NumberOfSQInstances%) DO (
xcopy “%MainSQLocation%” “%TempSQLocation%\strategyquant-temp\!arrayline[%%A]!” /E /Y
CD “%TempSQLocation%\strategyquant-temp\!arrayline[%%A]!”
rename StrategyQuant64.exe !arrayline[%%A]!.exe
start /LOW !arrayline[%%A]!.exe %SQParameters%

)

Hi gentmat

Great! Thanks for the modified .bat file code. It works very well.

Very much appreciated

AC1962

 i believe if the strategy rely on good quality history data test, it will not stable, i will fail the monte carlo for sure.

Hi gentmat

Great! Thanks for the modified .bat file code. It works very well.

Very much appreciated
AC1962

There was unused variables in the script now i deleted them.
It wasnt affection it but deleted it anyway to be more clean

Sent from my iPhone using Tapatalk

i used 2 cpu e5 2670 system to run this way but it crashed two times in less that 24 hours , it generate  error file like this

could anyone help on this, what is the cause of the problem.

Thank you .

total memory is 32G ,since it has 32 logical cores, i made 30 and 28 copies ,   is that caused by not enough memory? but each set the limit to 1G, 

yourrodstaff,

To put it simply, with 32 cores (threads?) 32GB is not enough memory a large quantity of SQ copies running at the same time. The first problem is that having two CPUs for a total of 32 cores is not the same as a single 32 core CPU.  A single E5-2670 v1 CPU is rated by PassMark with a score of 12,505 while a dual CPU E5-2670 v1 is rated at 18,472. There is a lot of overhead and inefficiency in sharing resources – memory, disk access etc. In this case a dual CPU system less than 150% of two independent E5-2670 computers.  I happen to use two of these machines and even though they are less powerful they were the best cpu processing power available for the money.

This inherent inefficiency with non-CPU resources is exacerbated when there not enough memory.  If you want get an idea of what your real “free” memory then boot up with nothing loaded and open the task manager.  That will show you the amount of memory being used.  The free memory is 100% – the amount shown in the task manager. For example my single cpu I7-4770 (that I used before I purchased the E5-2670s) with 16GB shows 7% memory utilization after a new boot up.  That means the maximum memory I have available is 93% of the 16GB or no more than 14.88 GB.  If you are allocating 1GB to each copy of SQ then the maximum number in theory that you should be able to run is 14 copies.  In reality I used to run 12 copies with 1 thread utilization and 1 GB allocated and lived with crashes about every 1 – 2 weeks.

If you use other programs like virus scanning or disk caching then the number decreases accordingly. If you are using your computer for other tasks then you need to allocate fewer copies.  Another factor is that memory usage is not linear with each instance.  For every copy of SQ that you open, SQ opens approximately 50 additional threads each taking up more system memory so there is no exact formula that I know of to find the maximum or optimum combination and they all need to coordinate with the OS,CPUs, memory controllers HDD controllers etc which requires more resources.  Think of it like kids in kindergarten the problems are exponential, not linear.

For maximum performance you will need to use a disk caching program like PrimoCache otherwise your computer will become bogged down re-reading data.  That program will also require about 15 – 25% of your RAM which means less instances but the performance improvement is worth it.  My systems were bogging down to an absolute crawl with 20 – 25 instances with a maximum CPU usage of 50% – 70% even though I had 128GB.  When I installed PrimoCache and allocated 32GB to it, I was able to load 40 instances with 2GB allocated to each, for a total of 80GB of memory usage plus the 32GB for the cache and the bottleneck switched to the CPU usage that is now 100% across all threads as opposed to the HDD throughput.  If you think about it, SQ reads the same data for every test and if that data can reside in RAM as opposed to an HDD platter and simply using PrimoCache (or any other cache program) will increase considerably.

I use two – 32 (e5-2670 v1) threaded systems each with 128GB both using PrimoCache at 32GB.  One computer is used 24/7 100% of the time generating strategies and is running 40 or more instances of SQ (one thread per instance) with 2GB allocated to each instance.  The second serves as my day to day computer, the household file server, backup server and video server for all of the multimedia in the house and still runs 24-28 instances of SQ at 2GB per copy. One thing – I have to re-boot each machine once every one or two weeks as there is a memory leak somewhere.

I hope that this helps.

Dave

Dave,

Thank you so much for the very detailed illustration of the problems and sharing the experiences. 

the reason i did that is in the first 12 hrs after start running the memory meter inside each job is almost flat, it was keeping about 700Mb and total 77% memory usage and the CPU is like 100% loaded. So I believed it was ok to keep like this, but after overnight it crashed.   

 So i reduced to 14 job each with 2gb allocated, so the CPU is now at 50% and memory about 40%, which should last longer than the first setup, but based on your experience seems like still not enough.   I am getting total 64Gb memory now also i will try the your suggestion about the disk cache memory.

Again I appreciate your help!

I ll send you email try to fix that . Adding GB is good idea for sure in anyway