The bottleneck of speed limitation

Hi,

is the harddisk the bottleneck.

Here is a screenshoot of my i7-4930

With windows 10.

 

 

 

Next days I will use a SSD and compare this.

 

thomas

When you load tick data or larger amounts of minute data SSD should really help to speed up

Tick data testing streams the data constantly from disk,  for non-tick testing, disk should not be a problem.

 

I turn off my antivirus as it seems to get the antivirus scanning log files slowing everything down.

Hi, I don´t use in this generation process tickdata.

 

This are Duka M1 Data.

 

thomas

Hi, I don´t use in this generation process tickdata.

 

This are Duka M1 Data.

 

thomas

 

I would dig deeper and see which process is using the disk when it is busy, I suspect it is not SQ, unless you are low on RAM and it’s swapping out to paging file constantly.

 

One suspect is antivirus scanning the SQ log file as this is written to a lot.

 

SSD is very cheap now, so put OS and SQ onto SSD. 

Problem I have with SSD solution is the size.  So when using tick data and many data feeds, could exceed the SSD amount.  So as another option: What about using 2 Hard drives and RAID – mirroring?  Would that speed things up if the hard drives are really the bottleneck?

 

-Stearno

I would do some I/O Tests on your current Disks with something like I/O meter or other Disk benchmark tool without SQ or anything else running.  This should give you a base line of what the system is capable of.  Also you will find that when dealing with raw tick data etc they are small file sizes and lots of them so the current underlying block sizing of the file system might slow things.  

 

There are a number of things you can do to improve Disk problems some cheap some not so cheap.

 

You didn’t say whether this drive is separate from the OS, I’m assuming that its all one drive.  To start I would look at having a separate drive from your Operating system and other files, also have it formatted to block size that caters for the lots of smaller files, so becomes a SQ and tick data only drive.  If not going down SSD road then go for fastest spindle speed drive you can afford, if more than 1 do Raid 0, just won’t be redundant if a drive fails, also how you connect them to your PC will affect speed i.e. Direct vs external drive.   

 

Other not so cheap solution is to look at second hand retired servers and disk storage, these are designed for high data throughput and you should be able to find some out on eBay or auction houses.  Usually these are leased equipment at the end of their lease and are sold off, still perfectly serviceable for what we want to do with SQ.  e.g. dell 2950 with SAS drives etc.  If running from home then adds some additional problems such as noise, power and cooling.

 

Also as @mikeyC says, disable or exclude your antivirus on SQ + data directories.  

 

Stephen

Great advice. I will try it.

Sent from my HUAWEI MT7-TL10 using Tapatalk