Btw Mark, have you ever used something like http://www.ej-technologies.com/products/jprofiler/landing.htmlto profile your code and look for the parts that can potentially create memory leaks / slow downs? JProfiler is dope in finding such things and you can sort them out right away. A great site in that relation always is: http://www.javaperformancetuning.com/
I´ve updated post #1 again as I have now switched to the serial garbage collector.
I´ve never even considered this one since it works single-threaded and not multi-threaded like the others and everyone said it is the slowest of all. However, I still thought it will be worth at least a test and wow, for SQ it seems to be the BEST one of all. Speed-wise as in terms of optimizations/generations performed by SQ, it is just as fast as the parallel garbage collector, but the RAM usage is actually 1/3 lower! When reading how this collector works, it´s not really good for latency as each time it works, it performs a full “stop the world” (e.g. the garbage collection holds any processing of the Java program and performs it´s “full cleaning”, then resumes the program). However, the advantage of this is that the garbage collection is super-perfect, since it holds the program and can efficiently clean ALL garbage completely, while the other garbage collectors perform their work while the program keeps on running and hence never can clean all garbage as perfect as the serial collector can do it.
With the updated settings you will save a lot of RAM usage especially during longer SQ runs but won´t compromise any of the speed during optimization / generating. Have fun:)
I was doing some general research on Machine Learning and ran across this.
Supercomputer performance on a PC via CUDA processing
Modern nVidia video display cards make their massive parallel processing power available to users via what they call a CUDA interface. The very best nonlinear models such as general regression neural networks can be extremely slow to train, making them impractical for very large problems. Programming CUDA implementations of the best models can speed training by a factor of hundreds, or even thousands, reducing training time from hours to seconds.
Have you considered possibly using CUDA to help the new Strategy Quant with Performance?
how about you use the Forum search function for that instead to hijack a unrelated thread on your very first post with something we have discussed several times already in dedicated threads?
Bryan, we were considering this long time ago and reconsidered it few weeks ago with the new version.
While it seems it may be technically possible to implement backtesting engine in GPU, it might not be faster and it would kill the flexibility we want to achieve with the new version.
As I wrote in other threads, backtesting is not ideal for parallel processing on GPU, because it is much more complicated than numeric and matrix operations, so even if it would be implemented in GPU, it might not be much faster.
But the main reason is that the program would be not open and extendable, which is one of the main goals of the new version – every indicator would have to be hardcoded in GPU code, and it would not be possible to extend the platform with your own indicators and building blocks.
Instead of using GPU we plan to support grid computing, so you could have SQ4 running on multiple computers.
But I’ll keep an eye on GPU as well, perhaps there will be some parts of the system that could use GPU to speed up some computations.
Thanks for the Java fine-tuning settings. They make a big difference.
I have also implemented your recommended “replace the JVM from Oracle to the one from Zulu”. Though this is working fine, I have found that SQ launch and initial operation with Zulu is much slower than previously with Oracle. Though, within 10-20min SQ performance is back to being similar to Oracle. This happens upon ‘each’ launch of SQ, not just the 1st launch after PC boot-up. Is this to be expected with Zulu JVM?
FYI, my PC is: Intel 4-core i7-4790 CPU, 16Gb RAM, running on Windows 10.
can´t really reproduce this, for me it´s faster than Oracle for launching and the first 20 minutes too, and especially in the longer run (40+ days of strategy generation just finished without any memory leaking like in the Oracle builds). Not sure what you might be doing wrong, maybe it has something to do with the sluggish Windows 10 MS crap, I am on “stripped to the bones” Windows 8.1. Maybe someone else with Windows 10 can comment. Sorry to not really being able to help with that.
Thanks for your quick response. It may well be something to do with Windows 10, or maybe because the PC I am currently using for SQ is a general purpose PC, probably slowed down due to the amount of software installed. I’ll look into doing some housekeeping on my PC, that may help improve it’s performance. The issue is not a big deal as SQ Build gets back up to good speed after a short while. Thanks again.
You must be logged in to reply to this topic.