Not logged in

Forums

Viewing 15 posts - 61 through 75 (of 94 total)

Forums>StrategyQuant (formerly named Genetic Builder)>General Discussion>[How To] Tune SQ for max performance and reduce memory usage by 4 to 5 times

  • #134592 |
    Customer
    574 Posts

    Great, I am really looking forward to a beta-test, can´t wait to bug-hunt everything:)

    #134606
    Customer
    574 Posts

    Btw Mark, have you ever used something like http://www.ej-technologies.com/products/jprofiler/landing.htmlto profile your code and look for the parts that can potentially create memory leaks / slow downs? JProfiler is dope in finding such things and you can sort them out right away. A great site in that relation always is: http://www.javaperformancetuning.com/

    #134611
    Customer
    574 Posts

    I´ve updated post #1 again as I have now switched to the serial garbage collector.

     

    I´ve never even considered this one since it works single-threaded and not multi-threaded like the others and everyone said it is the slowest of all. However, I still thought it will be worth at least a test and wow, for SQ it seems to be the BEST one of all. Speed-wise as in terms of optimizations/generations performed by SQ, it is just as fast as the parallel garbage collector, but the RAM usage is actually 1/3 lower! When reading how this collector works, it´s not really good for latency as each time it works, it performs a full “stop the world” (e.g. the garbage collection holds any processing of the Java program and performs it´s “full cleaning”, then resumes the program). However, the advantage of this is that the garbage collection is super-perfect, since it holds the program and can efficiently clean ALL garbage completely, while the other garbage collectors perform their work while the program keeps on running and hence never can clean all garbage as perfect as the serial collector can do it.

     

    With the updated settings you will save a lot of RAM usage especially during longer SQ runs but won´t compromise any of the speed during optimization / generating. Have fun:)

     

    Parallel Collector:

     

    parallel_collector.jpg

     

    Serial Collector:

    serial_collector.jpg

    #134841
    Customer
    5 Posts

    Mark,

     

    I was doing some general research on Machine Learning and ran across this.

     

    Supercomputer performance on a PC via CUDA processing

    Modern nVidia video display cards make their massive parallel processing power available to users via what they call a CUDA interface. The very best nonlinear models such as general regression neural networks can be extremely slow to train, making them impractical for very large problems. Programming CUDA implementations of the best models can speed training by a factor of hundreds, or even thousands, reducing training time from hours to seconds.

     

    Have you considered possibly using CUDA to help the new Strategy Quant with Performance?

     

    Thanks,
    Bryan Lee

    #134845
    Customer
    574 Posts

    Hi Bryan,

     

    how about you use the Forum search function for that instead to hijack a unrelated thread on your very first post with something we have discussed several times already in dedicated threads?

     

    https://strategyquant.com/forum/topic/2766-use-higher-end-gpu-for-numerical-processing/

    https://strategyquant.com/forum/topic/1455-cuda-please/

     

    Thanks.

    #134851
    Customer
    5 Posts

    Sorry Geektrader I will do that from now on, new here, I did’t mean to hijack, my apologies.  I see you have contributed much help here, you are very knowledgable with SQ.  I appreciate the fact that you give back to the community. Once again sorry.

    #134853
    Administrator
    3028 Posts

    Bryan, we were considering this long time ago and reconsidered it few weeks ago with the new version.

    While it seems it may be technically possible to implement backtesting engine in GPU, it might not be faster and it would kill the flexibility we want to achieve with the new version.

     

    As I wrote in other threads, backtesting is not ideal for parallel processing on GPU, because it is much more complicated than numeric and matrix operations, so even if it would be implemented in GPU, it might not be much faster.

     

    But the main reason is that the program would be not open and extendable, which is one of the main goals of the new version – every indicator would have to be hardcoded in GPU code, and it would not be possible to extend the platform with your own indicators and building blocks. 

    Instead of using GPU we plan to support grid computing, so you could have SQ4 running on multiple computers.

     

    But I’ll keep an eye on GPU as well, perhaps there will be some parts of the system that could use GPU to speed up some computations.

    Mark
    StrategyQuant architect

    #135241
    Customer
    2 Posts
    Hello to all,
     
    I have a xeon processor E5 1660 v3 running on windows 7,
     
    the processor utilization never goes over 20%.
     
    I set SQ to use 16 threads, and also using this tune does not go better.
     
    Do you have any advice?
    #135242
    Customer
    281 Posts

    Try to read this thread and youll find answer

    #135363
    Customer
    380 Posts

    Can you use the same tune up settings as in post one with analyzer and/or tickdatadownloader to reep performance improvements?

    #135381
    Customer
    574 Posts

    Yes, absolutely. I am doing this too.

    #135382
    Customer
    734 Posts

    Great question @ stearno

    #136123
    Customer
    85 Posts

    Hi Geektrader

     

    Thanks for the Java fine-tuning settings. They make a big difference.

     

    I have also implemented your recommended “replace the JVM from Oracle to the one from Zulu”. Though this is working fine, I have found that SQ launch and initial operation with Zulu is much slower than previously with Oracle. Though, within 10-20min SQ performance is back to being similar to Oracle. This happens upon ‘each’ launch of SQ, not just the 1st launch after PC boot-up. Is this to be expected with Zulu JVM?

     

    FYI, my PC is: Intel 4-core i7-4790 CPU, 16Gb RAM, running on Windows 10.

     

    Thanks again!

    AC1962

    #136157
    Customer
    574 Posts

    Hi AC,

     

    can´t really reproduce this, for me it´s faster than Oracle for launching and the first 20 minutes too, and especially in the longer run (40+ days of strategy generation just finished without any memory leaking like in the Oracle builds). Not sure what you might be doing wrong, maybe it has something to do with the sluggish Windows 10 MS crap, I am on “stripped to the bones” Windows 8.1. Maybe someone else with Windows 10 can comment. Sorry to not really being able to help with that.

    #136168
    Customer
    85 Posts

    Hi Geektrader

     

    Thanks for your quick response. It may well be something to do with Windows 10, or maybe because the PC I am currently using for SQ is a general purpose PC, probably slowed down due to the amount of software installed. I’ll look into doing some housekeeping on my PC, that may help improve it’s performance. The issue is not a big deal as SQ Build gets back up to good speed after a short while. Thanks again.

     

    AC1962

Viewing 15 posts - 61 through 75 (of 94 total)

You must be logged in to reply to this topic.