15% performance-boost and 40% less memory-usage using GraalVM
82 replies
geektrader
4 years ago #257329
Hi 🙂
I just wanted to share my “new” find. As always, I am exploring options about how to increase the speed of SQX for my daily workflow by tuning parameters, garbage collectors, etc. Lately, I´ve learned about GraalVM, which is a relatively new JIT-compiler that has maximum performance as its goal and has been shown sometimes to be up to 30x faster than if executing code in the standard JRE / OpenJDK runtimes that SQX is using right now. Over here is an excellent video about how it works, why it has been made and with some performance tests: https://www.youtube.com/watch?v=GinNxS3OSi0 For example: Twitter is running all their Java stuff on GraalVM instead of JRE / OpenJDK for a while now and have noticed a considerable drop in CPU and RAM usage.
And now the great part: using it with SQX is as easy as downloading the latest Windows binary from here https://github.com/graalvm/graalvm-ce-builds/releases
Simply unpack the ZIP file into the J64 of SQX (previously delete or rename the folder) and just continue working with SQX as you always did. Personally here is what I have experienced since replacing it and of course, using the same settings in SQX as before:
– about a 15% performance increase in strategy generation speed
– approximately a 40% drop in memory consumption (which is AMAZING given that I can now work with less RAM)
– no errors or crashes of SQX just works the same as with JRE / OpenJDK
As this is open-source, I would highly recommend that SQX is being shipped with GraalVM in its default configuration, as this seems to be amazing! There also is an Enterprise Edition available that seems to be even faster, but it´s 193 Euro/year and I haven´t tested it yet.
Have fun crunching strategies 😉
hankeys
3 years ago #260485
in this logfile is something stated
You want to be a profitable algotrader? We started using StrateQuant software in early 2014. For now we have a very big knowhow for building EAs for every possible types of markets. We share this knowhow, apps, tools and also all final strategies with real traders. If you want to join us, fill in the FORM.
geektrader
3 years ago #260494
Your command line is a mess, look at the log file:
Command Line: -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCIProduct -XX:-UnlockExperimentalVMOptions -XX:ThreadPriorityPolicy=1 -Djava.net.useSystemProxies=true -Djava.net.preferIPv4Stack=true -Xmx6g -XX:+UseParallelGC com.strategyquant.strategyquant.SQStarter
Stick to just using:
-server -Xmx26000m
and replace the 26000 with however many megabytes you want to assign. Should work fine after that adjustment.
hankeys
3 years ago #260496
i am not using any CL, i just only start SQX with icon – so this must be some devteam setting, not mine
with v20.0.0 i have no problem at all, but with the newest one v20.1.1 nothing happens and only the logfile is created in the SQX directory
i will try also on my server what will happen there, just trying on the second machine
You want to be a profitable algotrader? We started using StrateQuant software in early 2014. For now we have a very big knowhow for building EAs for every possible types of markets. We share this knowhow, apps, tools and also all final strategies with real traders. If you want to join us, fill in the FORM.
Conmariin
3 years ago #266913
Thanks Geektrader for this hint! 🙂
I changed to the open GraalVM on my VPS with Intel and can confirm the improvements: Faster creating strategies and 50% less RAM consumption! Great! 🙂
An error at starting up sqx with the original VM, but this didn’t have an effect on working with sqx it starts well though, vanish after implementing the new Graal VM. But I don’t know if it is related to.
Conni
Automatisches Handeln mit Expert Advisor
https://www.rabenesche.de
geektrader
3 years ago #266916
Great to hear that mein Freund 😉
Try the Enterprise Edition as described above, this is even faster.
Auf Wiedersehen 🙂
Csaba
3 years ago #266948
Hello Traders
I am using now GraalVM 20.2 (Java 11) Enterprise Edition and I can also confirm, that I see also an appr. +20-25% boost in the number of the hourly generated strategies.
Brg, Csaba
Conmariin
3 years ago #267617
New Version 20.3 out. Bugfixes and better performance
Automatisches Handeln mit Expert Advisor
https://www.rabenesche.de
ivan
3 years ago #267677
i reconfirm a 30% improvement in memory usage (less memory) on one of the most generated and profitable pair, XAUUSD, in working with 10.000 strategies in databank. This setting is one of the most demanding possible.
in the past, with 64GB, i had over 100% and occasional freezing, now with the latest graal versions, i have 45GB occupied out of 64GB when the databank is full
its a notable and very important upgrade, however, although its not critical, still 64GB memory is the minimum…32GB is not enough but given the memory prices, i doubt the price would be an inconvenient
Timisoara, Romania
3900X 3.8 Ghz 12 cores, 64GB RAM DDR4 3000Mhz, Samsung 970 EVO Plus M.2 NVMe
ivan
3 years ago #267718
after some 24 hours of installing a new graal version, i got a “makop” virus which encrypted everything
recovering is almost impossible so i will do a fresh install
i recommend great care in installing java related files, this is one of main sources of infection and the fastest easiest way is a fresh install
Timisoara, Romania
3900X 3.8 Ghz 12 cores, 64GB RAM DDR4 3000Mhz, Samsung 970 EVO Plus M.2 NVMe
hankeys
3 years ago #267721
where did you get the 20.3 – from oracle website or somewhere else?
You want to be a profitable algotrader? We started using StrateQuant software in early 2014. For now we have a very big knowhow for building EAs for every possible types of markets. We share this knowhow, apps, tools and also all final strategies with real traders. If you want to join us, fill in the FORM.
geektrader
3 years ago #267724
No such virus here and I highly doubt that it is related to that. Oracle is clearly not distributing viruses in their VM´s, nor have I ever seen any other Java distribution containing a virus in the last 20 years.
hankeys
3 years ago #267726
because of that i am asking, because from oracle its not probable
You want to be a profitable algotrader? We started using StrateQuant software in early 2014. For now we have a very big knowhow for building EAs for every possible types of markets. We share this knowhow, apps, tools and also all final strategies with real traders. If you want to join us, fill in the FORM.
ivan
3 years ago #267742
i got it from http://www.graalvm.org
i also had some doubts that it came from there but the coincidence is striking, i didnt had any problems in many years of intense personal torrent activity, movies, games….etc
i implemented several changes including securing the connection, installing a better antivirus and internet security and pay more attention
i cleaned the system but i have to run a repair on windows and reinstall everything else
the good news is the desktop is for SQ only so i had a barebone system with only the OS, minimal drivers and the SQ, nothing else. I didnt had any files or EAs stored. I formatted the partitions because the encrypted files are impossible to recover
Timisoara, Romania
3900X 3.8 Ghz 12 cores, 64GB RAM DDR4 3000Mhz, Samsung 970 EVO Plus M.2 NVMe
geektrader
3 years ago #267767
Keeping daily full drive backups might be a good idea for you, saved my life a few times over the last years. You just switch back to 24 hours ago with that in about 20 minutes. Did not have to reinstall Windows for at least 10 years now thanks to that. http://www.drivesnapshot.de/en/index.htm is your friend for that and it´s portable and just 500kb in size, a single EXE file. No ugly bloated installs for disk imaging software.
That being said, 20.3 is running here for days, with no virus or anything suspicious.
ivan
3 years ago #267773
yes, i do honestly believe you that probably graalvm has nothing to do with that virus
i have several laptops in the family and none of them got infected, from time to time i use windows remote desktop connection, on the default port, but now i made several changes and i will make a complete image of the disk C just in case. I read that its technically possible to get a virus on the default port of RDP
i had windows restore enabled but the virus crippled several functions including restoring, so in such a case, only a separate complete image is of practical use
Timisoara, Romania
3900X 3.8 Ghz 12 cores, 64GB RAM DDR4 3000Mhz, Samsung 970 EVO Plus M.2 NVMe