Two months ago we invited the db4o community to take part in the first db4o performance contest. Now the contest has been completed. Here are the results.
Three entries are eligible for prizes:
June 15
Andrew Zhang submitted a patch that improves the speed to sort queries.
June 19
Andrew Zhang submitted a UTF-8 string encoder.
July 17
Erik Putrycz submitted a patch to reuse query results in subsequent similar queries.
Andrew and Erik wrote own Poleposition circuits for their patches to be able to measure performance improvements. They have been incorporated into the db4opolepos project in our SVN as circuits "Hockenheim" and "Hungaroring".
The three patched jars that Andrew and Eric supplied have been renamed to db4o-7.2-sortedquery.jar, db4o-7.2-utf8.jar, and db4o-7.4-cachedqueries.jar to make it easy to distinguish the results. These jars have also been checked into the db4opolepos project. Anyone can easily reproduce the performance race by running org.polpos.AllRacesRunner from this project.
The results from running this benchmark on our hardware are attached to this posting as PerformanceContest.pdf.
Our impression from reviewing the results and the patches:
- The effect of the tuned quicksort algorithm on query sorting is absolutely amazing. Improvements are indeed in the range of 20x to 300x as you can see in Hockenheim#query_ascending and #query_descending on pages 52 and 53 of the attached Poleposition results. The patch is very straightforward and very nice, so it can immediately go into production code as is. A perfect submission.
- The UTF-8 string encoder reduces the required slot length for stored strings. Shorter slots result in faster writes as you can see in Nurburgring#write on page 24 of the attached Poleposition run. A performance improvement of about 20% is measurable. Incorporation of the patch code into the db4o core would require some more work: Probably we would add UTF-8 as a third configurable string encoding. Possibly we could make it the default. In any case we would have to continue to support the existing Unicode string encoding. The submission shows very nicely how easy a UTF-8 string encoder is. We should make one available soon.
- The idea to cache query results is great. A working implementation is certainly an order of magnitude more difficult to write than the other two submitted patches. Somehow the Hungaroring benchmark shows a slowdown instead of an improvement. After Erik's comments that our Trunk code got slower for concurrent queries, we have also compared running the patched Jar against Trunk and Trunk still turns out to be faster. Erik's submission also does not exactly follow the rules of the contest:
The patch is not applied to the 7.2 workspace, so a direct comparison run to other submissions is not completely fair. In any case we really liked the idea and we have to give credit for trying.
So how do we judge the three submissions and who gets first, second and third prize?
Andrew's submissions both are measurable and they are both in compliance with the rules that we have set up. Erik's submission is a great idea, but it is not 100% complete and we have not yet seen the improvement clearly documented in a Poleposition run.
The core team reviewers suggest to award prizes as follows:
1st USD 3000 Andrew Zhang
2nd USD 2000 Andrew Zhang
3rd USD 1000 Erik Putrycz
If anyone in our community has a different opinion, please post it to our performance forum.
Andrew and Erik: Thank you very much for your submissions. You have deserved the prizes!
For all other developers that tried to improve performance but did not complete a submission on time: Thank you for trying!
If you have suggestions how to improve the Performance Contest, please tell us about them. Maybe it would make sense to let the contest run for a longer period of time than 2 months? What do you think?
We are very happy to have received usable patches in this first db4o performance contest. Hopefully we will get even more contributions the next time.
It is great to work with such a nice community!