Assume for a moment, that you want to create the newest social web application, and you want to use an established distributed version tracking tool as data backend, so all data can easily be migrated and merged between seperate parts of the system, and even be edited offline and merged into the mainline at a later time.
(shut up and take me to the results)
If you want free tools (and since I believe in free software as the only ethical choice for any application, I won’t assume anything else), there are two main contenders: Git and Mercurial.
From knittl2010 we know that git is the clear winner in terms of commit speed if we omit the time needed for adding the data and the time for the frequently needed garbage collection runs, git needs to keep the repository sizes sane. If we take this into account, the question becomes harder to answer.
On a typical human workflow, for which most current performance tests are designed, the raw speed of a command doesn’t really matter, as long as it’s not too long, because the time the command takes to complete is far below the time for typing the command — even if you just type “C-r com ENTER”. The highest typing speed ever measured is 216 words per minute, which equals 1080 characters, or 56ms per keystroke.
On a server, though, the raw speed of the commands defines how many requests you can answer per second, or more generally, how many servers you need for supporting a given userbase.
As you will start out with limited resources (if you aren’t extremely lucky, you won’t be rich from the start), let’s make two assumptions:
Also you want to provide your users with as much storage space per server as possible, and you want to guarantee a certain maximum response time, so no user sits in front an unresponsive website (even 1 second load time is frustrating, and you need at least 300ms to get the data from your server to your users).
The following additional assumptions make it possible to design the simple test:
¹: To verify that assumption, you can check the public hourly access statistics of the 1w6 project (with almost entirely german content; Timezone GMT+1):
As final assumptions say, that you’ll implement your system in Python, calling git via subprocesses and calling Mercurial directly via its API, to avoid the start time of the python interpreter (about 30ms). That is the best way to designing a system which uses Mercurial, and it does not have a negative effect on git (see “Limitations” for details).
For testing, I read the changed files for each commit and the mean size of the change from different repositories (see Readme for details). They serve as usage pattern. Then I got about 8 MiB of arbitrary lines of code from the repositories as example data.
For each commit I then add/change the files which were really changed in the commit, but I use random lines from the example data. (Up to) half the lines are used to replace existing lines, the other half is appended.
All these tests were run on a quadcore amd64 (605e, 2.4Ghz) Gentoo GNU Linux system with git version 1.7.3.4, Mercurial version 1.7.3 and Python 2.7.1+ (release27-maint:87872, Jan 11 2011, 19:13:26) built with GCC 4.4.4.
The disk on which the repositories were created is a SAMSUNG SpinPoint F1 DT series: SAMSUNG HD103UJ; Serial Number: S13PJDWS513146.
On small tests (only the first 155 commits), git performed best on the change data of the mercurial repository. Since the full test took quite long, I stopped it after processing that change data. This partly favors git.
The other main limitation of this test is that the history is linear, while real history will be nonlinear with many merges.
Also Git is called from python via subprocess.call() which imposes a cost from two to three times 3ms (0.006s for normal commits, 0.009s for commits with garbage collection and for automatic garbage collection). The minimal time for git in this test is about 50ms. This could be reduced to about 44ms by calling it in other ways.
12980 commits, hg usage pattern.
First we’ll look at all data parsed together, except for Mercurial via dispatch() (because it offers no new information) and the aggressive garbage collection in git, because that is much too slow.:
The size shown in the legend shows the minimum size, which the repository has directly after a manual garbage collection and a size close to the maximum size. Since the data is not perfectly the same, the minimum size can deviate. I did not calculate the standard-deviation (for time reasons), so it can only be estimated. If the sizes differ by 3 MiB or less, treat them as equal.
Then we add aggressive garbage collection in git (git gc --aggressive):
Note: The garbage collection steps created a mean load of 100% for 3 of my 4 cores.
To get a clearer overview, let’s check the cumulative data, ignoring all details.
The total time of the commits as well as the repository sizes:
For getting the bigger picture, the cumulative time for all commits for Mercurial, git with gc every 10 steps and git with gc every 100 steps - including fit-functions.
Now it’s time to look at some more specific questions:
The actions with less then 0.4 seconds runtime (only the range 0 seconds to 0.4 seconds is shown, though the garbage collection in git takes significantly longer). This hides the time for garbage collection, showing only the fast operations:
Now we check if we can rely on the program to leave no user waiting. For that we compare Mercurial with Git using garbage collection every 10, 100 or 1000 commits:
There are strange spikes for git without garbage collection, which should not be there, because there is no specific program which should take time. Let’s investigate: Mercurial vs. git with no garbage collection on the full range:
Since git offers automatic garbage collection, we look for that, too. Keep in mind, that with automatic garbage collection the final repository was bigger than with garbge collection every 1000 commits, growing up to 5× the size of the repository after grbage collection (I did not investigate that further; I also did not check the absolute maximum sizes because that would have skewed the test by refreshing all disk buffers by reading all file sizes from disk, which does not happen in the webserver scenario).
Mercurial vs. git with automatic garbage collection (git gc --auto called after each commit, which makes the actions of Mercurial and Git feature equal):
From here on, the Mercurial code via commit() uses a stronger locking mechanism, which makes it faster and gets the commit time closer to a constant time.
If you look at the first few commits in the speed plots, you’ll notice that at the very beginning Mercurial seems to take only half the time of git. Let’s investigate:
Only the first 300 commits of Mercurial and git:
Finally a comparision of Mercurial called via mercurial.commands.commit(ui, repo, message=str(msg), addremove=True, quiet=True, debug=False) and via mercurial.dispatch.dispatch(["commit", "-q", "-A", "-m", message]):
![Mercurial commit() vs. Mercurial dispatch(['commit', …])](hg-com-vs-hg-dis.png)
As you can see, you should really use mercurial.commands.commit() with explicit locking instead of dispatch(), because commit() has a clear performance advantage.
Images with ¹ are run in a second run, with the data from the first.
If you need the fastest system and don’t mind occassional performance glitches and volatile repository sizes, git is the way to go. Note that its speed drops to half the speed of Mercurial (and less for big repositories, since the time for the garbage collection rises linearly) if Git is forced to avoid frequently growing and shrinking repositories by running the garbage collection every ten commits. Also note that git gc --auto allowed the repository to grow to more than 5 times the minimum size and that garbage collection created 100% load for 3 of my 4 cores which would slow down all other processes on a server, too (otherwise gc would likely have been slower by factor 3).
If you need reliable performance and space requirements, Mercurial is the better choice, especially when called directly via its API. Also for small repositories with up to about 200 commits, it is faster than git even without garbage collection.
Someone in the git mailing list complained, that the test was rigged to show that Mercurial is better.
The reality is, that the test is meant as a complement to the Bachlor Thesis knittl2010 linked in the intro. That thesis left out this usecase (wasn’t in the problem-space defined for the work), so I decided to test it myself. It struck me as the case where performance in the sub-second range actually counts.
Sidenote: Out-of-band gc only works if the load is very different from constant which at least for the german-only page whose statistics I linked in the intro is not true. For a multilingual page it will likely be even closer to constant load.
Another comment was very constructive and simply true: Yes, in branchy history, Mercurial would have been a bit worse (mostly the size would have been bigger). And yes, not requiring garbage collection is a big advantage - for which Mercurial pays the price of a bit bigger repository sizes.
PS: Also I now learned, that git might soon get pack format packv4, so I guess that git changes the repository format from time to time just like everyone else. I had been told a different story before, but that’s that. Good to hear that it evolves. Even though free software projects compete against each other for users, there’s the much more important competition against unfree software. In that we stand side-by-side.
hg vs. git for server applications - performance test-- 2011-12-17 00:01:04 --
Created with pyMarkdown Minisite
using the layout from the pybrary.