In Part 1 of this two-part series, we looked at how much server-side Git fetch performance, especially for CI, has improved in GitLab in 2021. Now, we will discuss how we achieved this.
Recap of Part 1
- In December 2019, we set up custom CI fetch caching automation for
gitlab-org/gitlab, which we internally called "the CI pre-clone
script". - In December 2020, we encountered some production incidents on GitLab.com,
which highlighted that the CI pre-clone script had become critical
infrastructure but, at the same time, it had not yet matured beyond
a custom one-off solution. - Over the course of 2021, we built an alternative caching solution
for CI Git fetch traffic called the pack-objects cache. In Part 1,
we discussed a benchmark simulating CI fetch traffic which shows
that the pack-objects cache combined with other efficiency
improvements reduced GitLab server CPU consumption 9x compared to
the baseline of December 2020.
The pack-objects cache
As discussed in Part 1, what we realized through the
production incidents in December 2020 was that the CI pre-clone script
for gitlab-org/gitlab had become a critical piece of infrastructure.
At the same time, it benefited only one Git repository on GitLab.com,
and it was not very robust. It would be much better to have an
integrated solution that benefits all repositories. We achieved this
goal by building the pack-objects cache.
The name "pack-objects cache" refers to git pack-objects, which is
the Git subcommand that
implements the packfile compression algorithm. As this Git commit message from Jeff King explains, git pack-objects is a good candidate for a CI fetch cache.
You may want to insert a caching layer around
pack-objects; it is the most CPU- and memory-intensive
part of serving a fetch, and its output is a pure
function of its input, making it an ideal place to
consolidate identical requests.
The pack-objects cache is GitLab's take on this "caching layer". It
deduplicates identical Git fetch requests that arrive within a short
time window.
At a high level, when serving a fetch, we buffer the output of git pack-objects into a temporary file. If an identical request comes in,
we serve it from the buffer file instead of creating a new git pack-objects process. After 5 minutes, we delete the buffer file. If
you want to know more about how exactly the cache is implemented, you
can look at the implementation
(1,
2).
Because the amount of space used by the cache files is bounded roughly
by the eviction window (5 minutes) multiplied by the maximum network bandwidth
of the Gitaly server, we don't have to worry about the cache using a
lot of storage. In fact, on GitLab.com, we store the cache files on the
same disks that hold the repository data. We leave a safety margin of
free space on these disks at all times anyway, and the cache fits in
that safety margin comfortably.
Similarly, we also don't notice the increase disk input/output
operations per second (IOPS) used by the cache on GitLab.com. There
are two reasons for this. First of all, whenever we read data from
the cache, it is usually still in the Linux page cache, so it gets
served from RAM. The cache barely does any disk read I/O operations.
Second, although the cache does do write operations, these fit
comfortably within the maximum sustained IOPS rate supported by the
Google Compute Engine persistent disks we use.
This leads us to a disadvantage of the pack-objects cache, which is
that it really does write a lot of data to disk. On GitLab.com, we saw
the disk write throughput jump up by an order of magnitude. You can
see this in the graph below, which shows disk writes for a single
Gitaly server with a busy, large repository on it: (the GitLab company
website). You can
clearly see the number of bytes written to disk per second jump up when we
turned the cache on.
This increase in disk writes is not a problem for our infrastructure because we have the
spare capacity, but we were not sure we could assume the same for all
other GitLab installations in the world. Because of this, we decided
to leave the pack-objects cache off by default.
This was a difficult decision because we think almost all GitLab
installations would benefit from having this cache enabled. One of the
reasons we are writing this blog post is to raise awareness that this
feature is available, so that self-managed GitLab administrators can
opt in to using it.
Again, on the positive side, the cache did not introduce a new
point of failure on GitLab.com. If the gitaly service is running,
and if the repository storage disk is available, then the cache is
available. There are no external dependencies. And if gitaly is not
running, or the repository storage disk is unavailable, then the whole
Gitaly server is unavailable anyway.
And finally, cache capacity grows naturally with the number of Gitaly
servers. Because the cache is completely local to each Gitaly server,
we do not have to worry about whether the cache keeps working as we
grow GitLab.com.
The pack-objects cache was introduced in GitLab 13.11. In GitLab 14.5,
we made it a lot more efficient by optimizing its transport using Unix
sockets
(1,
2). If
you want to try out the pack-objects cache on
your self-managed GitLab instance, we recommend that you upgrade to
GitLab 14.5 or newer first.
Improved RPC transport for Git HTTP
After we built the pack-objects cache, we were able to generate a much
higher volume of Git fetch responses on a single Gitaly server.
However, we then found out that the RPC transport between the HTTP
front-end (GitLab Workhorse) and the Gitaly server became a
bottleneck. We tried disabling the CI pre-clone script of
gitlab-org/gitlab in April 2021 but we quickly had to turn it back
on because the increased volume of Git fetch data transfer was slowing
down the rest of Gitaly.
The fetch traffic was acting as a noisy neighbor to all the other
traffic on gitlab-org/gitlab. For each GitLab.com Gitaly server, we
have a request latency
SLI. This is
a metric that observes request latencies for a selection of RPCs that
we expect to be fast, and it tracks how many requests for these RPCs
are "fast enough". If the percentage of fast-enough requests drops
below a certain threshold, we know we have a problem.
When we disabled the pre-clone script, the network traffic to the
Gitaly server hosting gitlab-org/gitlab went up, as expected. What
went wrong was that the percentage of fast-enough requests started to
drop. This was not because the server had to serve up more data: The
RPCs that serve the Git fetch data do not count towards the latency
SLI.
Below you see two graphs from the day we tried disabling the CI
pre-clone script. First, see how the network traffic off of the Gitaly
server increased once we disabled the CI pre-clone script. This is
because instead of pulling most of the data from object storage, and
only some of the data from Gitaly, the CI runners now started pulling
all of the Git data they needed from Gitaly.
Now consider our Gitaly request latency SLI for this particular
server. For historical reasons, we call this "Apdex" in our dashboards.
Recall that this SLI tracks the percentage of fast-enough requests from
a selection of Gitaly RPCs. The ideal number would be 100%. In the
time window where the CI pre-clone script was disabled, this graph
spent more time below 99%, and it even dipped below 96% several times.
Even though we could not explain what was going on, the latency SLI dips
were clear evidence that disabling the CI pre-clone script slowed down
unrelated requests to this Gitaly server, to a point which is
unacceptable. This was a setback for our plan to replace the CI pre-clone script.
Because we did not want to just give up, we set aside some time to try
and understand what the bottleneck was, and if it could be
circumvented. The bad news is that we did not come up with a
satisfactory answer about what the bottleneck is. But the good news is
that we were able to circumvent it.
By building a simplified prototype alternate RPC
transport,
we were able to find out that with the pack-objects cache, the
hardware we run on and Git itself were able to serve up much more
traffic than we were able to get out of GitLab. We never got to the
bottom
of what was causing all the overhead but a likely suspect is the fact
that gRPC-Go allocates memory for each message it sends, and with Git
fetch traffic we send a lot of messages. Gitaly was spending a lot of
time doing garbage collection.
We then had to decide how to improve the situation. Because we were
uncertain if we could fix the apparent bottleneck in gRPC, and because
we were certain that we could go faster by not sending the Git fetch data
through gRPC in the first place, we chose to do the latter. We created
modified versions of the RPCs that carry the bulk of the Git fetch
data. On the surface, the new versions are still gRPC methods. But
during a call, each will establish a side channel, and use that for
the bulk data transfer.
This way we avoided making major changes to the structure of Gitaly:
it is still a gRPC server application. Logging, metrics,
authentication, and other middleware work as normal on the optimized
RPCs. But most of the data transfer happens on either Unix sockets (for localhost RPC calls) or Yamux streams (for the regular RPC calls).
Because we have 6x more Git HTTP traffic than Git SSH traffic on
GitLab.com, we decided to initially only optimize the transport for
Git HTTP traffic. We are still working on doing the same for Git
SSH because, even though Git HTTP efficiency is more important for
GitLab.com than that of Git SSH, we know that for some self-managed
GitLab instances it is the other way around.
The new server-side RPC transport for Git HTTP was released in GitLab
14.5. There is no configuration required for this improved transport.
Regardless of whether you use the pack-objects cache on your GitLab
instance, Gitaly, Workhorse, and Praefect all use less CPU to handle
Git HTTP fetch requests now.
The payoff for this work came in October 2021 when we disabled the CI
pre-clone script for gitlab-org/gitlab, which did not cause any
noisy neighbor problems this time. We have had no issues since then
serving the Git fetch traffic for that project.
Improvements to Git itself
Aside from the pack-objects cache and the new RPC transport between
Workhorse and Gitaly, we also saw some improvements because of changes
in Git itself. We discovered a few inefficiencies which we
reported to the Git mailing list and helped get fixed.
Our main repository gitlab-org/gitlab has hundreds of thousands of Git
references. Looking at CPU profiles, we noticed that a lot of Git
fetch time was spent on the server iterating over these references.
These references were not even being sent back to the client; Git was
just scanning through all of them on the server twice for each CI job.
In both cases, the problem could be fixed by doing a scan over a
subset instead of a scan across all references. These two problems got fixed
(1, 2) in Git 2.31.0, released in March 2021.
Later on, we found a different problem, also in the reference-related
workload of Git fetch. As part of the fetch protocol, the server sends
a list of references to the client so that the client can update its
local branches etc. It turned out that for each reference, Git was
doing 1 or 2 write system calls on the server. This led to a lot of
overhead, and this was made worse by our old RPC transport which could
end up sending 1 RPC message per advertised Git reference.
This problem got fixed in Git itself by changing the functions that
write the references to use buffered
IO.
This change landed in Git 2.34.0, released in November 2021. Ahead of
that, it got shipped in GitLab 14.4 as a custom Git patch.
Finally, we discovered that increasing the copy buffer size used by
git upload-pack to relay git pack-objects output made both git upload-pack and every link in the chain after
it more
efficient. This got fixed in Git by increasing the buffer
size.
This change is part of Git 2.35.0 and is included in GitLab 14.7, both
of which were released in January 2022.
Summary
In Part 1, we showed that GitLab server performance when service CI Git fetch traffic has improved a lot in 2021. In this post, we explained that the improvements are due to:
- The pack-objects cache
- A more efficient Git data transport between server-side GitLab components
- Efficiency improvements in Git itself
Thanks
Many people have contributed to the work described in this blog post.
I would like to specifically thank Quang-Minh Nguyen and Sean McGivern
from the Scalability team, and Patrick Steinhardt and Sami Hiltunen
from the Gitaly team.
Related content
- Improvements to the client-side performance of
git fetch(although GitLab is a server application, it sometimes acts as a Git client): mirror fetches, fetches into repositories with many references - Improvements to server-side Git push performance: consistency check improvements
