| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
The figures in the README came from a single render of linux.git's tip,
which is an unusually large merge, and from a box that still had load on
it from the previous benchmark. Re-measured on an idle machine over 60
random commits: 12.6 ms to 2.0 ms rather than 14.83 to 3.13.
Also note that the esi-diff fragment does not change, which is the point
worth knowing: fragments carry no subject line and so never ask for
decorations at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
01 stopped add_ref_decoration() inflating every tag object, but left two
odb_read_object_info() calls per ref behind: one to classify the ref
target so lookup_object_by_type() can be called, and one to classify
what the ref peels to. Each is a binary search through the pack index.
On linux.git that index is 389 MB across 11.8 million objects, and a
commit page walks 946 refs, so it paid about 1900 of those searches per
request. Profiling put add_ref_decoration() at 78% of the whole frame
render, with odb_read_object_info() 76% and find_pack_entry() 52%.
Neither lookup is necessary. packed-refs only records a peeled line for
refs that point at a tag object, so peeled_oid being present already
says the type is OBJ_TAG. And the peeled object is only ever used as a
key for the decoration; nothing reads its type, so lookup_unknown_object()
gives the same struct a later lookup_commit() would fill in.
Frame render with cgit's cache off, on an i7-7700:
linux.git 14.83 ms -> 3.13 ms
git.git 6.49 ms -> 2.23 ms
A control run serving the same objects through a repository with one ref
instead of 946 costs 1.7 ms, so this closes most of the remaining gap.
Verified byte-identical: git log --decorate over 20000 linux.git commits
and the whole of git.git, for-each-ref and tag -l with peeled fields,
and 6000 cgit commit and diff pages. git's t4202-log, t3200-branch,
t6300-for-each-ref and t7004-tag pass, 976 tests between them, and
cgit's own suite passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
Preloads ESI fragments so the expensive half of every commit page is
already cached before anyone asks for it. Because fragments are shared by
every repository holding the same objects, walking one fork preloads the
diffs for all of them.
Three modes. "crawl" follows links from a starting URL the way a scraper
would, queueing commit and diff pages ahead of everything else. "commits"
walks one repository's log pages; it is denser, but cgit paginates the log
by walking the revision list from the start each time, so it slows down as
the offset grows. "oids" reads object ids from stdin and has no pagination
to walk, which makes it the one to use for a bulk preload: fed from
git rev-list it measured 249 req/s against 68 for the log walk.
The reported hit rate counts only the pages the crawler asks for. ESI
fragments are fetched by Varnish itself and a client cannot see whether
they hit, so a healthy run shows a low client-side hit rate together with
roughly half the expected backend requests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
02-cgit-output-buffering.patch was cut from a working tree that already
carried the ESI work, so it also contained the enable-esi config hook,
cache_key(), the repolink and cgit_fileurl fragment hunks and the
memrchr fix. That contradicted its README and made it fail to apply on
any tree that already had ESI, which is every tree in this repository.
Regenerate it with only the output buffering change: the html.c buffer
itself and the flushes in cache.c, filter.c, cgit.c and ui-shared.c.
Verified to apply cleanly both to stock cgit and on top of the ESI
commits here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
Two independent performance patches found while measuring the ESI split,
both of which help whether or not a cache sits in front of cgit.
01-git-peel-decoration.patch applies to the git submodule. git's
add_ref_decoration() calls parse_object() on every annotated tag to
follow it to its commit, inflating all of them, even though the ref
backend already hands the callback a peeled object id when packed-refs
has one. Using that takes commit and log pages on git.git from about
6 ms to about 2 ms. Output is byte-identical and git's own t4202-log,
t3200-branch and t6300-for-each-ref pass.
02-cgit-output-buffering.patch applies to cgit. html_raw() did one
write(2) per call: a 1.7 MB diff took 99151 writes averaging 18 bytes.
A 64 KB buffer takes that to 27. The buffer has to be flushed wherever
fd 1 is redirected or handed to another writer, which is the cache slot,
exec filters, and the end of the HTTP headers, because git's archive
code writes snapshots and clone data straight to fd 1.
Kept as patches rather than applied because the first belongs upstream
in git, and the second touches enough of cgit's output path to be worth
landing on its own terms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
Turns on ESI processing for frames carrying Surrogate-Control, hashes
esi-diff fragments on their query string so that every fork sharing a
commit shares one cached rendering, and refuses fragment requests that
do not come from the ESI processor so they do not become one more thing
for crawlers to enumerate.
Frames get a short TTL on purpose: they are cheap to render and there
are a lot of them, and they should not crowd fragments out of the cache.
cgit-esi.vtc exercises the VCL against a fake cgit backend, asking for
the same commit through two repository names and checking that the
backend saw exactly one fragment request.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
One of the most frequent questions on the mailing list relates to the
idle time in the repository list. The answer to this is to use the
"agefile" feature to calculate the time of the last change whenever the
repository receives changes.
Add a sample post-receive hook in a new "contrib" directory so that we
can just point people at the repository in the future.
Signed-off-by: John Keeping <john@keeping.me.uk>
|
