contrib/varnish: add a cache-warming crawlerHEADmaster

Preloads ESI fragments so the expensive half of every commit page is
already cached before anyone asks for it. Because fragments are shared by
every repository holding the same objects, walking one fork preloads the
diffs for all of them.

Three modes. "crawl" follows links from a starting URL the way a scraper
would, queueing commit and diff pages ahead of everything else. "commits"
walks one repository's log pages; it is denser, but cgit paginates the log
by walking the revision list from the start each time, so it slows down as
the offset grows. "oids" reads object ids from stdin and has no pagination
to walk, which makes it the one to use for a bulk preload: fed from
git rev-list it measured 249 req/s against 68 for the log walk.

The reported hit rate counts only the pages the crawler asks for. ESI
fragments are fetched by Varnish itself and a client cannot see whether
they hit, so a healthy run shows a low client-side hit rate together with
roughly half the expected backend requests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

0 files changed, 0 insertions, 0 deletions