summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2019-11-18built-in add -i: show unique prefixes of the commandsJohannes Schindelin1-11/+177
Just like in the Perl script `git-add--interactive.perl`, for each command a unique prefix is determined (if there exists any within the given parameters), and shown in the list, and accepted as a shortcut for the command. To determine the unique prefixes, as well as to look up the command in question, we use a copy of the list and sort it. While this might seem like overkill for a single command, it will make much more sense when all the commands are implemented, and when we reuse the same logic to present a list of files to edit, with convenient unique prefixes. At the start of the development of this patch series, a dedicated data structure was introduced that imitated the Trie that the Perl version implements. However, this was deemed overkill, and we now simply sort the list before determining the length of the unique prefixes by looking at each item's neighbor. As a bonus, we now use the same sorted list to perform a binary search using the user-provided prefix as search key. Original-patch-by: Slavica Đukić <slawica92@hotmail.com> Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-18built-in add -i: implement the main loopJohannes Schindelin1-2/+135
The reason why we did not start with the main loop to begin with is that it is the first user of `list_and_choose()`, which uses the `list()` function that we conveniently introduced for use by the `status` command. In contrast to the Perl version, in the built-in interactive `add`, we will keep the `list()` function (which only displays items) and the `list_and_choose()` function (which uses `list()` to display the items, and only takes care of the "and choose" part) separate. The `list_and_choose()` function, as implemented in `git-add--interactive.perl` knows a few more tricks than the function we introduce in this patch: - There is a flag to let the user select multiple items. - In multi-select mode, the list of items is prefixed with a marker indicating what items have been selected. - Initially, for each item a unique prefix is determined (if there exists any within the given parameters), and shown in the list, and accepted as a shortcut for the selection. These features will be implemented in the C version later. This patch does not add any new main loop command, of course, the built-in `git add -i` still only supports the `status` command. The remaining commands to follow over the course of the next commits. To accommodate for listing the commands in columns, preparing for the commands that will be implemented over the course of the next patches/patch series, we teach the `list()` function to do precisely that. Note that we only have a prompt ending in a single ">" at this stage; later commits will add commands that display a double ">>" to indicate that the user is in a different loop than the main one. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-18gitweb: escape URLs generated by href()Jeff King2-15/+19
There's a cross-site scripting problem in gitweb, where it will print URLs generated by its href() helper without further quoting. This allows an attacker to point a victim to a specially crafted gitweb URL and inject arbitrary HTML into the resulting page (which the victim sees as coming from gitweb). The base of the URL comes from evaluate_uri(), which pulls the value of $REQUEST_URI via the CGI module. It tries to strip off $PATH_INFO, but fails to do so in some cases (including ones that contain special characters, like "+"). Most of the uses of the URL end up being passed to "$cgi->a(-href = href())", which will get quoted properly by the CGI module. But in a few places, we output them ourselves as part of manually-generated HTML, and whatever was in the original URL will appear unquoted in the output. Given that all of the nearby variables placed into this manual HTML _are_ quoted, it seems like the authors assumed that these URLs would not need quoting. So it's possible that the bug is actually in evaluate_uri(), which should be doing a more careful job of stripping $PATH_INFO. There's some discussion in a comment in that function, as well as the commit message in 81d3fe9f48 (gitweb: fix wrong base URL when non-root DirectoryIndex, 2009-02-15). But I'm not sure I understand it. Regardless, it's a good idea to quote these values at the point of insertion into the HTML output: 1. Even if there is a bug in evaluate_uri(), this would give us belt-and-suspenders protection. 2. evaluate_uri() is only handling the base. Some generated URLs will also mention arbitrary refs or filenames in the repositories, and these should be quoted anyway. 3. It should never _hurt_ to quote (and that's what all of the $cgi->a() calls are doing already). So there may be further work here, but this patch at least prevents the XSS vulnerability, and shouldn't make anything worse. The test here covers the calls in print_feed_meta(), but I manually audited every call to href() to see how its output was used, and quoted appropriately. Most of them are esc_attr(), as they're used in tag attributes, but I used esc_html() when the URLs were printed bare. The distinction is largely academic, as one is implemented as a wrapper for the other. Reported-by: NAKAYAMA DAISUKE <nakyamad@icloud.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-18t/gitweb-lib.sh: set $REQUEST_URIJeff King1-1/+2
In a real webserver's CGI call, gitweb.cgi would typically see $REQUEST_URI set. This variable does impact how we display our URL in the resulting page, so let's try to make our test as realistic as possible (we can just use the $PATH_INFO our caller passed in, if any). This doesn't change the outcome of any tests, but it will help us add some new tests in a future patch. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-18t/gitweb-lib.sh: drop confusing quotesJeff King1-2/+2
Some variables assignments in gitweb_run() look like this: FOO=""$1"" The extra quotes aren't doing anything. Each set opens and closes an empty string, and $1 is actually outside of any double-quotes (which is OK, because variable assignment does not do whitespace splitting on the expanded value). Let's drop them, as they're simply confusing. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-18t9502: pass along all arguments in xss helperJeff King1-2/+2
This function is just a thin wrapper around gitweb_run(), which takes multiple arguments. But we only pass along "$1". Let's pass everything we get, which will let a future patch add an XSS test that affects PATH_INFO (which gitweb_run() takes as $2). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-15INSTALL: use existing shell scripts as exampleAndrei Rybak1-2/+2
Script git-pull.sh has been removed in commit [1]. Use command "request-pull" as an example of a shell script instead. Recently, many of shell script commands have been re-written in C, so tweak the wording of the sentence, while we're here. [1]: b1456605c2 (pull: remove redirection to git-pull.sh, 2015-06-18) Signed-off-by: Andrei Rybak <rybak.a.v@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-14t7004: check existence of correct tagMartin Ågren1-3/+3
We try to delete the non-existing tag "anothertag", but for the verifications, we check that the tag "myhead" doesn't exist. "myhead" isn't used in this test except for this checking. Comparing to the test two tests earlier, it looks like a copy-paste mistake. Perhaps it's overkill to check that `git tag -d` didn't decide to *create* a tag. But since we're trying to be this careful, let's actually check the correct tag. While we're doing this, let's use a more descriptive tag name instead -- "nonexistingtag" should be obvious. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-14built-in add -i: color the header in the `status` commandSlavica Đukić1-4/+37
For simplicity, we only implemented the `status` command without colors. This patch starts adding color, matching what the Perl script `git-add--interactive.perl` does. Original-Patch-By: Daniel Ferreira <bnmvco@gmail.com> Signed-off-by: Slavica Đukić <slawica92@hotmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-14built-in add -i: implement the `status` commandDaniel Ferreira1-1/+250
This implements the `status` command of `git add -i`. The data structures introduced in this commit will be extended later, as needed. At this point, we re-implement only part of the `list_and_choose()` function of the Perl script `git-add--interactive.perl` and call it `list()`. It does not yet color anything, or do columns, or allow user input. Over the course of the next commits, we will introduce a `list_and_choose()` function that uses `list()` to display the list of options and let the user choose one or more of the displayed items. This will be used to implement the main loop of the built-in `git add -i`, at which point the new `status` command can actually be used. Signed-off-by: Daniel Ferreira <bnmvco@gmail.com> Signed-off-by: Slavica Đukić <slawica92@hotmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-14diff: export diffstat interfaceDaniel Ferreira2-23/+36
Make the diffstat interface (namely, the diffstat_t struct and compute_diffstat) no longer be internal to diff.c and allow it to be used by other parts of git. This is helpful for code that may want to easily extract information from files using the diff machinery, while flushing it differently from how the show_* functions used by diff_flush() do it. One example is the builtin implementation of git-add--interactive's status. Signed-off-by: Daniel Ferreira <bnmvco@gmail.com> Signed-off-by: Slavica Đukić <slawica92@hotmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-14Start to implement a built-in version of `git add --interactive`Johannes Schindelin6-0/+37
Unlike previous conversions to C, where we started with a built-in helper, we start this conversion by adding an interception in the `run_add_interactive()` function when the new opt-in `add.interactive.useBuiltin` config knob is turned on (or the corresponding environment variable `GIT_TEST_ADD_I_USE_BUILTIN`), and calling the new internal API function `run_add_i()` that is implemented directly in libgit.a. At this point, the built-in version of `git add -i` only states that it cannot do anything yet. In subsequent patches/patch series, the `run_add_i()` function will gain more and more functionality, until it is feature complete. The whole arc of the conversion can be found in the PRs #170-175 at https://github.com/gitgitgadget/git. The "--helper approach" can unfortunately not be used here: on Windows we face the very specific problem that a `system()` call in Perl seems to close `stdin` in the parent process when the spawned process consumes even one character from `stdin`. Which prevents us from implementing the main loop in C and still trying to hand off to the Perl script. The very real downside of the approach we have to take here is that the test suite won't pass with `GIT_TEST_ADD_I_USE_BUILTIN=true` until the conversion is complete (the `--helper` approach would have let it pass, even at each of the incremental conversion steps). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-14stash: make sure we have a valid index before writing itThomas Gummerer2-5/+9
In 'do_apply_stash()' we refresh the index in the end. Since 34933d0eff ("stash: make sure to write refreshed cache", 2019-09-11), we also write that refreshed index when --quiet is given to 'git stash apply'. However if '--index' is not given to 'git stash apply', we also discard the index in the else clause just before. We need to do so because we use an external 'git update-index --add --stdin', which leads to an out of date in-core index. Later we call 'refresh_and_write_cache', which now leads to writing the discarded index, which means we essentially write an empty index file. This is obviously not correct, or the behaviour the user wanted. We should not modify the users index without being asked to do so. Make sure to re-read the index after discarding the current in-core index, to avoid dealing with outdated information. Instead we could also drop the 'discard_cache()' + 'read_cache()', however that would make it easy to fall into the same trap as 34933d0eff did, so it's better to avoid that. We can also drop the 'refresh_and_write_cache' completely in the quiet case. Previously in legacy stash we relied on 'git status' to refresh the index after calling 'git read-tree' when '--index' was passed to 'git apply'. However the 'reset_tree()' call that replaced 'git read-tree' always passes options that are equivalent to '-m', making the refresh of the index unnecessary. Reported-by: Grzegorz Rajchman <rayman17@gmail.com> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13send-pack: check remote ref status on pack-objects failureJeff King1-4/+4
When we're pushing a pack and our local pack-objects fails, we enter an error code path that does a few things: 1. Set the status of every ref to REF_STATUS_NONE 2. Call receive_unpack_status() to try to get an error report from the other side 3. Return an error to the caller If pack-objects failed because the connection to the server dropped, there's not much more we can do than report the hangup. And indeed, step 2 will try to read a packet from the other side, which will die() in the packet-reading code with "the remote end hung up unexpectedly". But if the connection _didn't_ die, then the most common issue is that the remote index-pack or unpack-objects complained about our pack (we could also have a local pack-objects error, but this ends up being the same; we'd send an incomplete pack and the remote side would complain). In that case we do report the error from the other side (because of step 2), but we fail to say anything further about the refs. The issue is two-fold: - in step 1, the "NONE" status is not "we saw an error, so we have no status". It generally means "this ref did not match our refspecs, so we didn't try to push it". So when we print out the push status table, we won't mention any refs at all! But even if we had a status enum for "pack-objects error", we wouldn't want to blindly set it for every ref. For example, in a non-atomic push we might have rejected some refs already on the client side (e.g., REF_STATUS_REJECT_NODELETE) and we'd want to report that. - in step 2, we read just the unpack status. But receive-pack will also tell us about each ref (usually that it rejected them because of the unpacker error). So a much better strategy is to leave the ref status fields as they are (usually EXPECTING_REPORT) and then actually receive (and print) the full per-ref status. This case is actually covered in the test suite, as t5504.8, which writes a pack that will be rejected by the remote unpack-objects. But it's racy. Because our pack is small, most of the time pack-objects manages to write the whole thing before the remote rejects it, and so it returns success and we print out the errors from the remote. But very occasionally (or when run under --stress) it goes slow enough to see a failure in writing, and git-push reports nothing for the refs. With this patch, the test should behave consistently. There shouldn't be any downside to this approach. If we really did see the connection drop, we'd already die in receive_unpack_status(), and we'll continue to do so. If the connection drops _after_ we get the unpack status but before we see any ref status, we'll still print the remote unpacker error, but will now say "remote end hung up" instead of returning the error up the call-stack. But as discussed, we weren't showing anything more useful than that with the current code. And anyway, that case is quite unlikely (the connection dropping at that point would have to be unrelated to the pack-objects error, because of the ordering of events). In the future we might want to handle packet-read errors ourself instead of dying, which would print a full ref status table even for hangups. But in the meantime, this patch should be a strict improvement. Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13t6120-describe: correct test repo history graph in commentSZEDER Gábor1-9/+10
At the top of 't6120-describe.sh' an ASCII graph illustrates the repository's history used in this test script. This graph is a bit misleading, because it swapped the second merge commit's first and second parents. When describing/naming a commit it does make a difference which parent is the first and which is the second/Nth, so update this graph to accurately represent that second merge. While at it, move this history graph from the 'test_description' variable to a regular comment. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13completion: tab-complete "git svn --recursive"James Shubin1-0/+1
Signed-off-by: James Shubin <james@shubin.ca> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13promisor-remote: remove fetch_if_missing=0Jonathan Tan2-17/+32
Commit 6462d5eb9a ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from the lazy-fetching mechanism in promisor-remote as well. But doing so reveals a bug - when the server does not send an object pointed to by a tag object, an infinite loop occurs: Git attempts to fetch the missing object, which causes a deferencing of all refs (for negotiation), which causes a lazy fetch of that missing object, and so on. This bug is because of unnecessary use of the fetch negotiator during lazy fetching - it is not used after initialization, but it is still initialized (which causes the dereferencing of all refs). Thus, when the negotiator is not used during fetching, refrain from initializing it. Then, remove fetch_if_missing from promisor-remote. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13clone: remove fetch_if_missing=0Jonathan Tan2-4/+2
Commit 6462d5eb9a ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from clone as well. But doing so reveals a bug - when the server does not send an object directly pointed to by a ref, this should be an error, not a trigger for a lazy fetch. (This case in the fetching mechanism was covered by a test using "git clone", not "git fetch", which is why the aforementioned commit didn't uncover the bug.) The bug can be fixed by suppressing lazy-fetching during the connectivity check. Fix this bug, and remove fetch_if_missing from clone. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13parse-options: avoid arithmetic on pointer that's potentially NULLRené Scharfe1-1/+2
parse_options_dup() counts the number of elements in the given array without the end marker, allocates enough memory to hold all of them plus an end marker, then copies them and terminates the new array. The counting part is done by advancing a pointer through the array, and the original pointer is reconstructed using pointer subtraction before the copy operation. The function is also prepared to handle a NULL pointer passed to it. None of its callers do that currently, but this feature was used by 46e91b663b ("checkout: split part of it to new command 'restore'", 2019-04-25); it seems worth keeping. It ends up doing arithmetic on that NULL pointer, though, which is undefined in standard C, when it tries to calculate "NULL - 0". Better avoid doing that by remembering the originally given pointer value. There is another issue, though. memcpy(3) does not support NULL pointers, even for empty arrays. Use COPY_ARRAY instead, which does support such empty arrays. Its call is also shorter and safer by inferring the element type automatically. Coccinelle and contrib/coccinelle/array.cocci did not propose to use COPY_ARRAY because of the pointer subtraction and because the source is const -- the semantic patch cautiously only considers pointers and array references of the same type. Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13mingw: use COPY_ARRAY for copying arrayRené Scharfe1-1/+1
Use the macro COPY_ARRAY to copy array elements. The result is shorter and safer, as it infers the element type automatically and does a (very) basic type compatibility check for its first two arguments. Coccinelle and contrib/coccinelle/array.cocci did not generate this conversion due to the offset of 1 at both source and destination and because the source is a const pointer; the semantic patch cautiously handles only pure pointers and array references of the same type. Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13test-tool: use 'read-graph' helperDerrick Stolee8-82/+58
The 'git commit-graph read' subcommand is used in test scripts to check that the commit-graph contents match the expected data. Mostly, this helps check the header information and the list of chunks. Users do not need this information, so move the functionality to a test helper. Reported-by: Bryan Turner <bturner@atlassian.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13test-lib: don't check prereqs of test cases that won't be run anywaySZEDER Gábor1-6/+6
With './t1234-foo.sh -r 5,6' we can run only specific test cases in a test script, but our test framwork still evaluates all lazy prereqs that the excluded test cases might depend on. This is unnecessary and produces verbose and trace output that can be distracting. This has been an issue ever since the '-r|--run=' options were introduced in 0445e6f0a1 (test-lib: '--run' to run only specific tests, 2014-04-30), because that commit added the check of the list of test cases specified with '-r' after evaluating the prereqs. Avoid this unnecessary prereq evaluation by checking the list of test cases specified with '-r' before looking at the prereqs. Note that GIT_SKIP_TESTS has always been checked before the prereqs, so prereqs necessary for tests skipped that way were not evaluated. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13t4215: use helper function to check outputDenton Liu1-111/+97
When git commands are placed in the upstream of a pipe, their return codes are lost. In this particular case, it is especially bad since we are testing the intricacies of `git log --graph` behavior and if we hit an unexpected failure or segfault, we want to know this. Extract the common output checking logic into check_graph() where we redirect the output of git commands upstream of pipe into a file and have sed read from that file so that git failures are detected. This patch is best viewed with `--color-moved`. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13fsmonitor: do not compare bitmap size with size of split indexJunio C Hamano1-2/+4
3444ec2e ("fsmonitor: don't fill bitmap with entries to be removed", 2019-10-11) added a handful of sanity checks that make sure that a bit position in fsmonitor bitmap does not go beyond the end of the index. As each bit in the bitmap corresponds to a path in the index, this is the right check most of the time. Except for the case when we are in the split-index mode and looking at a delta index that is to be overlayed on the base index but before the base index has actually been merged in, namely in read_ and write_fsmonitor_extension(). In these codepaths, the entries in the split/delta index is typically a small subset of the entire set of paths (otherwise why would we be using split-index?), so the bitmap used by the fsmonitor is almost always larger than the number of entries in the partial index, and the incorrect comparison would trigger the BUG(). Reported-by: Utsav Shah <ukshah2@illinois.edu> Helped-by: Kevin Willford <Kevin.Willford@microsoft.com> Helped-by: William Baker <William.Baker@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-13hex: drop sha1_to_hex()Jeff King4-24/+5
There's only a single caller left of sha1_to_hex(), since everybody that has an object name in "unsigned char[]" now uses hash_to_hex() instead. This case is in the sha1dc wrapper, where we print a hex sha1 when we find a collision. This one will always be sha1, regardless of the current hash algorithm, so we can't use hash_to_hex() here. In practice we'd probably not be running sha1 at all if it isn't the current algorithm, but it's possible we might still occasionally need to compute a sha1 in a post-sha256 world. Since sha1_to_hex() is just a wrapper for hash_to_hex_algop(), let's call that ourselves. There's value in getting rid of the sha1-specific wrapper to de-clutter the global namespace, and to make sure nobody uses it (and as with sha1_to_hex_r() in the previous patch, we'll drop the coccinelle transformations, too). The sha1_to_hex() function is mentioned in a comment; we can easily swap that out for oid_to_hex() to give a better example. Also update the comment that was left stale when we added "struct object_id *" as a way to name an object and added functions to convert it to hex. The function is also mentioned in some test vectors in t4100, but that's not runnable code, so there's no point in trying to clean it up. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-12completion: learn to complete `git rebase --onto=`Denton Liu1-0/+4
In 2b9bd488ae ("completion: teach rebase to use __gitcomp_builtin", 2019-09-12), the completion script learned to complete rebase using __gitcomp_builtin(). However, this resulted in `--onto=` being suggested instead of `--onto `. Before, when there was a space, we'd start a new word and, as a result, fallback to __git_complete_refs() and `--onto` would be completed this way. However, now we match the `--*` case which does not know how to offer completions for refs. Teach _git_rebase() to complete refs in the `--onto=` case so that we fix this regression. Reported-by: Paul Jolly <paul@myitcv.io> Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-12pack-objects: avoid pointless oe_map_new_pack() callsJeff King3-6/+19
This patch fixes an extreme slowdown in pack-objects when you have more than 1023 packs. See below for numbers. Since 43fa44fa3b (pack-objects: move in_pack out of struct object_entry, 2018-04-14), we use a complicated system to save some per-object memory. Each object_entry structs gets a 10-bit field to store the index of the pack it's in. We map those indices into pointers using packing_data->in_pack_by_idx, which we initialize at the start of the program. If we have 2^10 or more packs, then we instead create an array of pack pointers, one per object. This is packing_data->in_pack. So far so good. But there's one other tricky case: if a new pack arrives after we've initialized in_pack_by_idx, it won't have an index yet. We solve that by calling oe_map_new_pack(), which just switches on the fly to the less-optimal in_pack mechanism, allocating the array and back-filling it for already-seen objects. But that logic kicks in even when we've switched to it already (whether because we really did see a new pack, or because we had too many packs in the first place). The result doesn't produce a wrong outcome, but it's very slow. What happens is this: - imagine you have a repo with 500k objects and 2000 packs that you want to repack. - before looking at any objects, we call prepare_in_pack_by_idx(). It starts allocating an index for each pack. On the 1024th pack, it sees there are too many, so it bails, leaving in_pack_by_idx as NULL. - while actually adding objects to the packing list, we call oe_set_in_pack(), which checks whether the pack already has an index. If it's one of the packs after the first 1023, then it doesn't have one, and we'll call oe_map_new_pack(). But there's no useful work for that function to do. We're already using in_pack, so it just uselessly walks over the complete list of objects, trying to backfill in_pack. And we end up doing this for almost 1000 packs (each of which may be triggered by more than one object). And each time it triggers, we may iterate over up to 500k objects. So in the absolute worst case, this is quadratic in the number of objects. The solution is simple: we don't need to bother checking whether the pack has an index if we've already converted to using in_pack, since by definition we're not going to use it. So we can just push the "does the pack have a valid index" check down into that half of the conditional, where we know we're going to use it. The current test in p5303 sadly doesn't notice this problem, since it maxes out at 1000 packs. If we add a new test to it at 2000 packs, it does show the improvement: Test HEAD^ HEAD ---------------------------------------------------------------------- 5303.12: repack (2000) 26.72(39.68+0.67) 15.70(28.70+0.66) -41.2% However, these many-pack test cases are rather expensive to run, so adding larger and larger numbers isn't appealing. Instead, we can show it off more easily by using GIT_TEST_FULL_IN_PACK_ARRAY, which forces us into the absolute worst case: no pack has an index, so we'll trigger oe_map_new_pack() pointlessly for every single object, making it truly quadratic. Here are the numbers (on git.git) with the included change to p5303: Test HEAD^ HEAD ---------------------------------------------------------------------- 5303.3: rev-list (1) 2.05(1.98+0.06) 2.06(1.99+0.06) +0.5% 5303.4: repack (1) 33.45(33.46+0.19) 2.75(2.73+0.22) -91.8% 5303.6: rev-list (50) 2.07(2.01+0.06) 2.06(2.01+0.05) -0.5% 5303.7: repack (50) 34.21(35.18+0.16) 3.49(4.50+0.12) -89.8% 5303.9: rev-list (1000) 2.87(2.78+0.08) 2.88(2.80+0.07) +0.3% 5303.10: repack (1000) 41.26(51.30+0.47) 10.75(20.75+0.44) -73.9% Again, those improvements aren't realistic for the 1-pack case (because in the real world, the full-array solution doesn't kick in), but it's more useful to be testing the more-complicated code path. While we're looking at this issue, we'll tweak one more thing: in oe_map_new_pack(), we call REALLOC_ARRAY(pack->in_pack). But we'd never expect to get here unless we're back-filling it for the first time, in which case it would be NULL. So let's switch that to ALLOC_ARRAY() for clarity, and add a BUG() to document the expectation. Unfortunately this code isn't well-covered in the test suite because it's inherently racy (it only kicks in if somebody else adds a new pack while we're in the middle of repacking). Signed-off-by: Jeff King <peff@peff.net> Reviewed-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-12doc: improve readability of --rebase-merges in git-rebaseNaveen Nathan1-2/+2
When --preserve-merges was deprecated in 427c3bd28a a sentence was introduced describing the difference between --rebase-merges and --preserve-merges which is a little unclear and difficult to parse. This patch improves readability while retaining original meaning. Signed-off-by: Naveen Nathan <naveen@lastninja.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11hex: drop sha1_to_hex_r()Jeff King3-23/+0
There are no callers left; everybody uses oid_to_hex_r() or hash_to_hex_algop_r(). This used to actually be the underlying implementation for oid_to_hex_r(), but that's no longer the case since 47edb64997 (hex: introduce functions to print arbitrary hashes, 2018-11-14). Let's get rid of it to de-clutter and to make sure nobody uses it. Likewise we can drop the coccinelle rules that mention it, since the compiler will make it quite clear that the code does not work. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11sequencer: reencode commit message for am/rebase --show-current-patchDoan Tran Cong Danh3-1/+32
The message file will be used as commit message for the git-{am,rebase} --continue. Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11sequencer: reencode old merge-commit messageDoan Tran Cong Danh3-1/+63
During rebasing, old merge's message (encoded in old encoding) will be used as message for new merge commit (created by rebase). In case of the value of i18n.commitencoding has been changed after the old merge time. We will receive an unusable message for this new merge. Correct it. This change also notice a breakage with git-rebase label system. Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11bundle-verify: add --quietRobin H. Johnson2-4/+7
Add --quiet to git-bundle verify as proposed on the mailing list [1]. Reference: https://www.mail-archive.com/git@vger.kernel.org/msg182844.html <robbat2-20190806T191156-796782357Z@orbis-terrarum.net> Signed-off-by: Robin H. Johnson <robbat2@gentoo.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11bundle-create: progress output controlRobin H. Johnson4-10/+65
Support the progress output options from pack-objects in git-bundle's create subcommand. Most notably, this provides --quiet as requested on the git mailing list per [1] Reference: https://www.mail-archive.com/git@vger.kernel.org/msg182844.html <robbat2-20190806T191156-796782357Z@orbis-terrarum.net> Signed-off-by: Robin H. Johnson <robbat2@gentoo.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11bundle: framework for options before bundle fileRobin H. Johnson1-45/+145
Make it possible for any of the git-bundle subcommands to include options: - before the sub-command - after the sub-command, before the bundle filename There is an immediate gain in support for help with all of the sub-commands, where 'git bundle list-heads -h' previously returned an error. Downside here is an increase in code duplication that cannot be trivially avoided short of shared global static options. Signed-off-by: Robin H. Johnson <robbat2@gentoo.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11merge-strategies: fix typo "reflected to" to "reflected in"Dominic Jäger1-1/+1
Signed-off-by: Dominic Jäger <dominic.jaeger@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11sequencer: reencode squashing commit's messageDoan Tran Cong Danh2-4/+14
On fixup/squash-ing rebase, git will create new commit in i18n.commitencoding, reencode the commit message to that said encode. Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11sequencer: reencode revert/cherry-pick's todo listDoan Tran Cong Danh1-1/+4
Keep revert/cherry-pick's todo list in line with rebase todo list. Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11sequencer: reencode to utf-8 before arrange rebase's todo listDoan Tran Cong Danh2-2/+2
On musl libc, ISO-2022-JP encoder is too eager to switch back to 1 byte encoding, musl's iconv always switch back after every combining character. Comparing glibc and musl's output for this command $ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 | iconv -f utf-8 -t ISO-2022-JP | xxd glibc: 00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B 00000010: 0a . musl: 00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B 00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B 00000020: 1b24 4224 551b 2842 0a .$B$U.(B. Although musl iconv's output isn't optimal, it's still correct. From commit 7d509878b8, ("pretty.c: format string with truncate respects logOutputEncoding", 2014-05-21), we're encoding the message to utf-8 first, then format it and convert the message to the actual output encoding on git commit --squash. Thus, t3900::test_commit_autosquash_flags is failing on musl libc. Reencode to utf-8 before arranging rebase's todo list. By doing this, we also remove a breakage noticed by a test added in the previous commit. Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11t3900: demonstrate git-rebase problem with multi encodingDoan Tran Cong Danh1-0/+29
We're using fixup!/squash! <subject> to mark if current commit will be used to be fixed up or squashed to a previous commit. However, if we're changing i18n.commitencoding after making the original commit but before making the fixing up, we couldn't find the original commit to do the fixup/squash. Add a test to demonstrate that problem. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-11configure.ac: define ICONV_OMITS_BOM if necessaryDoan Tran Cong Danh1-0/+49
From commit 79444c9294, ("utf8: handle systems that don't write BOM for UTF-16", 2019-02-12), we're supporting those systems with iconv that omits BOM with: make ICONV_OMITS_BOM=Yes However, configure script wasn't taught to detect those systems. Teach configure to do so. Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-10The first batch post 2.24 cycleJunio C Hamano3-2/+70
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-10Merge branch 'bc/hash-independent-tests-part-6'Junio C Hamano16-154/+276
Test updates to prepare for SHA-2 transition continues. * bc/hash-independent-tests-part-6: t4048: abstract away SHA-1-specific constants t4045: make hash-size independent t4044: update test to work with SHA-256 t4039: abstract away SHA-1-specific constants t4038: abstract away SHA-1 specific constants t4034: abstract away SHA-1-specific constants t4027: make hash-size independent t4015: abstract away SHA-1-specific constants t4011: abstract away SHA-1-specific constants t4010: abstract away SHA-1-specific constants t3429: remove SHA1 annotation t1305: avoid comparing extensions rev-parse: add a --show-object-format option t/oid-info: add empty tree and empty blob values t/oid-info: allow looking up hash algorithm name
2019-11-10Merge branch 'js/update-index-ignore-removal-for-skip-worktree'Junio C Hamano6-4/+42
"git stash save" in a working tree that is sparsely checked out mistakenly removed paths that are outside the area of interest. * js/update-index-ignore-removal-for-skip-worktree: stash: handle staged changes in skip-worktree files correctly update-index: optionally leave skip-worktree entries alone
2019-11-10Merge branch 'pb/pretty-email-without-domain-part'Junio C Hamano5-62/+111
The custom format for "git log --format=<format>" learned the l/L placeholder that is similar to e/E that fills in the e-mail address, but only the local part on the left side of '@'. * pb/pretty-email-without-domain-part: pretty: add "%aL" etc. to show local-part of email addresses t4203: use test-lib.sh definitions t6006: use test-lib.sh definitions
2019-11-10Merge branch 'hw/remove-api-docs-placeholder'Junio C Hamano4-40/+0
Docfix. * hw/remove-api-docs-placeholder: documentation: remove empty doc files
2019-11-10Merge branch 'sg/commit-graph-usage-fix'Junio C Hamano1-1/+0
Message fix. * sg/commit-graph-usage-fix: builtin/commit-graph.c: remove subcommand-less usage string
2019-11-10Merge branch 'dl/apply-3way-diff3'Junio C Hamano2-27/+30
"git apply --3way" learned to honor merge.conflictStyle configuration variable, like merges would. * dl/apply-3way-diff3: apply: respect merge.conflictStyle in --3way t4108: demonstrate bug in apply t4108: use `test_config` instead of `git config` t4108: remove git command upstream of pipe t4108: replace create_file with test_write_lines
2019-11-10Merge branch 'sg/dir-trie-fixes'Junio C Hamano3-60/+74
Code clean-up and a bugfix in the logic used to tell worktree local and repository global refs apart. * sg/dir-trie-fixes: path.c: don't call the match function without value in trie_find() path.c: clarify two field names in 'struct common_dir' path.c: mark 'logs/HEAD' in 'common_list' as file path.c: clarify trie_find()'s in-code comment Documentation: mention more worktree-specific exceptions
2019-11-10Merge branch 'jc/am-show-current-patch-docfix'Junio C Hamano1-1/+1
Doc update. * jc/am-show-current-patch-docfix: doc: am --show-current-patch gives an entire e-mail message
2019-11-10Merge branch 'wb/midx-progress'Junio C Hamano6-27/+149
The code to generate multi-pack index learned to show (or not to show) progress indicators. * wb/midx-progress: multi-pack-index: add [--[no-]progress] option. midx: honor the MIDX_PROGRESS flag in midx_repack midx: honor the MIDX_PROGRESS flag in verify_midx_file midx: add progress to expire_midx_packs midx: add progress to write_midx_file midx: add MIDX_PROGRESS flag