summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2022-03-13attr.c: delete duplicate includeElia Pinto1-1/+0
dir.h is included more than once Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-13t6423-merge-rename-directories.sh: use the $(...) constructElia Pinto1-5/+5
The Git CodingGuidelines prefer the $(...) construct for command substitution instead of using the backquotes `...`. The backquoted form is the traditional method for command substitution, and is supported by POSIX. However, all but the simplest uses become complicated quickly. In particular, embedded command substitutions and/or the use of double quotes require careful escaping with the backslash character. The patch was generated by: for _f in $(find . -name "*.sh") do shellcheck -i SC2006 -f diff ${_f} | ifne git apply -p2 done and then carefully proof-read. Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-13name-rev: use generation numbers if availableJacob Keller2-14/+175
If a commit in a sequence of linear history has a non-monotonically increasing commit timestamp, git name-rev might not properly name the commit. This occurs because name-rev uses a heuristic of the commit date to avoid searching down tags which lead to commits that are older than the named commit. This is intended to avoid work on larger repositories. This heuristic impacts git name-rev, and by extension git describe --contains which is built on top of name-rev. Further more, if --all or --annotate-stdin is used, the heuristic is not enabled because the full history has to be analyzed anyways. This results in some confusion if a user sees that --annotate-stdin works but a normal name-rev does not. If the repository has a commit graph, we can use the generation numbers instead of using the commit dates. This is essentially the same check except that generation numbers make it exact, where the commit date heuristic could be incorrect due to clock errors. Since we're extending the notion of cutoff to more than one variable, create a series of functions for setting and checking the cutoff. This avoids duplication and moves access of the global cutoff and generation_cutoff to as few functions as possible. Add several test cases including a test that covers the new commitGraph behavior, as well as tests for --all and --annotate-stdin with and without commitGraphs. Signed-off-by: Jacob Keller <jacob.keller@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-13reflog: don't be noisy on empty reflogsÆvar Arnfjörð Bjarmason2-0/+11
Fix a regression in my daf1d8285ee (reflog expire: don't use lookup_commit_reference_gently(), 2021-12-22), in changing from lookup_commit_reference_gently() to lookup_commit() we stopped trying to call deref_tag() and parse_object() on the provided OID, but we also started returning non-NULL for the null_oid(). As a result we'd emit an error() via mark_reachable() later in this function as we tried to invoke parse_commit() on it. Reported-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Tested-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-12userdiff: add builtin diff driver for kotlin language.Jaydeep P Das15-0/+167
The xfuncname pattern finds func/class declarations in diffs to display as a hunk header. The word_regex pattern finds individual tokens in Kotlin code to generate appropriate diffs. This patch adds xfuncname regex and word_regex for Kotlin language. Signed-off-by: Jaydeep P Das <jaydeepjd.8914@gmail.com> Acked-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-12t0030-t0050: avoid pipes with Git on LHSShubham Mishra2-28/+50
Pipes ignore error codes of LHS command and thus we should not use them with Git in tests. As an alternative, use a 'tmp' file to write the Git output so we can test the exit code. Signed-off-by: Shubham Mishra <shivam828787@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-12t0001-t0028: avoid pipes with Git on LHSShubham Mishra4-14/+16
Pipes ignore error codes of LHS command and thus we should not use them with Git in tests. As an alternative, use a 'tmp' file to write the Git output so we can test the exit code. Signed-off-by: Shubham Mishra <shivam828787@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10core.fsync: new option to harden the indexNeeraj Singh3-6/+15
This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10core.fsync: add configuration parsingNeeraj Singh3-5/+82
This change introduces code to parse the core.fsync setting and configure the fsync_components variable. core.fsync is configured as a comma-separated list of component names to sync. Each time a core.fsync variable is encountered in the configuration heirarchy, we start off with a clean state with the platform default value. Passing 'none' resets the value to indicate nothing will be synced. We gather all negative and positive entries from the comma separated list and then compute the new value by removing all the negative entries and adding all of the positive entries. We issue a warning for components that are not recognized so that the configuration code is compatible with configs from future versions of Git with more repo components. Complete documentation for the new setting is included in a later patch in the series so that it can be reviewed once in final form. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10core.fsync: introduce granular fsync control infrastructureNeeraj Singh15-33/+97
This commit introduces the infrastructure for the core.fsync configuration knob. The repository components we want to sync are identified by flags so that we can turn on or off syncing for specific components. If core.fsyncObjectFiles is set and the core.fsync configuration also includes FSYNC_COMPONENT_LOOSE_OBJECT, we will fsync any loose objects. This picks the strictest data integrity behavior if core.fsync and core.fsyncObjectFiles are set to conflicting values. This change introduces the currently unused fsync_component helper, which will be used by a later patch that adds fsyncing to the refs backend. Actual configuration and documentation of the fsync components list are in other patches in the series to separate review of the underlying mechanism from the policy of how it's configured. Helped-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10core.fsyncmethod: add writeout-only modeNeeraj Singh13-8/+184
This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10wrapper: make inclusion of Windows csprng header tightly scopedNeeraj Singh3-11/+7
Including NTSecAPI.h in git-compat-util.h causes build errors in any other file that includes winternl.h. NTSecAPI.h was included in order to get access to the RtlGenRandom cryptographically secure PRNG. This change scopes the inclusion of ntsecapi.h to wrapper.c, which is the only place that it's actually needed. The build breakage is due to the definition of UNICODE_STRING in NtSecApi.h: #ifndef _NTDEF_ typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING; typedef LSA_STRING STRING, *PSTRING ; #endif LsaLookup.h: typedef struct _LSA_UNICODE_STRING { USHORT Length; USHORT MaximumLength; #ifdef MIDL_PASS [size_is(MaximumLength/2), length_is(Length/2)] #endif // MIDL_PASS PWSTR Buffer; } LSA_UNICODE_STRING, *PLSA_UNICODE_STRING; winternl.h also defines UNICODE_STRING: typedef struct _UNICODE_STRING { USHORT Length; USHORT MaximumLength; PWSTR Buffer; } UNICODE_STRING; typedef UNICODE_STRING *PUNICODE_STRING; Both definitions have equivalent layouts. Apparently these internal Windows headers aren't designed to be included together. This is an oversight in the headers and does not represent an incompatibility between the APIs. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10block-sha1: remove use of obsolete x86 assemblybrian m. carlson1-17/+0
In the block SHA-1 code, we have special assembly code for i386 and amd64 to perform rotations with assembly. This is supposed to help pick the correct rotation operation depending on which rotation is smaller, which can help some systems perform slightly better, since any circular rotation can be specified as either a rotate left or a rotate right. However, this isn't needed, so we should remove it. First, SHA-1, like SHA-2, uses fixed constant rotates. Thus, all rotation amounts are known at compile time and are in fact baked into the code. Fortunately, peephole optimizers recognize rotations specified in the normal way and automatically emit the correct code, including a preference for choosing a rotate left versus a rotate right. This has been the case for well over a decade, and is a standard example of the utility of a peephole optimizer. Moreover, all modern CPUs, with the exception of extremely limited embedded CPUs such as some Cortex-M processors, provide a barrel shifter, which lets the CPU perform rotates of any bit amount in constant time. This is valuable for many cryptographic algorithms to improve performance, and is required to prevent timing attacks in algorithms which use data-dependent rotations (which don't include the hash algorithms we use). As a result, even though the compiler does the correct optimization, it isn't even needed here and either a left or a right rotate is equally acceptable. In fact, the SHA-256 code already takes this into account and just writes the simple code using an inline function to let the compiler optimize it for us. The downside of using this code, however, is that it uses a GCC extension, which makes the compiler complain when using -pedantic unless it's prefixed with __extension__. We could fix that, but since it's not needed, let's just remove it. We haven't noticed this because almost everyone uses the SHA1DC code instead, but it still shows up for some people. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09Merge branch 'pw/single-key-interactive' into pw/add-p-single-keyJunio C Hamano3-6/+33
* pw/single-key-interactive: add -p: disable stdin buffering when interactive.singlekey is set terminal: set VMIN and VTIME in non-canonical mode terminal: pop signal handler when terminal is restored terminal: always reset terminal when reading without echo
2022-03-09test-lib: declare local variables as localMichael J Gruber1-0/+2
131b94a10a ("test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34", 2022-03-04) introduced "local" variables without declaring them as such. This conflicts with their use in some tests (at least when running them with dash), leading to test failures in: t0006-date.sh t2002-checkout-cache-u.sh t3430-rebase-merges.sh t4138-apply-ws-expansion.sh t4124-apply-ws-rule.sh Declare those variables as local to let the tests pass again. Signed-off-by: Michael J Gruber <git@grubix.eu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09The tenth batchJunio C Hamano1-1/+27
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09Merge branch 'ab/help-fixes'Junio C Hamano5-24/+187
Updates to how command line options to "git help" are handled. * ab/help-fixes: help: don't print "\n" before single-section output help: add --no-[external-commands|aliases] for use with --all help: error if [-a|-g|-c] and [-i|-m|-w] are combined help: correct usage & behavior of "git help --all" help: note the option name on option incompatibility help.c: split up list_all_cmds_help() function help tests: test "git" and "git help [-a|-g] spacing help.c: use puts() instead of printf{,_ln}() for consistency help doc: add missing "]" to "[-a|--all]"
2022-03-09Merge branch 'ab/c99-variadic-macros'Junio C Hamano8-246/+65
Remove the escape hatch we added when we introduced the weather balloon to use variadic macros unconditionally, to make it official that we now have a hard dependency on the feature. * ab/c99-variadic-macros: C99: remove hardcoded-out !HAVE_VARIADIC_MACROS code git-compat-util.h: clarify GCC v.s. C99-specific in comment
2022-03-09Merge branch 'hn/reftable-no-empty-keys'Junio C Hamano7-19/+136
General clean-up in reftable implementation, including clarification of the API documentation, tightening the code to honor documented length limit, etc. * hn/reftable-no-empty-keys: reftable: rename writer_stats to reftable_writer_stats reftable: add test for length of disambiguating prefix reftable: ensure that obj_id_len is >= 2 on writing reftable: avoid writing empty keys at the block layer reftable: add a test that verifies that writing empty keys fails reftable: reject 0 object_id_len Documentation: object_id_len goes up to 31
2022-03-09Merge branch 'jc/cat-file-batch-commands'Junio C Hamano3-22/+333
"git cat-file" learns "--batch-command" mode, which is a more flexible interface than the existing "--batch" or "--batch-check" modes, to allow different kinds of inquiries made. * jc/cat-file-batch-commands: cat-file: add --batch-command mode cat-file: add remove_timestamp helper cat-file: introduce batch_mode enum to replace print_contents cat-file: rename cmdmode to transform_mode
2022-03-09Merge branch 'pw/xdiff-alloc-fail'Junio C Hamano4-48/+51
Improve failure case behaviour of xdiff library when memory allocation fails. * pw/xdiff-alloc-fail: xdiff: handle allocation failure when merging xdiff: refactor a function xdiff: handle allocation failure in patience diff xdiff: fix a memory leak
2022-03-09Merge branch 'en/present-despite-skipped'Junio C Hamano19-127/+310
In sparse-checkouts, files mis-marked as missing from the working tree could lead to later problems. Such files were hard to discover, and harder to correct. Automatically detecting and correcting the marking of such files has been added to avoid these problems. * en/present-despite-skipped: repo_read_index: add config to expect files outside sparse patterns Accelerate clear_skip_worktree_from_present_files() by caching Update documentation related to sparsity and the skip-worktree bit repo_read_index: clear SKIP_WORKTREE bit from files present in worktree unpack-trees: fix accidental loss of user changes t1011: add testcase demonstrating accidental loss of user modifications
2022-03-09clone: fail gracefully when cloning filtered bundleDerrick Stolee2-0/+25
Users can create a new repository using 'git clone <bundle-file>'. The new "@filter" capability for bundles means that we can generate a bundle that does not contain all reachable objects, even if the header has no negative commit OIDs. It is feasible to think that we could make a filtered bundle work with the command git clone --filter=$filter --bare <bundle-file> or possibly replacing --bare with --no-checkout. However, this requires having some repository-global config that specifies the specified object filter and notifies Git about the existence of promisor pack-files. Without a remote, that is currently impossible. As a stop-gap, parse the bundle header during 'git clone' and die() with a helpful error message instead of the current behavior of failing due to "missing objects". Most of the existing logic for handling bundle clones actually happens in fetch-pack.c, but that logic is the same as if the user specified 'git fetch <bundle>', so we want to avoid failing to fetch a filtered bundle when in an existing repository that has the proper config set up for at least one remote. Carefully comment around the test that this is not the desired long-term behavior of 'git clone' in this case, but instead that we need to do more work before that is possible. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09bundle: unbundle promisor packsDerrick Stolee2-0/+6
In order to have a valid pack-file after unbundling a bundle that has the 'filter' capability, we need to generate a .promisor file. The bundle does not promise _where_ the objects can be found, but we can expect that these bundles will be unbundled in repositories with appropriate promisor remotes that can find those missing objects. Use the 'git index-pack --promisor=<message>' option to create this .promisor file. Add "from-bundle" as the message to help anyone diagnose issues with these promisor packs. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09bundle: create filtered bundlesDerrick Stolee2-11/+90
A previous change allowed Git to parse bundles with the 'filter' capability. Now, teach Git to create bundles with this option. Some rearranging of code is required to get the option parsing in the correct spot. There are now two reasons why we might need capabilities (a new hash algorithm or an object filter) so that is pulled out into a place where we can check both at the same time. The --filter option is parsed as part of setup_revisions(), but it expected the --objects flag, too. That flag is somewhat implied by 'git bundle' because it creates a pack-file walking objects, but there is also a walk that walks the revision range expecting only commits. Make this parsing work by setting 'revs.tree_objects' and 'revs.blob_objects' before the call to setup_revisions(). Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09rev-list: move --filter parsing into revision.cDerrick Stolee2-11/+7
Now that 'struct rev_info' has a 'filter' member and most consumers of object filtering are using that member instead of an external struct, move the parsing of the '--filter' option out of builtin/rev-list.c and into revision.c. This use within handle_revision_pseudo_opt() allows us to find the option within setup_revisions() if the arguments are passed directly. In the case of a command such as 'git blame', the arguments are first scanned and checked with parse_revision_opt(), which complains about the option, so 'git blame --filter=blob:none <file>' does not become valid with this change. Some commands, such as 'git diff' gain this option without having it make an effect. And 'git diff --objects' was already possible, but does not actually make sense in that builtin. The key addition that is coming is 'git bundle create --filter=<X>' so we can create bundles containing promisor packs. More work is required to make them fully functional, but that will follow. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09bundle: parse filter capabilityDerrick Stolee6-22/+50
The v3 bundle format has capabilities, allowing newer versions of Git to create bundles with newer features. Older versions that do not understand these new capabilities will fail with a helpful warning. Create a new capability allowing Git to understand that the contained pack-file is filtered according to some object filter. Typically, this filter will be "blob:none" for a blobless partial clone. This change teaches Git to parse this capability, place its value in the bundle header, and demonstrate this understanding by adding a message to 'git bundle verify'. Since we will use gently_parse_list_objects_filter() outside of list-objects-filter-options.c, make it an external method and move its API documentation to before its declaration. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09list-objects: handle NULL function pointersÆvar Arnfjörð Bjarmason3-5/+36
If a caller to traverse_commit_list() specifies the options for the --objects flag but does not specify a show_object function pointer, the result is a segfault. This is currently visible by running 'git bundle create --objects HEAD'. We could fix this problem by supplying a no-op callback in builtin/bundle.c, but that only solves the problem for one builtin, leaving this segfault open for other callers. Replace all callers of the show_commit and show_object function pointers in list-objects.c to call helper functions show_commit() and show_object() which check that the given context has non-NULL functions before passing the necessary data. One extra benefit is that it reduces duplication due to passing ctx->show_data to every caller. Test that this segfault no longer occurs for 'git bundle'. Co-authored-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09MyFirstObjectWalk: update recommended usageDerrick Stolee1-28/+16
The previous change consolidated traverse_commit_list() and traverse_commit_list_filtered(). This allows us to simplify the recommended usage in MyFirstObjectWalk.txt to use this new set of values. While here, add some clarification on the difference between the two methods. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee5-31/+29
Now that all consumers of traverse_commit_list_filtered() populate the 'filter' member of 'struct rev_info', we can drop that parameter from the method prototype to simplify things. In addition, the only thing different now between traverse_commit_list_filtered() and traverse_commit_list() is the presence of the 'omitted' parameter, which is only non-NULL for one caller. We can consolidate these two methods by having one call the other and use the simpler form everywhere the 'omitted' parameter would be NULL. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee5-20/+14
Now that all consumers of prepare_bitmap_walk() have populated the 'filter' member of 'struct rev_info', we can drop that extra parameter from the method and access it directly from the 'struct rev_info'. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09pack-objects: use rev.filter when possibleDerrick Stolee1-2/+3
In builtin/pack-objects.c, we use a 'filter_options' global to populate the --filter=<X> argument. The previous change created a pointer to a filter option in 'struct rev_info', so we can use that pointer here as a start to simplifying some usage of object filters. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09revision: put object filter into struct rev_infoDerrick Stolee2-14/+19
Placing a 'struct list_objects_filter_options' within 'struct rev_info' will assist making some bookkeeping around object filters in the future. For now, let's use this new member to remove a static global instance of the struct from builtin/rev-list.c. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09list-objects-filter-options: create copy helperDerrick Stolee2-0/+23
As we add more embedded members with type 'struct list_objects_filter_options', it will be important to easily perform a deep copy across multiple such structs. Create list_objects_filter_copy() to satisfy this need. This method is recursive to match the recursive nature of the struct. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-09index-pack: document and test the --promisor optionDerrick Stolee2-1/+11
The --promisor option of 'git index-pack' was created in 88e2f9e (introduce fetch-object: fetch one promisor object, 2017-12-05) but was untested. It is currently unused within the Git codebase, but that will change in an upcoming change to 'git bundle unbundle' when there is a filter capability. For now, add documentation about the option and add a test to ensure it is working as expected. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-08gitweb: remove invalid http-equiv="content-type"Jason Yundt2-3/+14
Before this change, gitweb would generate pages which included: <meta http-equiv="content-type" content="application/xhtml+xml; charset=utf-8"/> When a meta's http-equiv equals "content-type", the http-equiv is said to be in the "Encoding declaration state". According to the HTML Standard, The Encoding declaration state may be used in HTML documents, but elements with an http-equiv attribute in that state must not be used in XML documents. Source: <https://html.spec.whatwg.org/multipage/semantics.html#attr-meta-http-equiv-content-type> This change removes that meta element since gitweb always generates XML documents. Signed-off-by: Jason Yundt <jason@jasonyundt.email> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-08comment: fix typoJason Yundt1-1/+1
Signed-off-by: Jason Yundt <jason@jasonyundt.email> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07submodule: move logic into fetch_task_create()Glen Choo1-47/+52
get_fetch_task() gets a fetch task by iterating the index; a future commit will introduce a similar function, get_fetch_task_from_changed(), that gets a fetch task from the list of changed submodules. Both functions are similar in that they need to: * create a fetch task * initialize the submodule repo for the fetch task * determine the default recursion mode Move all of this logic into fetch_task_create() so that it is no longer split between fetch_task_create() and get_fetch_task(). This will make it easier to share code with get_fetch_task_from_changed(). Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07submodule: extract get_fetch_task()Glen Choo1-25/+36
get_next_submodule() configures the parallel submodule fetch by performing two functions: * iterate the index to find submodules * configure the child processes to fetch the submodules found in the previous step Extract the index iterating code into an iterator function, get_fetch_task(), so that get_next_submodule() is agnostic of how to find submodules. This prepares for a subsequent commit will teach the fetch machinery to also iterate through the list of changed submodules (in addition to the index). Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07submodule: store new submodule commits oid_array in a structGlen Choo1-18/+34
This commit prepares for a future commit that will teach `git fetch --recurse-submodules` how to fetch submodules that are present in <gitdir>/modules, but are not populated. To do this, we need to store more information about the changed submodule so that we can read the submodule configuration from the superproject commit instead of the filesystem. Refactor the changed submodules string_list.util to hold a struct instead of an oid_array. This struct only holds the new_commits oid_array for now; more information will be added later. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07submodule: inline submodule_commits() into callerGlen Choo1-16/+6
When collecting the string_list of changed submodule names, the new submodules commits are stored in the string_list_item.util as an oid_array. A subsequent commit will replace the oid_array with a struct that has more information. Prepare for this change by inlining submodule_commits() (which inserts into the string_list and initializes the string_list_item.util) into its only caller so that the code is easier to refactor later. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07submodule: make static functions read submodules from commitsGlen Choo1-10/+19
A future commit will teach "fetch --recurse-submodules" to fetch unpopulated submodules. To prepare for this, teach the necessary static functions how to read submodules from superproject commits using a "treeish_name" argument (instead of always reading from the index and filesystem) but do not actually change where submodules are read from. Submodules will be read from commits when we fetch unpopulated submodules. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07t5526: create superproject commits with test helperGlen Choo1-50/+45
A few tests in t5526 use this pattern as part of their setup: 1. Create new commits in the upstream submodules (using add_upstream_commit()). 2. In the upstream superprojects, add the new submodule commits from the previous step. A future commit will add more tests with this pattern, so reduce the verbosity of present and future tests by introducing a test helper that creates superproject commits. Since we now have two helpers that add upstream commits, rename add_upstream_commit() to add_submodule_commits(). Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07t5526: stop asserting on stderr literallyGlen Choo1-61/+56
In the previous commit message, we noted that not all of the "git fetch" stderr is relevant to the tests. Most of the test setup lines are dedicated to these details of the stderr: 1. which repos (super/sub/deep) are involved in the fetch 2. the head of the remote-tracking branch before the fetch (i.e. $head1) 3. the head of the remote-tracking branch after the fetch (i.e. $head2) 1. and 3. are relevant because they tell us that the expected commit is fetched by the expected repo, but 2. is completely irrelevant. Stop asserting on $head1 by replacing it with a dummy value in the actual and expected output. Do this by introducing test helpers (write_expected_*()) that make it easier to construct the expected output, and use sed to munge the actual output. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07t5526: introduce test helper to assert on fetchesGlen Choo1-55/+84
Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy: * The tests have extra logic in order to reproduce the expected stderr literally, but not all of these details (e.g. the head of the remote-tracking branch before the fetch) are relevant to the test. * The expect.err file is constructed by the add_upstream_commit() helper as input into test_cmp, but most tests fetch a different combination of repos from expect.err. This results in noisy tests that modify parts of that expect.err to generate the expected output. To address both of these issues, introduce a verify_fetch_result() helper to t/t5526-fetch-submodules.sh that asserts on the output of "git fetch --recurse-submodules" and handles the ordering of expect.err. As a result, the tests no longer construct expect.err manually. Tests still consider the old head of the remote-tracking branch ("$head1"), but that will be fixed in a later commit. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07rev-list simplify tests: don't ignore "git" exit codeÆvar Arnfjörð Bjarmason1-5/+7
Change a fragile test pattern introduced in 65347030590 (Topo-sort before --simplify-merges, 2008-08-03) to check the exit code of both "git name-rev" and "git log". This test as a whole would fail under SANITIZE=leak, but we'd pass several "failing" tests due to hiding these exit codes before we'd spot git dying with abort(). Now we'll instead spot all of the failures. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07checkout tests: don't ignore "git <cmd>" exit codeÆvar Arnfjörð Bjarmason1-16/+35
Change a fragile pattern introduced in 696acf45f96 (checkout: implement "-" abbreviation, add docs and tests, 2009-01-17) to check the exit code of both "git symbolic-ref" and "git rev-parse". Without this change this test will become flaky e.g. under SANITIZE=leak if some (but not all) memory leaks revealed by these commands are fixed. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07apply tests: don't ignore "git ls-files" exit code, drop sub-shellÆvar Arnfjörð Bjarmason1-1/+2
Fix code added in 969c877506c (git apply --directory broken for new files, 2008-10-12) so that it doesn't invoke "git ls-files" on the left-hand-side of a pipe, instead let's use an intermediate file. Since we're doing that we can also drop the sub-shell that was here to group the two. There are a lot of these sorts of patterns in the test suite, and there's no particular reason to fix this one other than in a preceding commit all similar patterns except this one were fixed in "t/t4128-apply-root.sh", so let's fix this one straggler as well. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07gettext tests: don't ignore "test-tool regex" exit codeÆvar Arnfjörð Bjarmason1-3/+13
Amend a prerequisite check added in 5c1ebcca4d1 (grep/icase: avoid kwsset on literal non-ascii strings, 2016-06-25) to do invoke 'test-tool regex' in such a way that we'll notice if it dies under SANITIZE=leak due to having a memory leak, as opposed to us not having the "ICASE" support we're checking for. Because we weren't making a distinction between the two I'd marked these tests as passing under SANITIZE=leak in 03d85e21951 (leak tests: mark remaining leak-free tests as such, 2021-12-17). Doing this is tricky. Ideally "test_lazy_prereq" would materialize as a "real" test that we could check the exit code of with the same signal matching that "test_must_fail" does. However lazy prerequisites aren't real tests, and are instead lazily materialized in the guts of "test_have_prereq" when we've already started another test. We could detect the abort() (or similar) there and pass that exit code down, and fail the test that caused the prerequisites to be materialized. But that would require extensive changes to test-lib.sh and test-lib-functions.sh. Let's instead simply check if the exit code of "test-tool regex" is zero, and if so set the prerequisites. If it's non-zero let's run it again with "test_must_fail". We'll thus make a distinction between "bad" non-zero (segv etc) and "good" (exit 1 etc.). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-07rev-list tests: don't hide abort() in "test_expect_failure"Ævar Arnfjörð Bjarmason1-4/+9
Change a couple of uses of "test_expect_failure" to use a "test_expect_success" to positively assert the current behavior, and replace the intent of "test_expect_failure" with a "TODO" comment int the description. As noted in [1] the "test_expect_failure" feature is overly eager to accept any failure as OK, and thus by design hides segfaults, abort() etc. Because of that I didn't notice in dd9cede9136 (leak tests: mark some rev-list tests as passing with SANITIZE=leak, 2021-10-31) that this test leaks memory under SANITIZE=leak. I have some larger local changes to add a better "test_expect_failure", which would work just like "test_expect_success", but would allow us say "test_todo" here (and "success" would emit a "not ok [...] # TODO", not "ok [...]". So even though using "test_expect_success" here comes with its own problems[2], let's use it as a narrow change to fix the problem at hand here and stop conflating the current "success" with actual SANITIZE=leak failures. 1. https://lore.kernel.org/git/87tuhmk19c.fsf@evledraar.gmail.com/ 2. https://lore.kernel.org/git/xmqq4k9kj15p.fsf@gitster.g/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>