mirror of
https://github.com/git/git.git
synced 2026-10-25 15:15:20 +01:00
Patrick Steinhardt
cbfd4cb86f
Originally, when designing pluggable object databases the goal was that
the object database can have multiple sources, and every source attached
to it could use a different backend. This would have allowed for quite a
lot of flexibility, as you could trivially mix and match different kinds
of object storages in whatever way you like.
But while well-intentioned, this design led to a bunch of conceptual
problems:
- We're now trying to read objects in source order, whereas we
previously tried to read objects via packfiles before trying to read
them via loose objects. This led to a performance regression when
using alternates or when using a quarantine directory.
- Some data structures are supposed to only ever exist once, like for
example bitmaps and commit graphs. At the same time, those data
structures also span across the union of all objects, so they may
cross sources.
- It is unclear how we can extend GIT_OBJECT_DIRECTORY or
GIT_ALTERNATE_OBJECT_DIRECTORIES to become backend-agnostic in a
backwards-compatible way. In general, introducing an object storage
extension into the current status quo where alternates may have to
be extended to become generic was proving to be painful.
- Some mechanisms of alternates assume way too much about how exactly
their backends work. Alternate refs for example assume that the
alternate is backed by a filesystem path, and that this filesystem
path may also allow us to read references. This is not a given
though, as backends may not even have local data at all.
In short, there are a bunch of conceptual mismatches when we have
alternates and pluggable object databases coexist. So while the original
idea was nice, it does not result in a system that is easy to reason
about.
Correct course by moving alternates into the "files" source itself so
that it becomes an implementation detail thereof so that we can avoid
all of these shortcomings. While it's unfortunate that we cannot easily
mix and match sources now, that ability doesn't go away. It's still very
much feasible to introduce a new backend that allows for exactly that
use case, and such a backend may also be a lot more flexible as we can
now add new logic to determine which objects should be stored where. So
the original motivation for having per-source backends can still be
realized with the new architecture.
Note that as part of this move, we also handle the GIT_OBJECT_DIRECTORY
and GIT_ALTERNATE_OBJECT_DIRECTORIES environment variables in the
"files" backend. This may be surprising at first, but object directories
are very much a concept of that backend, too. So these variables would
have bad interactions with other backends, and they create a bit of a
mismatch with the eventual object storage extension that we plan to
introduce.
Note that this commit is way larger than I'd like it to be. I'm sorry, I
couldn't find a way to split it up further. That being said, most of the
changes are straight-forward conversions that go from iterating over
sources to iterating over object directories. The more involved changes
are moving the infrastructure to track individual sources from "odb.c"
and moving them into "odb/source-files.c".
This also serves to show a bit of a who's-who of commands that don't
work properly with pluggable object databases. Almost all of these
commands are related to housekeeping though -- some of them will be
converted eventually, like for example commit graphs or MIDXs. But many
of them will stay incompatible going forward as they are simply too
specific to the "files" backend.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
110 lines
3.2 KiB
C
110 lines
3.2 KiB
C
#ifndef ODB_SOURCE_FILES_H
|
|
#define ODB_SOURCE_FILES_H
|
|
|
|
#include "odb/source.h"
|
|
|
|
struct odb_source_loose;
|
|
struct odb_source_packed;
|
|
|
|
/*
|
|
* A single object directory that encapsulates access to both the loose and
|
|
* packed backend. This can either be the primary or an alternate object
|
|
* directory.
|
|
*/
|
|
struct odb_files_dir {
|
|
/* Absolute path to the object directory. */
|
|
char *abspath;
|
|
|
|
/* List of alternate object directories. */
|
|
struct odb_files_dir *next;
|
|
|
|
/*
|
|
* Entry in the files source's map of directories, keyed by this
|
|
* directory's path.
|
|
*/
|
|
struct hashmap_entry by_path_entry;
|
|
|
|
/* The two sources derived from this object directory. */
|
|
struct odb_source_loose *loose;
|
|
struct odb_source_packed *packed;
|
|
|
|
/*
|
|
* Whether this is the local object directory of the owning
|
|
* repository. Directories added via alternates are not local.
|
|
*/
|
|
bool local;
|
|
};
|
|
|
|
struct odb_files_dir *odb_files_dir_new(struct object_database *odb,
|
|
const char *path, bool local);
|
|
void odb_files_dir_free(struct odb_files_dir *dir);
|
|
|
|
/*
|
|
* The files object database source uses a combination of loose objects and
|
|
* packfiles. It is the default backend used by Git to store objects.
|
|
*/
|
|
struct odb_source_files {
|
|
struct odb_source base;
|
|
|
|
/*
|
|
* List of all object directories; the main directory is first (and
|
|
* cannot be NULL after initialization). Subsequent directories are
|
|
* alternates.
|
|
*/
|
|
struct odb_files_dir *dirs;
|
|
struct odb_files_dir **dirs_tail;
|
|
|
|
/*
|
|
* Map of object directories, keyed by their respective paths. This
|
|
* map is used to detect the case where the same directory is
|
|
* registered multiple times.
|
|
*/
|
|
struct hashmap dirs_by_path;
|
|
|
|
/*
|
|
* Whether directory paths shall be compared case-insensitively, as
|
|
* determined by "core.ignoreCase".
|
|
*/
|
|
int dirs_paths_icase;
|
|
};
|
|
|
|
/* Allocate and initialize a new object source. */
|
|
struct odb_source_files *odb_source_files_new(struct object_database *odb,
|
|
enum odb_new_flags flags);
|
|
|
|
/*
|
|
* Optimize the files object database source by repacking loose objects and
|
|
* packfiles as needed. Returns 0 on success, a negative error code otherwise.
|
|
*/
|
|
int odb_source_files_optimize(struct odb_source *source,
|
|
const struct odb_optimize_options *opts);
|
|
|
|
/*
|
|
* Check whether optimization of the files object database source is required
|
|
* given the provided options. Returns true if optimization should be
|
|
* performed, false otherwise.
|
|
*/
|
|
bool odb_source_files_optimize_required(struct odb_source *source,
|
|
const struct odb_optimize_options *opts);
|
|
|
|
/*
|
|
* Cast the given object database source to the files backend. This will cause
|
|
* a BUG in case the source doesn't use this backend.
|
|
*/
|
|
static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
|
|
{
|
|
if (source->type != ODB_SOURCE_FILES)
|
|
BUG("trying to downcast source of type '%s' to '%s'",
|
|
odb_source_type_to_name(source->type),
|
|
odb_source_type_to_name(ODB_SOURCE_FILES));
|
|
return container_of(source, struct odb_source_files, base);
|
|
}
|
|
|
|
/*
|
|
* Find "files" directory by its object directory path. Returns a `NULL`
|
|
* pointer in case the object directory could not be found.
|
|
*/
|
|
struct odb_files_dir *odb_source_files_find_dir(struct object_database *odb, const char *obj_dir);
|
|
|
|
#endif
|