summaryrefslogtreecommitdiff
path: root/include/git.sh
AgeCommit message (Collapse)Author
2024-07-17git.sh: re-try git pull three timesLeah Rowe
mitigate jittery internet connections Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17trees: auto-delete+auto-rebuild if project changesLeah Rowe
actual source code is not scanned, but config directories are scanned. simply get the checksum of each file under config/ pertaining to a given project/tree, and also for the given target. coreboot utilities are also handled. if it changes, in any way, delete and re-build automatically. such deletions should probably still be done manually, as part of understanding the build system, but this change should make the build system much easier to use during development. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17git.sh: don't download to src/project/project/Leah Rowe
re-use repo/project/ this means that single- and multi-tree projects now have a unified cached git repo location, as per the new rules, thus saving on disk space usage. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17git.sh: cache git downloads to repo/Leah Rowe
do it based on the URL, e.g. https://review.coreboot.org/coreboot becomes repo/coreboot the downside is if you have two projects with repo urls specifying the same string at the end, but this isn't the case at the moment and likely won't be the case, but it's a theoretical issue. this saves on bandwidth when downloading identical submodule repos between multiple trees within the same multi-tree project for example, coreboot 3rdparty/vboot is no longer downloaded more than once, instead cloned locally on subsequent downloads. if repo/DIR exists, git-pull is attempted, but errors do not result in a non-zero exit, by design. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10trees: general code cleanupLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10remove executable permission on include/Leah Rowe
files under include/ should never be executed directly Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: simpler for loop in git_am_patches()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: merge for loops in git_am_patches()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30trees and git.sh: tidy up global variablesLeah Rowe
some of the variables only initialised in git.sh are also used in the trees script, which is technically ok because git.sh is included from the trees script, but it makes more sense to declare them in the latter. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: simplified initialisation of "loc"Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: condense fetch_targets() a bitLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: short git_prep command in fetch_targets()Leah Rowe
loc is already set, and will correspond to the same path, so we can quite conveniently use it. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: merge prepare_new_tree with fetch_targetsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: remove duplicate "xtree" variableLeah Rowe
it's already defined inside the trees script Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30trees, multi: download bare project *before* treesLeah Rowe
when downloading multi-tree projects, the rev can be reset to HEAD instead of the actual rev for a given target. this occurs when the bare repo (e.g. src/coreboot/coreboot) does not exist and has to be downloaded first. bare repository downloading does not rely on target.cfg, in this context, only pkg.cfg, but it uses the same variable names (e.g. "rev"). instead of using a separate variable name, thus increasing code complexity (which is the exact opposite of what i want to do), do the bare repository download first. this means that the git.sh script is much cleaner now, for multi-tree projects, in that it *only* copies the bare repo then runs git_prep; in that context, the bare repo is cloned directly by calling the relevant function from script/trees, which is the same behaviour as when cloning single-tree project sources. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30trees: unified multi-tree configuration handlingLeah Rowe
the same function that loads configurations for single-tree projects has been merged with the function for multi-tree configs in git.sh, and that functionality has been removed from git.sh; now it is all unified in the trees script. as the saying goes: write one program to do one thing well. the purpose of git.sh is to download source code, but not to handle configuration files; the latter is meant to be handled by the trees script, which then calls into git.sh before running the build logic for that given project. additionally: the "seen" files are no longer handled, at all. the logic there was added ages ago, because at the time, i was considering whether to separate configuration into a new repository, so that users could more easily make their own configuration, so it was a guard against misconfiguration. however, that decision was canceled and we're always very careful not to introduce a loop; if a loop does occur, the worst that can possibly happen is you waste some CPU cycles. Instead, print (on standard output) what config file is being used, so the operator can see when an infinite loop occurs. ALSO: remove _setcfgarg in load_project_config() it was used to skip when a target.cfg file didn't exist, specifically on single-tree projects, but this is now handled using -f instead, on the while loop inside that function, so _setcfgarg is now a redundant variable. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29git.sh: rename Fetch_project_trees fetch_targetsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29git.sh: rename fetch_project_repo to fetch_projectLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-28tidy up some setvars listsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-27remove use of _xm variable in the build systemLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-27git.sh: remove unnecessary checkLeah Rowe
the trees script itself will check that the directory exists, and exit with zero status if it does, without doing anything else other than the return. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-25lib.sh: simplify use of environment variablesLeah Rowe
don't have a separate variable for them. just export them directly and use them directly. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-22use backticks on eval commands, not subshellsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-22lib.sh: more unified config handlingLeah Rowe
replace it with logic that simply uses "." to load files directly. for this, "vcfg" is added as a variable in coreboot target.cfg files, referring to a directory in config/vendor/ containing a file named pkg.cfg, and this file then contains the same variables as the erstwhile config/vendor/sources config/git files are now directories, also containing pkg.cfg files each with the same variables as before, such as repository link and commit hash this change results in a noticeable reduction in code complexity within the build system. unified reading of config files: new function setcfg() added to lib.sh setcfg checks if a config exists. if a 2nd argument is passed, it is used as a return value for eval, otherwise a string calling err is passed. setcfg output is passed through eval, to set strings based on config; eval must be used, so that the variables are set within the same scope, otherwise they'd be set within setcfg which could lead to some whacky results. there's still a bit more more to do, but this single change results in a substantial reduction in code complexity. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: revert modification to for loopLeah Rowe
i tried to be clever with this one, but it just made the script exit with an error. revert back to the old check (check whether one of either repo or repo backup is set) Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20minor code cleanup in the build systemLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: general code cleanup in fetch_submodule()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: reduced indentation on repo/file checkLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: simplified repo/backup checkLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-14unified checks for variable initialisationLeah Rowe
new function chkvars() does the job Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09minor code cleanup in the build systemLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: hide e() output on for loopLeah Rowe
this for loop is a hack to make sure that all the sources get nuked (using nuke.list files). hide the messages so that they do not appear when running just any command in the trees script. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: download xtree *before*, not afterLeah Rowe
downloading it after means that if an error occurs when downloading the xtree project, the main project will still be there and nothing will mandate the downloading of the xtree project. whereas, if we grab the xtree project first, then the main project won't get saved to src/ this makes the build system a bit more resilient under fault conditions, but otherwise doesn't change behaviour. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: fix deletion path in nuke()Leah Rowe
i accidentally forgot to include src/ in the prefix Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: further simplify nuke()Leah Rowe
it's a very compact nuke Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: simplify link_crossgcc()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: simplify nuke()Leah Rowe
do not over-engineer such a trivial thing. seriously. all we're doing is nuking some files. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-08git.sh: support downloading *files* as submodulesLeah Rowe
when we download coreboot, we currently don't have a way to download crossgcc tarballs, so we rely on coreboot to do it, which means running the coreboot build system to do it; which means we don't get them in release archives, unless we add very hacky logic (which did exist and was removed). the problem with coreboot's build system is that it does not define backup links for each given tarball, instead relying on gnu.org exclusively, which seems OK at first because the gnu.org links actually return an HTTP 302 response leading to a random mirror, HOWEVER: the gnu.org 302 redirect often fails, and the download fails, causing an error. a mitigation for this has been to patch the coreboot build system to download directly from a single mirror that is reliable (in our case mirrorservice.org). while this mitigation mostly works, it's not redundant; the kent mirror is occasionally down too, and again we still have the problem of not being able to cleanly provide crossgcc tarballs inside release archives. do it in config/submodules, like so: module.list shall say the relative path of a given file, once downloaded, relative to the given source tree. module.cfg shall be re-used, in the same way as for git submodules, but: subfile="url" subfile_bkup="backup url" do this, instead of: subrepo="url" subrepo_bkup="backup url" example entries in module.list: util/crossgcc/tarballs/binutils-2.41.tar.xz util/crossgcc/tarballs/gcc-13.2.0.tar.xz util/crossgcc/tarballs/gmp-6.3.0.tar.xz util/crossgcc/tarballs/mpc-1.3.1.tar.gz util/crossgcc/tarballs/mpfr-4.2.1.tar.xz util/crossgcc/tarballs/nasm-2.16.01.tar.bz2 util/crossgcc/tarballs/R06_28_23.tar.gz the "subrev" variable (in module.cfg) has been renamed to "subhash", so that this makes sense, and that name is common to both subfile/subrepo. the download logic from the vendor scripts has been re-used for this purpose, and it verifies files using sha512sum. therefore: when specifying subrepo(git submodule), subhash will still be a sha1 checksum, but: when specifying subfile(file, e.g. tarball), subhash will be a sha512 checksum the logic for both (subrepo and subfile) is unified, and has this rule: subrepo* and subfile* must never *both* be declared. the actual configuration of coreboot crossgcc tarballs will be done in a follow-up commit. this commit simply modifies the code to accomodate this. over time, this feature could be used for many other files within source trees, and could perhaps be expanded to allow extracting source tarballs in leiu of git repositories, but the latter is not yet required and thus not implemented. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-08git.sh: remove unnecessary line breakLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: fix submodule pathLeah Rowe
i accidentally cloned to tmpdir rather than tmpgit oops! Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: simplify prep_submodules()Leah Rowe
copying the module list into tmpdir/ no longer makes sense, because it was only done before when we supported either running the list from "git submodule update", or module.list. since we only support handling of module.list, we can greatly simplify this function. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: unified handling of git clone/reset/amLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: simplify submodule handlingLeah Rowe
there were stragglers remaining, from when we used to actually run "git submodule update", but this was removed. clean up the submodule functions and merge them together. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: provide feedback for repository downloadsLeah Rowe
otherwise, it's not clear to the operator what's happening i'm normally against such verbose feedback, because it's bloat, but this minimal amount of feedback will make the build system more pleasant to use, especially during testing. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: download "depend" projects *before*Leah Rowe
don't do it after, because that means the main project is saved under src/ before we know whether the subrepo was downloaded. the "depend" variable (in config/git/) is no longer used for projects that go in subdirectories of a parent; now, we use config/submodules/ for this type of dependency. download the "depend" projects (as per config/git/) first. this way, if they fail, the main one will fail, but if they succeed and main fails, you can just run the main download again and it won't fail. this fixes a bug where, depending on how you download a set of projects and depending on the order which you do so, a given project can become un-downloadable on current design, because git will complain that a directory already exists. this fix is done not only in code (by this commit), but by prior configuration changes. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: reduced indentation in fetch_submoduleLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: reduced indentation in prep_submodulesLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: *never* run git submodule updateLeah Rowe
only use config/submodules/ which the build system then uses to run git clones manually, replicating the submodules feature. we must never use a project's own gitmodules feature, because we can't easily control it. better to let it break first, and then figure out what modules to add manually, so that we have only what we need for each project. it's done this way, because git's own submodules feature doesn't have very good error checking in general, nor does it have good redundancy. with the current design, we can declare backup repositories for each submodule. we replicate it precisely. for example: 3rdparty/vboot this is a coreboot submodule, and we handle that in the coreboot trees. however, our current design also allows you to do this even if the upstream repository does not contain a .gitmodules file Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-06git.sh: use singletree() to decide submodulesLeah Rowe
now it no longer hardcodes a check for whether the project name is coreboot. this maintains the same behaviour but will now work for other multi-tree projects; in practise, the other multi-tree projects did not use .gitmodules files anyway, but some of them used config/submodules/ in our build system. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-02make GRUB multi-tree and re-add xhci patchesLeah Rowe
Re-add xHCI only on haswell and broadwell machines, where they are needed. Otherwise, keep the same GRUB code. The xHCI patches were removed because they caused issues on Sandybridge-based Dell Latitude laptops. See: https://codeberg.org/libreboot/lbmk/issues/216 The issue was not reported elsewhere, including on the Haswell/Broadwell hardware where they are needed, but the build system could only build one version of GRUB. The older machines do not need xHCI patches, because they either do not have xHCI patches, or work (in GRUB) because they're in EHCI mode when running the payload. So, the problem is that we need the xHCI patches for GRUB on Haswell/Broadwell hardware, but the patches break Sandybridge hardware, and we only had the one build of GRUB. To mitigate this problem, the build system now supports building multiple revisions of GRUB, with different patches, and each given coreboot target can say which GRUB tree to use by setting this in target.cfg: grubtree="xhci" In the above example, the "xhci" tree would be used. Some generic GRUB config has been moved to config/data/grub/ and config/grub/ now looks like config/coreboot/ - also, the grub.cfg file (named "payload" in each tree) is copied to the GRUB source tree as ".config", then added to GRUB's memdisk in the same way, as grub.cfg. Several other design changes had to be made because of this: * grub.cfg in memdisk no longer automatically jumps to one in CBFS, but now shows a menuentry for it if available * Certain commands in script/trees are disabled for GRUB, such as *config make commands. * gnulib is now defined in config/submodule/grub/, instead of config/git/grub - and this mitigates an existing bug where downloading gnulib first would make grub no longer possible to download in lbmk. The coreboot option CONFIG_FINALIZE_USB_ROUTE_XHCI has been re-enabled on: Dell OptiPlex 9020 MT, Dell OptiPlex 9020 SFF, Lenovo ThinkPad T440p and Lenovo ThinkPad W541 - now USB should work again in GRUB. The GRUB payload has been re-enabled on HP EliteBook 820 G2. This change will enable per-board GRUB optimisation in the future. For example, we hardcode what partitions and LVMs GRUB scans because * is slow on ICH7-based machines, due to GRUB's design. On other machines, * is reasonably fast, for automatically enumerating the list of devices for boot. Use of * (and other wildcards) could enable our GRUB payload to automatically boot more distros, with minimal fuss. This can be done at a later date, in subsequent revisions. Signed-off-by: Leah Rowe <leah@libreboot.org>