summaryrefslogtreecommitdiff
path: root/include/git.sh
AgeCommit message (Collapse)Author
2024-07-18git.sh: warn when a cached clone failsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-18git.sh: fix typo in git commandLeah Rowe
the || : condition should be used, whereas i just wrote : by mistake. this was done in a previous change. fix it now. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-18git.sh: fix lack of error exits on faultLeah Rowe
a previous change made it more redundant, falling back on old behaviour (direct downloading, not cached), but the way it's done means that the function never returns an error condition in practise. this patch fixes it. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17unify caching of files into cache/Leah Rowe
hash/ becomes cache/hash/ repo/ becomes cache/repo/ Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17git.sh: remove previous tmprepo if args>5Leah Rowe
if doing a retry, the directory may still exist, which would make git clone yield an error response; the existing directory will have been the one that failed to reset, so let's delete it. the one deleted is not the cache (repo/PROJECT/), thus otherwise maintaining current behaviour. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17git.sh: try direct clone if cached git failsLeah Rowe
normally, a project is cached at repo/PROJECT/, and cloned from there to the final destination. errors lead to a calling of $err, but this will result in a return if done from inside a subshell, of non-zero value, so use this to re-try with a 6th argument when calling tmpclone(). in most cases, this fallback will never kick in, but it will kick in resetting or patching the cached clone fails; specifically, we are interested in the reset part. a given project name may change repositories in lbmk at a given time. if this happens, and the old one is cached, the overall result of this patch is that lbmk will fall back to the old behaviour, where git urls are tried directly, without caching. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17git.sh: re-try git pull three timesLeah Rowe
mitigate jittery internet connections Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17trees: auto-delete+auto-rebuild if project changesLeah Rowe
actual source code is not scanned, but config directories are scanned. simply get the checksum of each file under config/ pertaining to a given project/tree, and also for the given target. coreboot utilities are also handled. if it changes, in any way, delete and re-build automatically. such deletions should probably still be done manually, as part of understanding the build system, but this change should make the build system much easier to use during development. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17git.sh: don't download to src/project/project/Leah Rowe
re-use repo/project/ this means that single- and multi-tree projects now have a unified cached git repo location, as per the new rules, thus saving on disk space usage. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-17git.sh: cache git downloads to repo/Leah Rowe
do it based on the URL, e.g. https://review.coreboot.org/coreboot becomes repo/coreboot the downside is if you have two projects with repo urls specifying the same string at the end, but this isn't the case at the moment and likely won't be the case, but it's a theoretical issue. this saves on bandwidth when downloading identical submodule repos between multiple trees within the same multi-tree project for example, coreboot 3rdparty/vboot is no longer downloaded more than once, instead cloned locally on subsequent downloads. if repo/DIR exists, git-pull is attempted, but errors do not result in a non-zero exit, by design. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10trees: general code cleanupLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-07-10remove executable permission on include/Leah Rowe
files under include/ should never be executed directly Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: simpler for loop in git_am_patches()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: merge for loops in git_am_patches()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30trees and git.sh: tidy up global variablesLeah Rowe
some of the variables only initialised in git.sh are also used in the trees script, which is technically ok because git.sh is included from the trees script, but it makes more sense to declare them in the latter. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: simplified initialisation of "loc"Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: condense fetch_targets() a bitLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: short git_prep command in fetch_targets()Leah Rowe
loc is already set, and will correspond to the same path, so we can quite conveniently use it. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: merge prepare_new_tree with fetch_targetsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30git.sh: remove duplicate "xtree" variableLeah Rowe
it's already defined inside the trees script Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30trees, multi: download bare project *before* treesLeah Rowe
when downloading multi-tree projects, the rev can be reset to HEAD instead of the actual rev for a given target. this occurs when the bare repo (e.g. src/coreboot/coreboot) does not exist and has to be downloaded first. bare repository downloading does not rely on target.cfg, in this context, only pkg.cfg, but it uses the same variable names (e.g. "rev"). instead of using a separate variable name, thus increasing code complexity (which is the exact opposite of what i want to do), do the bare repository download first. this means that the git.sh script is much cleaner now, for multi-tree projects, in that it *only* copies the bare repo then runs git_prep; in that context, the bare repo is cloned directly by calling the relevant function from script/trees, which is the same behaviour as when cloning single-tree project sources. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-30trees: unified multi-tree configuration handlingLeah Rowe
the same function that loads configurations for single-tree projects has been merged with the function for multi-tree configs in git.sh, and that functionality has been removed from git.sh; now it is all unified in the trees script. as the saying goes: write one program to do one thing well. the purpose of git.sh is to download source code, but not to handle configuration files; the latter is meant to be handled by the trees script, which then calls into git.sh before running the build logic for that given project. additionally: the "seen" files are no longer handled, at all. the logic there was added ages ago, because at the time, i was considering whether to separate configuration into a new repository, so that users could more easily make their own configuration, so it was a guard against misconfiguration. however, that decision was canceled and we're always very careful not to introduce a loop; if a loop does occur, the worst that can possibly happen is you waste some CPU cycles. Instead, print (on standard output) what config file is being used, so the operator can see when an infinite loop occurs. ALSO: remove _setcfgarg in load_project_config() it was used to skip when a target.cfg file didn't exist, specifically on single-tree projects, but this is now handled using -f instead, on the while loop inside that function, so _setcfgarg is now a redundant variable. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29git.sh: rename Fetch_project_trees fetch_targetsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-29git.sh: rename fetch_project_repo to fetch_projectLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-28tidy up some setvars listsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-27remove use of _xm variable in the build systemLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-27git.sh: remove unnecessary checkLeah Rowe
the trees script itself will check that the directory exists, and exit with zero status if it does, without doing anything else other than the return. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-25lib.sh: simplify use of environment variablesLeah Rowe
don't have a separate variable for them. just export them directly and use them directly. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-22use backticks on eval commands, not subshellsLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-22lib.sh: more unified config handlingLeah Rowe
replace it with logic that simply uses "." to load files directly. for this, "vcfg" is added as a variable in coreboot target.cfg files, referring to a directory in config/vendor/ containing a file named pkg.cfg, and this file then contains the same variables as the erstwhile config/vendor/sources config/git files are now directories, also containing pkg.cfg files each with the same variables as before, such as repository link and commit hash this change results in a noticeable reduction in code complexity within the build system. unified reading of config files: new function setcfg() added to lib.sh setcfg checks if a config exists. if a 2nd argument is passed, it is used as a return value for eval, otherwise a string calling err is passed. setcfg output is passed through eval, to set strings based on config; eval must be used, so that the variables are set within the same scope, otherwise they'd be set within setcfg which could lead to some whacky results. there's still a bit more more to do, but this single change results in a substantial reduction in code complexity. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: revert modification to for loopLeah Rowe
i tried to be clever with this one, but it just made the script exit with an error. revert back to the old check (check whether one of either repo or repo backup is set) Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20minor code cleanup in the build systemLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: general code cleanup in fetch_submodule()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: reduced indentation on repo/file checkLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-20git.sh: simplified repo/backup checkLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-14unified checks for variable initialisationLeah Rowe
new function chkvars() does the job Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09minor code cleanup in the build systemLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: hide e() output on for loopLeah Rowe
this for loop is a hack to make sure that all the sources get nuked (using nuke.list files). hide the messages so that they do not appear when running just any command in the trees script. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: download xtree *before*, not afterLeah Rowe
downloading it after means that if an error occurs when downloading the xtree project, the main project will still be there and nothing will mandate the downloading of the xtree project. whereas, if we grab the xtree project first, then the main project won't get saved to src/ this makes the build system a bit more resilient under fault conditions, but otherwise doesn't change behaviour. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: fix deletion path in nuke()Leah Rowe
i accidentally forgot to include src/ in the prefix Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: further simplify nuke()Leah Rowe
it's a very compact nuke Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: simplify link_crossgcc()Leah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-09git.sh: simplify nuke()Leah Rowe
do not over-engineer such a trivial thing. seriously. all we're doing is nuking some files. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-08git.sh: support downloading *files* as submodulesLeah Rowe
when we download coreboot, we currently don't have a way to download crossgcc tarballs, so we rely on coreboot to do it, which means running the coreboot build system to do it; which means we don't get them in release archives, unless we add very hacky logic (which did exist and was removed). the problem with coreboot's build system is that it does not define backup links for each given tarball, instead relying on gnu.org exclusively, which seems OK at first because the gnu.org links actually return an HTTP 302 response leading to a random mirror, HOWEVER: the gnu.org 302 redirect often fails, and the download fails, causing an error. a mitigation for this has been to patch the coreboot build system to download directly from a single mirror that is reliable (in our case mirrorservice.org). while this mitigation mostly works, it's not redundant; the kent mirror is occasionally down too, and again we still have the problem of not being able to cleanly provide crossgcc tarballs inside release archives. do it in config/submodules, like so: module.list shall say the relative path of a given file, once downloaded, relative to the given source tree. module.cfg shall be re-used, in the same way as for git submodules, but: subfile="url" subfile_bkup="backup url" do this, instead of: subrepo="url" subrepo_bkup="backup url" example entries in module.list: util/crossgcc/tarballs/binutils-2.41.tar.xz util/crossgcc/tarballs/gcc-13.2.0.tar.xz util/crossgcc/tarballs/gmp-6.3.0.tar.xz util/crossgcc/tarballs/mpc-1.3.1.tar.gz util/crossgcc/tarballs/mpfr-4.2.1.tar.xz util/crossgcc/tarballs/nasm-2.16.01.tar.bz2 util/crossgcc/tarballs/R06_28_23.tar.gz the "subrev" variable (in module.cfg) has been renamed to "subhash", so that this makes sense, and that name is common to both subfile/subrepo. the download logic from the vendor scripts has been re-used for this purpose, and it verifies files using sha512sum. therefore: when specifying subrepo(git submodule), subhash will still be a sha1 checksum, but: when specifying subfile(file, e.g. tarball), subhash will be a sha512 checksum the logic for both (subrepo and subfile) is unified, and has this rule: subrepo* and subfile* must never *both* be declared. the actual configuration of coreboot crossgcc tarballs will be done in a follow-up commit. this commit simply modifies the code to accomodate this. over time, this feature could be used for many other files within source trees, and could perhaps be expanded to allow extracting source tarballs in leiu of git repositories, but the latter is not yet required and thus not implemented. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-08git.sh: remove unnecessary line breakLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: fix submodule pathLeah Rowe
i accidentally cloned to tmpdir rather than tmpgit oops! Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: simplify prep_submodules()Leah Rowe
copying the module list into tmpdir/ no longer makes sense, because it was only done before when we supported either running the list from "git submodule update", or module.list. since we only support handling of module.list, we can greatly simplify this function. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: unified handling of git clone/reset/amLeah Rowe
Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: simplify submodule handlingLeah Rowe
there were stragglers remaining, from when we used to actually run "git submodule update", but this was removed. clean up the submodule functions and merge them together. Signed-off-by: Leah Rowe <leah@libreboot.org>
2024-06-07git.sh: provide feedback for repository downloadsLeah Rowe
otherwise, it's not clear to the operator what's happening i'm normally against such verbose feedback, because it's bloat, but this minimal amount of feedback will make the build system more pleasant to use, especially during testing. Signed-off-by: Leah Rowe <leah@libreboot.org>