git lfs clone for submodules

[larsxschneider]

I realized that the git lfs clone speed up does not work with Git Submodules and I wonder if anyone has a clever idea to approach this problem. Here is my first wrapper script attempt:

git-lfs clone $@
if [[ -z $2 ]]; then
    CLONE_PATH=$(basename ${1%.git});
else
    CLONE_PATH=$2;
fi
pushd "$CLONE_PATH"
    git submodule init
    wt_prefix=$(git rev-parse --show-prefix)
    git submodule--helper list --prefix "$wt_prefix" | {
        while read mode sha1 stage sm_path
        do
            name=$(git submodule--helper name "$sm_path") || exit
            url=$(git config submodule."$name".url)
            if ! test -d "$sm_path"/.git && ! test -f "$sm_path"/.git
            then
                git submodule--helper clone --prefix "$wt_prefix" --path "$sm_path" --name "$name" --url "$url"
                pushd "$sm_path"
                    git -c filter.lfs.smudge=cat checkout -q $sha1 || exit
                    git-lfs pull || exit
                popd
            fi
        done
    }
popd

This works by taking the relevant pieces from the git submodule command. However, this is probably a pretty fragile solution. I would prefer to use higher level Submodule commands. @sinbad @stefanbeller any better idea or do you think this is the way to go?

[larsxschneider]

I started a discussion on the Git core mailing list in addition: http://thread.gmane.org/gmane.comp.version-control.git/292466

[sinbad]

Man, submodules always throw a spanner in the works.

In 1.2.1 the --recursive option is supported which presumably means the submodules are cloned but the -c options are not passed along, meaning the submodules work but don't benefit from the speed-up, is that correct?

To support a manual cascade like this we'd firstly have to catch & suppress the --recursive and --recurse-submodules options in the call git clone. After that wouldn't a git submodule update with the same -c filter.lfs... options work instead of manually configuring? If so the remaining issue would be doubly (or more) nested submodules meaning some recursion. I'd need to test this but would maybe be more portable/future proof.

[larsxschneider]

Your 1.2.1 --recursive statement is correct. From my point of view that would be OK for now as it is just slow but not broken.

Your "catch & suppress --recursive" solution could work but @peff pointed me to an easier solution. We could just pass the filter.lfs... git config on to the submodule checkout:
http://thread.gmane.org/gmane.comp.version-control.git/292466/focus=292563

I will try to prepare a patch for Git core. If this gets accepted then git lfs clone could support fast submodule clones starting with Git 2.9 easily...

[sinbad]

That sounds ideal, thanks @larsxschneider!

[larsxschneider]

Look what landed in next: git/git@89044ba

If this patch makes it into Git 2.9 then we can do this:

git -c filter.lfs.smudge= -c filter.lfs.required=false clone --recursive $URL $REPO_PATH
cd "$REPO_PATH"
git-lfs pull

git submodule--helper list |
while read mode sha1 stage sm_path
do
    if test -e "$sm_path"/.git
    then
        (
            cd "$sm_path"
            git-lfs pull
        )
    fi
done

Thanks to @gitster , @dscho , and @peff for making this work!

[sinbad]

Nice! Thanks so much for organising this.

[larsxschneider]

@sinbad: Do you have plans to update the git lfs clone code with the snipped above?

[sinbad]

Yeah, I was initially waiting to see if it made it into 2.9 and have been on vacation. I have some other transfer work to do first but it's on my list.

[sinbad]

Looking at this now, wouldn't it be better to use git submodule foreach --recursive instead of git submodule--helper list? That way nested submodules would work automatically.

[larsxschneider]

Nice idea! I haven't tested it but this should work, yes!

[sinbad]

Yeah, this is what I'm going with. PR soon.

[knight2010]

git lfs is not stable, but git submodule is much stable.

[moltob]

@larsxschneider Is it possible that git/git@89044ba is not effective on Windows in the below scenario?

LFS files are still downloaded during execution of:

git -c filter.lfs.smudge= -c filter.lfs.required=false submodule update --init

If I enter the submodule dir, the two options are again set to their original (LFS enabled) values. What is really weird: That even seems to happens after git lfs uninstall...

The only thing that seems effective is setting GIT_LFS_SKIP_SMUDGE=1.

(Version: 2.16.2.windows.1)