The Problem

As the complexity of our project has grown, developers have been experiencing problems with cloning and pulling our repository due to the time it takes to download all the git lfs content and their https connections timing out. While one answer is to use SSH, the setup is complex and this does not help speed up the download, neither does it allow developers to filter the lfs content based on their area of interest, e.g animators want the Maya files, character artists want the Zbrush files, concept artists want the photoshop files. Ideally, developers should only clone what they need to begin with.

Solutions

There are various options available to resolve the problems, some of those are:

option 1:

.gitconfig

  • .gitconfig file should live in the root of your repository.
  • Any user can clone it without worrying about what it’s set to.
  • Any git config settings overwrite any settings from the .gitconfig file.

option 2:

.lfsconfig

  • To prevent the repo from auto-fetching any lfs files add a .lfsconfig file in the repo with fetchexclude=*
  • Using git fetch –all or git fetch –recent will override the default pattern.
  • Use any of the lfs commands clone, fetch or pull with the -X flag and a pattern of files to exclude, or the -I flag to include.

option 3:

skip smudge

  • Install with the skip-smudge option [2] and then configure as required.

GIT_LFS_SKIP_SMUDGE=1 git clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

option 4:

git lfs clone

  • git lfs clone does a clone first without the lfs smudge filter (see option 3) then does a git lfs pull
  • The git lfs api is called less than with git clone for up to 10x improvement in clone times [3]

option 5:

git -c

  • git lfs clone does the following:

git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git
git lfs pull

  • There is the option to add extra options to the clone command or more permanent configuration changes before the git lfs pull.

Tests

In order to verify the results I created a new repo in the tcg.hackspace at https://bitbucket.org/TeaClipper/test_git_lfs_clone.git and then set out to solve a series of tests. The repo is setup to have 2 lfs tracked file types *.junk and *.huge, across the root, include/, exclude/ and misc/ folders. The files are empty data created using fsutil.

fsutil file createnew <filename> <length>

Test 1

clone all files

clone:

git clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

better:

git lfs clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

Test 2

clone without getting any large files

setup:

git config -f .lfsconfig lfs.fetchexclude “*”
git add .lfsconfig git commit -m “added .lfsconfig”
git push

clone:

git clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

no setup:

git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

Test 3

clone only lfs tracked files in include/

setup:

git config -f .lfsconfig lfs.fetchinclude “include/”
git add .lfsconfig
git commit -m “added .lfsconfig”
git push

clone:

git clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

no setup:

 git -c lfs.fetchinclude=”include/” lfs clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

better no setup:

git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git
git config lfs.fetchinclude “include/”
git lfs pull

Test 4

clone all but lfs tracked files in exclude/

setup:

git config -f .lfsconfig lfs.fetchexclude “exclude/”
git add .lfsconfig git commit -m “added .lfsconfig”
git push

clone:

git clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

no setup:

git -c lfs.fetchexclude=”exclude/” lfs clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

better no setup:

git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git
git config lfs.fetchexcluide “exclude/”
git lfs pull

Test 5

clone all but *.junk lfs tracked files

setup:

git config -f .lfsconfig lfs.fetchexclude “*.junk”
git add .lfsconfig git commit -m “added .lfsconfig”
git push

clone:

git clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

no setup:

git -c lfs.fetchexclude=”*.junk” lfs clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

better no setup:

git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git
git config lfs.fetchexcluide “*.junk”
git lfs pull

Test 6

clone all include except *.junk

setup:

git config -f .lfsconfig lfs.fetchinclude “include/”
git config -f .lfsconfig lfs.fetchexclude “*.junk”
git add .lfsconfig git commit -m “added .lfsconfig”
git push

clone:

git clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

no setup:

git -c lfs.fetchinclude=”include/” -c lfs.fetchexclude=”*.junk” clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

better no setup:

git -c lfs.fetchinclude=”include/” -c lfs.fetchexclude=”*.junk” lfs clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git

best no setup:

git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://bitbucket.org/TeaClipper/test_git_lfs_clone.git
git config lfs.fetchinclude “include/”
git config lfs.fetchexcluide “*.junk”
git lfs pull

Conclusion

In the end I’ve gone for a .lfsconfig option using sensible defaults, which should reduce the error rates across the team, the number of tech calls and downtime to restore local repositories to a workable state. The sensible defaults will also help with situations where we don’t have full control over cloning and pull options, such as Unity Cloud Build, where we can remove the unnecessary sync of source content.

Resources

[1] https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-install.1.ronn
[2] https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-config.5.ronn
[3] https://developer.atlassian.com/blog/2016/04/git-lfs-12-clone-faster/
[4] https://github.com/git-lfs/git-lfs/issues/227
[5] https://unity3d.com/services/cloud-build