I’ve been spending a lot of time recently trying to get used to the new world of the virtual office. My knowledge of source control using Perforce or SVN and game engines such as idTech or Unreal Engine are being replaced by the flavours of the month technologies – Git and Unity. This transition has brought a lot of joys and a lot of new ground to cover as I get used to fully distributed source control, posix like tools and C# coding at a high level of abstraction, without being able to peak under the hood to the murky bits.
Along the way I seem to be hitting my head and knocking my knees of every bleeding edge there is. With that in mind, if I can save someone else some of the pain of my ignorance then it’s all been worth it. I’ve started working on a Unity template git repo available on the tcg.hackspace for ease of forking (the repo is currently setup to provide Unity meta file support but I will look to add default .gitignore and Git LFS attributes in the future).
There are a couple of setup steps that are a MUST for effective functioning of a Git repo and the hopes of not bursting the download limits of https and the hard timeouts of your cloud hosting provider.
Enable Git LFS from the start
Git Large File Storage (LFS) is essential for any kind of development that requires large or binary files such as images, videos or model data to be used as part of the build process. Git LFS allows these large files to be stored in a seperate content cache and only downloaded when required, reducing the size of the repository and making cloning and pulling of large projects manageable. github and bitbucket both offer Git LFS storage as part of their offerings.
- Go to https://git-lfs.github.com/ to download and install the Git LFS extension.
- Open git bash and type:git lfs install
- Make sure your .gitattributes has all the file types for big files you use in your project. Don’t know what they are? Then use something like windirstat on your project folder to find out. Here’s an example .gitattributes file to start out with:
proj_root_dir/.gitattributes
*.fbx filter=lfs diff=lfs merge=lfs -text
*.FBX filter=lfs diff=lfs merge=lfs -text
*.mb filter=lfs diff=lfs merge=lfs -text
*.ma filter=lfs diff=lfs merge=lfs -text
*.exr filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.tif filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.mov filter=lfs diff=lfs merge=lfs -text
*.avi filter=lfs diff=lfs merge=lfs -text
*.bnk filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
*.wem filter=lfs diff=lfs merge=lfs -text
*.akd filter=lfs diff=lfs merge=lfs -text
*.cache filter=lfs diff=lfs merge=lfs -text
*.dat filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bundle filter=lfs diff=lfs merge=lfs -text
*.pdb filter=lfs diff=lfs merge=lfs -text
*.exe filter=lfs diff=lfs merge=lfs -text
*.dll filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.chm filter=lfs diff=lfs merge=lfs -text
Don’t commit more than you need
If something doesn’t need to be in the Git repo then it probably shouldn’t be. Make sure you use .gitignore file properly to filter out files that need to be shared from those that don’t. Here’s an example .gitignore file to start out with:
proj_root_dir/.gitignore
# 3rd party files
#
# wwise intermediate files
**/.cache/
*.akd
#maya intermediate files
**/incrementalSave/
# Unity generated
#
Temp/**
Obj/**
UnityGenerated/**
Library/**
build_ios/**
build_android/**
build_html5/**
build_win/**
project_android/**
# Ignore: name DOT meta
*.meta
# Include: name DOT extension DOT meta
!*.*.meta
# Visual Studio / MonoDevelop generated
#
ExportedObj/
*.svd
*.userprefs
*.csproj
*.pidb
*.suo
*.sln
*.user
*.unityproj
*.booproj
# OS generated
#
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
Local/*
# misc
#
Set Unity’s asset serialisation TO ‘Force Text’
The second thing to mention is Unity Text Serialisation – Git has a very hard time handling binary assets so switch to ‘Force Tex’ mode to enable YAML encoding of Unity assets.
- Open the project folder in Unity
- Goto ‘Edit > Project Settings > Editor‘
- Check ‘Version Control > Mode‘ is set to ‘Visible Meta Files‘
- Check ‘Asset Serialization > Mode‘ is set to ‘Force Text‘
- Save everything and commit the changes to Git.
Don’t trip over Unity meta files
Forrest Smith has written about Unity meta files well enough that I don’t need to repeat his words here but do follow this advice to handle them properly in Git and save yourself a lot of headaches in trying to reconstruct your unity scenes from the ground up.
[UPDATE – turns out the 3pjames’ git hooks clash with the ignore folder meta files pattern, as such I’ve added the tcg.hackspace.unity-git repo instead]
Add the tcg.hackspace.unity-git repo as a submodule to your git repo:
$ git submodule add https://bitbucket.org/TeaClipper/tcg.hackspace.unity-git
Directory meta files should be ignored from the repo:
$ ./tcg.hackspace.unity-git/scripts/ignore-dir-meta-files.sh
I’d then recommend going into git bash and removing any old folder meta files before installing the git hooks:
$ git rm – cached *.meta
$ git add *.meta
$ git commit -m “[-] removed folder meta files from repository”
Install git hooks via git bash to check that assets and their corresponding meta files are kept in sync:
$ ./tcg.hackspace.unity-git/scripts/install-hooks.sh
The easy part
Now all you have to do is go and make a game! …using continuous integration, testing, deployment and analytics…
Did you intentionally leave out of .gitattributes *.prefab, *.asset and *.unity files for some reason?
Thank you!
Hi Amaltheia,
Thanks for your question. In terms of .gitattributes ommissions for *.prefab, *,asset and *.unity, there are a few factors behind the decision of whether to track files in Git LFS:
1. Are the files easy for Git to process?
2. Are the files small enough not to blow Git storage limits on our cloud service?
3. Are the files small enough that clone times won’t be overly affected?
In terms of file formats that Git can process, using Asset Serialisation ‘Force Text’, *.prefab, *.asset and *.unity files are stored in YAML which is human readable, highly structured and easy to diff, meaning Git can efficiently store any changes to the files.
In terms of file size relative to cloud limits, we are using Bitbucket (https://bitbucket.org/TeaClipper) which has a soft limit of 1GB and hard limit of 2GB per repo with additional Git LFS storage available. Taking a snapshot of my current Unity working directory – out of 8.3GB we have 49.9 MB *.assets, 8.8 MB *.unity, *.meta 8.5MB, 7 MB *.prefab., *.cs 5.1MB. These are projected to be more than manageable for our project size.
Are these files going to slow down a Git pull or clone excessively? The total size is acceptable for a clone. We aren’t seeing excessive turnover in scenes or assets, while the changes we do see to the YAML diffs are relatively small.
If we do reach excessively large text asset sizes then we might have to review storage and do a clean of the git history (see https://rtyley.github.io/bfg-repo-cleaner/) but that is not currently projected to be an issue over the project’s life time.
The files we really need to get in the Git LFS are our source content files, currently standing at 1.6GB *.wem, 765MB *.tif, 213 MB *.fbx, 179MB *.png and 119 MB *.psd. These files are projected to increase 10x or more over the project lifetime as we get more content coming on board.
I hope that helps.
N.B. All file sizes gathered by windirstat (https://windirstat.net/) – one of the first tools I install on any workstation.
Just ran across this. Thanks for the resource!
I did run into an issue though to be aware of. Unity Asset Bundles typically have no extension, so your system treats them as directories for the purpose of removing the .meta file, but then complains that it is missing the .meta file upon commit 🙂
Error: Missing meta file.
Asset
Assets/BundleData/BundleData' is added, butAssets/BundleData/BundleData.meta’ is not in the git index.$ ls -la Assets/BundleData
total 48
drwxr-xr-x 1 jwvan 197609 0 Mar 29 19:16 ./
drwxr-xr-x 1 jwvan 197609 0 Mar 29 19:17 ../
-rw-r–r– 1 jwvan 197609 1537 Jul 4 2018 BundleData
-rw-r–r– 1 jwvan 197609 190 Jul 4 2018 BundleData.manifest
-rw-r–r– 1 jwvan 197609 177 Jun 5 2017 BundleData.manifest.meta
-rw-r–r– 1 jwvan 197609 174 Jun 5 2017 BundleData.meta
Ah yes, that is a very good detail to mention. As with anything, there are often caveats and workarounds but hopefully this is a good 95% solution to common setup problems.