Unity + Git, Friends Forever – Pt 4 : Recycling

Any developer or team ends up building a base of code or assets that they reuse in multiple projects. Your team (using Git and a workflow like I outlined in the last few posts) has made new stuff so quickly you don’t know what to do with it. While Unity makes it easy to drag and drop files between projects, it has no built in ability to manage these reused frameworks. As these frameworks become bigger or more important, you need to start managing changes to them more robustly. If you fix a bug while working on one project, your other projects using the same code should get fixed too. You should be able to revert to a past version of a framework without untangling it from the rest of your project. What might help with this? Why, Git, of course! By using Git’s submodules you can get all the benefits of a Git repository plus the ability to add that repo to any project.

 

Submodules aren’t perfect. They’re rather fragile little things and unlike most other Git operations, if you trip-up you often have to start over. But not to worry! Follow these simple steps and you’ll be sharing assets so much you won’t know what to do with them. As before, I’ll detail the process in both SourceTree and with the command line.

Collecting Assets

Possibly the most important part of the process. Deciding what assets to bundle-up into a shareable submodule. A submodule needs to be project-independent and self contained, so it can be dropped into a new project without causing errors or needing to be modified. Non-code assets are easy, they can generally be added to any project without issue. Scripts, however, often rely on scripts elsewhere in a project. A useful framework is one that is relied on, rather than relying on other code, so if you can separate a portion of code from the rest of your project easily its a pretty good sign that it should be a framework. In any case, collect all the assets you want for your submodule and place them in a single folder. To ensure that the assets are self contained, copy them into a fresh Unity project and check that there aren’t any errors.

Here’s my example submodule (a handy group of useful script tools), sitting in a new Unity project all by itself with no complaints.

Clean project

Note: make sure that the Unity project you’re making the submodule from is using meta-data and text assets (as detailed in Git setup), this makes sure that object links such as in prefabs or materials are shareable between projects.

Making the submodule

A submodule is just one Git repo within another, so making one is much the same as making a Git repo for your full game projects. The important difference is that you don’t add any Unity project files, just the asset folder you made in the last step. This is also why it’s important to put the folder with your future submodule into a new Unity project rather than creating it from an active project, so that Git doesn’t get angry when you try to make a new repo while still inside another.

SourceTree

Follow the steps as detailed in the first post, but instead of choosing the root folder of your Unity project, choose the asset folder that contains your submodule files. eg: MyUnityProject/Assets/MyAmazingSubmoduleFolder. Set up a remote and push to it.

Make submodule SourceTree

 

Command Line

The same as in part one, but make the git repo in the folder containing the submodule files, rather than the whole Unity project. Push to a new remote.

 

Adding a submodule to a project

This is where things start getting fiddly. While the steps are straightforward, simple mistakes like mistyping a path or forgetting a step can really mess things up (if only temporarily). If something goes wrong while fetching a submodule, it’s almost always easiest to delete all references to it and start over (detailed later)

When you add a submodule to a Git repo, you aren’t actually adding the submodule’s files. Instead, Git adds the submodule’s local path and URL to a file called  .gitmodules, and only stores the SHA-1 ID of a certain commit of the submodule to checkout (the tip of master by default). You then pull the actual files from that commit into your project.

Note: If you’re adding a submodule to a project that already contains the samefiles, make sure to delete the existing files before adding the submodule. As long as the files have the same .meta files, asset references shouldn’t be lost.

SourceTree

SourceTree is pretty good about managing submodules, you can add them to a project by right-clicking the sidebar, and choosing Add Submodule… In the panel that opens, enter the Git URL of the submodule’s repo, and the path that you want to keep it (Make sure it’s in Assets) I put all my submodules into a Assets/SharedCode. You can also type in the name of the branch you wish to check out, if not master.

sourcetree add submodule

sourcetree add submodule details

SourceTree will add the submodule to the repo, then clone it into the folder chosen. It may ask you for your username/password a couple times, as it tries to access the submodule repo. Once it’s done, you’ll notice your submodule appear in the left hand bar. You can double click the entries here to open up the submodule’s repo, and right-click to remove or change the URL.

Speaking of which, you’ll need to make one change if you want other people to use your project. Since a submodule is only saved as a URL and a commit ID, other users need to be able to access that URL to download the actual files. By default, SourceTree attaches your username to the front of submodule URLs, so other users won’t be able to access it without your password. Right-click the submodule you just added, and select Change Source URL… In the box that comes up, delete any username that may be at the front of it (anything after the https:// up to and including an @). Click OK to update the URL. The pictures below show the URL for one of my submodules, UtiliKit, before and after the URL was de-usernamed.

Note: Make sure to check this now, since if anyone tries to update their copy of the project, and a submodule URL is set to your username, it can be tricky to reverse.

sourcetree change url

sourcetree default url

sourcetree new url

Now that your new submodule is set up, you should commit. If you hadn’t made any other changes since the last commit, it should look something like the picture below.submodule sourcetree added

 

Command Line

Adding a submodule with the command line is a two step process. First, you need to add the address of the submodule to your repo. Do this with git submodule add submoduleURL localPath where submoduleURL is the address of your submodule’s repo, with no username included (so other users can use the same address). This should complete with no message.

Then you just need to initialise the submodule and update it. That is, download the needed files into your project. Use git submodule update –init –recursive. This goes through all submodules, initialises them if needed, then updates their files. The recursive flag makes sure that any submodules within these submodules are also updated. Git may ask for a username/password several times as it tries to access each submodule’s repo.

Once this completes successfully, commit and push the new submodule so others can access it.

command line add submodule

 

Change a submodule

Say while you’re working on Project 1, you add some needed feature to a submodule. Of course, you want Project 2 to have these changes too. After you commit the changes to the submodule and push, you’ll be able to update the referenced commit in Project 2 to point at the new commit. After committing and pushing this change in Project 2, every other user will be updated to the new version of the submodule when they pull.

A submodule is really just a link to a commit in another repo. This means that you can treat the submodule within as a standalone repo (you can commit, push, pull, branch, etc) and other users of your project will get the same changes. On the other hand, it means that if you commit changes to your submodule without pushing them, other users won’t be able to use your project, since the linked commit only exists on our computer.

Another thing to watch out for is that submodules are prone to getting into a detached head  state, which just means that they’re not on a branch. This is what happens to normal repos when you revert to a past commit. Submodules end up in this state when another user changes the commit reference and you pull down the changes. The submodule’s files will be updated to the new commit, but the local branch structure won’t change. Just make sure to either checkout the correct branch or make a new one before committing.

SourceTree

If you make changes to any files in a project’s submodule, SourceTree will prompt you to commit and push the submodule when you commit/push the project. This makes it pretty hard to break things for other users (though make sure you’re not on a detached head!) When you want to commit manually, you can open up the submodule like a normal repo. In the right hand panel under Submodules, look for your submodule and double click on it. A new window/tab will open on your submodule.

sourcetree submodule to update

If you’re on a detached head, the current (ticked) branch will be HEAD, as below. To fix, just click on the HEAD branch to select the commit it corresponds to (usually the latest). Then just checkout that commit (right-click and choose Checkout…).

SourceTree updated submodule

Sourcetree undetach head

You’re then to change the submodule, commit, and push to your hearts desire.

If the submodule was updated from another project, you can pull the changes just like a normal repo. Just checkout the branch you want to pull from and hit pull.

sourcetree upstream submodule changes

sourcetree updated submodule

Any changes to the referenced commit, be they to the local copy or just pulling down a different commit, will appear in the staging area. They need to be committed and pushed before other users of the master project will get the updated submodule.

source tree commit submodule update

 

Command Line

On the command line, submodules are even more similar to normal repos. To change a submodule, just cd to the submodules index. Any git commands will act on the submodule. You can commit and push local changes (make sure you’re not on a detached head!), or pull down changes from other projects.

After any changes are made, git will add the updated references to the staging area. Just commit -m ‘updated submodule’ and push to let other users get the same changes.

If you are one of these other users, you need to take one extra step to have your local submodule update itself. After running git pull as normal, you need to run git submodule update to have the submodules checkout the new commits(if changed).

Command line pull submodule changes

 

And that’s about it for the basics of submodules! They may be a little strange to grasp, but once you do you won’t be able to go back. No really, I’m a little worried about how reliant I am on them. If you have any questions or suggestions, let me know!

 

EDIT: I’ve made the example submodule in this post, Utilikit, open source. Use it to test your submoduling mastery, and feel free to use it in your own projects. Get it at https://bitbucket.org/eddiecameron/utilikit.git

Check back soon for the (hopefully) final part, on quickly starting new projects through forking.