Git Branching Workflows in SaaS Development and the Review ASAP Policy

In the last few years we have built up a software service from zero, which is reachable today all over the world with good response time, and we reached to a point where downtime almost never happens.

This is probably the reason why many people are asking me about what technologies they should choose, how to do continuous integration, how to write tests and about software development in general.

I always try to emphasise that the basis of all of these is a good workflow, which includes a good branching model if you use git. (And you should.) It means that throughout the software development process, everything should be based on git and git related events and tools.

In this chapter you will learn about how we use git to control every aspect of our software development process. I am going to show you the workflow which evolved within our company, which I can summarize in a set of rules.

I am also going to show you the tools which help us in our work or even enforces some rules which are essential to the workflow.

Git Concepts You Should Be Familiar With

To understand the rest of this post, you must be familiar with at least the basic concepts of git. There are many great guides and tutorials around, so learning it should not be a problem.

If you know the following git concepts, then you are ready to go on:

The concepts above are the very basics. The interesting part starts beyond the basics. One very cool feature in git is that you can create hooks on certain events on the client and on the server side.

We use two of the client side hooks: the pre-commit and the post-merge hook. On the pre-commit hook, we simply run our tests:

#!/bin/sh
exec npm test

It prevents developers from committing something which would break the tests. You have to copy the commands above into your repositories' pre-commit files: (your_repo/.git/hooks/pre-commit)

On the post-merge hook, we run npm install and npm prune when the package.json has changed. (Yes, we use npm and node.) It helps developers to always be up-do-date with the dependencies of the project. (You have to copy the contents below to: your_repo/.git/hooks/post-merge)

#/usr/bin/env bash

changed_files="$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD)"

check_run() {
    echo "$changed_files" | grep --quiet "$1" && eval "$2"
}

check_run package.json "npm install && npm prune"

If the post-merge hook above is set up, then every time when you pull a change from the origin and the package.json has changed, the dependencies will be updated automatically. I think it's a huge help. It happened many, many times that someone forgot to update the dependencies after pulling and could not figure out for a while why things don't work...

The drawback of the client side hooks is that you have to set it up for all of your repositories, which is quite cumbersome. After you set it up, everything will be much smoother. You can set up these hooks globally, but it only works when all of your projects are very similar. (Eg.: the test command has to be npm test in every case.)

Git hooks on the client side are great, but the greatest things happen when you start to take advantage of the server side hooks. You can hook up your git server with a CI tool, for example, Jenkins or CodeShip.

CI tools help you to run tests after every push, and you can automatically deploy your application when a push happens to a certain branch. You can be notified of each and every event on a Slack channel, email, Chrome notification or any other channel. It helps your team to stay up-to-date about the current changes. (So, for example, everybody will be notified when something is rolled out to production.)

Git Workflows

There are some problems with Git, for example sometimes people forget to pull before merge, and they can mess up a lot with conflicting merges, etc.

The best thing in git is that all of the branches are just branches; there are no distinct branches at all. Although it gives you freedom, it's also the biggest problem with git. Everyone can do whatever they want with branches without any restrictions. It is very, very easy to mess up everything by merging things you should not.

That is why we need conventions like that the master branch should reflect the latest stable version of the software. Besides the conventions, people started to come up with branching models (or workflows) to give developers guidelines how to create and merge branches.

The two most famous are the git workflow and the GitHub flow. Both of them are great, because they give you very definite guidelines about when you should create and merge branches. Also, they give specific meaning to certain branches which is essential to have a clear and understandable workflow.

Review As Frequently As Possible

In the last few years we tried a slightly modified version of the git workflow and the Github flow as well. Our biggest problem with both was that we had huge code review sessions which made it hard to release something. Even if you think you are ready, there is the code review... and if someone requests a lot of changes, then you need much more time than expected to roll out a new feature.

That is why we defined our own workflow which focuses on frequent code reviews.

In this set up you have two special branches:

The master branch reflects the actual stable state of the application. If you merge anything into it, then that change is automatically rolled out to our production environment in minutes.
The staging branch is a kind of "just before release" branch. Everything you merge into this branch is automatically rolled out into our staging environment. Our staging environment is exactly the same as the production environment, except it only contains dummy/test data. It's a great way to do a final inspection before releasing something into production.

The branches above are protected branches. It means that no one can push into them directly. You have to open a pull request if you want your changes to be applied on those branches. On top of that, we only accept pull requests to the master branch from the staging branch and hotfix branches.

A side note: We have many repositories which are libraries and used by many other repositories. These are typically published to (a private) npm and these repositories don't have a staging branch. It's because you'd need a staging npm environment environment for that and we did not want to tinker with it. Also, since you can set the exact version number of your dependencies, this kind of staging environment is not needed when we are talking about base libraries or any other npm packages.

We use the staging branch and staging environment in user-facing repositories where we use continuous deployment. It can be an API or a client side component. The common thing is that we want to have a production-like environment where we can check if everything goes well.

Besides our special branches, there are three types of branches:

feature branches: Whenever you start a new feature, you create a feature branch from the staging branch. You should prefix feature branches with feature-, so it will be self explanatory that it's a feature branch. When a feature is done, you have to open a pull request from the feature branch to the staging branch. If the reviewer approves and merges the changes, then you are happy. If not, then you have to satisfy the change requests.
task branches: Each and every task within the feature has to have a task branch. You create these branches from feature branches. When you are done with the task, you open a pull request to the feature branch. This way someone else can review the changes you made and when they think that everything is okay, they will merge them back to the feature branch. Task branches should be prefixed with task-.
hotfix branches: these branches are created based on the master branch, merged back to the master branch and then the master has to be merged into the staging branch. Hotfixes should be small fixes only which are not needed to review in the staging environment! Hotfix branches should be prefixed with hotfix-.

From the description above, you can see that we code review a lot. Actually it happens every time when you want to merge something back to its original branch. We do it through pull requests. That is the platform for code reviews and to discuss the details of the changes.

This workflow is a mix of the git workflow and the Github flow and it adds the concept of the task branches. It helps to do code reviews continuously and when you review a bigger feature, every change should already be familiar.

We use Github and ZenHub together, which gives us lot of tools which help us with our workflow. I will talk about them in a different post, so if you are interested, please sign up to our newsletter.

A Simple Example

All of the following examples start with the two special branches: master and staging. As you can see from the figure below, the staging was just merged into the master branch, so they are logically the same.

If you start clicking on the figure below, you will see the following steps:

someone creates a feature branch
creates a task branch from the feature branch
commits something to the task branch
sends a pull request to the feature branch; someone approves the changes and merges the task branch to the feature branch -> code review
after it's merged, someone starts to work on another task
commits several things into that task branch
and again when the task is done the task branch is merged through a pull request into the feature branch -> code review
when every feature related task is merged into the feature branch, then you can send a pull request to the staging branch -> code review
after it's merged into the staging and everything seems to be fine in the staging environment, staging can be merged into the master
now, the new feature is released!

Click on the figure above to see the steps.

Parallel Tasks Example

The previous example was very ideal, because there were no parallel tasks. In reality, many people can work on the same feature but they do different tasks.

The following example illustrates this scenario. It starts similar to the previous one, but two features are done in parallel.

Click on the figure above to see the steps.

Things get a little bit complicated when you want to merge (through a pull request) the second task into the feature branch, because the first task is already done and merged.

In this case you have to merge the feature branch into the task branch. You can resolve conflicts, and if you send a pull request to the feature branch, only the things you added will be shown in the diff.

Dependent Features Example

There are many cases when features are dependent on each other. In most of these cases they are only dependent on some part of the other feature. It means that the development of the second feature can be started before the first feature is merged into the staging branch.

Well, you might say that in these cases, the dependency should be merged into the staging branch, but in reality this scenario happens a lot. Sometimes it's only because people can't finish code reviews fast enough.

Anyways, the following example illustrates that the second feature is dependent on one of the tasks in the first feature.

Click on the figure above to see the steps.

You might have observed the same pattern which was illustrated in the previous example. If you open a pull request to a certain branch, first you have to merge all of the changes from that branch to your branch. This way you can ensure that the conflicts are resolved, so your pull request can be merged. The other important aspect is that every relevant change will be included in your branch.

This is a very simple rule to follow. If you combine it with the things mentioned previously, you will be able to build a workflow in your company where many things happen automatically, and huge errors will almost never happen.

Summary

In many cases when people have to review huge chunks of changes they tend to just approve it because of several reasons. One of them is that it can be very tiresome after a while, another is that there can be a pressure on the developer that they have to release the new feature very soon.

Smaller and more frequent code reviews are better, because the reviewer is able to do it in a more detailed way and it's easier to handle small chunks of changes.

This article described a workflow which was created with the "review ASAP policy" in mind. Hopefully the examples help you to understand the workflow better. Based on this workflow you can introduce continuous integration and automatic deployments, which can speed up the development process radically.

If you have any questions or ideas, please leave a comment below. In the following articles we are going to inspect other aspect of SaaS development, for example coding patterns, project management and tons of other interesting topics.