The Tool is the Easy Part – What about the Processes?
The FishEye team was the first team at Atlassian to make the switch to DVCS, and while some Atlassians had previous DVCS experience, quite a few had not yet used it in the workplace with a medium-size team of developers before. We looked for help around the web, but there wasn’t a lot of people sharing their experiences at the time. We found many resources like “How do I push a branch in Git?” or “How do I pull from multiple remotes in Mercurial?”, but we found few resources that explained “What is the best way for a 15-developer team to move from SVN to DVCS?”
Rather than diving in and prematurely using every DVCS feature out there, the FishEye team decided to keep our old SVN workflows until we got the hang of DVCS. After all, we were still trying to ship new features to our customers as fast as possible, and we couldn’t risk slowing down our development learning completely new workflows.
So, we continued to develop new code on trunk up until a couple of weeks before a release. We would then create a release branch to manage last minute features and bug fixes. Once we created our release branch, we would begin new development of the next release back on trunk. When a new version was ready for release, some poor developer was then assigned the time-consuming and painful task of merging the release branch back to trunk using SVN. If we needed any immediate fixes, it was normal to cherry-pick them rather than to merge the whole branch.
In practice, our development team could have one or more release branches (e.g 2.5 and 2.6) that they were maintaining at the same time with occasional bug fixes or crucial security fixes. In SVN, the easiest way to manage small bug fixes was often to make the same changes 2 or three times over these release branches. Occasionally, these bug fixes either would not get merged into our latest version, get misapplied, or get merged in once the actual code had been refactored, making it extra difficult to figure out where the fixes should be applied.
Copying the SVN workflow with Mercurial, we pushed all of our development on default – Mercurial’s version of trunk. We also strongly advised our developers against using any history modifying commands like histedit or rebaseuntil we were extremely familiar with the tool and the concepts that Mercurial provided.
The immediate downside of this workflow for a team of 14 active developers was that by the time somebody pushed his/her commits to our Bitbucket hosted master repository, it was highly likely that someone else had already pushed his/her own changes. This meant you had to hg pull; hg merge; hg commit to integrate their changes, then hg push and hope someone else hadn’t done the same before you! Generally, following our old SVN habits, people started off trying to push after every commit, meaning we had a lot of merges! There were several people in the team agitated that Mercurial created more work!
Feature Branches versus Continuous Integration
Luckily, branching and merging are much faster in DVCS than in SVN. We had already experienced this difference in the merges we were doing on default, and once we became more confident in our own DVCS abilities, we started doing more work on feature branches to solve our workflow problems. For those unfamiliar with feature branches, feature branches allow developer(s) to code features unhindered from the other changes going on around them. They allowed a R&D platform for our developers, so that they could create features without worrying about breaking code for their other teammates. They also meant less merging and fewer distractions every time another developer executed an svn update. The branch would stay active as the feature went through development, testing, code review, etc.
Feature branches provided an elegant solution for our development, but we ran into a big problem with our Bamboo builds. The builds for our default branch were no longer building against these feature branches; the devs had to run the full test suite manually or wait until they merged the feature branches back to default. Either workaround meant that the devs weren’t coding while they were waiting to reconcile the builds!
Before the advent of Bamboo Branch Builds, we actually cloned the default build for longer running feature branches, but Bamboo 4 has made this super-easy. We could now use a branch build for each feature branch that we have.
If you are not using a CI server with this capability, the approach we took before Bamboo 4 was released can help. We configured a clone of our main build to only run manually and took a build parameter for the branch field in the source repository configuration. This meant that without configuring a separate plan in Bamboo, we could build any branch we wanted to on demand (Note: in Mercurial, a commit hash also works, so it doesn’t even need to be the head of a branch). We actually still use this approach for smaller branches that need a one-off build. We can then link the build result onto the Crucible review so that reviewers can see that the code in question actually has a successful build before they approve the changes.
Lastly, we were not integrating our code until features were complete. However, Mercurial’s easy merging made this problem manageable. Each team would merge from default as often as they wanted to make sure they were integrating with the latest stable code on default.
Hardening Time
The weeks leading up to a release were a great time to let my inner dictator come out and play. We combined the above techniques to make sure we had a green build on our release branch as much of the time as possible. As the release manager, I would not let anyone else commit on or merge to the release branch. All work had to be done on feature branches, and I would only merge branches which had been reviewed in Crucible, had closed JIRA issues, and passed all their builds. To track whether a branch met each of these criteria, we wrote the FishEye Release Report, which could not only report the state of all the code between two releases, but could also show the status and linked JIRA issues of all the commits that would be merged from the feature branch.
To further ensure the success of a merge to the release branch, I mandated that all merges have no (even automatic) conflicts. If an hg merge feature-branch and hg resolve -l (which lists all the merged files, not just conflicting ones) reported any file level merge conflicts, the branch would go back to the developer, who would have to merge from the release branch himself and resubmit to Bamboo for a green build before I’d consider it again.
While it sounds time consuming for the developer, this process meant that every other developer who was working on a bug-fix could have absolute confidence on the integrity of the build system. Any build breakage on their feature branch could be attributed to their own code changes, as the release branch was always green when they started work. This actually saved a lot of time for the team as a whole, as it took a lot of the investigating out of the development workflow.
Since that release, we have evolved to a somewhat less dictatorial style where we try to mix a little more flexibility while maintaining the wins we gained from keeping a green build on the release branch. The FishEye Release Report and the review builds are crucial to this.
Bug Fix Merging Made Easy
Another problem we found easier to deal with in DVCS was the problem of bug fixing older releases and making sure those fixes got into every subsequent release. In our worst case scenario, we had two release branches and a development branch that we were maintaining (e.g. 2.6, 2.7 and default). Every fix for 2.6 had to get into the 2.7 branch as well as the default branch.
The easiest way to do this in DVCS is to not backport fixes to older releases (which often happened in SVN days), but to fix the bug in the oldest branch which needed the fix. Anyone who commits a bug fix feature branch to a release branch is then responsible for making sure that that release branch is merged into the development branch and every future release branch. This step is especially important for merging bug fixes between versions that were released three or four months apart, as the fix in question may differ quite considerably due to refactored code. And who better to determine how it should be merged than the developer who coded it. This process works much better than the SVN model, where we would either cherry-pick or leave it to the unlucky developer to fix every now and then!
But what about changes in a release branch that you don’t want in later releases? Won’t they get merged and cause problems? Our solution to this was the same as above — anyone who made a change to a release branch (like bumping a version number in a pom.xml file) was responsible for merging that change all the way down the chain to the developer branch. When developers merged, it was their responsibility to make sure they reverted the actual changes in the files in question, and then commit. This measure ensured that no one could ever merge these changes downstream accidentally, for according to the DVCS commit graph, they had already been merged!
Typically, this policy is an easy one to follow. In Mercurial, when merging what should just be one’s own work to a downstream release branch, one can check before executing the actual merge using the -P option to merge. Mercurial will list the actual commits a developer is about to merge and let him validate that he’s not merging anyone else’s code. If this is the case, he can go and grab the other person and get him to merge his commits first.
If you don’t use this model of managing release branches, DVCS has another solution for you. For a bug fix or other change that needs to go to two or more disparate branches, create a feature branch from the oldest common ancestor (Git and Mercurial both have one line commands in order to determine the oldest common ancestor). That way you can safely merge the bug fix to any or all of the target branches without merging any other commits.
DVCS FTW!
DVCS has definitely made us more productive – apart from giving us more flexibility in how we develop, we have evolved workflows that let us keep a higher quality in our release branches. DVCS also prevents our developers from getting distracted at crucial times by other work that is happening around them, while allowing them to control when they integrate their code and ensure that these integrations work. Lastly, merging across release branches ensures that each and every bug fix gets integrated into all subsequent releases.
Takeaways
- Use feature branches!
- Merge from your stable branch to your feature branches as often as you want to make sure you’re integrating with the latest stable code!
- Use Your CI server to make sure your feature branches are good before you merge to your stable branch
- Have a clear model of which active branches are “downstream” and always fix common bugs on the most upstream branch.
For more resources on DVCS, check out these links below: