Hervé Lourdin, Benoit Lafontaine
Transcript (Translated)
Hello everyone.
Hello.
Welcome. Ah, we're in the middle of... This isn't good.
Huh?
This isn't well done.
Pardon?
I'll show you this. So, we're doing quite a bit. There.
Woman, boss, boss.
It's not a big deal.
This, this worked.
Almost. Come on.
We'll do it like this. And here, we're beating the thing.
There, that's great.
Google is good.
Good.
Well, no, but no, it's coming back to me.
Okay, we're going to talk to you about... It's going to be a press event somewhat in continuity with those who were already in this room earlier, a bit more technical, less purely methodological, but still very connected to that, which will discuss versioning strategies and their impact on team dynamics. So we will be two to give this presentation.
Benoît Lafontaine, Technical Director at Octo Technologies.
And Arnaud Urbain, CTO at Vite Dressing.
So, why this session? No, we'll start right away. How did we come up with this idea?
Do you know who this is?
Who is it?
That's right, that's what she said, it's Melvin Conway. So this gentleman, in 1968, had a great idea, a big revelation, he formalized his law. which is as follows, stating that organizations that create systems, let's say software, to make it easier for us, are constrained to produce systems that copy or reproduce their communication systems. So if we simplify all these words a bit, if we make it less complicated, we often say, in short, organizations replicate, well, we find our organizations in the structures, in the design of the code we produce. In any case, our organization has a strong impact on the way we write our code. So this has inspired us a lot, it has inspired many people. And it has led to quite a few reflections, in particular we have understood well that the way we organize ourselves when we do a project, when we create several agile teams, was going to have a fairly direct impact on the way we would write our code, on our time to market. And so, as a result, we really tried to chase after, to find the most elegant fault lines between our teams, to constitute
so-called cross-functional teams, in which we will rather find all the business functions, from product to development, testing, to production, to avoid creating these too strong silos which will have an impact on the very way we design our code. Well, this gave us some various and varied ideas. Benoît and I said to ourselves, but in fact, these stories of future teams, component teams, etc., you may have heard of them, it's good, but there must also be other things that influence the way we work, in the speed at which we produce software and the way we communicate together. And we think that versioning strategies, code management, probably also have an impact on the way we communicate. So here, it's a bit the other way around. And we said to ourselves, well, let's investigate this path. We don't really know what we're going to find. But we think it must have some impact. So, very humbly, we reached out to people we know or meet. So we went to meet many people we knew in our circle or with whom we had the chance to work.
And who have worked on quite a few different projects, in different companies. So we didn't study in detail all the practices of each company, but each time we learned from the projects we heard about, on which people had worked. So these are small excerpts from different projects of the companies. And we may not go into detail about each company, we won't necessarily name them, because sometimes we don't have the right to name them, but we tried to look at a fairly wide range of projects.
What is quite interesting is that it combined our little links from the web, mobile, quite varied technologies. As a result, we had a panel that we find... representative, we want to continue, we learned things. And that's what we propose to share with you, to look at what we learned and share it with you.
But to start, what is a branch? So that's the first question, if we want to talk about branching, versioning, echo, and all these barbaric, awful words, what could they possibly be?
So, being well-mannered, we'll first look at Wikipedia. So on Wikipedia, it says the following. In software configuration management, a branch is a divergence in the history of the evolution of configuration items. A branch is an evolution of an item originating from a specific version, producing a version branch. A version branch corresponds to an axis of version evolution. It is attached to a source branch and can branch into several sub-branches. The management of all branches and versions of a product constitutes versioning and is the subject of configuration management. Okay. In progress, a branch is this. So, there you have several versions that will follow one another. Then, at a given moment in the life of your project, for various and sundry reasons, your code will diverge into another version, an alternative version. So here, we took the case where, at a given moment, we started wanting to prepare for the future and then work on a version 2.1+. And we made a branch.
So, if you look on the web for what branches are, there are endless debates about what they are, how they work, what should be done, etc.
We will try to address the different branching models that exist, because we will see that it's not necessarily just that. So, Benoît and I... Arnaud has another clean definition of branches, which is to say a branch is an alternative version of the source code of a software.
Very simple. At a given moment, the software has an alternative version of itself, enhanced or modified, or whatever, and for the simplicity of this presentation, we will agree that this is what a branch is. Why? Because we will see the very formal branches as we just saw them, but there are other types of branches. There are branches that we will call implicit, which are for example a version of the software that lives locally on a developer's workstation, but which has not yet met another version, which has not yet been confronted with a version on a centralized server or with that of another developer. And then there are still other types of branches which are those much more tricky ones, which we will discuss a bit more in detail, so we will call them branches by abstraction. So exploiting an alternative version of the software in production. So two alternative versions of the software will live at a given moment. So you can start thinking about different use cases for this, we'll see them just after. But there you have it, just to tell you that in the end, it's not just a branch that we pulled in an SVN or Git server, it can be other things. So we will focus on saying that a branch is an alternative version of the code. Thank you.
So, second question, why do we branch actually? Why do we do that? Not just because we feel like complicating our lives. Presumably, most people who have been led to do this all answer for the same reasons. The first is because, after all, we would like to be able to work a bit quietly and not have to bother with the version of the colleague who is doing something different from me.
That's the first reason. And then often, when we start digging, another reason is simply, I would like to be able to deliver things gradually, without having to bother taking into account what
my neighbor is doing, but he hasn't finished, so I won't wait for him, I'll be able to send it right away. And so, as a result, this is one of the major reasons that also lead us to want to create this alternative version of the code in a slightly more isolated place.
The problem with branches is that when you talk about branches, it means that at some point, you will have to merge. And that is the big moment when everyone meets. It's the moment when we say, okay, there are alternative versions coexisting together, and we will have to make them meet. And that is the most important moment. So we will talk a lot about branches, but we will also talk a lot about merging, meaning making the two alternative versions meet to end up with only one, the one that will be executed first.
Once we have shared this vocabulary, we can continue. And we will look at the different versioning/branching models in the broadest sense that we have encountered with these different players, and what impact they have on communication.
First of all, I will present to you what we call the trunk. Ultimately, it is a branching strategy that aims to avoid branches, precisely. In fact, we will try to work as much as possible on the latest up-to-date version of the code, so sharing everything. Concretely, that's it. So if we have two developers working together, I'll take two for simplicity, we have a first version, and then developer B makes a few commits, or rather makes some changes, and will push them to the central master. At that point, the other developer and the other developers must retrieve this version to fix it locally, and to be able to work on their side again, and to push their development later. What is interesting to see is that, at some point, we will meet to say, 'Well, we would like to release this to production, or at least show it somewhere.' And how does this work in teams that often use this technique? It's that, in the end, you raise your head and say, 'I would like to push, are we all agreed that it works?' 'And then everyone looks at each other and says, 'Yeah, I'm good, I'm done, or it's a rather stable state, so we can go ahead.' Maybe someone says, 'Oh no, wait a bit, because I just pushed something that doesn't necessarily work, so wait for me, I'll commit in an hour and then we'll start again.' So, we are talking about communication. Ultimately, what we see with this pattern is that it is often quite small teams, 10 people or fewer. Sometimes more, but most of the time that's it, who are generally co-located, or at least who communicate very easily with each other, to be able to resolve all these conflicts that they have quite regularly, quickly and in a fairly collegial way.
Most of the time, this is called trunk-based, which comes more from the SVN repository, and afterward, in the slides, we often see 'master,' which is more the Git definition, but otherwise, it's the same thing, it's really having a continuous flow of commits, and everyone, as development progresses, tries to push. So what we also see in teams is that, necessarily, since we have continuous integration, and we don't want to bother everyone, developers still force themselves to push changes that are relatively stable, and in any case, that do not break all the unit tests, otherwise it bothers everyone and everyone is blocked. Which is why Hervé was talking about implicit branches. If we have a typical developer who does a big task and never commits, we have a branch that starts to last over time on their local machine. No matter what version control system we are on, it's the same.
Experience with this kind of skidding is that it actually works if it's a small team that communicates a lot, or we've seen it in contexts where there were many, many people, but it worked because there was a test suite that was extremely developed with tests in all directions, where the overall goal was to never, ever, ever break it. And so, as a result, one, having tooling, and two, having extremely fine-grained commits, to succeed in integrating into the flow. So it doesn't seem very simple, and avoiding branches at all costs, in a way, but it has a consequence. It works well with not many, or with many, you need to have execution rigor in the size of the commits and in the way of... highly advanced build tooling.
Another thing is that all projects start with this. I haven't yet seen a project that starts with a more elaborate strategy than this. But on the other hand, after a while, and this is typical, we have an internal project at Octo that does this, it's that for a very long time, they were three, it worked well. And then, they meet other developers, or rather, other developers join the team. And then they say, ah, we no longer have the same trust we had with our current team. And as a result, they set up other branching strategies.
So, it's a bit of a story, it's that at some point we say, well, it would be good if we followed our own path for a while before meeting again.
The first pattern, which is rather an anti-pattern in the end that we have seen, is ultimately trying to isolate developers from each other or teams of developers. So the idea is to say, we still have a lot of things to do, there are two teams that keep stepping on each other's toes, so we will... isolate them on two different branches, so they can live their lives rather peacefully, and we will meet at the next release, so we're talking about cycles usually of several months, that's more what we see, we try to meet in several months, and then at that point we will do this merge step, code fusion, and put it into production. So that's more the theory, because in practice what happens is rather this. Well, first of all, those two months slipped because they went over to 3-4 months, and at some point, there is still a team that manages somehow to have its functional scope covered, and to have something rather stable. But there is the other team that, maybe it was roughly stable a little before, but now it has started development and so it is in a somewhat complicated situation. So maybe it hasn't finished. But in addition, when it tries to merge its code with the other team, team B, it doesn't work. So they say, ultimately, it's a bit complicated to finalize, it's a bit complicated to merge, never mind, we'll postpone our integration date to next time.
So already, no one is very happy, but that's what happens. Next time, so 2, 3, 4 months later, or more, we still have to do this merge, because both teams have still reached the end of their functional scope. And then, it's really the merge of death, meaning that when you have two alternative versions of code that have diverged for several months, it becomes very, very complicated. And especially since no one...
Everyone was focused on their functional scope. Ultimately, no one had taken into account the cost of the merge afterward. And every time, maybe there are developers from time to time who come and say, it would still be good to merge. Every time, the organization often says, 'We'll see about that later, you're wasting time now.' A bit like the reflections around TDD that we saw a little earlier. But often that's it. So when you see people starting to go for team branching, fortunately it happens less and less, it remains an anti-pattern because we will never be able to remerge the teams.
It's a continuation of trunk-based. In fact, we encountered a context where there was user branching. In the end, no one had taken into account the cost of the merge afterward. And every time that... Maybe there are developers from time to time who say, 'Ah, it might still be good to merge.' Every time, the organization often says, 'No, we'll see about that later.' Right now, you're wasting time. A bit like the reflections around TDD that we saw a little earlier. But often, that's how it is. So, when you see people starting to use team branching, fortunately, it happens less and less, but it remains an anti-pattern because we'll never be able to re-merge the teams.
And another...
It's a somewhat alternative branch that we encountered.
It's a continuation of trunk-based development. In fact, we encountered a context where there was user branching. So what does that mean? It means that at a given moment, all users are pulling a version of the trunk, of the master. They have it locally. They don't create any branch. They live with it for a very long time. And in fact, they will create another repo, another master, but which will be a team master. And they will play a game of individual layers toward this team master. So, it's not officially a branch, but it's as if, in fact, we had decided to fork the master for a given team, to integrate together within this team repo to be able to deploy it in an alternative environment. And exactly for the same reasons that Benoît described, the teams will continue to work together around this repo space, which is a kind of virtual team branch. And then at some point, the two teams or three or N that need to go into production together to make a single version of the software will try to meet and do the merge. So it's a way of branching, of team branching, that is a bit more implicit but is based on this. In this case, this kind of not-so-great pattern works remarkably well with technologies like the ones mentioned. Because as a result, you can really live your life with the duplicate of the repo, do push-pull between the repos, between the users, and then manage to find a stable team version. Deploy it on this small environment, small repo of team repositories, which will later become a team integration server. And what does this tell us about team dynamics? It's that, in fact, it's a very vicious pattern because the team itself doesn't realize it's taking risks because within the team, communication is generally rather good, the angels talk to each other, they commit around their team branch, they live their life. And the problem is when teams A, B, X, Y, Z want to communicate with each other. And that's when the murder of death will happen, as it's called. So, it's a vicious communication pattern. We feel good, we're safe, we communicate very well within the team. However, cross-team communication is poor. And yet, we can have very agile teams. We need to have teams that live their own lives. And at some point, these teams A, B, and C will have to come together.
So, ultimately, what are we trying to... The other, at least the other general branching pattern, which will also be the basis for the slightly more complex ones we'll see later, is the future branch. On paper, it looks roughly like a team branch, except that instead of separating by people and teams, we separate by features. So there's a small team that takes charge, either one person or several, but who takes charge of this feature and works on it.
Same on the other side, and at some point, as before, once we're done, and that's where it's interesting, we didn't try to strictly limit the time. We limited it around the feature. This morning in the keynote, he was also talking about cadence. Here, we manage to have cadence around the feature.
So once we have this feature finished, we can push it, at least test it, validate it, push it to production or before. And during that time, we have another feature that was living its life, which, just when feature B arrives in production or on the master, feature 1 must do this merge of this feature and is obliged, well, that's still good practice, to do it, to merge from the master to be able to rebase afterward. Why is this possible in this case and not possible, or at least we see it less often with team branching, is that again, we have cadence around the feature and we can ensure that our organization produces small-sized features. Earlier I was talking about it often being several months, at least what we've seen is several months of development, here we're talking about a few days or two weeks maximum. That's roughly the timeframes we expect. And so as long as we manage to have these branches of a slightly shorter duration, it works. So my example from earlier where there was a small team of three people who integrated others, the idea is really this, it's to say, typically there's the red developer over there who just joined, because they don't yet know all the development standards, who often when they commit, breaks the continuous integration. So we need to... still accompany them a little. And so, having these feature branches allows us to isolate the different developments. And as soon as we have a feature that's OK, we can push it to production. And we accompany the other developer to integrate the best practices, correct, etc.
What's good with this pattern is that it's not a technical pattern at all, because in the end, we see that we're very close to having done the same thing as in the previous example. It's more about how you're going to do your sprint planning, or if you're guided by directives, your user stories, with what level of detail you're going to break them down, and how you're going to, among developers, divide the work. You use the branching system, a technical solution as rudimentary as team branching, there's nothing super advanced behind it. However, the constraint is that you're only allowed to deviate from the common trunk at the time scale of a functionality. Ideally, a user story, if we want to take it a bit further. So here, it's not the technique that will dictate this model, but rather the methodological and team communication practices and work sharing that will support this pattern. And for that matter, it's the most basic shelter we'll find later in most of the others.
And to finish on this pattern, the advantage is that afterward, we can rotate teams and ensure dynamics and communication between people. Whereas earlier we were ultimately freezing the teams.
So we move right on to a big piece which is ultimately the Git Flow. Why do we go directly to it? Because in fact, the person who put this on paper didn't invent anything, they just put together a lot of good practices that were used on different projects. The advantage is that here, we have a global view. So what is it? It starts from the observation that on the master, we only have the reflection of production, so only complete deliveries, and we have a branch called develop on which we make all our code changes as we go along.
So we use exactly the same pattern as the future branch on this side. As soon as we have a story, we create a branch, code on it, and bring it back to this develop branch.
At that point, it's when the team says, now that we have a functional scope that holds together roughly, it would be good to put it into production. Except that most of the time, at that point, it takes a few weeks, a few months. So we have an integration and fine-tuning phase. Manual tests to verify that everything is fine and that we haven't forgotten any gaps. So we create a release branch that allows part of the team to fine-tune this release, make the final adjustments, while other developers start the releases for the next version or continue features that would take a little longer. So we have this release branch, and once the team is happy, it's put into production. And the cycle continues. And we have a final branch which is hotfix, because so far I've talked about 1, 2, 3 months, but in real life, there are bugs regularly in production, depending on your... Depending on production, but in real life, there are some from time to time. And so, we have a special branch to say, here, I'm fixing the bug, I stop everything, I fix a bug, I create this branch on the side, and I push directly to production, and of course, bringing back the modification to our develop branch. So that's quite comprehensive. If you're looking for a base to cover almost all possible cases, it's not bad, and that's why there's quite good adoption in teams. What's interesting to see, there are two things.
These practices were historically established, often by the products we delivered. Physically on CD, etc. Not so much for the web, because we have these release branches that can be quite strong, quite marked.
Nowadays, there's a lot of web, so sometimes we wonder, what's the point of release branches? There's still one case we see, and every time, they use this pattern, it's ultimately everything related to mobile. Because even in mobile, even if we want to deliver continuously, we're limited by the stores and the teams are limited by the stores. Or for example, every time we release a new version on the Apple Store, your ratings go back to zero. As a result, to keep decent ratings on the Apple Store, we're forced to have a timeframe of about two months. On the store, well, that's what's generally practiced. So force the release cycle to two months. So even if the team is quite mature, quite structured, etc. To have small features that are developed every day and that integrate into this development branch very regularly, they force the use of release branches to precisely pace according to their intentions.
What's interesting is that... In fact, if we say to ourselves, release branches—maybe we don’t need them because we manage to have small enough features and our organization tests these features as we go. And ultimately, as soon as we merge into Develop, we deploy to production. This is what we’ve seen with some teams that went down this path and said, 'Hey, why don’t we just deploy our features directly to production?' So, they end up removing this release branch because they’ve managed to integrate and refine releases during that time. They’ve succeeded in incorporating these actions into their feature and feature development cycle. So once release branches are removed, there are other teams that say, 'Well, actually, hotfixes and releases are basically the same thing.' Because I calibrate my features to take a few days—usually a few days—while hotfixes take half a day, so in the end, it’s pretty much the same. And so, in this dynamic, Hotfix and Feature Branch are the same thing. So ultimately, we removed those three because Develop becomes the Master. And in fact, we return to a classic case of Feature Branch or GitHub Flow, which we’ll look at next.
The interesting thing about GitFlow is that it’s usually not small teams that start using it. It’s often that teams start hitting limits in how they synchronize with other agile teams. I’m an agile team, I mean, I’ve started doing feature branching, or I’ve stayed within a branch, so why not. Then after a while, we need to synchronize. And sometimes, for reasons that aren’t even technical—just marketing—we need to introduce the concept of a release because we’re going to launch cross-team features simultaneously for an announcement effect. And that’s why we start looking into processes like this one. Otherwise, it’s often too sophisticated, a bit too complicated to consider at first glance. In fact, what we’ve observed is that teams needing synchronization—those working in agile with multiple teams—start looking at a pattern like GitFlow to work together.
Being able to remove the release branch or the hotfix branch is generally a sign of high maturity because it means the team is very likely moving toward continuous delivery. So they have testing dynamics, they’re able to push fresh code very frequently without fear of breaking the developments they’re working on at that moment.
The story Benoît just told applies as soon as a team feels comfortable with its versioning models and has advanced practices in build and automated testing. However, that’s it. It allows everyone to understand it fairly well because, while this model is a bit dense, it’s still quite simple to grasp, and it brings teams together, though they can’t always synchronize. However, we advise against adopting this from scratch if you’re starting a new project without having experimented with simpler models first, because otherwise, it can become a bit of a factory. The funny thing about this model is that if, instead of looking at it horizontally, you view it vertically, some interesting things appear. What I like about it is that you can start to see that feature branches actually have a progression—these are all the features I’m working on. So we have an indicator: the more feature branches I have, the more likely I’m dealing with a significant WIP, but overall, it shows me what I’m working on at any given moment.
If you look, the development branch is probably your baseline. It means you’ve started reconciling your code within the team’s common trunk. So if your definitions of done are well established, you end up in the development branch. And then, if you have subsequent AT phases where you thoroughly test all your developments to see if everything is functionally consistent, that resembles a release branch. And finally, live. So we find it quite interesting that if your agile process replicates this, it’s a model—a GitFlow model—that’s quite interesting to implement. And additionally, if you have expedited fixes, you can always push them through a hotfix branch, which also elegantly resembles what you’d do in a well-followed Kanban model.
Another pattern we’ve seen that’s quite trendy right now, Before discussing the GitHub Flow model, I’ll first talk about what GitHub brought to Git.
Because, well, not widely used in open source at first, today it’s become the big reference for all open-source projects—and not just those. But what they fundamentally brought that was different is the fork. It’s this ability to say, there’s a project I see—probably open source—
With a single click, I’ve cloned this repo, and I can work on improving it. Either with the goal of dethroning the king and becoming the new reference, but more likely to improve the project.
And contribute to it. And behind the fork, as we said earlier, after the fork comes the merge. And what GitHub also introduced is this unique feature of having pull requests and the review process. So the pull request in GitHub is: I’ve forked a project, I think I’ve significantly improved it, and so I ask the repository owner So either with the aim of dethroning the king and becoming the new reference, but probably more to improve
this project and contribute to it. And behind the fork, as we mentioned earlier, after the fork comes the merge. And what GitHub also brought is this particular feature of having pull requests and this review process. So the pull request in GitHub is: I've done my fork, I think I've significantly improved a project, and so I ask the repository owner
to integrate my code into theirs, because it's really great. And then, the project maintainer sees all our modifications, so they have small tools like this to highlight and see what has been changed, and most importantly, can start a discussion. And that is really the important part. If we wonder why the person did it, or if we have comments about their code, the standards aren't met, there aren't enough tests, etc., we can communicate with them and exchange ideas.
To eventually complete this merge step at the end. And in the end, GitHub, like many other solution providers, created their own workflow. And GitHub Flow, ultimately, is... That's it. So it's exactly the same as a feature branch, except we have this pull request step where we ask someone to integrate their code into the master. And most importantly, we have this code review part.
So we see this on all open-source projects on GitHub or GitLab or Bitbucket, but also what we see in teams, even small ones for that matter, who say that code review is still an important part of our activity, we're not doing it well enough yet, so we're adopting this tool and this workflow, to ultimately enforce this code review, this exchange among us about the code to take ownership of it, etc. Before finally pushing it to the master.
The funny thing is that originally, it was really GitHub that set the trend through open source, but now most private companies are starting to implement it internally. So, either they take a private account on GitHub, or they set up their own servers. So, there's Atlassian, the makers of Jira, who have Bitbucket Server, or GitLab, which has an open-source project that is also hosted. And they integrate this within the company to get everyone to adopt this exchange model. And this is quite new because before, you only really saw it in the open-source version, since people are normally very distant. But now within companies, we're replicating this open-source pattern that fosters communication at the time of the merge. which makes the tool incredible, a kind of global forum around a file. Well, it's not a technological revolution. But on the other hand, we have a developer exchange convention that encourages discussion before performing this merge. So for distributed teams, it works really well. But not only that. Whereas before, two people could arrange a merge between themselves, now everyone sees what's happening, the conversation with colleagues. And we permanently replicate this pattern that was rather... Reserved for the open-source world.
And that's really it, because there's a small team at Octo, where I worked, where before we were trunk-based, and code reviews were part of our process, but we did them when we could, so to speak, and afterward, we had to untangle all the commits since everyone had committed on top of each other. So it was a bit more complicated. And so now, what this team has done is that they've set up these arcane tools and fabricators, but the idea is the same: they do their development on their own branch, and then they enforce the code review step. On their side.
So it's really to enforce this communication. So we were talking about communication and branch strategy. Here, the really important thing is that by highlighting this pull request and code review, we force this communication. A somewhat advanced pattern for people not far from here, the Ferrets.
So they do something quite complex that we'll stay at a high level on, but very interesting. Because in fact, they do...
Ultimately, they do feature branches. I'm not giving you a history lesson, but roughly, they used to have a release every month, and they thought, every month is too long. And so, they tried to speed that up. So, they automated a lot of things. And then they set up a process around feature branches that is a bit complex. So here I shamelessly copied their DevOps slide that we can't see, but it's okay. But it's online with them, no problem. What happens with them is that they wanted to keep the advantage of feature branches to be able to deploy branches individually. But at the same time, they wanted to keep a single place where there is always the latest version of all branches simultaneously to manually test this integration. So how did they do that? So they take the Master Branch,
I have a feature Ticket 3 to develop, I create this branch, I work on it, and at some point, I push my Ticket 3 branch to a central repository called Features, which contains all the features currently in development. And then, there's a pretty funny automaton that takes all these features under development, regardless of their state, whether they're finished or not, and performs this big merge, which is a bit of a unique, temporary merge that only exists to be there, and pushes it to the integration environment. This allows all developers and testers to have this latest version in integration and to continuously test this integrated version in an integration environment. Once the developer and the team responsible for the tickets are okay, they decide it goes to production, through another action that is the same.
Which pushes these features, these feature branches, to another repository. And then, there's the same automaton that ultimately performs this merge, but only of validated tickets, to send it to production and bring this code back to the master branch. So the last arrow you see there. So this is quite clever, quite nice.
There are still two things for which it works pretty well. It's that their organization also means that tickets are relatively separate. And so there are quite few conflicts between tickets. And the magic of Octopus is that when there are conflicts, developers are alerted because the automaton can't guess them on its own. So developers are alerted that there's a conflict, for example, between ticket 2 and ticket 3, resolve the conflict in Git like any conflict resolution, and push it to this automaton. And this automaton, when it redoes this integration, sees ticket 2, ticket 3, conflict, oh, I see a developer has already resolved this conflict, so it applies this resolution. And so they always manage to have this consolidated version at any time. But again, it's really interesting because even if technically they manage to resolve conflicts, their organization already has teams that are separate and work mostly on separate components. So these teams inherently have very few conflicts, and the teams work with a relatively reduced WIP, so there are quite few tickets that can conflict with each other. In fact, if we step back on the combination of practices they have, they actually do two interesting things.
They say, we're going to feature branch in a strange way. That is, as soon as we do something, we feature branch. We create an environment, a kind of virtual development branch where everyone can deploy their code, even if it's not quite finished, to see how it will coexist with the rest of the other features currently being developed.
And so, we automatically merge everything that comes in. And in fact, this brings a rather... interesting and unexpected indicator, as we discussed with someone from Les Furets earlier, which is that today they have an organization that is quite independent, meaning they've created feature teams with low overlap. So in fact, they almost never have conflicts. Why? Because when they develop a feature, it only affects a relatively well-segmented area, so there's no conflict, the merge happens smoothly. When there actually is a conflict, people look up from their keyboards, they receive a message, wait it's not working, come on let's talk. So there's a conversation that arises, and they resolve the conflict. Very well. The indicator that appears here is that the more conflicts there are, the more information we ultimately have about the organization. Two possibilities. Either we have a feature that is very cross-cutting, but it's occasional. It's a very cross-enterprise project. And yeah, that one will be a bit tough, because with this automatic merge story, let's just say there's no going back. The way out is ahead, we think about merging all the time. So they'll talk often, often, often, often. But it might only stop with the project. If that's not the case, it means that overall, the company may be growing, features are changing, the team's scope is becoming less and less adequate with how features are being developed. So a kind of technical indicator starts to say, be careful, your organization, which was rather elegant and didn't have too much overlap between teams, allowing for an interesting type of market, is starting to degrade because code conflicts are appearing more and more frequently. So what we found elegant in this pattern, even if it doesn't apply easily to every context, is that a very technical indicator ultimately provides a health indicator on inter-team communication modes and possibly on the current organization that is starting to show signs of strain. is constantly moving within a company. So there you have it, we found this pattern interesting. Again, there are many prerequisites for this. And if you were using Scrum, it would be a bit tough. At Les Furets, they use Kanban, so they work with continuous flow. So in fact, they don’t have a time unit that strongly constrains them to synchronize with each other, unless there was a particular rhythm unit, as I mentioned earlier, for marketing reasons. So when you're in continuous flow with Kanban, that works well. If it were Scrum, I think it wouldn’t work as well. And the second thing is that the stories, the features, are rather small. The granularity is low. That’s one of the reasons why it works. So it’s not magic, but the pattern is quite unique.
Second pattern, well, last pattern, we’ll start reviewing other things. Earlier, we were talking about ways to branch by abstraction, which is feature toggling or feature flipping. So what is this thing, actually? So it’s a pattern that has been, as they say, brought back to the forefront by Martin Fowler; he’s written quite a bit about it, and all the people who do continuous delivery are actually trying to do this. Because in a way, the five levels of continuous delivery would be to go back to the first slide of the presentation, which would be to say that we only do trunk-based development, so we don’t have complexity related to these... branches that need to merge together. But still, sometimes our code isn’t finished. So, if we say we only have one version at a given time and we push it to production, what do I do with my unfinished code? And that’s where this pattern comes in, which is the idea of saying, if my code isn’t finished, I’ll wrap it in a condition, basically an if statement—we’re talking high-tech here—and I’ll deactivate it. So, we officially say that this part of the code won’t be executed in production. So, it’s a bit scary, meaning that overall, we’re sending unfinished code to production, but it won’t be executed. Now, it turns out that feature flipping or flag flipping has other benefits, like A/B testing, the ability to execute code on a particularly well-segmented population, because in the IF statement, we can put more than just deactivation clauses because the code isn’t finished. But in this talk, we’re mainly discussing this practice. So, another somewhat sneaky way of branching is to say, okay, I’ll include code that isn’t finished yet, that’s still under construction, for which I may not have done all the checks, all the necessary things, and I’ll deactivate it. So that’s quite interesting because it helps minimize the complexity of branches. However, from experience, and in this case, we have a joint project with Benoît where we used this, on the product side, it’s great, everyone is happy to have this kind of superpower at the product level—no, it’s too risky, I push to production, deactivate this part, let’s go, we really need to have superpowers. On the other hand, you might end up with code that has a succession of diffs that will need to be maintained, and as much as you... If you go too far down this path, it’s no joke. It’s not very pleasant to handle. It’s a pattern that’s widely used, especially by people who want to do continuous deployment. It’s interesting, but if you push it too far, it can become a bit tricky to handle because there are too many things. So you need to stay very alert in the code. So, in a large team, if you cross paths with a succession of if statements, and you weren’t part of the team that coded it, you’ll need to quickly go talk to the corner.
Another somewhat cross-cutting pattern—well, the idea here is to tell you that we’ve seen patterns further ahead, and there are a few others that we find to be anti-patterns we’ve encountered that aren’t tied to a specific workflow but are global. The first one.
The first one is the drifting branch. Almost everyone knows this one. And no matter the single-version strategy, whether it’s trunk-based, feature-branch, or user-branch, as soon as you have a drifting branch, it’s ultimately a symptom of a problem in the organization. Earlier, when we were doing team branches, we saw that the organization pushed for drifting branches. So we had that problem. We physically create teams that don’t talk to each other.
I recently saw this with a team I worked with, where I had to develop a feature, and since we had to integrate with external services, I was forced to test in integration. And in integration, I didn’t understand; I was forced to do force-pushes, well, push-force, In fact, I was forced to overwrite what was there.
And so that was a bit unfortunate; I didn’t understand. And I saw that there was another person on the team doing the same thing on their side. And at some point, I thought, this situation isn’t normal; why? In fact, I realized that she had her local branch that had drifted for quite some time, that she hadn’t merged for several days or maybe several weeks. And after digging a little deeper, it was someone who had just arrived and hadn’t been properly supervised. Meaning, I thought, who’s supervising this person? Because on top of that, it was an intern. And then the team looked at each other sheepishly, saying that ultimately, no one was really handling the technical supervision. So it’s really, again, a symptom that points to deeper problems.
So here I was talking about a new person and an intern, but there’s a colleague, Chocto, whom I respect a lot, who’s very good, but who told me about this pattern—the lone coder—and he himself did it. Meaning, he set up his little team, there were 5-6 people, they had strong standards, code reviews, TDD, etc. And at some point, he thought, hey, there’s this feature, it’s complicated, it’s going to be my sacred mission, and I’m going to climb my mountain, and I’m going to tackle it alone head-on. Since I don’t want to slow everyone down, I’ll go alone. So he did that; he did it for, I think, two weeks—it’s not very long—but after two weeks, he came back, he descended from his mountain saying, 'Yeah, it’s good, I’ve got it all figured out, it works, yippee!' And the rest of the team, during the code review, said, 'Well, yeah, but listen, we don’t really understand much, and also, I think this test doesn’t work, and on top of that, it doesn’t meet the standards you wrote.' " So he said, 'Oh yeah, okay, it doesn’t work.' And so, the lone coder mode is another incarnation of a drifting branch, meaning he did everything he was supposed to, he committed when he was supposed to, but he stayed alone in his corner for a while, so it’s another form of branch.
In terms of communication, there are two or three things that can still help with the idea of a lone coder mode: at the stand-up, overall, it’s been a long time since he’s said the same thing or he hasn’t spoken. Either he always says, 'Yes, I’m still there, I’m still there,' or on the contrary, he says nothing at all, he plants himself in his corner. If that happens, you should try to catch him because the moment of the mess is going to become complex.
That’s it. And we were talking about feature flipping; it’s the same thing, actually. Even if we push our code every day, several times a day, and even if it’s in production but deactivated, and no one has reviewed it, no one has looked at it, no one has validated it,
we’re still in this pattern of a drifting branch, even if it’s not visible. And again, this is more visible at the stand-up or on the Kanban board—there’s always that post-it that doesn’t move.
So, a drifting branch is a symptom of organizational discomfort.
Fortunately, I’m also starting to see this less and less often: the integration team. And the integration team is often a remedy for drifting branches. And often, when you’ve done team branches with two teams that don’t talk to each other, no one wants to take responsibility for the merge because it’s complicated. And so, we say, well, since merging is something technical, that’s often what the higher-ups say, merging is technical, so, do it at the end. But ultimately, since it’s not that technical, we say we need an integration team. And since it’s really not that simple, we often put the best people on it because they’re the only ones who understand everything that’s going on. So in the end, we’ve put together a team, often of seniors and experts, who are there solely to perform a task that was initially considered technical. So it’s really an anti-pattern, and it’s certainly not a solution to the problems. And it’s true that every time we’ve seen it, whether it’s a team or sometimes just one person, one person in charge of integration, it never works. So that’s another symptom that if you see this being set up or you see it being implemented, you can stop right away and dig deeper.
Well, we’re going to wrap up.
Two conclusions, two messages we have after our various meetings and discussions. The first is that we haven’t seen a single solution. It’s not like we leave the room, implement GitFlow, and everything will be fine. In fact, that’s not it. Overall, there are quite a few strategies. Often, we change them, and we see that they’re adapted to the context. So basically, in most cases, our conclusion is to take models that seem quite interesting for your issue at a given time and adapt them to your context. There’s no ultimate solution. However, we see that there are things that regularly don’t work well, or things that work well at first but stop working later. And so, just as an organization evolves, versioning strategies must evolve too. However, you need to monitor how you communicate. And on this last point, we ultimately realized—it’s quite simple—but merging is discussing. In fact, from the moment there is a merge, it is the atomic communication element between developers. You can play alone for a long time, but the moment of the merge is the moment that triggers communication. So the whole challenge is to trigger this communication at the right time. It's not that there aren't any more, because that would mean you've found the ultimate organizational zen without conflict, but we know that doesn't exist. So maybe I'll remember the organization now, but I think we haven't seen it. And so, it's more about knowing when to raise this communication and trigger that merge. And in any case, the later the merge is pushed, the more complex the resolution will be.
Take your current process, look for merge points, check if they are too frequent, if it's problematic, and if you could have done without them. Perhaps there are factors of organizational change or process that can help you. But overall, it's by observing when merges occur, the volume, and the difficulty of conflict resolution that you will find the lessons to adapt a model that can help you.
That's about all we've observed. We don't have a magic formula; we tried to create this little library of patterns.