Thierry De Pauw - Non-Blocking Continuous Code Reviews, a Case Study

[00:00:05] So, I'm Thierry from Belgium. Land of chocolates.
[00:00:13] And uh, I'm a pretty shy speaker and an introvert. So, this is kind of horrible.
[00:00:24] Yeah, yay.
[00:00:32] Um, before I forget, um, I am French-speaking. Well, uh, it's my mother tongue. Oh, the idiot, he's doing his presentation in English.
[00:00:48] Well, yes, there you go. Uh, well, because I I I do most of my presentations in English and there you go, that that I've done it a few times in French, but well, it takes an enormous amount of time to translate. Uh, so there you go, sorry. But, you can ask your questions in French at the end if you want. I will try to translate.
[00:01:14] It doesn't always work very well, but so be it. All right. So, I've just said that people may ask questions in French and I'll try to translate. Which doesn't always work very well. So now, let's go. Um, so I want to make the case for non-blocking continuous code reviews as opposed to the classic, commonly accepted and pervasive pull requests. It's a thing we did back in 2012 with a fairly novice team. Um, I haven't talked a lot about this. This is only the fourth time I'm presenting this since last year. It's only last year I started. Um, I've occasionally mentioned this on social media. But other than that, I didn't do a lot of that, because, well, I didn't realize this thing was actually a thing. It's only um, when Martin Mortensen from Denmark wrote his experience in an article in 2020 that I started to realize, oh, this seems to be something. And he also coined the term non-blocking uh code reviews. So, and and actually, it's it's only Martin and I that that that speak about this. So, it starts to get traction. Um, so I've seen Ken back mention it sometimes.
[00:02:39] And then somewhere in 2023, on a Sunday morning, I was cooking.
[00:02:46] There was suddenly a message popping on my on my phone and you're like, well, oh, the world has something to tell me. Must be very important. Well, actually, it was important because yeah, Dave Farley was reaching out. Wow, flattering. Um, telling me, oh yeah, you mentioned those non-blocking code reviews and and I actually like the sound of it and I might do a video about this. Can you point me to an article? Right. Um, I didn't have an article at that time, so I was like, yeah. Wait a minute. Give me five. Well, five became four hours that night. But eventually I wrote this down and so here I am. Now, okay, turning on. Now, before moving on, um, I'd like to explore why we are running code reviews. What is its purpose? What do we want to achieve? And and here often lies the first problem teams often face, that they make a conscious decision about what they want to achieve with code reviews. And if they did, um, did they make it clear what good or bad design means and how does good and bad code look like? So before even starting with code reviews, well, this should be made clear right from the start so that we can have valuable code reviews. Now, back then with that team, we had several reasons to introduce code reviews. And the first one was, well, it's a mentoring tool. It's a learning opportunity. So I was hoping that somehow team members would learn something from reading code from other team members. It also allows us to share knowledge, because, well, the team was responsible for 20-something applications and services and not everyone was involved with all applications and services. So, it allowed the team to have a certain awareness of this different part of the codebase. And well, this creates a collective ownership and a shared understanding, something that extreme programming identified as very important to get to better quality. It also allows us to check on readability and maintainability. So this is not about imposing a coding standard and coding guidelines. So now, for this we have tooling. We have linters and static code analysis. So it avoids silly conversations during code reviews about missing whitespace. I have seen this in teams, really. So now, all discussions regarding uh spaces or tabs, well, it's spaces, right? Or or curly braces at the end of the line or at a new line. Well, it's at a new line. You agree, right? So these things should be defined right from the start. And then we have tooling that will enforce that. So we have a machine that will control that and then it breaks the build if if violations are found. Computer says no. So now, here readability and maintainability is really about, can we read the code like a novel? Do we have the right abstractions at the right levels? And lastly, well, I used to think it helps in maintaining a certain consistency in design. Now, in hindsight, I realized this was a mistake. Um, spotting design problems during a code review is far too late.
[00:06:27] Um, we should detect this this way more earlier. And so, well, a better approach would be to adopt um design workshops before starting the work so that we reduce chances of of of rework.
[00:06:42] But then again, this wasn't really a big deal for for the team, because it never, the rework never really introduced any delays because we practiced non-blocking code reviews and we will get back to that. So these were our reasons to have code reviews in place. Now, there are other reasons that people mention and the number one reason that is mentioned mostly in regulated industries is, well, the four eyes principle. We need this to implement the four eyes principle. Now, from what I see, this is the killer excuse to slow everything down and get a team to a halt with fairly little value other than reassuring auditors that compliance compliance tick boxes are are are ticked. Um, so not very helpful.
[00:07:33] So, research is pretty clear about this. Um, process gates drive down quality. So, the exact thing that we are introducing that is supposedly there to increase quality does the exact opposite. It will reduce quality. Additionally, it introduces a huge um uh trust, trust issue that is way more, way more important. Now, often times organizations conflate their approach to regulation with regulation.
[00:08:07] Not the same thing at all. So most regulations is about, do we do what we say we do? And the most rigorous regulations says, well, we need to have two people to look at it and we need to have an audit trail of uh uh of what happened. So, a better approach to implement four eyes principles is adopting pair programming or software teaming. And Dave Farley reported that even in highly regulated industries like health care, finance or gambling, this is good enough to implement four eyes principle.
[00:08:44] The other reason I repeatedly hear is, I don't trust my code. So, we want to have a sense of security, not true security, a sense of security, an illusion of control.
[00:08:58] Now, often times there are very valid reasons for engineers to dilute this responsibility. And this this most often happened in environments with loads of legacy code. Engineers don't understand the code and so they have very little faith in in their changes on whether it broke the application or not. Or it happened in dysfunctional organizations with lots of blaming. Um, and then, while engineers want to protect themselves, and so they lean to other engineers to gain some more faith in their code. Now, I don't think that any process will fix for that. Um, and certainly not a code review.
[00:09:44] It's it's a cultural problem. It's a system problem and system problems can only be solved by management. Damming is pretty clear about this. So if you want to hear more about this, about the relation of management with system problems, well, I invite you to read out of the crisis from from Demon, um, where he will slam you in the face during 400 pages. Um, now what code what code reviews should absolutely not be, well, it should not be an exercise of power, which happens easily when code reviews are run through tooling asynchronously,
[00:10:24] as opposed to running code reviews face-to-face synchronously. And that's why it's it's not a good idea to introduce any hierarchy in in code reviews where only the team lead or only the the architect or the seniors may perform code reviews. Now, we should have true peer code reviews in place where everyone can review the code of everyone. We should also not assign code reviews to untrained senior engineers that lack the mentoring skills to or that lack the mentoring skills or the ability to clearly communicate and and and and and to to communicate constructively. I've I've experienced this um at one organization. And I was I was fairly senior at that time. So I had enough baggage to to protect myself. Still, it's a horrible experience. So, there was one core component that fell under the responsibility of a chief something. Um, and anytime we touched the code, well, it had to be reviewed by the chief something. Now that person was particularly bad in written communication, like really horrible. And people kept telling me, yeah, but they are a kind person, you know. Yeah, well, can be, but I never met the person. I only uh know them from communication through the them tool. I fundamentally disliked the person, and in the end, I didn't stay very long in the organization. It's one of my shortest missions and I'm pretty sure they lost quite some engineers because of of that behavior. Code review should also not vary in content based on who wrote the code. There is a bias problem in here that can disproportionately harm. Um, so it should not be blaming and harming. So if there is no mechanism in place to curate and train effective code reviews, it is unlikely the process will be free from abuse and and and bias. So, reviewers should listen and reflect on the code without judgment and without blame without blaming. We should provide a willing ear so that others can do a better job at solving a problem. And so anyone who can't do that should not run code reviews. So, in the end, it shouldn't be a miserable experience. And so if we avoid exercise of power, untrained senior engineers and blaming and harming, well, we already get a long way forward. It should also not be a gatekeeper.
[00:13:13] Process gates introduce delays and are negatively correlated with with quality. And lastly, it should not be a must.
[00:13:24] It's an option. It's a bloody choice of the team whether to run code reviews or not. And we are not a more mature team if we are running code reviews. I have seen so many teams that run code reviews and that are absolutely not mature and that lack the proper engineering skills. And you are also not a bunch of rag tags if we don't run code reviews. We are not unprofessional. I've seen highly skilled engineers, very performant teams, not running code reviews. So, in 2010, I've worked in in a product organization where we decided to only run code reviews on bug fixes, not on features, because we thought that bug fixes were the most risky activities. Now, bugs were also quite tricky in that organization because most of the time they were related to race conditions. Kind of problem that take you a day to reproduce the problem. And then you're super happy because you've spent a boring day trying to to to reproduce the problem and then while you know, oh, tomorrow I will be able to fix it. Yay.
[00:14:35] So, back in 2012, I had the opportunity to start a technical coaching mission with a fellow coach, so we were the two of us, with the purpose to upskill the engineering skills of a rather novice team inside of a fairly conservative um organization in the finance industry. Big central player in the finance industry. Um, it's quite funny what we achieved there because well, it created like this contradiction between the team and then the and then the conservative organization. So, I've already mentioned this team before and I'm pretty proud of anything we did with that team and that was possible because we had this amazing team manager, Stefan De Moorloze, who who was open to any of the ideas that my fellow coach and I proposed to him. He was like, oh yeah, yeah, yeah, good idea, go, go, go, do, do, do. And uh, so his take for his team was like like, look, this is the team mission. You do whatever you want. I don't care as long as it fits with the team mission. I'm going to do the marketing of the team, which he was pretty good at. Um, now, for some team members, this was difficult. This way of managing was difficult, because they were used to to to micromanagement. And here you are, you have a management that don't tell you what to do. You have to find it out. So, with that team, we used subversion, because we thought that get was a bridge too far. Too complex, too complicated, too many concepts to grasp for a novice team. And because it was said that branching is painful with subversion, well, we also went for no branches at all. Also because, well, back then, I didn't understand much about branching strategies. Um, and so, that worked pretty well. Everyone was committing straight into main line and worked pretty well, because, well, we also introduced right from the start the practice of continuous integration and team commitment that any change had to be covered by an automated test, preferably a unit test.
[00:16:47] Now, because the team was novice, we wanted to have some mentoring in place. But,
[00:16:54] pair programming was difficult. The team was not particularly open to the uh to the idea. And well, just telling the team, oh, we're going to start pairing, well, doesn't work. You they haven't signed for this. Although we did some pairing from a coaching perspective, so we were sitting, my fellow coach and I, next to team members to show them how things could work. It wasn't part of the culture of the team, so it wasn't an instituted practice. So the next best thing we came up was, uh, uh, was code reviews. And back then, I was a fervent believer of code code reviews. That has changed by now, but, so I was a fervent believer of code reviews influenced by books like um code complete and facts and fallacies of uh software engineering and also by the articles of Joe Spolsky. Anyone knows Joe Joe Spolsky?
[00:17:50] Right. So, um, say something about our age, whatever.
[00:17:56] So, in the early 2000, he was a an entrepreneur in New York. He's still an entrepreneur, but he he also created um Trello and and and Stack Overflow together with um the the person from coding horror,
[00:18:12] um, the other blog. Now, because the team was was um novice, um we also decided that every single commit had to be reviewed.
[00:18:25] And because we didn't have branches in place, well, we ended up running non-blocking code reviews, which was the result of applying lean principles and keeping things simple. Now, we didn't realize that this way of working was actually a performant way of running code reviews. And and and to be honest, back then, I was fairly embarrassed by our way of working, because, well, the rest of the world looked very professional using pull requests and we didn't we did not. Well, in hindsight, it seems that well, we weren't that unprofessional actually. We were actually quite professional. Because the team was was novice, um we also decided that every single commit had to be reviewed. And because we didn't had branches in place, well we ended up running non-blocking code reviews, which was the result of applying lean principles and keeping things simple. Now, we didn't realize that this way of working was actually a performant way of running code reviews. And and and to be honest, back then, I was fairly embarrassed by our way of working. Because, well, the rest of the world looked very professional using pull request and we didn't we did not. Well, in hindsight, it seems that, well, we weren't that unprofessional actually. We were actually quite professional. Now, the review process was based on two key ideas. And the first key idea was to establish a continuous review process that consists of regular reviews as part of the development flow. So, the team practiced Kanban and had a Kanban board to track work. And so every ticket on the Kanban board was a feature or was a bug, but not a task. These were whole features. And this has the benefit that it puts a constraint on the size of the feature because we want to move those tickets quickly from left to right. And so, well, we were forced to keep those features small. Now, the unit of work on the Kanban board was the feature. But the feature was not the unit of integration as it happens with feature branching and with pull requests. Now, the unit of integration was the commit. And the unit of release was a commit or a series of commits. Because, well, we still did, um, we still decided manually whether to release or not. Um, so we practiced continuous delivery and not continuous deployment.
[00:20:08] Now, in order to ensure that reviews happened, well, we introduced the two review column to track reviews. Meaning, well, this feature is ready, it's finished and it's ready to be reviewed. Now, Dave Farley reports, he has seen this before in other teams, but it never really worked because it was seen as a gate. And, well, process gates introduced delays and all sorts of problems and and and dysfunctions and reduces quality. But in our case, it wasn't, it was different. It wasn't experienced as a gate. The feature growth, commit by commit, on mainline, but reviews occurred on the whole feature on mainline. And so when a feature was finished and ready to be reviewed, well, it was moved into the 'To Review' column. Now, to avoid context switching, well, we decided that anyone who finished work before starting new work or at the start of the day, well, they would first look at the 'To Review' column to spot if anything is waiting to be reviewed, and if it was, well, they would review it. And so this ensured we had a rolling review in place, and so we had a continuous code review.
[00:21:29] We only had one single rule. No one should review a feature they were involved in in writing. But there were no rules regarding hierarchies and who may review the code of who. Anyone could review the code of anyone. So seniors reviewed code from seniors and juniors, we had juniors reviewing the code of juniors and seniors. Now, I wasn't expecting many comments coming from juniors. I was hoping they would learn something from reading the code from from more experienced, um, engineers. However, I've seen quite some interesting code reviews coming from juniors because they asked different questions, they asked more outsider questions. More the why questions, which are necessary triggers for, um,
[00:22:17] the the the for necessary improvements. So if we do not practice pull requests or we do not practice pair or pair programming or software teaming, well, we end up having unreviewed code in the remote mainline. And so we have both reviewed code and unreviewed code sitting alongside each other on mainline. And so this code is releasable. And this code is not releasable at the same time. How do we keep them apart? And what does releasable actually mean in that case? And this is the second key idea in the in in in this review process is that, well, stop the review being definitive for the release. Stop using the code review to decide whether a codebase is releasable or not. So the code review is not a gate anymore, it's not a go no go decision anymore, it doesn't block delivery anymore. So the team practiced true continuous integration with space development that later evolved to words continuous delivery with a deployment pipeline. And every commit that went through that went successfully through all stages of the deployment pipeline and that have been thoroughly tested had the potential to arrive into production if we decided to release. So the Kanban board and the reviews happened at the level of the feature, but the software grew small commit by small commit on mainline. And any commit that went through the pipeline could arrive in production.
[00:23:47] So this means that unreviewed code can arrive in production, whether the feature is finished or not, and whether the feature has been reviewed or not. Now, we can argue that, well, we run the risk of having low quality code sitting in production. Yes, that's true, and that happened many times.
[00:24:07] But that wasn't really a problem because, well, bad quality does not mean a bug, and code reviews are not there to catch bugs. For this, we had automated tests in place and manual exploratory testing in place. So we had team commitment that any change had to be covered by an automated test, preferably unit tests. We had automated acceptance test in place and we additionally, we also applied something I call peer manual testing. So we didn't had engineers in the team that identified as test engineers, unfortunately, and this happens quite often with the teams I'm working with. And so the next best thing we came up was was, well, peer testing. So someone not involved in implementing the feature would do some exploratory testing on the feature. And then we had a decent amount of static code analysis that would break the build on any rule violation. So the risk of working like that is not so much that the code would be buggier or not, but that we might deliver lower quality code into production. But code quality does not have much impact on product delivery and even less on um, on on functionality. But code quality does has impact on the speed of change. And that's the reason why we want to have high code quality. Because it allows us to introduce changes and add new functionality quicker and more easily, and so it reduces delays and reduces time to market. In the end, this is code that works. Because the test says so. It passed all tests. But that could be better. But, well, somehow that's always true. Code reviews only reduce the chances of delivering low quality code. But that depends on the quality of the code review, but it doesn't eliminate bad quality altogether. Now, this worked because, while we agreed as a team that every single commit would be reviewed at some point in time. And we were sure that this was going to happen because we had this 'To Review' column, and and and as long as the review didn't happen, well, the feature was not done, although it might already be in production. According to our Kanban board, it was not done. And whenever an issue was raised, so a quality issue was raised, it had to be handled with the highest priority. And that also we were sure it would happen because well, the the the the ticket stayed in the 'To Review' column as long as those issues were not fixed. So what are the benefits? Now, to understand the benefits, we need to understand transaction cost from lean manufacturing. So the transaction cost is the cost of moving a batch of work to the next stage. Now, this is important because the higher the transaction cost is, well, the more inventory is created in front of the next stage to compensate the transaction cost. So let's say we want to order some goods online, and the transaction cost for delivering the goods is 3 euros.
[00:27:13] So, in order to have the goods delivered, it will cost us 3 euros. Well, in that case, we are not going to order one single good at 1 euro. No, we are going to order a set of goods to end up with a total cost that is a multiple of the transaction cost so that we can compensate the transaction cost. And so we are going to create a bigger batch of items to deliver.
[00:27:38] So, if it takes us 10 minutes to make a change and the change is blocked until a review happens. And it takes two hours to get a review, well, this means that the change is waiting for 92% of its total lead time. Now, this is a fairly expensive process. So this signals us that the code review is pretty costly. And in that case, to compensate the transaction cost, well, we are incentivized to ask less often for code reviews. Now, what happens if we ask less often for code reviews? Right, we are going to make more changes. And and and adding more changes in a bigger batch before even asking for a code review. And so the process discourages us to work in small incremental steps and it discourages us to adopt refactoring. And if we don't refactor, well, we don't pay back technical debt. And so if we don't pay back technical debt, it impacts codebase quality and so it impacts the speed of change. And so it introduces delays and increases cost of delay.
[00:28:48] So, because we are waiting for someone to get work done, because we are waiting for for a review, well, we are also incentivized to create more work in progress. So engineers want to feel productive and it's fairly uncomfortable to just wait and do nothing. I invite you to do that and then look at the reactions of your leaders.
[00:29:11] So what happens?
[00:29:14] Well, we keep busy and we start new work. And here's the start of lots of context switching and we know where this goes and it's not pretty. Now, because of Little's Law, more work in progress means bigger lead times, bigger delays. And so we end up with delayed delivery, delayed feedback. Learning will decelerate, we become risk averse and innovation falls flat and we will go back to conservative solutions. And we see this with organizations that practice pull requests. They are unable to adopt the performant engineering practices which would allow to unlock a myriad of possibilities, they would they are even not aware this would be possible. And so they are stuck in what they know and so they are stuck in in in in in a tunnel vision and they go back to the things to to to conservative solutions. So, to get things out sooner, well, we need to reduce the transaction cost. So we need to minimize um we we need to minimize the cost of code reviews in order to reduce the work in progress. And because of Little's Law, this will drive down lead time, drive down delays and will will shorten our time to market and will accelerate feedback. And consequently will lead to better quality. So what are the benefits? Second time. Well, because the transaction cost is nearly zero, the code review had almost no impact on on the total delivery lead time. Commits already went through the deployment pipeline and if it went successfully through the deployment pipeline, it might already be in production before the code review even started. And so there is no reason to make more changes in bigger batches before asking for a review. And so we are incentivized to work in small incremental steps, and naturally, refactoring will happen. Which will improve codebase quality and improve speed of change, and as a result, we are optimizing engineering time. To introduce change and to add new functionality to the codebase, which again reduces our time to market.
[00:31:31] So we are never slowed down anymore. We don't have any gates. We never have to wait for someone to get work finished. So changes are never blocked. And so we commit and we commit and we commit and every commit that goes through the deployment pipeline arrives into production if we decide to release. And this is what enables early and fast feedback, which allows us to take new decisions and run new experiments in production in order to find new ways to delight our users. So the fact that unreviewed code could already be tested or was already deployed into production before a review even happened was surprisingly a significant benefit. Because we already received feedback from the user, we could see how the user was using the feature. Or maybe he was not using the feature. And in that case, we even didn't have to run the code review. Well, we should remove the code in that case. With pull requests, we don't have that feedback. As long as the review didn't happen, the feature is still is is just blocked. And and we have zero production feedback. So we don't know if the thing that we've implemented is the right thing. We might be completely wrong, but we don't know it.
[00:32:47] So we can now work on a functionality from start to finish. And so we never have to start new work before finishing ongoing work because we want to feel productive and we are waiting for someone to continue. So that never happened anymore. And so as a result, people could focus on one thing at a time and and and work on on on on functionality from start to end. And so there was no context switching anymore, no multitasking anymore, so no productivity productivity killers anymore. And so because we are not starting new work, well, we have way less work in progress. And so because of Little's Law, we have reduced lead times, reduced delays, shorter time to market and faster feedback. Which allows us to take more decisions and run more experiments in production to find more unmet needs of our customers and as a result, well, make more money.
[00:33:48] Now, sometimes it happened that we noticed that the design was completely wrong during a code review. Design was completely off.
[00:33:57] These were moments where I wished we were practicing pair programming so that we would detect these things earlier.
[00:34:04] But then again, this wasn't really a big problem because, well, the feature was found to work. It passed all tests. It was delivered in production, it was in the hands of the users, the users had the value of the feature already. We received already feedback on whether the feature was the right one or not. And so we had all the time to redo the design. There was no delivery pressure because it was already delivered. And what's more, we could even incorporate the feedback we received from production so that we do a better redesign. Now, with pull requests, we have two options if we find that the design is wrong. Either we impose a fix and during that a fix is is is is implemented, while the feature is still not delivered. We still don't have production feedback, we still don't know if the thing that is waiting there is the right thing. Might be that we are investing a fix that is not needed because maybe it's not the right thing. Or because there is there is delivery pressure, well, we decide to cut corners and we say, oh, we'll fix it later. And we know where this goes. And here starts the technical debt and the impact on on the speed of change. So in our case, no pressure. No stress, no fatigue, no burnout.
[00:35:29] Now, because unreviewed code could arrive in production, while if the code went successfully through the deployment pipeline, well, it puts a lot of trust in the hands of the team, and so we are expecting from the team they are going to do the right thing.
[00:35:46] Instead of thinking that they are stupid and that we need a process to fix for that. So now, we were expecting the team to add automated tests, to add enough automated tests and that they would add the right automated test and high quality automated tests. So this is applying management theory, uh, management theory Y over management theory X. And even when something went wrong in production, that too wasn't really a big problem because we had a process that allowed us to recover quickly. Continuous delivery, well, allowed us to release within minutes. So I like to say, I don't care about bugs in production. They will happen anyway, regardless of how many testing we do because of all the unknowns we don't know. Um, oh, the users are using it this way. We weren't expecting that. These kind of things.
[00:36:45] So it's more important to have a process in place that allows us to recover quickly, to fix those problems quickly.
[00:36:56] Now, in all honesty, there is one gotcha.
[00:36:59] With non-blocking code reviews.
[00:37:02] It might be difficult to implement in regulated industries. Many regulated industries require a gate before deploying into production. And the gate is often implemented using code reviews. So, in that case, it might be difficult. However, I can imagine of certain ways we can implement part of this non-blocking code reviews. But I never tested it. So I never worked for healthcare, for instance. So, I won't mention anything of this. So I might be wrong.
[00:37:35] But still, for regulated industries, pair programming and software teaming are a way better approach. If we do pair programming, well, the pair could sign off the code review at the moment that they commit the change and tag the change with the identifiers of both engineers involved in the pair. So, in conclusion,
[00:38:01] I still think that pair programming and software teaming are superior delivery options. But, in many organizations, this is a cultural stretch for many valid reasons. Um, you can't just change a culture because, while you are the coach and you are arriving there. You need to be mindful. People haven't haven't necessarily signed for this and they are not necessarily open to the idea.
[00:38:32] In this situation, non-blocking review strategies are significantly better than any of the alternatives because, well, there's no gating. So it allows for fast delivery without delays. Um, so it enables the fast flow of work through our value stream, which is a critical practice to adopt to get to better quality because it accelerates feedback. Because we are never waiting, while we do not need to start new work just to feel productive. So we have way less work in progress, and because of Little's Law, we are reducing delays, reducing lead times, reducing time to market, reducing cost of delay. And naturally, while we accelerate feedback and we get to better quality.
[00:39:19] Um, because we can finish work from start to finish in one go, well, we don't have context switching anymore, there's no multitasking anymore, no productivity killer anymore. the fast flow of work through our value stream, which is a critical practice to adopt to get to better quality because it accelerates feedback. Because we are never waiting while we do not need to start new work just to feel productive. So we have way less work in progress and because of Little's Law, we are reducing delays, reducing lead times, reducing time to market, reducing cost of delay. And naturally, while we accelerate feedback, then we get to better quality.
[00:39:19] Um, because we can finish work from start to finish in one go, while we don't have context switching anymore, there is no multitasking anymore, no productivity killer anymore.
[00:39:35] There's no urgency to fix quality issues when they are discovered. We have all the time because, well, the feature was found to be working, automated tests said so. Um, it's, it's delivered in production, it's in the hand of the users. Um, so, well, we have all the time to fix it, so no pressure, no stress, no fatigue, no burnout.
[00:39:56] Naturally, while it enables us to grow the software incrementally, small commit by small commit, and, while refactoring happens naturally. allowing us to get to better code base qualities and enabling the speed of change. Thank you for your time.
[00:40:24] Any questions?
[00:40:34] Bonjour.
[00:40:35] Bonjour. Je te laisserai traduire.
[00:40:38] J'ai plusieurs questions. C'est bien.
[00:40:41] Ouais.
[00:40:42] L'usage des feature flags dans votre contexte. À quel, est-ce que vous en, enfin est-ce que c'est rendu nécessaire, est-ce que c'est utile? Enfin, j'aimerais savoir comment vous les utilisez.
[00:40:52] Pendant longtemps, So, question was,
[00:40:57] What about feature flags? that we use it, was it useful?
[00:41:02] Um, well, so during a long time, we didn't use feature flags, uh, and then at a certain point, we started using feature flags and yes, this was useful.
[00:41:12] Uh, to because, well, we had to enable features at a certain time or something else, uh, needed to be deployed before we could enable the feature, well, these kind of things. So to decoule deployments of different applications so that we don't have a coordinated deployment.
[00:41:33] Et du coup, ma question c'était plutôt par rapport, enfin, merci pour cette réponse. Par rapport justement à la partie test où l'équipe produit aurait envie de tester donc du coup, est-ce que vous avez un environnement de staging, c'était pareil, comment vous déployez-tu si vous en avez un et où c'est qu'on teste?
[00:41:48] All right, so, question is regarding, yeah, um, that we use it for testing, and if we used it to facilitate product teams to test the application. Now, I have to, I have to admit, it's far away, so I don't remember everything anymore. Um, I have a vague memory that we did do that, uh, like enabling a feature in the test environment. But that wasn't enabled in the production environment, so that, uh, uh, product teams could, could see the effect.
[00:42:23] Merci.
[00:42:28] here.
[00:42:29] Um, thank you, here. The, uh, your explanation was very clear, uh, both about the in, uh, impacts and the rational and principles behind your practice. Uh, do you have any measure of any kind of efficiency whether it's time to market before, after, uh, lead time, quality in any?
[00:42:51] Oh, it's, it's more subjective measure, it's an observation. So when we arrived, so my, my, my, my, well, fellow coach, now friend, uh, Martin and I, when we arrived, um, the team was really in an operational mode, like making sure things didn't break. So, I think 80% of their time was like operations and 20% was features. And we just switched that around, like 80% uh, value creation and only 20% operations. Because, well, there was lots of automation, but also lots of automated tests. And, and we were pretty sure that when something went in production, it, it, it, it, it, it was going to go right.
[00:43:38] And, uh, a short, maybe second question. Uh, you mentioned, uh, automated tests and unit tests. Do you think they are a prerequisite for this practice to work or was this the case in your experience?
[00:43:53] So, we were lucky because we had a crazy manager that said from the start, we're going greenfield. Everything that exists, um, is going to be replaced. So, that allowed us to just grow the software with automated tests right from the start.
[00:44:15] Is it necessary? Well, it helps, sure.
[00:44:19] I think it depends on your, um, risk appetite as a team.
[00:44:25] Um, because, well, saying, yeah, we need to have a certain level of test before adopting trunk-based development, yeah, well, sure, you can do that, but when will you have the right level of test and do the switch? Um, I'm, I'm a bit, um, um, afraid that because you have feature branches and pull requests that you can hide behind it and, and keep doing the wrong practices. So, if I would be a, a, a leader, I would just say, yeah, well, never mind, we do trunk-based development and we'll fix the things as they come.
[00:45:01] But it, it puts quite some pressure on the team, so needs some sort of mindset.
[00:45:08] Thank you.
[00:45:10] Merci pour le talk. J'ai une petite question en fait d'un point de vue pratique, comment est-ce que vous faisiez pour, ben, des gens en terme d'outils et en terme de comment choisir quoi reviewer, par qui? Est-ce que c'est à la, est-ce que c'est la personne qui écrit le code qui demande à certaines autres personnes de le reviewer ou est-ce que c'est les gens qui lisent le trunk petit à petit?
[00:45:33] So, the question is about, well, tooling and how we organized the code reviews, like did the person that implemented the feature asked specifically for a review or were people just reading the code on trunk? and going over the commits. So, yeah.
[00:45:51] The very first version of this, of this slide deck also had some slides about the tooling. Um, so back then, there was tooling in place to do this. Most of it has been discontinued by the tooling vendors in favor of pull requests. So back then, Atlassian had, um, uh, Jira together with, um, something with a C. I forgot the name. Um, that allowed to, um, so it connects on the version control system, collects all the commits and then you can start filtering on the commits based on keywords. And so we used ticket numbers. So the ticket number of the feature and then we grouped all the commits, these are the commits for the feature, here's the code review. Um, Jet Brains also had a, a tool like that, it was called Upsource. And then we had, uh, Facebook who had Fabricator, I guess, uh, but they discontinued that also. Um, so yeah, um, nowadays, there is a company, Axion it that is practicing trunk-based development since 10 years, facing the same problem, we want to do code reviews, there's no tooling. So they, they, they implemented their own code review tool, it's called X-Reme, and it's available on GitHub, I never used it, so I can't say whether it's good or not. Um, so that's, that's for the tooling.
[00:47:24] Um, we, we never asked explicitly for code reviews. It was really like, feature is finished, moved into the to review column and then someone will pick it, the first person available that finished work would pick it. Um, and so we avoid having, yeah, to poke people, like, oh, please, please, please, can you review something I see in, in organizations where on Slack, they are like, oh yeah, yeah, can you quickly review this pull request because it's blocking a fix for production, for instance? It's crazy. So, while poking will not make your teams faster, the problem is work in progress.
[00:48:06] There was a question here in, in front.
[00:48:17] Hello, Jory.
[00:48:21] First, I would like to thank you for this presentation. I found it very insightful. It's the presentation I learned from the most today.
[00:48:29] Thank you.
[00:48:31] And, uh, my question is about the use of, uh, large language models in code review. I had the occasion to work on a solution that may, that makes code audits on GitHub repositories.
[00:48:46] So it, it uses an LLM to review the non-respect, for example, of coding best practices, data leakage, security, uh, security, uh, problems. So how do you see the usage of LLMs in code review and could it, for example, solve some of problems, some of the flaws of human code review?
[00:49:08] Yeah, so, um,
[00:49:11] I'm, I'm a bit skeptical.
[00:49:14] especially knowing the recent developments in LLMs, especially in code generation LLMs like
[00:49:22] Yeah, so I'm, I'm a bit skeptical. So I'm, I'm, I'm quite impressed by how co-pilot completes the code in my editor and I'm, I'm really thankful about that. But then again, I know about coding and I know how good code and bad code looks like, so I can make a, uh, conscious, um, decision about us.
[00:49:48] I'm not sure we can just hand over reviews in, in into the hands of machines. Because there is also, I, I, well, there's also lots of context that they, that they miss, um, to do a proper code review, uh, I guess. But, well, call it being old school, probably, I'm not particularly young. So, might be that it works. We'll see what, what the future will give, but I'm a bit skeptical about the whole hype around LLMs, that's just that. And I'm not the only one being skeptical, so.
[00:50:30] Thank you.
[00:50:31] Thank you.
[00:50:34] Yes, I, oh.
[00:50:37] j'ai j'ai une question mais c'est peut-être plus à propos du trunk based development. Euh comment vous j'ai j'ai peut-être pas bien compris. Vous aviez une feature one to three, feature three four five euh et vous décidez à un moment de passer en production. Comment vous faites pour la feature three four five qui n'est pas terminé?
[00:50:59] So, the question is about trunk-based development because there is one slide that shows like several commits. with different identifiers of features and they are like mingled in between them, which actually happened, so we had multiple people working on features and so we had commits coming in for different features. And so the question is, yeah, but what happens when you decide to release and, and feature one, so you want to release feature one, two, three, but four, five, six is not finished.
[00:51:28] Well, yeah, we released and we made sure it didn't break. It's, it's, um, it's, yeah, it's, it's an idea people have, yeah, you can only release finished functionality, that's not true. It's okay to have unfinished functionality sitting in production, um, even behind publicly accessible URLs, you just need to make sure to hide the functionality so that users don't discover the functionality and that it also doesn't create any impact, like bugs or certain things. But that requires testing to make sure that this doesn't happen. All right.
[00:52:11] Hello. Uh, I mean, I wanted to talk about, you know, design code quality and the no pressure to finish the quality before shipping to users.
[00:52:18] Uh, no pressure to finish the quality before shipping to users. So I can see how that could be an advantage to get the functionality in front of users as fast as possible. Um, but then I can see cases where we never go back even after the review, never go back to fix the internal quality of the code because we have other features to deliver and we want them to be in front of users as quick as fast as possible as well. So, how, the question is, um, how will we transform the no pressure to fix the quality before releasing to some pressure to actually address the code quality issue?
[00:52:55] Ah yeah, but, but the, The thing is, we didn't had to have no pressure. Yeah, so the thing is, we didn't had to fix the code quality before releasing. We could release.
[00:53:06] And we knew that the fix was coming later on, because, well, we, we tracked that the fix was, was, was happening. Feature was not finished according to us, although it was already in production and released, but according to our, our tracking tool, our Kanban board, feature was not finished, so it was going to happen. Now, on the other side, well, sometimes it's also okay to have a bit of lower quality code because other things are more important to happen. What I, what I started to do is just adding comments, like yeah, this thing needs to be improved someday, and then someday we come back because something needs to be changed in this area and we just do that.
[00:53:55] And so you also have natural, natural improvement. So, um, I have a friend who, who says, yeah, we, I do refactoring to prepare the code to add the new feature. So it's a necessary thing that needs to happen as part of the development.
[00:54:16] Thank you. You're welcome.
[00:54:21] Okay, we got only time for one last question.
[00:54:25] Thank you. I, I really like the, um, challenging approach and the challenging idea. Especially for, uh, people who live in industries where there is a huge concern for quality of having the sense of control over the quality. which might be two different things. Um, I'm not sure if it's a question or more than a reaction and it's quite, uh, connecting to the previous question. Where when you have a product which seems to be pretty much available to the user, pretty much finished, uh, you may have situations where you have a product owner who is feeling like, okay, let's go next, code review, we don't really need to focus on that. That's fine, it works. Let's move on to the next, you know. And I was feeling like it might be very helpful in such situation to consider the work done not when it's actually available to users, but after having some time to actually measure the impact of the value that you may benefit from your, uh, deliverable.
[00:55:32] Well,
[00:55:34] we were lucky because the product owner was actually the team. So we did the management of the product because we, so we had a couple of engineers. who came from the departments for whom we were developing the software and so they very well understood what they needed. Um, so that was one thing.
[00:55:59] Um, and we, we put, well, we, we thought that quality was really important and so, well, that situation never happened. Again, because according to us, it was part of the definition of done, like, it's in production, right, but it's not finished. There's still work to be done and we need to finish that before starting new work.
[00:56:27] Thank you.
[00:56:29] Thank you Thierry for your insightful session. Thank you. And we love now a break time.

Transcript