Pascal Rieux
Transcript (Translated)
This is the Toulouse room track here, since Nicolas, previously, also comes from this city.
I am going to talk to you about organization. I am going to tell you a story, part of the history of an organization that was an R&D department. At Sierra Wireless, which develops a product. I will talk about it more precisely. And as is often the case in stories we can read in books, at the beginning, there are a number of quotes, so often a quote at the start of the work, or even a quote at the beginning of each chapter. I will not fail in this practice and will start with a quote, which is Conway's Law. Some of you may know it. There are some who know it, Conway's Law, among you. So Conway's Law is not really a law, it is more of an observation than a law. It means that an organization, so the structure of an organization, mirrors the structure of the product it develops.
So there is a parallel drawn between the architecture of a product being developed and the communication structure within the company. Conway stated... this law in an article some years ago already, since it dates back to 1968, and it is based on his own experience as a computer developer at the time.
And it is something that can still be found quite often today.
It is something, among other things,
that we find and that we found within the department where I work.
I will talk about structure and I will talk about the evolution of structure, both product structure and organizational structure.
So in an organization, the screen is a bit... There is a way to adjust the zoom differently.
Otherwise, we risk losing some information.
No, but there is a zoom button.
I think I am at the limit.
That's good.
So in fact, the story I am going to tell you is the parallel evolution of the organization's structure and the structure of the product that this organization designs. So I will present two major stages, the beginning and the end. That is, roughly, three years ago, in what situation we found ourselves in terms of organization and products, and the current situation, that of today.
So, initially, the company I work for, Sierra Wireless, designed a service platform called Arvantage, to which objects, communicating boxes, connect via GSM, 2G, 3G, or even 4G systems today. So that was what was developed 3-4 years ago. Today, what is being developed,
is the same thing, we continue to develop this product, but we have improved it. In three years, we can hope that this product has improved. We have also improved the way we develop it. We have acted on both the product structure and the organization structure.
So first, I will explain or show you how it was organized, both in terms of organization and products. And secondly, I will show you the current state of the product and the organization. Then we will see... How the different evolutions are implemented, by what means, with what tools, and what the perspectives are from today. So previously, the product consisted roughly of three layers.
So a low-level component layer, database, communication with the communicating boxes, a business layer, which managed business rules and communication via APIs with third-party applications. And then a user interface layer. So a fairly classic organization in terms of product architecture. Parallel to that, so that is the product structure, remember Conway's Law, product structure in three layers. Parallel to that, we had three teams. So a first team that handled architecture, work on low-level components, and communication, everything related to communication with communicating systems, with communicating boxes. First team. Second team that managed the entire back-end part, business logic, and APIs. And then a third team, specialized in user interface, which developed the user interface. You see a division into technical layers.
That prevailed at that time. In addition to these three teams, two entities outside, at the red border, which is the R&D department I mentioned at the very beginning, were responsible, one of these teams, for administering the platform, since we operate in hosted mode, and a second external team, still part of the R&D team, for validation. So a fairly classic QA team.
dynamically, how these teams were organized, so in relation to a roadmap, dictated by marketing,
the things to do, the stuff, moving from backlog to backlog, so a product backlog at the start, then a sprint backlog, then a sprint backlog per team, since we had three teams, Then a validation backlog, handled by the validation team, external to the development team. And finally, a backlog of things put into production.
The activities, quite classically, were estimation. It was necessary, to be able to predict what we were going to put into production, to estimate the size of each of the things we were going to put into production. We had arbitration.
Meetings to estimate, meetings to arbitrate, meetings to prioritize.
Meetings to dispatch the different tasks to be carried out in each of the teams. Based on a marketing request, it was necessary to properly direct this request into one or another of the backlogs, or even into several, thus dividing the features according to the technical layers.
So also, specific and regular meetings.
packaging to be able to validate, and then a final meeting to say, okay, everything that was done in the package, we are good to put it into production.
Okay? So, a lot of meetings, development cycles, which took about three months. We had iterations of three months, so every quarter, we delivered a set of features into production. We could deliver a little more often, but not much more than every month, for example. Every month, we could deliver bugfix releases. But generally, the big features were delivered every three months.
And with a mode that was a Scrum mode, with planning poker for estimates, preparation meetings to dispatch the different items, prioritize, arbitrate. And then a number of meetings that involved what we call a core team, so the different stakeholders who gave the red light or green light to go into production. A set of fairly heavy mechanisms, let's say, but still called agile, since we were in a pseudo-Scrum mode, let's say.
Today, so after. The organization looks a bit like this. It is more of a network organization, let's say.
With members of a team and subjects.
The team being a broad-sense team, meaning it integrates both developers, development skills, operational skills, and validation skills. So there has been a change in terms of organizational structure to bring together the different skills into a single team.
So, I will talk about it. There are a number of triggers, but it is important to understand that I am talking about a period of about three years, between what I just presented and what I am presenting to you today.
No, here we have subjects. I will clarify this notion later. We have subjects and the people from the extended team, so from the R&D department, organize around these subjects.
There is no directive to assign a certain number of subjects to people based on their skills.
So if we look at the product side, do you remember the structure in technical layers? We still find this structure in technical layers, since we cannot avoid it, we still need low-level components, to manage data, to historize.
We still need business components, even user interfaces, even programmatic interfaces, APIs. But the distribution is done differently today. I talked about subjects in the organization of the teams. The subjects are found a little in the form of vertical components. So we will no longer divide the product into technical layers, but into functional blocks, which can group different technical levels up to the user interface.
So, a topic, in fact,
It's a set of features to develop.
with a homogeneous set of features to develop, We find the concept of an epic, which is quite similar in the Scrum methodology, for example. And we can attach a functional component, thus vertical, to a topic, We can attach the same vertical component to several topics. So, there is no strong coupling, I would say, between the concept of a topic and the way the product is divided. But the product is much more functionally divided than it was previously. So, Conway's Law also applies here in the parallel between the organization of the structure and the way it is designed. One of the components.
There you go.
So, for the product organization part.
So, dynamically, how does it work? So, we still have a roadmap, because outside the R&D team, there is a marketing team. It is outside, but it is really very close; we work in close collaboration. So, this roadmap is brought by the marketing team, but it is also completed or amended by the technical teams.
And alongside that, we have a release train.
Which has notably accelerated. So, it has accelerated thanks to a number of techniques, but roughly speaking, we have moved from quarterly releases to weekly releases. So, today we release every week, or even several times a week. So, how did this become possible? It became possible by, on the one hand, compartmentalizing the product into vertical blocks, thus into features.
It became possible by adopting a number of tools, which we will see later. And thus by working on topics on which each team member can position themselves. We will also see how later.
So, how does it work when something to do is extracted from a topic? Marketing and the technical team decide that there is a topic on which we must work. This topic is divided into several small features, or even tasks. And for each of these tasks or features, we will create a development branch. On which the team assigned to the topic will work.
At the end of the work on the development branch, we have a mechanism called a pull request, which we could also discuss
in terms of tools, that allows us to give approval to merge what has been developed on the parent branch, on the master or the trunk. The technical team decides there is a topic on which we must work. This topic is broken down into several small features, or even tasks. And for each of these tasks or features, we will create a development branch on which the team assigned to the topic will work.
At the end of the work on the development branch, we have a mechanism called a pull request. We could also discuss this at the tool level, which allows us to give approval to merge what has been developed on the parent branch, on the master or the trunk.
From what has been developed on the trunk recently, we will
aggregate everything that has been recently developed on the trunk to perform a validation pass, with the validation skills now integrated into the R&D team.
And then, we always have a...
A go/no-go decision to make when performing validation, but this is done without a specific meeting. So, it is the validation team that gives the green or red light at the end of its validation phase, which lasts from one to three days, but not much longer, for deployment to production.
So, we do this cycle every week.
Thus, the cycle of taking everything that has been aggregated on the master, bringing it to a platform for validation, validating it, and deploying it to production, is carried out every week. On a fixed date. So, we decided it would be Tuesday. We evolved. Initially, it was Thursday. And then there were some people who were not necessarily available at the right times for it to be done reliably. So, we moved the date to Tuesday. And every Tuesday, we deploy a version. To production, which includes. The things to do, which can be features, functionalities, but can also be fixes.
Well, that's all well and good, we saw the flow of what was deployed. How will we synchronize?
Because previously, we had estimation, preparation meetings. Planning, distribution in the different backlogs, go/no-go to decide jointly with the core team whether it's good, we're at the end of the quarter, are we okay to deploy, are there no risks, etc. Today, we no longer have all that, but we still need to synchronize.
in a certain way. So how do we synchronize?
So, we synchronize every Monday morning.
And for each topic, preferably, we will do a demo.
In the worst case, we give a status update.
So, each micro-team gathered around a topic will demo what has been developed during the week or give a status update.
We also have a 5-minute slot for a roadmap update. So, this is where marketing participates in this meeting and provides an update if necessary on the roadmap points. This is generally a slot that is occupied... since there are still, after all, announcements made to our various clients that are not made every week. There is a series of announcements that is more on the order of the quarter, month, or quarter. So, at the beginning of a quarter, we have more frequent updates on the roadmap. Than throughout the quarter.
And finally, we have a 5-minute slot for a status update on deployments. You remember the green or red light in the release train.
So, this also results in 5 minutes during this Monday morning meeting. To say, okay, we deploy, or there is a bug blocking the deployment, which needs to be worked on urgently. Generally, we do not wait for the Monday meeting to try to unblock a release that is blocked by a bug. We do it in a fairly interactive way. Synchronization is essentially done through this Monday morning meeting, which lasts one hour, maximum one hour I would say. Quite frequently, it lasts less than an hour. The entire team is gathered, so depending on the Monday, between 20 and 30 people. With all the skills of the R&D team, plus marketing people, plus sometimes internal clients as well, since we have an operations service that is in contact with clients, and who come to listen or see in the demonstrations what will be brought into the product during the week. So, this is the preferred meeting for synchronization. Yes?
Full-time?
It's 25 to 30 people.
So, it's quite rhythmic, since for each topic, there are currently, I would say, and most of the time, about twenty topics being carried out more or less in parallel. Some are on pause, but can resurface, since we have... We work with mobile telephony operators, so we can have topics that require dependencies with stakeholders or third parties, so they can go on pause while other topics are brought forward. But roughly, we have a backlog of about twenty topics and 25 to 30 people assigned to these different topics.
So, we do not try to do a very formal review of each topic during this one-hour meeting. We prioritize demonstrations. And then, if there are important things to say, specifically a change in status, then this is the place, the moment to communicate it to everyone.
So, there is the entire team, the developers, the test engineers,
the system administrators, marketing, product marketing, and internal clients, essentially people from the operations service, who are in contact with external clients. Who manage the back office and must respond to clients when there is a problem on the platform, respond to client requests in general.
Exactly. And there is a 5-minute slot to do it.
There is a leader in the topic team who will do the demonstration. It is not necessarily always the same person, by the way. There is no concept of team lead as it could have been, I will come back to it a bit later, as it could have existed previously in the more static organization where we had three teams, each with its team lead. Here, we have topics, there are somewhat natural leaders who will emerge on each topic, who can change from one week to the next. And who will do the demo. But this is the method we have adopted that allows us to pace development and production deployment and to give a rhythm to the work as well.
We do not have stand-ups, we no longer have stand-ups, for example. We no longer have daily stand-ups.
We have this meeting.
Of course, we also have ad hoc meetings. If a team needs to gather on a topic to discuss a design, an implementation, interact with marketing, call clients because we need information, then we do not prohibit meetings at all. Simply, we have much fewer formalized meetings, except for the Monday morning one, which is limited to one hour.
We keep retrospectives. Previously, we had retrospectives, we had retrospectives by team, so each of the three teams had retrospectives at the end of each three-week sprint. Today, we have much fewer retrospectives. We do quarterly retrospectives, but they bring together the 25 or 30 people.
We have also set up a day that does not necessarily serve for synchronization at the product level, but which we called a Make It day.
In other organizations, this is also called a Gold Card. It is a day every three weeks during which all developers can... Carry out or work on a project of their choice, a personal project, certainly related to the platform we develop, but it can be quite distant from the platform we develop. It is a personal project within the framework of the company, on which each person can work one day every three weeks. And which will result in a demo. Here too, every three weeks, we reserve a one-hour slot to demo the Make It projects. These are projects independent of the topics we saw previously. This does not mean that they cannot become topics.
A number of ideas like this, personal projects that have become topics or pre-sales tools, for example, quite naturally. So, this allows us to catalyze innovation, in fact, and to keep... It allows team members to take a step back and let ideas flourish that may help advance and develop the product further. And especially to have fun.
We also have a number of special events.
This summer, we had the World Cup, so we held a bug-fixing championship over one day with a trophy. And by team, the development team competed to win the trophy for which team would fix the most bugs. More recently, we held a code retreat day, a day more focused on coding practices, coding exercises. So, we also keep time, we try to keep one day per quarter for events of this type. Nothing is planned very far in advance. It's a bit based on ideas we can pick from retrospectives, or from people who tell us they'd like to do katas or things like that.
So, there was a transition, an evolution in the product organization and in the product structure, as well as in the organization's structure.
This evolution, unlike the first diagram I showed you, did not happen in a continuous or linear way. There were plateaus.
The last plateau we crossed was at the beginning of this year, early 2014, in which we... As part of this plateau, we really moved to a network organization. We grouped the teams. This was the last stage in the evolution of this team. There were a number of objectives to achieve in this plateau. A first objective was to be able to manage topics that were hidden. So, there were a number of topics raised by the development team, mostly technical topics, that were regularly set aside because they were deemed non-priority. And this generated frustration.
So, organizing the service into teams around topics allows us to list all the topics and make them visible in order to manage the previously hidden topics. visible so that we can manage the topics that were previously hidden.
By moving to weekly releases, of course, we drastically minimized the time to market or time to production. And since the product is largely based on feedback, meaning the input brought by customers, more or less directly in the user interface. There is a button that allows customers to make requests directly. We also have the operations service I mentioned earlier, which is the interface that handles customer tickets as well as phone calls. So, we have requests coming in that way.
And so, we can react very quickly within a week. If the new things to develop fit within this timeframe, we can react much faster than before.
We also wanted to have a less monolithic architecture.
Previously, we really had...
A rather heavy package to manage.
So now, the package has been split into several components that can use different technologies. Not necessarily the same languages, not necessarily the same technologies, We'll see that this is one of the reasons pushing us to go further, breaking down the product structure into microservices.
So, for that, we need a scalable architecture. And this is what has been developed over these three years. What was also desired was the concept of self-organization around topics. So, having an extended team with more skills, different skills within the team, and not just a team limited to development. And there was a strong need for self-organization.
So, these different points, to break down the product structure into microservices.
So for that, a scalable architecture is needed. And this is what has been developed over these three years.
What was also desired was the notion of self-organization around topics. So, having an extended team with more skills, more diverse skills within the team, and not just a team limited to development. And there was a strong need for self-organization.
So, these different points,
The last one is much greater visibility than before. It's the visibility provided by the Monday meeting, where we can review all the topics, but everyone in the team, all 30 people, have a view of all the topics simultaneously.
And these different needs, or more precisely objectives, were raised through various retrospectives, for example, or through informal discussions. So, this was not something imposed by management.
It was something done in agreement with management, or even pushed by management, but it also came up from the teams.
And as we were discussing earlier, there was also the need for leaders to be more like designated leaders, that there no longer be a team lead
with an official title, but that people who want to advance a topic can do so without constraints.
So, how did this happen?
So, it happened over time.
I've mentioned it several times, so it took three years and several stages to bring about this organizational change. It also happened by starting from where we were.
We are at a conference on Kanban, among other things. So, we started from where we were. We didn't try, three years ago, to want... To bridge too large a gap all at once. So, we really started from where we were, we used feedback, we had support from management, including the service director who rather pushed for the organization to be free to change. There were no constraints.
something driving at that level.
There was strong collaboration with product marketing, which is sometimes a pitfall, and this was somewhat raised in the very first retrospectives, meaning that product marketing was quite absent
in defining features, in interactions with the teams. So, this has greatly improved. And today, no problems are being raised. Collaboration happens very naturally now. I would say that, if anything, the team is stretched. We could extend it to product marketing.
I would say that in the official organizational chart, product marketing is separate from the development team, but on a daily basis, that boundary does not exist. There are strong interactions with product marketing. I think this is one of the key points.
We evolved incrementally, not necessarily continuously, but in stages.
So, this is also the Kanban approach.
More than the... In Kanban, we had adopted a number of elements some time ago, including visual management. That has somewhat fallen into disuse. The teams have somewhat abandoned visual management, except for the Monday meeting. So, we have visual management there. We also use visual management for retrospectives, or even for certain events. But we no longer have stand-ups and not necessarily a board. On the wall, we no longer have those boards. However, from Kanban, we kept incremental evolution and the fact of starting from where we are.
Not wanting to change roles and functions completely and overnight. And finally, we relied on the people. It is really the individuals who participated in the evolution of this organization and the evolution of the product in terms of practices, tools, and metrics. So, we will quickly go over the tools.
So, I'm talking about the tools, I've talked a bit about the practices and how we evolved.
It is not the tools that forced the evolution of the team or the evolution of the organization that led us to adopt new tools. We cannot say there is a cause-and-effect relationship. There was simply a parallel evolution of the tools. Which certainly triggered a number of changes in behaviors or in the way individuals in the team interact. And then there are a number of practices within the teams that led us to adopt different tools. There is no cause-and-effect relationship that we can note. So, we adopted a version control system... Decentralized version control, Git, to name it. This changed the way, among other things, of working on features, by creating feature branches, working through pull requests, so pull requests that replaced
the code review meetings we used to have, which could be quite formal, Code reviews were something that had been introduced by the very first retrospectives, also. We had felt the need to have these code reviews or to do... design reviews as well. And then, by relying on the tool, we arrived at a system that allows us to both do code and design reviews, and manage the merging of features or bug fixes into the master branch.
We worked a lot on automated tests, but not only on test automation, really on what we wanted in terms of testability for the project, for the product.
So, pull request reviews, I mentioned that. We also more recently worked on containers.
You are familiar with container technology, Docker, you know. So, we set up a... A system, well, an entire organization around Docker containers that allow developers to have the entire platform on their dev workstation. And beyond that, we deploy containers in the cloud to test feature branches. So, this has also greatly facilitated our lives for... Improving the deployment process. Of course, we worked on continuous integration and continuous deployment. And then we set up metrics. So, David Anderson was talking about metrics earlier in the keynote.
We did not set up... very complicated metrics, we set up very simple metrics, focusing on bugs,
Why focus on bugs? Because it is much easier to find metrics on bugs than on features today.
We are in a fairly exploratory domain in terms of functionalities. We take a lot into account customer feedback by mixing it with ideas brought by marketing or technology.
Features can be quite informal when we start developing them. So, we focused the metrics on bugs because it still gives a good indication of how we execute the process.
And we count the bugs, the number of open bugs over time, and we look at the evolution of the bug cycle time. These are the two metrics we track, and they still give us a good indication of how we develop and how we improve the product.
No. We have these indicators, and these indicators can alert us to the fact that we need to work on bugs as a priority or that we can continue to work quietly on topics, on features. But we have these indicators available in real time, so they are updated daily.
They are updated automatically on a daily basis, and this allows us to raise alarms. We keep an eye on these indicators. And we can, in a fairly reflexive way, say, hey, the bug count indicator is showing an increasing trend.
It's time to work on the bugs.
We also have indicators, well, not indicators, but we also have a...
an indication of blocking bugs through the inability to perform in production. And in that case, the reflex is more than immediate.
So, I've almost finished the presentation. There are a few avenues we will certainly explore. I talked about microservice architecture. In the product organization, as you recall Conway's Law, there is the product organization. The structure of the product and the structure of the organization. In the product structure today, we have a few vertical components, 4 or 5. This corresponds to 4 or 5 web applications that communicate with each other or with the outside.
One of the avenues we would like to explore is a microservice architecture, which would be an even greater split in terms of web applications, in terms of independent components.
Why? To go further in scalability. We are working on a product that is in a strong adoption phase.
The number of users is rising significantly.
We are asking questions about scalability. One way to address scalability is...
Microservices.
We are also asking quite a few questions, and there are inevitably conflicting opinions among a group of 30 people on how to test.
Should we do BDD, ATDD? What is TDD?
Positions or ways of thinking differ from one individual to another. So, this is a project we need to pursue, which we have been working on for many months, but it is not yet completed. In my opinion, it never will be, but we have questions to address in this area. So, what tests should we do? Should we do regression tests? Should we automate them or not? Should we prioritize the speed of test execution? Should we prioritize the comprehensiveness of the tests?
Should we only test the features used by customers? Which features are used by customers? So, there are a whole bunch of... A whole bunch of issues revolving around testability. This is one of the avenues we need to explore.
I was talking about automation at the testing level, but we are not yet fully in a... In an organization where everything is automated, from the moment we create a feature branch until the moment we deploy the feature, In production, we still have areas of manual intervention.
And on this, there is a project that has already started some time ago, which is the cloudification of the application. For now, we operate in a hosted mode, but internally. And we are working on a mode where we will offload hosting to a public cloud.
And there, of course, we will have to automate deployments even more. This is an avenue that may also evolve the organization's structure in parallel. In particular, the notions of on-call duty that were handled by system administrators, even if they are part of the team.
We can ask the question of... Can everyone in the team start doing on-call duty?
But essentially, we will continue to work on individuals, people, and interactions within the team. This is a crucial point. In fact, none of these evolutions I have shown would have been possible without the people in the team feeling involved and adhering to these changes. And sometimes, it requires courage. There are changes that are not easy. There are viewpoints we must challenge.
So, it takes courage, and it is actually ongoing work to work with people, to understand how people work together.
And also to work on each person's skills.
So, I will end with a quote. I started with a quote and will end with a quote from Jean-François Jagodensky, who, during the Agile Tour Toulouse,
said this phrase that I find quite beautiful: a team is cultivated like a garden.
I did not talk about my role at the beginning. I am somewhat of a team cultivator in the team whose evolution I have presented to you.
I wanted to finish with that.