Dimitar Bakardzhiev
Transcript
Okay, so let's start. Hello everyone and thank you for attending this presentation. My name is Dimitar and today I'll try to show you how you can forecast your next project using just Excel. Hopefully there will be time for this at the end. First we'll go quickly through the slides and then focus on Excel.
Something about me, it's a bit long, but in general I'm managing a small software development company in Bulgaria.
I am AKT, which stands for a created Kanban trainer. And also I am publishing some books in Bulgarian, for instance the Kanban book. Which I see you have it in French, we have it in Bulgarian since like two years ago for you, and books by Elie Godrat and W. Edwards Deming, hopefully you know the names.
So what we face always, at least myself, is
software vendor is that clients come to us and with an idea for a new product and they always ask the questions how long it will take us and how much it will cost us to deliver.
They need a delivery date and a budget estimate no matter what they say. experience. Reality is uncertain, yet we as software developers are expected to deliver new projects with certainty.
And to increase our chances of success, we should acknowledge uncertainty and incorporate it in our planning and also exploit it.
My goal with the work that I'll show you is that I believe that we can't even though we can't control the waves of uncertainty we can learn how to surf. It's poetry. And this aligns very well with the no estimates paradigm. For me no estimates mean Two things. First, effortless estimates, which means estimates without any effort, done without effort, and also using no estimates of effort.
These days, deterministic planning, which is in use, forces certainty on the uncertain situation and masks uncertainty instead of highlighting it.
It is based on the first principle of scientific management. By Frederick Taylor, which says namely that in principle it is possible to know everything we need to know in order to plan. That's the basis of the deterministic planning and it manifests itself in
modeling projects as a network of activities packaged in work packages into the project's work breakdown structure. So project management paradigm believes that uncertainty does play a role, indeed, but it could be eliminated by a more detailed planning. That's why we have these networks, effort estimates,
etc., etc. We will challenge the project management paradigm.
And we will suggest that for planning purposes, it is better to model projects as flow of work items through a system.
Here's the definition. That we'll be using, the definition of a project. A project is just a batch of work items, each one representing independent customer value that must be delivered on or before due date. That's our definition for planning projects.
When we do this, we don't estimate the size of each and every work item.
Actually, we will consider there are only two sizes.
Small enough and too big, and too big should be split and not allowed to enter backlog.
Here is a quick animation to show how actually we envision a project flowing through a development system. It's a Kanban board and we have some items in the backlog and we start working, the system works on them and then they travel through the board and eventually they are delivered. That's our model of a project.
What we will try to reach is to have a high level plan using probabilistic planning. So the high level plan plans only the initial budget and the range of the time frame for the project. It does not include any details.
Schedules, for instance, are created upon the execution of the plan.
And we try to keep the focus on the project intent by making a series of small choices. And then we evaluate our actions and hopefully there are unexpected successes that we will exploit. So that's the execution part of the high level plan. Today we will focus only on the making of the high-level plan. The execution part is well covered in the Kanban method, the Scrum framework, and anything like that.
Our probabilistic planning method will be based on the reference class forecasting, which is based on the work of Daniel Kahneman, a psychologist from Princeton, who
won the Nobel Prize in Economics in 2002. So reference class forecasting promises more accuracy in forecast by taking an outside view on the projects being forecasted based on the knowledge about actual performance in a reference class of comparable projects. So the important thing here, the most important thing from my point of view and from Kanban point of view, is that we will be taking an outside view. We will not be looking how the projects were executed, will be executed. We are looking from the outside. And we will be...
Trying to put the new project into a distribution of comparable projects. It sounds complicated, but actually, again, the idea is that all the math is in a simple Excel file, and you will just use it. Here is, let's say, the proof of the method. So reference class forecasting, it consists of three steps. First, identification of a relevant reference class of past similar projects. The class must be broad enough to be statistically meaningful, but at the same time, it must be narrow enough in order to be comparable with the specific projects that we will be working on. In the future. The second step is to establish a probability distribution for the selected reference class. We'll see how to do this. And the third step is to compare our new projects that we need to plan with the reference class distribution. In order to establish a more likely outcome for the new project. So first step is to identify a reference class of similar projects.
We do this by comparing the projects by specific characteristics. First of each is the team structure. So we are looking into do we have projects with a comparable team structure. Again, we are taking an outside view. So most probably in our system, we do have teams that actually work on projects, and they are quite stable, relatively stable. So we can compare the last project the team worked on with some other projects. Next step, next characteristic to compare, are the technologies used comparable?
Next step is are the development processes comparable, and especially the process for preparing requirements or breaking down the work into work items or stories.
And also, are the client types comparable?
Like, it's the same team.
We'll have a different performance if it works for a startup or for Fortune 500 corporation due to the different way
of collaboration with the client.
Hopefully you can see that we are again taking an outside view and we are considering the context into
which the projects were executed and will be executed. Again, we are taking an outside view. Of course, we looked into the team structure, but again, we are looking from the outside, how the team structure is. Also, we are looking at the business domains comparable. There will be a huge difference between developing the same team, developing embedded software for Arduino or a new ERP system. Different business domains. So assuming we have such a
reference class of comparable projects. Next step, we should establish a probability distribution for this. And that's the tricky part because it is up to the planner or up to us what kind of parameter to use.
And the metric that we should use should allow us to take again an outside view, the development system. Also, it should allow us to compute, to calculate the delivery time, because again we try to ask two questions, how long and how much, and usually in our business, how much is proportional to how long. I believe so. And also the metric should make sense from a client perspective. So we should be able to go to the client and present this metric and it should make sense.
Usually, just like off-topic, story points make no sense. But time... Usually makes sense. And the metric we will be using is called tag time. Tag time is defined as the average time between two successive deliveries. It comes from manufacturing, but in our context, we'll define it as an average time between two successive deliveries to the customer.
That's the definition.
In what unit of time will be measured in time? In manufacturing, they measure tack time in minutes, in hours, even in seconds. In mass production, we'll be measuring tack time in days. For the knowledge work, it will be in days.
So here is a simple animation, what actually time is. This is the delivery curve for a fictitious project. Each yellow box represents a work item delivered at a particular time during the project. Again, we are not interested in the size of the work item or anything, how they were executed or anything like that. It is just an outside view.
When and how much work items were delivered. So at the left hand side we have the start time of the project and on the right hand time we have the finish date of the project. And we can see that after five days, the first work item was delivered and its tack time was obviously five days, right? Because its tack time again is the time between two successive deliveries. After seven days, we have another, we had another work item delivered. But in the same day, we have a third one.
Because we are measuring tack time in days and they are delivered in the same day, on the same day, the second work item will have a tack time of zero days. Again, it's seven days, so seven plus zero is seven, right?
After another two days, three additional work items were delivered. And again, following the definition, the first one had a tag time of two days, but the other two tag time of zero days. And we can see how it went and what we delivered. And the very important thing here, again, is that when we sum up all the values for tag time for each of the work items, and you can see them below, it makes the delivery time for the project. In this case, 22 days. Again, our metric will allow us to calculate the delivery time, just like that.
And here is the histogram of this fictitious project, and it is just to illustrate that we have work items with different tag time. Lots of work items with tag time of zero, which means they were delivered in parallel at the same time with something else. It's just for us to take better understanding on the pattern. back time for each of the work items and you can see them below it makes the delivery time for the project in this case 22 days again our metric will allow us to calculate the delivery time just like that
And here is the histogram for this fictitious project, and it is just to illustrate that we have work items with different tag time. Lots of work items with tag time of zero, which means they were delivered in parallel at the same time with something else. It's just for us to take better understanding on the pattern.
So this is the formula for the tack time if it is calculated by dividing the delivery time for the project by the number of work items delivered. And a simple example, for our fictitious project the delivery time was 22 days and we delivered 10 stories. So on average, between two successive deliveries, we had 2.2 days.
Using the same formula, we can actually calculate the delivery time for the next release or another project. If we assume that for the next project, we will have the same tack time, and we know the number of work items that should be delivered, We can multiply them and if we have, let's say, 45 stories to deliver next time,
it will take us 99 days, right? It's quite simple. The important point here is that this average value, the average time between two successive deliveries is indeed average, and it is unqualified average without any variance.
which is something we should not use.
In principle, but because we are following the reference class forecasting, it mandates us to have a probability distribution of this value. So we need this value, the distribution of tag time, somehow we need this in order to follow the method. So again, mathematically, it's not credible calculation to use only the average value. But even from a practical perspective, we need the distribution. It's quite difficult because for our fictitious project, we had only 10 work items. And these of you who are aware of... Statistically, you should know that 10 events are not enough, right, to claim a distribution.
We will use a technique called bootstrapping. So the bootstrapping, it was invented by Bradley Efron. It is well-known mathematical method. It is based on an assumption that a random sample is a good representation of the unknown population, which in our case means that if we have these 10 work items, we assume that they are proper good representation of the thousands of work items. we would have if our project had more than 10 work items, let's say 2,000 work items. Again, we are taking an outside view on the system right here. We are not looking into what's going on.
Bootstrapping does not replace or add data to the original data. That's the mathematical method. You can Google it and you can look for more details.
And it is based on the assumption of independence, which is a huge topic here. I will not spend time here on this thing, but if your user stories were prepared using the invest mnemonic, who knows invest mnemonic? Independent, okay. So they will be considered independent.
So this is the method for applying bootstrapping in order to prepare the distribution of the average tag time. So first we have this sample.
The presentation is on internet and it will be available. So in general we have the sample of tag time values. We have the number of work items delivered. And we draw at random with replacement. Values from this sample size, we calculate the delivery time for each of the sampling that we have.
And we repeat this many times.
Here is a quick animation, just to get it more easier. So TTI, there, you see 0, 0, 0, 1, 1, 1, 5, 7, that's our sample from our fictitious project. That's what we have. And it contains only 10 data points. We want at least 1,000.
When we sum them up, the delivery time was 22 days and the average time between two successive deliveries in our case was 2.2 days. What we'll do, you'll see in Excel as you'll see later, we draw one sample with replacement. Drawing a sample with replacement means we draw at random 10 times from this array. Of 10 events
And here is one sample, what we can do, could happen. And then we use this formula and we calculate the average time between two successive deliveries for this particular sampling. So we draw 10 items and we return them. It's with replacement, right? We return them in the bucket. Then we do this 998 times.
And eventually a thousand times. And the thousand times is exactly the same. We draw with replacement 10 of these events. We calculate the delivery time for the project. We divide by the number of work items and we get the average value. So the idea here is that this sample and this sample here above, they represent a probability delivery time for our project.
Again, we are drawing probable delivery times for our project. That's why we needed the metric to allow us to calculate the delivery time for our project. So when we sum up all the times, all the tag time values, we end up with the delivery time. Any questions here?
Yes?
When you make the sample, do you draw?
From the array over there, right? 0, 0, 0, 1. Totally at random. Totally at random. Using uniform, if you ask, like Excel, by default, is using random distribution for generating random numbers. So what I do is I have 10 numbers, and I at random pick one. And just note the value, pick another 10 times, and I sum them up, and it will make 21 days in this time. And I do it almost like 9, 900, and then 1,000 times again the same. So it's very easy. Excel is quite good at it, even though the random number generator is not that good. But anyway, for our purpose, it's okay. And we have a probability distribution of the delivery time for our project. Again, we are taking an outside view. Okay?
Yes, sure.
10 elements. 10 elements, yeah. You draw exactly the same number.
Exactly the same number, yeah. That's the bootstrapping variation that we will be using, right? Okay. I'll show you in Excel. And the result, what we have is actually this histogram shows the distribution of the average time. And we can see that the average time is 2.19 something days, which is almost 2.2 days. But along with the average, we have some percentiles, and we have the median, for instance, and the standard deviation.
So right now we completed the second step of required by the reference class forecasting method, right?
What should we do with this histogram? Again, the histogram actually represents a thousand values. That's it, thousand numbers. We should put it in a library.
Note the context where this particular data came from. Again, the team, the client, the technology.
And these two things together, we can call it stochastic information packet. It comes from the book Flow of Averages. It's quite a good name. Again, every stochastic information packet contains an array of possible outcomes, numbers, and the context where the numbers came from. So this is kind of a fingerprint of your development system from the outside.
Okay? It's just a good name. Now the third step of the reference class forecasting method is to compare a new project with the reference class. Right? And calculate the delivery time, right?
If you get back and recall a bit, this formula assumes a linear delivery rate. Right? 22 days, 10 work items, straight line. Usually, at least in my project, that's not the case. We have quite a bit different delivery rate. This is from a real project. And you can see that it started slow and then it ran very fast. And at the end, it was again slower.
This concept is called Z-curved and comes from a book from David J. Anderson, his first book.
In short, it says that most of the projects can be, the delivery rate can be divided in three phases or legs. And during the first leg, usually, it's around 20% of the project time, but we deliver only 5% of the work. The third step is something like that. Again, another 20% of the time. Again, another 5% of the work. And the real work is inside. 60% of the time, but 90% of the work. Again,
your projects might be different. Might have only two legs. Without any defects, for instance, right? But, or may have a linear delivery rate next release of the same project, right? The same product. Let's look into each of the legs. So each leg of the Z-curve is characterized by, first, a different work type. You'll see what difference there is, different level of variation, very important. And different staffing in terms of headcount and level of expertise. So the second leg of the Z-curve represents the system itself. It's a common cause variation we see here. The first and the third leg, these are special events that happen to the project. For instance, so the first step, usually, you can call it a setup time. Usually the team is climbing the learning curve or conducting experiments to cover the most riskiest items or is doing innovation. Setting up environments, adapting to clients'culture and procedure, understanding new business domain, mastering new technology, etc. That's why little work is delivered, right? A lot of thinking, less delivery. The second leg, it's the productivity period, and if the project is scheduled properly, it should be like clockwork, sustainable pace, no stress, no surprises. And the third leg is for cleaning up. The team will clean up the battlefield, fix some outstanding defects, support the transition of the deliverable into production. And if we...
Sum up the times for each of the legs, we will end up with the total time of the project, right? It's logically simple. So the first leg, the duration of the first, plus the duration of the second, plus the duration of the third leg, actually gives us the delivery time for the whole project. And having the formula for the tack time, when we substitute the delivery time for each of the legs, we can calculate the delivery time for the project using the number of work items delivered into the first phase multiplied by the tack time for the first phase plus the number of work items delivered in the second phase times the tack time for the second phase. And of course, plus the number of work items delivered during the third phase times the time for the third phase, right? But again, what we have here is like the average values for the tag time, right? Which is incorrect. Mathematically, it's incorrect. We should not use this.
We will be using Monte Carlo simulation in order to sum up this random variables, which actually are represented by the average tag times.
Let's say we have a, it's an example of a,
planning a new project. Let's say we have a new project that is the same with, we have a reference class of such projects in our database.
And the new project is for the same company, the same client, the same organization is using it. Remember the team structure and everything. It is the same technology. And we can safely assume that the development organization will deliver this project the way it delivered the previous ones. by the average times.
Let's say we have a, it's an example of calculating, planning a new project. Let's say we have a new project that is the same with, we have a reference class of such projects in our database.
And the new project is for the same company, the same client, the same organization is using it. Remember the team structure and everything. It is the same technology. And we can safely assume that the development organization will deliver this project the way it delivered the previous ones.
These are the distribution of the tag time for each of the legs of the Z curve. That's in our stochastic database. We have them. And we assume that the new project will follow the same pattern. Again, we are taking an outside view. We assume that the development organization will deliver the same way.
And what we have to do is actually to sum them up in a way. Important thing is that because it's our new project, we have to decide how many, the amount of work that will be delivered during the first leg, during the second leg, and during the third leg. And for this particular planning, I'll assume that 12 stories will be delivered during the first leg, 70 stories, the majority of the work, will be delivered during the second leg, and 18 stories will be delivered into the third leg. Important thing is that you should account for cost of delay, other work items for that matter, and failure load. But it is outside of the topic of this simulation. How you will be breaking the work down is outside of this. I believe that every organization has some kind of a procedure that is using on a regular basis, a method, and there is a way to break down the scope. So we will not be looking into how.
That's a visualization of what we will try to do. So we have to deliver 12 work items during the first phase of the Z curve, the innovation phase, the setting up phase, 70 work items during the second leg, the productivity phase, and 18 during the third leg. Again, our project might have, we might decide that there will be no innovation. We just start working, right? No first phase of the Z curve. Zero work items. We just don't care. Or no defects, nothing. We can zero here, right? But that's the general case. We have three legs, okay? Any questions here? So what we have to do, we have to sum up
random variables, these distributions. Although mathematically it's not correct to say we cannot sum up distribution, but we can sum up probability generating functions. So that's when we calculate this, it will give us the distribution of the delivery time for our new project.
Here is how we do it, right? We have tack time SIP stands for the distributions that we saw before for each of the legs.
We draw from the distribution one at a time, one value at a time. We use the formula that we saw with average tag time value.
And we end up with one probable delivery time for our project. We do this 49,090, 98 times, and eventually 50,000 times.
Again, almost like the bootstrapping. We just calculate a probable delivery time. In this case, we will end up with 50,000 probable delivery times for our project.
Which we can present as a histogram.
So the important point here is that the mean delivery time is 78 days. This is based on real data. The distributions you saw are from my reference classes.
So the mean time is 78 days. The 85th percentile is 90 days.
And it's up to you now how to use this histogram, this data. You can say we will commit the 85th percentile to the client, or we can go to the client with this distribution and say, that's our prediction forecast for the delivery time for your project. Let's talk about it, right? And there are lots of presentations how to use this. It's very useful.
Again, we are taking an outside view when forecasting a new project, and it will produce more accurate results.
faster, much faster, than using the deterministic inside view.
Again, because we have the reference classes data and we just use it, right?
These are the books that it's good to pay attention to. Kanban book, David J. Anderson's first book about the Z-curve and some other stuff. Flow of Averages, very nice book. And a book from Professor Fleaberg about the reference class forecasting applied to forecasting projects.
Now, let's see how we have 30 minutes, I believe.
That's my reference class data.
The one I showed you, this is from real project. You see, first leg Z curve and a range of values. Second leg Z curve, range of values. Third leg of the Z curve, range of values. Average tack time for each of the legs, that's real data. I got this using Bootstrap, and I'll show you how it's done in Excel, but let's say we have this data, and we want to forecast our project quite fast, as fast as possible. So the data, you see, 1,000 values. I have 1,000 values for each of the legs.
That's my stochastic information package. It is unique per my context. Right? A thousand. Just to show.
A thousand values, right?
Well, what we want to forecast is this new project. We decided that we will be delivering 12 work items during the first leg, 70 work items during the second leg, and 18 during the third leg. That's my input. That's the only thing I will enter here. That's why it's highlighted in yellow. That's the only thing I will enter.
And what I'll do, I want this
think, right? Which is, yeah, you can see the median, the standard deviation, the average and stuff. That's what I need. How I can do this? I go to calculation and I say calculate sheet.
I run it 50,000 times. I can run it a million times, and Excel is not that good in it, right?
The numbers are a bit changing, if you see. But pay attention to the average and the 85th percentile, how much it will change.
85% it changes a bit, right? Because we have only 50,000 runs of this simulation. If there are a million, it will not change at all, right? Let's go and change our decision. Actually, we will be delivering 15 work items during the first leg.
We will have a little less defects this time. That's our decision, right?
And let's go back.
Let's see what will happen.
Finished. Oh, quite fast, right?
It changes a bit, right? It's changed a bit, but just a bit. And that's based on my data. I really don't know. I'm not looking into how the team will work, right? I'm taking an outside view. See, I believe it's fast enough. Let's put here 100 work items.
So I presented you the mat, but the idea is that the mat is inside a tool, and we just use it. See?
Okay, any questions here? There is time for questions.
For me the problem is, it's a lot of work to have enough data to make the calculation.
Right, that's why we have Excel.
You have to collect data in the same condition, so you have to have the same team to work.
Comparable, like the most important thing is comparable, right? It might be the same, and again, it's not the team, the structure, right?
But you have a lot of parameters too.
Right. Yes, the context should be comparable, right? If I come to your company and start planning instead of you using my data from Bulgaria, it will not make sense, right? So I believe it's natural to take the context into, yeah?
It's your organization. So the idea here and the reference class forecasting is based on the notion that we are managing organizations. And the next project the organization will be delivering will be... Well, it's the same organization, right? If the technology is comparable, if the team structure is comparable, if the client is comparable, if the technology is comparable, and of course we've done some analysis here, how much work items will be delivered, right? We do some work here, right? It takes like two seconds to come out with the delivery time. And it's the fastest possible way I am aware of, right? But again, we have some data here to collect, right? We have here thousands of items. Usually, a project, so how How we can do this? I believe we have 20 minutes, so we have... So that's my initial data. That's from a real project from like three years ago. What I need in order to prepare this stochastic package is only the delivery time, like the dates, and how many work items we delivered to this. So you can see in this particular project, we delivered 78 times.
78 times.
That's what I need as an input. Again, I'm taking an outside view. And again, very important point. This method is not the gospel, right? It is by taking an outside view. Then you can go, prepare the network of the project, run a simulation or anything like that, come up with a different number, Compare both numbers, right? But this is from an outside perspective. Again, in Kanban, we don't estimate. We are taking a system perspective here. There might be other perspectives. Inside, we can look inside if we can, if we have time. We can look inside and compare. It's just another tool, again, right? So that's our input.
What we are interested in is how many of the work items were delivered. Like we need, remember, the time between two successive deliveries, right? We have the deliveries, the number of deliveries. We have the data, the date when they were delivered. So it's a very simple, you can see the formulas. And again, this is an Excel that I share, is how to calculate the attack time. And here we have two days, four days, four days, and here I calculate just the number of items delivered in parallel because the attack time is zero, right? And what I end up with, if you can recall, is this histogram. That's for the real project. 160 work items delivered in parallel. But we have one delivered like 21 days. After the previous one, right? It was in the first phase, an experiment that we run a prototype. So it's quite different.
And what I do, I say, well, let me calculate the distribution only for the first leg. And because I know the project, I go down, down, down, and I say, well, up to this point, that was our first leg, right? See, it's a different highlighting. I know the project, I'm preparing the data for it, right? After that, we started just like the second phase, right? And what I do, I copy this data from here, real just copy, and paste it into this So it's quite different. And what I do, I say, well, let me calculate the distribution only for the first leg. And because I know the project, I go down, down, down, and I say, well, up to this point, That was our first leg, right? See, it's a different highlighting.
I know the project, I'm preparing the data for it, right? After that, we started just like the second phase, right? And what I do, I copy this data from here, real just copy, and paste it into this
into this, like, how many do we have? 42 work items, right? The second in the first leg. And using Excel, it's a very simple formula. And if you check what Excel is doing here,
I have 42 events and I am sampling 42 times on average at random. And the formula in Excel is like, what I'm doing is that I am saying Excel give me one of these 42 values at random.
and place it here, and here, and here again. It's one sample, right? It's one sample. And here,
That's the formula I know that actually, the formula that I, when I have this sample, I sum it up. You can see that's the sum of the sample. And you can see a different timeframe for this first leg.
It could be 66 days or 150 days, right? And I do it a thousand times.
And here is the distribution of the average time, another 1,000 values. And how I can calculate this, that's what I'm looking for.
Where it is.
Delivery time.
PDF. Yeah, so that's the distribution of the tag time for the first leg. And here is the frequency, here, the left hand side. And what I should do, I just go and say calculate the sheet, and it starts simulating a thousand times. See? And I can do it,
The best is to do it 50,000 times, right? But because it's for illustration purposes, it's not. And I can do it several times, right? And you can, if you check here, the curve is changing, but
The statistical parameters are not changing. If I run it again, see, that's for the average. So it will, yeah, it changes a bit, right?
But not that much. I do it the same for the second leg.
Yeah, this is for the second leg, and then I do it for the third leg.
I take this thousand values from here, right here, and I copy and I paste it in a new file here.
The bootstrapping.
Which, what?
The resampling.
Okay, so here in this resampling, what we are doing, we are selecting one value at random from this column, right? That's our data.
Those are real values.
Those are real values, and we are taking one at random, okay?
No, random from this list. You see, run between one. So what I'm saying, say, Excel, here are the sequence numbers of my sample.
Tell me which sequence number, just one number at random. Excel says, well, 10.
The value for the 10th value is 1. Next time, 18. The value is 0, right? So that's why the random generator in Excel is using the uniform distribution, which means... That we have a similar distribution, right? Between 1 and 42.
It's built in. And when we have this, the list, we sum the list up. In this case, it is 36. It is one of the probable
time frame for this
curve, right? For this leg of the project, of the Z curve. And we go here, and this is a function in Excel called table, which one may say like, do a dissumation, which is the values from here to here. Do it for me like automatically.
Yeah.
Did you compare the simulation result at the end of the project? Did you compare the simulation with the actual duration of the project?
Yeah, and it's a good question. Yeah. So here you can see the duration of the first leg of the Z curve.
Right? The duration of the first leg. The delivery time. This is the distribution of the delivery time for the first leg. But we know actually how much days the first leg took. Right? And here we can see that the average time out of the distribution should be 31.6 days.
Let's see how actually it took.
It took 32 days. That's the real thing.
Our simulation actually, again, the bootstrapping does not add or remove any data.
It just uses the available data in order to prepare the distribution.
And it is expected that it will be like that, right? The average will be still the same. The average value should stay the same. It is expected.
And if we go here,
I have it in another file for a real project, but I have had cases where the whole duration of the project actually, when I prepare the probability distribution for the whole project and then I go back to the real thing, And I found that actually from a probability perspective, we had only 40% chance to deliver in the time we delivered, right? Because what's very important here is that people, we often forget about this, is that
So this project
The median value is 88 days. The 85th percentile is 130 days. The median means that 50% of the time, we will deliver less or equal to 88 days. When we say 50% chance of delivering, and we actually, what we hear is actually 50% chance of failure, right? It's like we hear failure, 50% of the time we'll fail. But the truth is that 50% of the time we will be less than 88 days.
And we can use this information depending on the situation, right?
I usually use the 85th percentile when I have to make a commitment or anything like that. Because the 95th percentile is quite big.
It's too conservative, right? And again, if you heard Don Reinerson, that actually variability is where we make money, right? So if we always say, well, we will deliver definitely in 115 days, definitely we will be delivered, right? But who will make money, right? We will not be making money. So we should use this information, right, that 50% of the time will be less than 88 days, right? And you can use it in a different way. There are contracts which are called fixed. Plus incentive. So you may say, well, let's fix it on 103 days. It's an example, right? It's up to you. But if we deliver for less than 88 days, we have a bonus.
It's just an idea, right? It depends on your situation. But having this data, it's much better than having nothing, right? And again, this data is supposed to be based on the actual performance of your organizations.
So this method is first time it was used. It was for forecasting projects. It was not for software. It's for forecasting.
Construction projects in the range of hundreds of millions pounds.
If you go and read the idea behind the reference class forecasting, it tries to avoid some cognitive biases, mainly a bias called planning bias.
But it's off topic here. Again, that's why we are taking an outside view. We don't go into it, we don't make any effort estimates, we don't go to people and asking them how much time it will take or anything like that. We may go after having this and say, well, how much time do you think this project will take? And maybe the expert, our expert will say, well, around 90 days, right? That's my gut feeling, right? And it's another forecast. But this one, again, is based on a solid mathematical ground.
Supposed to be using your real data, which is really simple, right? Remember, we need only when and how much you delivered. That's it.
Sorry, I'm just more like technical, but where are you using tag time and then for example, through time, cycle time, or it's another?
It's not in this method, right? But throughput is the reciprocal value of tag time, right? So you can safely replace it, but it's the same results, right? If you like throughput best. I like tag time because I can sum it up really easy, right? And summation, we can replace.
The things, we can remove one of the things, not to sum it up. Like it's really safe from a statistical perspective to sum things up. Dividing thing, multiplication, it's not safe all the time. But summation is always safe. It's the safest way. And again,
Just to...
Please go to SlideShare.
When you type no estimates, this presentation will be top of the list. If you type reference class forecasting, it will be third of the list.
There are the links to both files I just showed you. The Excel files are there. I have prepared videos how to use these files. Like if you forget probably how to, it's possible, right? You can check the books, especially this one.
And the other one, it's, yes?
If you don't like this version, which I don't see why you wouldn't, but Troy McGinnis has another version which he's put up on GitHub. So if you have a look at Troy McGinnis'GitHub account, he has another one. You could run both of them and compare the results. That would be fun.
The other thing I was going to say is that I really like systems that are easy to game and in a way where the gaming actually produces the sort of outcome that you're looking for. So it's pretty obvious to see from the development teams.
You just slash your story in a small time. We want that. That's a good thing. So it's kind of encouraging the sort of value that we actually need. So I'm all for things that are easy to gain if they produce the right sort of results.
Right.
One last thing if I may. This is the theory at the top of the list for no estimates, but it's actually it. Your Z curve makes a bloody good case for why you shouldn't bother with projects. Exactly.
Right, right.
Just about this gaming the system. Actually, even though, if you think about it, even though the developer decides to break, to produce twice work items, right? Break the product scope in twice more work items, from the outside perspective, there will be no change, provided actually you take some data after they decide to make smaller work items, right?
But Joshua is correct. For instance, we have data when they used method for breaking work done, let's say, which method? Product sushi. Okay, if you are aware. One method. And then they decide to use another method. If you go back, it actually will invalidate our data because the method for breaking work done is changed. Our data is invalid from that moment on. We have to collect some more data, right?
We only need a little data.
Yeah, yeah, but our data is invalid. So context is very important. Here it is shown that context is the king.
So it will lag, the new method will lag by 11 days.
Yeah, yeah.
And you can use it with Scrum planning, release planning. We can use it with XP, Kanban systems, waterfall, anything. Why? Because we are taking an outside view. Again, outside view. We don't care how the system works internally. We are just looking from the outside. And again, this is just one method. You can use anything like that, apparently for just like something different, but this is something very simple, very fast, and it is statistically solid.
Any other questions?