Secrets of High Quality Development with Scrum Co - Creator, jeff sutherland

Hi, everyone. My name is Steve Miller. And I'm the vice president of ALM Solutions at SmartBear Software. I want to welcome you today to today's webinar, "The Secrets of High Quality Development Using Agile, Scrum, and Lean." We're really fortunate today. Our speaker is Dr. Jeff Sutherland. He is one of the co-founders of Scrum, and he's also the CEO of Scrum, Incorporated. A little bit about our agenda today. We're going to talk a little bit about our housekeeping here, the things that we can do during the webinar. And then we're going to dive right on into the demonstration and the webinar. And we're going to talk about the secrets of high-quality development using Agile, Scrum, and Lean. Dr. Sutherland will talk about how to implement Scrum, key metrics that you can use for hyper-productive teams. He'll talk about the best Scrum teams, how they deliver five to 10 times faster than waterfall teams. And then he'll finish up talking about common practices for high-performing development teams. We'll allocate enough time at the end, probably about 10 to 15 minutes, for Q&A. So make sure that, as we're going through this, that you ask as many questions as you feel that you need to. Now from a housekeeping perspective, you'll be able to use the question box in the bottom left area of your GoToWebinar and-- that's right below your help area-- and from here ask any questions. We'll see these questions flow in. And we'll make sure that we answer your questions as we go along. Also, be sure to know that we're sharing this event live with a #ScrumCEO hashtag. Additionally, we'll be conducting a follow-up session at the end of this with the same hashtag. So be sure to be looking for that. And finally, from a recording perspective, we will be recording today's webinar. And we will make that available out at www.smartbear.com within 24 hours of the following event. Also, any attendees here today will also receive a follow-up email with instructions on how to download and access the recording. So at this time I'd like to introduce Dr. Jeff Sutherland. And I'll turn it over to you for the presentation. Thank you. Thank you very much. We're going to talk about Scrum. And we're going to talk about Software in 30 days. So there are books available, recently published, The Power of Scrum is a novel about how a CTO of a company rescued a project that was at high risk. And Ken Schwaber and I are using Wiley for publishing the book Software in 30 iDays./i Next month it'll come out. And this is a business-oriented book for Agile leaders. My background is as a fighter pilot and as a professor in a medical school, which is maybe an unusual background for someone who invented a process. But the Scrum is based on what I learned as a fighter pilot flying over North Vietnam. Because what I discovered in software is that projects are very high risk, and many of the risk-avoidance strategies that I used in flying are embedded in Scrum. The other thing is after my last tour at the Air Force Academy, I joined the University of Colorado School of Medicine. I spent 11 years funded by the National Cancer Institute doing mathematical modeling of biological systems. So that got me really knowledgeable about complex systems, and how do they work. And that knowledge has been brought to bear on how to work with teams building software. Since my time at the medical school, I've been in 10 or 11 companies; most of them have been Scrums. Since we created the first Scrum in 1993, they've all been Scrum. One of the interesting things that's going on now is part of my time I spend as the senior adviser to a venture capital firm. And 18 of our companies are doing Scrum. And they are a really good laboratory to see what works really well, and what doesn't work so well. So we're in there coaching our venture companies all the time on how to do Scrum better. Scrum is Agile. Agile is actually a conceptual framework that consists of some values and principles. A group of us met many years after Scrum was created in Utah. And we spent a day and a half really arguing and discussing how to better improve the software development process. And about the middle of the second day, Martin Fowler went to the whiteboard-- you can see him in this photo-- and said, "Is there anything that we, as inventors of processes and consultants and authors, is there anything we can agree on that everybody should be doing in software development today?" And the first thing we agreed on was that the way teams worked, the way they communicate, is absolutely critical. And even though many of us were involved with processes and tools, and maybe even selling tools, that if you can't get the team communicating well, the process and tools don't matter. We had over 10 years of research at Bell Labs, with lots of data, showing clearly that the communication patterns and teams is what drives productivity and quality. The second thing we agreed on was that working software in small, short iterations was a lot more important than documentation is. Getting software in front of the users so that you can get feedback on regular intervals, that's what drives quality. That's what drives speed. That's what built the internet. That's what's building mobile applications today. The third thing we agreed on was that getting the customer involved in the software development process was critical. Holding a customer at arm's length through a contract negotiation actually disrupts the development cycle and produces poor software. So one of the interesting things that the head of the FAA development says is the unique thing about Scrum that's different than his traditional project development is that in Scrum we have the customer embedded in the team as products owners. So that's one of the major differences of all the Agile practices. We have either the customer or a representative of the customer on the team that's doing the development. And finally, we know from data from the Standish Group and elsewhere, surveys over the last 20 years, that a huge amount of requirements change occurs during development. The average in the last survey the Standish Group did was 65%. So if 65% of requirements change, that's going to cause traditional development projects to be late and to be over budget. And in order to try to control that, they've set up change control boards. And what happens is during development the customer comes up with better ideas. They come up with new ideas, things they need. Their business changes. And these change control boards prevent them from getting what they want. So at the end of the day, there's a huge amount of software that's never used. The average worldwide for features not used after delivery is over 65%, are never or rarely used after delivery. So there's a huge amount of waste in the software process and development system worldwide. And one of the major thrusts of Agile is to try to remove some of that waste. It's not to say that we don't think processes and tools and documentation are useful. And we actually do more planning in Agile than you do in a traditional project. But those things are not what makes great software fast. And it's not what makes users really happy. We need a much more Agile process to do that. When we talk about Agile, most people when they're talking about Agile today mean Scrum. This is a survey from version one, over 6,000 responses, over 90 countries. Scrum is worldwide. And you can see that 52% of the responders who say they're doing Agile, say they're doing Scrum. The fastest teams we see are actually doing extreme programming practices inside the Scrum. That's 14%. So it turns out that engineering practices are really important. Actually we had implemented virtually all the XP practices in the first Scrum team when we started Scrum in '93. But when Ken and I got together and wanted to formalize the process, Ken felt strongly that we would get much faster adoption if we didn't require the engineering practices but created a framework for team functioning where the core of the framework was inspecting and adapting continuous improvement. And once you start doing that, the team would see immediately that they needed to improve their engineering practices. And they would pull them into the Scrum. The remainder of what's called Agile, 9% of the companies have some hybrid process they're using internally. And 8% really are not clear about what they're doing. So it's quite dominant Scrum out there. The other thing that's interesting about Agile is recently I've been meeting with the CEO of the Standish Group that has done more surveys, probably, than any other organization on software development, all the way back to the early '90s, maybe even back to the '80s. And as they follow this, they realized recently, as they summarized the data from most of the last decade, that those organizations that say they're doing Agile, their success rate on projects is three times the success rate on traditional project development. And the data is so overwhelming that the Standish Group now takes the position that Agile should be used for all projects. That traditional project development, the error rate is so high that it doesn't make sense to use it for any project. And an even more forceful statement was made by Gartner in this year's professional advice to companies doing application delivery. And they're saying that with the kind of applications that people are using today on the iPhone and the iPad or the Android phones, users are losing patience with traditional, old-style IT. And, basically, Gartner Group is saying to all their customers, say goodbye to waterfall. It's over. You need to move away from a project-view of the world and view your portfolio of projects like products, where you actually try to improve them, try to enhance, and support them properly through their life cycle. To do that, you're going to have to build teams that have cross competencies, the collaboration across teams, which is an actual practice. And you're going to have to be much better on the usability side. And finally, the key point is start removing some of the technical debt, which is the old code, the bad code, the stuff that slows you down, the stuff that disrupts users. There needs to be a plan for systematically rehabilitating old systems and moving them into the current technologies. Now it's interesting to look at where Scrum comes from because, as you can see, most of what is Agile is Scrum. But, actually, most of, or maybe all of, what is Lean is just looking at Scrum from the other side. Let me describe why that is the case. In 1993, we were looking at the best way to formalize a process for delivering new tooling, object oriented development tooling, round-trip engineering tools that would allow you to design and generate code and then re-gen design from code. And we found the best approach to formalize a process for this was articulated by two Japanese professors at the Harvard Business School. Ikujiro Nonaka, who is now viewed by many as replacing Peter Drucker as the leading management guru on the planet was working with Professor Takeuchi, who was dean of the most important business school in Japan for over 20 years, recently had to step down because of retirement age and did not retire but came back to the Harvard Business School. So I've been working with him this past year. And last year I visited Professor Nonaka in Tokyo. And I found that the Japanese view process and managing the flow of work in a little different framework than Westerners. Their original paper was published in the Harvard Business Review. It was called "The New New Product Development Game." They had this chart in that paper-- this is back in 1986. And the chart in the upper right, at the top, is what they call Type A project management. And that example was actually derived from NASA. They brought the senior management of Fuji Xerox to NASA. And they took a look at how they would have built the space shuttle. First they had years of requirements definition, then some years of analysis, years of design, and years of coding, and then our implementation, then years of testing, and then some years of deployment. And so after 10 or more years, they launched the space shuttle. Well, Fuji Xerox thought that was best practice. So they went back and tried to implement that in Tokyo. But what they found out very quickly is that the failure rate was very high. And because they had been trained by people like Edwards Deming after World War II to do a root cause analysis of these failures, they determined that the failures were caused primarily by large documents that would be handing off between silos, siloed teams. So they decided to abolish those silos and also the big documents and get the teams working together. And that became what Takeuchi and Nonaka called Type B project development. When Takeuchi and Nonaka look at a company like Honda or Toyota, they see teams working together doing everything all at once. They're doing a little bit of requirements, analysis, design, coding, testing, all at the same time. They're delivering working prototypes of cars and evolving those in short intervals. Takeuchi and Nonaka called this Type C project management. And as they looked at many companies around the world that we're doing this, they said it reminded them of the game of rugby, and particularly the Scrum formation in rugby. Scrum is a short form of scrumage, a rugby term and also of football, American football term. So when we read this paper and saw how the teams were put together, we thought it was a really good formal model. And because Takeuchi and Nonaka called it Scrum, we called it Scrum. One of the first things that happened was the team said, well, what will we call the team leader. And then the team said, well, we should call the team leader the scrum master. And Nonaka pointed out that all the great teams they'd seen were characterized by autonomy. They were in charge of their own destiny. They had a higher purpose, a higher goal. So, for example, the Prius team at Toyota, they had to build a car that got twice the gas mileage in half the time. And their goal was to actually help with the oil crisis, help make the world a better place by building a better car. And to do that, they worked with teams where there was cross fertilization, cross learning, working together constantly to move the Scrum down the field. Takeuchi and Nonaka noticed some other key things that they saw in these teams. Initially, a built-in instability, the management would give them strong goals and challenges and then they would back off, almost like investors investing in a company. And the team would then have to figure out how they would do often an impossible task, like building the Prius. And because they didn't know what to do, they had to figure it out. And that caused them to self organize this team. They had to figure out how to organize themselves to move as fast as possible. And so when they did that, they had overlapping phases of development. They had cross learning within the team. The management didn't totally release control. But it's a very subtle control that management has on these teams. And the team is also responsible for organizational transfer of learning. Building a new product, they need to teach the rest of the company their process, their tools, their product, how to deploy that product to the field. So this is what Takeuchi and Nonaka we're seeing. And as I worked with them last year, I noticed they never mentioned the word "lean." In fact, I even went to Takeuchi's book on Toyota. And I searched it for the word "lean." And the word "lean" is not in the book. So I was fascinated. About a month ago, I was invited to go over and talk with the Lean Enterprise Institute. We're almost right on the MIT campus. And right across the street, the next building, is the Lean Enterprise Institute. This was founded by Professor Womack in the '80s. He had an MIT project to figure out why were the Japanese winning in the car industry against the Americans. And as he did that, they wrote a book-- the top, right book on the slide-- they wrote a book called The Machine that Changed the World, which became a very popular book. And some years later, Womack wrote the book Lean Thinking, which was really the original definition of Lean. It really is a Westerner's view of what goes on at Toyota. But when Takeuchi wrote his book, Extreme Toyota, he was focused on a completely different aspect, the product creation aspect. The Lean people were focused on the assembly line and the production techniques. But Takeuchi and Nonaka focus on the knowledge generation of the teams that build the new products. So when I went over to the Institute, and we met with Steve Bell, who's their IT guy there. He's written books on IT. I asked Steve why is it I hear nothing but Scrum from Takeuchi, but we hear nothing but Lean from Womack. And they're both talking about the same company. How could that be? And Steve said, well, Scrum is about knowledge generation. And he drew the yin-yang on the board. He said, the left side of the yin-yang is knowledge generation. That white dot in there is Lean. And the right side of the board is the production techniques, which we call Lean. And the black dot in there is the Scrum inside the Lean teams on the production line. So he said they're really the same thing. It's just looking at a different side of the coin. So Agile, Scrum, and Lean, they're all the same thing. That's one of my key points. And just to reinforce that point, John Shook, who is the CEO of the Institute, he kept on coming into our meeting with Steve. He was in another meeting, but he just couldn't resist coming in. The second time he broke into the meeting he gave me the book on the lower right of the page, Lean Production and Process Development. He said this is the best book on Lean production, Lean product development. And the guy that wrote it, Allen Ward, he was a genius. And he wrote down what the best companies in the world, particularly Toyota, do for Lean product development. So I opened up to the first chapter of the book, and I read, here's what Lean process development is. They have an entrepreneurial system designer, what Toyota often calls the chief engineer. And as you read a description of that, it sounds a whole lot like the Scrum product owner. And the second thing they have is teams of responsible experts. When you read that, it sounds like a Scrum team. They do set based concurrent engineering, which we use some of those strategies in the first Scrum team. Apple uses this all the time. Most software teams don't do a very good job of that. But that's a key component of Lean product development. And finally, they're driven by cadence, pull, and flow. Cadence in Scrum is the sprint, the iteration, the cycle. Pull, Scrum teams pull work into the sprint. They're self-managing teams. And flow in Scrum is the velocity of the teams. So I sent this back to John Shook and Steve Bell. And I said, hey, I think we're both talking about the same thing here. And so far they're in total agreement. So what is it about Scrum that makes things radically different? Well, it's like if you take a traditional software development project, and you took a Toyota team and put them to work on that project, it would look a lot different. Back when I started into software, coming out of the medical school, I was hired to be a vice president of advanced systems for a large banking company. We had 150 banks. And I noticed, as soon as I went in from the medical school, that the projects tended to be late. People were upset. The developers were put under pressure. They had to work nights. They had to work weekends. They went on death marches. But no matter how hard they worked, the customers were still unhappy. And coming from the medical school background and never seeing commercial software development, I thought this is crazy. And they particularly wanted me to straighten out their automated teller systems, which they were deploying widely in North America. So I went into the CEO's office, and I said, hey, I want everybody that touches that product. I want sales, marketing, support, all the engineers, the systems engineers, the deployment people, I want them in one group. We're going to run them as a small company within the company. And he was a senior management team, you can be my board of directors. And once a month I will present to you a profit and loss statement and brief you on the status of the company. But other than that, you leave us alone because in order to get this thing really working, we have to run it in an Agile Scrum Lean way. We didn't have the terminology then. And to my surprise the CEO actually agreed. And we took a company that was a loss leader with negative 30% margins when we started. In less than six months, they were over 30% positive margins, the most profitable unit in the bank. And the bank started investing millions of dollars in that division. And the reason is that we-- by using a team process, getting people aligned, all of our projects started to come in early. I positioned the management so that they were supporting the teams and positioned the environment so that people were having fun. And our customers were really happy. So they were willing to pay us more money. And we made more revenue. Now, one of the things that happens when you start this up-- and you'll see this in Scrum-- as you get the teams aligned, and then they have to figure out how to pull their work into the system and figure out how to do their work, you'll see this built-in instability that Takeuchi and Nonaka talk about. The team will move out of a stable state, which is the late, upset, pressure, unhappy state. And they will move into an unstable region. In any systems there are only a few stable states. And most of it is unstable areas. So as the team moves into instability, you need to put certain things in place to get them to cross into the next level where you want them to be. In Scrum, we need to get the backlog ready in a really ready state. And we need to get the software done. Often, teams have trouble getting software done, tested, and usable by end users in a sprint. And so things are better, but not where they could be. So if you look at some of the big implementations, the one at Yahoo, for example, most of the teams where not done at the end of the sprint, but they had a 37% reduction in cost of software development. And it was saving them over $200 million a year just for the 100 or so teams that implemented it. It was only part of Yahoo. But there were about 50 teams that really had implemented the process well. And they were getting 10 times the improvement. So you're going to see a lot of different pictures out there when you look at Scrum. If you go to a company that is Lean, like Systematic in Denmark is a CMMI maturity-level five company that spent seven years moving up the maturity ladder of CMMI, and then decided that they needed to remove a lot of the overhead they generated in process and reporting. And they were going to do it by implementing Lean. And they invited Mary Poppendieck to talk about Lean tooling. And they ordered the implementation of those tools-- so the left row is value. What they did initially was good refactoring and testing on the team. They trained the management in eliminating waste. They trained them in value stream analysis to see where the waste was. And Mary Poppendieck told them if you're going to implement Lean, you implement Scrum in software because a good Scrum is Lean. And this was the summary of their results. They spent seven years going from CMMI one, on the left, to CMMI five, in the middle. And in those seven years they reduced the project cost by 31%, got much better predictability, and much better quality. They then implemented a series of pilot projects with Scrum for six months. And at the end of the six months, they found that the cost of projects were cut in half. The defects were cut almost in half. So the net, by flipping to Scrum-- and remember these guys can do a perfect traditional project, perfect waterfall-- and they took the same guys doing the waterfall, and they put them on Scrum teams. They said, OK, do the same project, but do it in a Scrum way. And on the average they got a 80% reduction in planning cost. Project management overhead was cut by 80%. 40% reduction in defects. 50% reduction in rework. 100% increase in productivity. They have such good data they could immediately start bidding two prices for a project, 10 million Euro for the waterfall version, 5 million for the Scrum version. The only difference is the Scrum version will have 40% fewer bugs. They drove this doubling of production by using Lean tooling. In this particular case, we have a control chart where every red dot is a defect. And what they found is that if they could fix every defect in less than nine hours, every team would double. So they have the best data in the world. And they've simplified it to the point where even a brand new Scrum team can say, hey, if we get all our bugs fixed during the sprint, and we have usable software at the end of the sprint, we are guaranteed to double production, just like Systematic. So what you'll see, just going back to the Gartner technical professional advice for 2012, what they're saying is you need to focus on iterations. You need to focus on testing. You need to focus on continuous integration. You need to focus on refactoring. So these are the engineering practices that when put inside the Scrum make it work really well. For those of you who may not be familiar with Scrum, it's a very simple framework. It only has three roles-- a product order that builds a backlog, a Scrum master that's a coach for a team that's building the software. It has three primary meetings-- a sprint planning meeting at the beginning of a sprint; a daily meeting every day, no more than 15 minutes; a sprint review at the end. The sprint review is followed by a retrospective. And the sprint planning is usually preceded by product grooming. So often you'll see five meetings in Scrum teams. The artifacts are real simple. Scrum was designed to be self-reporting. So the product backlog needs to be visible. And you'll see charts on the wall. You'll see what we call Scrum boards. They're combine boards coming out of Lean production in Japan. And out of that product backlog, teams will pull part of the backlog into the sprint. That's the sprint backlog. And then they will measure their progress by using burn-down charts. So a very simple framework. Here is the dynamic model. This is what we use in the venture company. Here's what we expect. We expect that teams will get their software done at the end of the sprint. And they're going to have it tested. And it's going to be usable by the users. And in order to get it done, they're going to have really good backlog at the beginning of the sprint. And in the middle of the sprint, the team is going to be moving blocks. Every day they're going to be inspecting and adapting to go faster. And if we can do that, we can get significant speed. So you can see that while Scrum is a simple framework that does not specify engineering practices, in the venture group we have found we absolutely have to have continuous integration, automated testing in the build during that integration, code reviews or peer programming. All these things that-- why SmartBear is supporting this webinar is because they've built a complete tool kit to do all of these things. And these are fundamental. So from the point of view of our investments, at the board level having a good implementation of Scrum is critical. And inside that Scrum, having continuous integration, automated testing, and good code reviews is critical to be competitive in the global market. So we never implement Scrum without those things. So let me show you some of the things that happen when we get working with a venture company. Here is a paper that the Japanese call the A3 Process. Toyota uses it for everything-- for strategy, for reporting, for fixing problems. And it has a format that starts with-- it's all on one page. It has a format starting with a title. So one of our venture companies had teams that were failing sprints. And they had over 50 developers and seven teams. And sometimes they would all fail together because of common components. So teams were not getting software done and tested. There were critical components failing every other sprint. And if you dug into the current condition, you found that engineers weren't working together in the right way. They did not have continuous integration implemented, so they were unable to effectively test, causing failures. The estimated cost of this was 2 million Euro a year, which was a significant proportion of our investment that was being totally wasted. So, obviously, as investors we were excited about that. The target condition was to get tested code working at the end of the sprint. We felt we cut 90% of the wasted effort out of the system. And that would save us at least 1.8 million Euro a year for those teams. Now the way the Japanese approach this, they ask why does the current condition prevent us from reaching the target. And they have many ways of doing root cause analysis. But one of the simplest is the five whys. So in this case, I asked the teams, why did they fail. They said, well, the engineers had different ideas about the design of components. And they didn't work together at end of the sprint. I asked them how could that be. Why? And they said, well, the Scrum master should have made sure the teams were talking about one another, but they weren't. So I said, well, this happened last month, and we beat up the Scrum master. You think we can beat up the Scrum master again and fix this? They said, no. We don't think that will work. I said, well, what about continuous integration. We talked about that. And they said, well, we can't get continuous integration because their product owner will not put in the backlog any engineering improvement and practices. He wants new features, new features, new features. I said who's your product owner. They said the CEO of the company. I said, oh, well, I see you have a problem. So what we need to do-- I said I'm going to meet with our board member, which I did. And we got on a conference call with the CEO. And our board member was not going to let the conference call end until the CEO committed to implement, that very month, continuous integration. I was to go over and see what happened. I did. At the end of the month, it was working. The VP of engineering told me the average production of the team went up 20%. And, well, just that improvement in velocity from turning on the build process was worth 1.7 million Euro a year. So I went to the CEO, and I said how much did you spend for implementing continuous integration. He said 3,000 Euro. I said, wow, well that's probably a bigger return on investment than you've ever seen in your life. And he said absolutely. And what happened next was, just by implementing automated testing, they cut their delivery cycle from four to six weeks down to two weeks. That cut their support calls in half. It increased their revenue and so forth. Here's just another example. We went through A3 with the leading web company in the Netherlands. Competitor Amazon can't get into the Netherlands because this company owns it. Their biggest problem when they started Scrum was it took a week to build their system. And that was a problem because they couldn't get their testing done with the build that slow. And the teams didn't know how to fix that. They had no idea. It was delaying deployment, delaying revenue. So, again, we come back to these engineering tooling. And how do you get good testing and integration? We want the build to run multiple times a day. So why doesn't it do that? Well, you have many people involved in the build. Even one person could have up to 45 manual steps. Not good management of the build process. No one person really owns it. So the intervention was get management's commitment to assign an expert to go through that, understand in detail the process, and systematically automate it to get a fast build. In less than two weeks, one consultant got the build to run in four hours. So the point of these-- you can see from this Lean stuff, you go through this, and you have a problem that's been hampering the company for years. And in two weeks it's gone by one person spending a few hours fixing it. The power of this is unbelievable. The challenge, though, is that the nature of the company has to change. When start-ups start off, they're organic-- down in the lower right quadrant. As they get bigger, Yahoo, for example, when they got bigger, they got over 1,000 developers. They hired some consultants to come in and give them traditional project management. That introduced a lot of bureaucracy, a lot of documentation they didn't like. Many years later, they looked at Scrum. And they said, hey, Scrum is a way to scale the way we were working in the beginning. But that requires empowered employees, managers as leaders, all these things. So the company starts to change its nature. And Takeuchi and Nonaka talk about this. The self organization of teams, the teams choose their work. Management has to back off and support the teams. And what the management is looking for is the energy of product creation on the teams, which the Japanese call the ba. That's what you'll see Takeuchi and Nonaka talking about. Where is the ba on the teams? That's where the product comes from. And management can't tell the teams to get back. They have to facilitate an environment of trust and commitment where the teams start generating it. And then when they generate it, they need to do the kind of things that Steve Denning talks about in the book. They need to organize the work in short cycles. They have to avoid interrupting the team. They need to get the team reporting to the client or the client representative, in Scrum's case the product owner, to get good software. The team's doing the estimation. They're deciding how much work and how to do it. They measure their own performance and so forth. If the goal, from our investors' point of view, is we want at least four times the performance. That's what Toyota got over General Motors. And this is a slide that shows the velocity of ScrumInc. And these are weekly sprints. And you can see it went up four times. By the end of last year, we started down at 40 points, and we were doing 250 points. And this was a whole company. What happens if a whole company goes four or five times as fast? Well, here's what happens. We implemented it at Pegasystems, a publicly traded company. And during the implementation the stock price went up 400%. Well, the investors want even more than stock price. They want market share. And here's how Citrix Online, who's hosting this webinar, implemented Scrum. These dots are time to market of a release of a product. So you see in the early days it was a five, 10 months. As they got more and more old code, legacy code, you'll see in the middle of the chart, it goes up over 40 months, almost four years to get a release out. They had to do something. They implemented Scrum. But they implemented it not only at the team level, but at the enterprise level. And the enterprise team prioritized their projects and releases just like a Scrum team would prioritize their stories. So the enterprise Scrum drove the projects out so fast that they dropped the release time for new releases to under 10 months, as fast as a start up but, of course, many more releases. So at the end of the day, if you want to multiply your productivity, if you want to ramp your stock price, and if you want to grab market share, which is what Citrix is doing, what Takeuchi and Nonaka will tell you is the wise leader implements Scrum. It's both idealistic and pragmatic. It's based on transparency, trust, community of practice. Managers have to turn into leaders to lead the troops forward. It accelerates innovation and new product development. And the transcendent goals tend to make the world a better place. And so what they would like to see is Scrum implemented everywhere. If you want to talk about this, you can contact Scrum, Inc. Our website is on there-- scruminc.com. info@scruminc.com is the email. And I think I'll turn it back over to Steve at this point. All right. Well, thanks, Dr. Sutherland. We have a lot of questions that have come in. Everybody seems to be really charged about this topic. And we have kind of a rock star here in our presence here today. So we're very lucky to have Dr. Sutherland here to answer these questions for us. I'm sure that we won't get through all the questions today. But we will make the answers available to all of the questions in due time through email. So I'll just go ahead and jump into some of these questions here, and I'll ask Dr. Sutherland to respond to them. So we got a question com in that wants to know how you measure velocity. Could you expand on that a bit? Well, this is a deep subject. In the 1940s, the Rand Corporation was asked by the Department of Defense how to estimate big projects. And they determined that, through their research, that humans were really bad at estimating hours. So they told the Department of Defense never use hours. What humans are really good at is estimating relative sizing, particularly if those relative sizes are in a growth pattern you see everywhere. So those relative size things we measure in points. So points are a measure of feature delivery. Whereas hours are measured in cost. And what we want to do is measure the rate of feature delivery. If we know that we need 100 features, and we can deliver 10 of them every two weeks, then we know it's going to take 10 two-week sprints to get those 100 features. That's what we want to know. That's the velocity. So the velocity of feature delivery is what Scrum is based on. Just to follow up on that, how do you measure velocity. Do you do it in story points? How do you physically do that? The way we do it is if you look-- the growth pattern that humans see everywhere, they see it in flowers. They see it in vegetables. They see it in the human body. They see it in galaxies. The pattern is the next stage of growth is the sum of the previous two stages. This can be measured mathematically by the Fibonacci Series. It goes one, two, three, five, eight, 13. So the way we size stories is we pick a small reference story. And we give it, say, a three. And then we look at the next story. And we say, well, how big is that compared to the three? Is it smaller? Maybe it's a one or a two. Or is it bigger? Maybe it's a five or an eight. And then we have a process for having the team collectively-- this is a team-based estimate with everybody participating-- the team collectively zeroing in on the best estimate. Microsoft has data that shows that this is-- they get 20% error, where in the early stages, the big waterfall teams have a 400% error at the beginning of the project. As they go along, it narrows. But the error band of Microsoft is so small with points compared to the traditional curve, it's mind boggling. Microsoft won the lead paper at a recent conference for IEEE just showing this data. It's unbelievable. Thanks for that follow up. We have another question here asking about disparate teams. Because a lot of people now are outsourcing, they have teams located in disparate locations. How do you do the daily stand-ups whenever there are big time differences from time zone perspective? There are several published papers on my website at scrum.jeffsutherland.com. If you'll click on Jeff Sutherland's papers on distributed Scrum, and you'll see that the fastest teams in the world actually have teams, say, in the Netherlands and India. And we've even seen the same result in San Francisco and India. Where they are meeting every day for 15 minutes, today it's using Skype Video or Google Hangup. So it's the same Scrum meeting that they have, except they do it by Skype Video. If you read the paper on San Francisco to India, you'll see they had to adjust some of the meeting times because of the 12 1/2- hour time difference. But we have one of our venture companies in Los Angeles that just this week is launching three Scrum teams in India. So they have three Scrum teams in Los Angeles, three Scrum teams in India. They're going to use this model where all teams, six teams, will be half LA, half India. They will all meet in the daily meeting, 15 minutes at the same time. It's going to be probably around 7:00 PM in LA when they have the Scrum meeting. It'll be in the morning in India. So it's doable. You have to run a really good Scrum to make it work. What Microsoft has found is that if you distribute , you will always, unless you're running a really good Scrum, you will always increase the risk of the project, increase the cost of the project, increase the dysfunction of the team, and delay the deliverables. 100% of the time, Microsoft has found, that happens. So you have to have a really good reason for doing that. A lot of people don't. And if you're going to do it, you need to run a really good Scrum to overcome the problems. We have a follow-up to that, too. Somebody was asking how about the different frequencies of you stand up. Should that be five days a week? Or could you do three days a week? Is there any leniency in that? Well, my wife was a Unitarian Universalist minister. She's written a paper on Scrum in church, which, actually, one of the biggest banks in Europe adopted her paper to help run the bank. But in a church, she said to me I can only get these people together once a week. They're part-time. They're volunteers. Is it worth even have the meeting? And I said, well, having it once a week will get 20% of the benefit. So what I can say is you will never have a high-productive team with a daily meeting one or two days a week. If you want them to go really fast, they're going to have to meet every day. Got it. Now, a lot of people have asked the question about doing fixed-price development. And they have clients that are kind of beating on them for an estimate, that kind of thing. And with Scrum, because it's kind of an evolving situation from a requirements perspective, how do you handle that if you have a client that's wanting that estimate upfront? Well, you should read the papers on Scrum and CMMI because Systematic does almost 100% fixed-price, upfront bid. And what they do is they break down all the backlog in the RFP in the stories. The ones that are in closer, the smaller stories, the things that are going to be further out, are bigger. And then they estimate them using points. Now, there's also some unique opportunities with Agile because they can tell the customer, in the past, your projects were usually late and over budget because you had change requests. And in Scrum, we can adopt to change requests. In fact, we can give you all your changes for free Because we're implementing the most valuable things first. And as soon as you decide something is most valuable, we'll implement that right away, as long as you'll throw away the most low-value things that are at the end. And as long as what you throw away is of equal amount of work as what you're putting in, the changes are all free. So what we're actually doing is coaching the customer to throw away some of those 65% of features that are never or rarely used. And in return, they get all the changes for free. It's guaranteed to come in on time and on budget. So Agile provides a tremendous competitive advantage. Every Agile consultancy I'm working with is giving change for free. It's no risk. It's easy to do. The good ones are even executing a process we call money for nothing. Any customer, or any CIO, knows that if a project does not meet a return on investment criteria, you will not do the project. So we say tot he CIO, when you are doing a feature-- in Scrum we do the most important features first, so the value keeps going down along the timeline. As soon as the next feature costs more than the value, you should terminate the contract and deploy the software. The contract will say in it the vendor gets 20% or 30% of the remaining contract value, and the customer gets the rest. Well, I have a consultancy with thousands of people that took this on as a strategic initiative. We found that a third of the CIOs of the big companies in the world are ready to do this immediately. They're tired of being late and over budget. They're tired of paying people who are slow. So what they want to do is get together, do a small pilot to establish trust in the team to build estimation practice, to establish a velocity. Then they want to pay for velocity, so that if the vendor goes slower, then the vendor has to eat the cost. Well, in most cases, the vendor can actually increase the velocity over the time of the contract. And because they're not billing hours-- they're billing points-- they actually make more money per hour by increasing velocity. So we have groups in the world now that are charging a premium for a project that allows the customer to terminate, 30% up front. If they finish early, they get 30% of the remaining contract value. And then they anticipate they can increase velocity 30% during the contract phase. They're making 30% three ways in this contract. If you figure out the margins on this, what you find is that, instead of consulting margins being 10%, 15%, you can get margins there are 50% or 60%, if you're really good at this. So this is a retirement-early strategy. And you can only do it with Agile. Well, it sounds like prioritization's the key to all that. So we have about three more minutes, probably time for one more question. This comes from an automation engineer. Wanted to know how automated testing fits within Scrum and when you automate. In other words, do you do manual tests up front and then automate those? Do you do regression? How do you handle that? OK. In a couple of minutes, I'm going to have to shorten this quite a lot. Let me tell you the most advanced venture company we have in automation. They have one tester that works with the product owner. And the product owner specifies the acceptance test from the user's point of view. And the tester will write those in a natural language script using a tool called Cucumber. Those acceptance tests are all implemented before the sprint starts. So when the sprint starts, all tests are red. And if at the end of the sprint, all tests are green, they're done. Now I asked the chief product owner, I said, well, how are you feeling about this because they started working on this a year ago. And he said at the end of a sprint now, all test to green, I feel confident in pushing the button, and it goes immediately to production. That's what we want to see Scrum teams. Precious few can get there. So here's the way to start. You look at what are the biggest problems in deployment after the sprint. And then you prioritize those. And then you systematically create automated tests to make those problems go away one by one. And you implement them in the build process so that they run every build. What happened in the company I mentioned earlier in the presentation, when they started doing this after employing continuous integration, it took them three months. They only had one person running tests. They only had 120 tests. But it actually cut the time from the end of the sprints to production, it cut it. It was four to six weeks. It cut it to two weeks. And, as I said, it cut the support calls in half. The customers were much happier. It generated a lot more revenue. So careful selectively prioritizing the implementation of automated acceptance tests is the key to making this really work. All right. Well, I really appreciate that. Well, it looks like we're at the top of the hour. And I do appreciate everybody that sat in on the presentation today. A lot of people were asking if this presentation was going to be available afterwards. Absolutely. We have recorded today's session. And it will be available out at smartbear.com. And the slide deck will be available, as well. So I do appreciate everybody's participation. I know everybody's time is valuable. And we do appreciate you guys attending today. Thank you very much. And thanks to Jeff. Thank you