Jeff McKenna's Agile Development Blog

Agile Webinar Q&A

Yesterday, John Scumniotales, Paul Dupuy and I participated in a Webcast for The Agile Journal. We could not answer all the various questions on line. I thought it might be interesting to see some of them and our answers.

You can see the Webcast yourself by visiting: http://w.on24.com/r.htm?e=155782&s=1&k=A98F0FB3D0AD7D354E60F0C65DB31BB7

Here are the questions and answers:

Question: What if the business process isn't defined how does Agile handle business reengineering?

Answer: When the business process is not defined, an agile approach provides an incremental model to explore and define it! Using Scrum for instance, you would work with your Product Owner (customer) to elaborate user stories that define some aspect of the business process. Don't boil the ocean. Pick a small "chunk" that can be easily described and understood. If the end goal is to have a system that is implemented using software, then you should strive to implement this process as well. To the extent possible, this should be accomplished in the same sprint. Avoid using the sprints to only document the process. Your goal should be to produce working software as early as often as possible.

Q: Thanks for this presentation. One issue we struggle with is using scrum teams that do all of the things you are describing but we have trouble bidding competitively on small professional services projects. There is some overhead with scrum when applied to smallish projects (lasting 2 - 6 weeks) that we struggle with. If you have any suggestions or examples for how to insulate the customer from our internal costs of using scrum on small projects I would love to hear it. We've been using scrum for about 4 years and it works great for our product team. Any thoughts you have would be appreciated.

A: When using Scrum on professional services engagements, we highly recommend giving your customers visibility (and getting their buy in!) to the process you use to deliver the requested features. If they've agreed to the approach, then this should minimize the impedance mismatch and associated overhead costs for spinning up the project. Also, try and keep your teams together. Remember, with Scrum, we focus on building and optimizing teams. If your building up and tearing down teams for each project, this will certainly introduce overhead and also decrease the predictability of teams velocity as it can take up to three sprints for a team's velocity to normalize.

Q: Which is more important between Individuals/Interaction and Processes/Tools? What I should do if my team members do not have cross-functional skills?

A: In agile we always favor direct communication. Whenever and where ever possible get people in a room together hashing through the problems and issues. The reality of "agile in the enterprise" is that teams are distributed by time and distance. This is where tools are essential in enabling team collaboration. When teams are first formed, its common that there is some specialization of skills. This moderates over time. Pairing, an extreme programming best practice, is an excellent tool to use to help cross train team members.

Q: How can you help a large, compliance-oriented, traditional development organization move to a more agile approach without jumping all the way into Scrum?

A: If you don't have expert help on staff, we recommend getting help with the organizational change management issues that you will deal with in adopting agile. That is why Serena partnered with Valtech, a leading provider of Agile transformational services, with our Serena On Demand product.
Also set it up for early success. Identify a few of your best teams that are starting up new projects. Plant an experienced agile coach on the team, get them norming and storming. You'll be able to use the experience from this team to seed (and motivate) others.
A tool like Serena Agile On Demand helps as well. It provides a prescriptive Scrum framework for the teams to follow.

Q: What tools to you offer to facilitate yearly budgeting and goal setting in a way that could feed into incremental deliveries while supporting financial accountability within key cost centers?

A: The "team" is the key unit for agile projects. We want to fund teams and keep them stable for as long as possible. This allows us to optimize predictability. So you want to fund as many teams as possible, keep them together for as long as possible, and bring the work (projects) to them. At the portfolio level, its also helpful to consider relative investment levels for key initiative requested from the business units. This will dictate the mix of projects that you direct to the teams.

Q: Can we implement these tools based on traditional methods and then seamlessly transition to agile?

A: No. Adopting agile has significant impacts on the people and processes within an organization.

Q: How does this approach handle complex systems integrations with third parties and systems involving end-to-end financial settlement processes such as GL posting.

A: This is hard regardless of whether your following a traditional or agile approach! When we scale agile, we typically have many teams collaborating on a single appliation/product/service. The integration of the work developed by these teams of course needs to be closely managed. From a program management perspective, we can use the "Scrum of Scrums" to manage the inter-dependencies between teams and other cross team issues. We can also use the "Super Scrum" as longer range product planning tool to make sure that work is appropriately sequenced so that cross team dependencies will be satisfied.

Q: What if the business perception is that small, incremental releases do not generate adequate business value to warrant business change, training, and communication required? Can this process handle small internal increments that culminate into large releases?

A: Absolutely. Agile (and Scrum) are based on incremental approaches that produce potentially shippable code. Whether or not the increment is shipped is a business decision based on product, market, and customer readiness.

Q: (Loaded question) What if the user stories to be delivered in a sprint get delayed, and the spring demo cannot be completed on time?

A: We love loaded questions! The duration of a sprint never changes. If a story is not completed, its not accepted and not delivered as part of the sprint. The story will either be returned to the product backlog or moved into the next sprint.

Q: How is Serena differentiated from competition (such as Rally)?

A: Serena Agile On Demand was built from the ground up to deal with agile at the enterprise scale. Multiple teams, multiple releases, multiple products. We offer unique capabilities in our ability to deal with enterprise complexity. Serena Agile On Demand comes fully loaded for prescriptive Scrum development. While complete in its ability to deal with Scrum, its hyper-configurable to meet your specific needs as well. Serena Agile On Demand not only delivers the product features as a service, but it also delivers on demand agile coaching through Coach Live in cooperation with Valtech.

Q: How can the tool support allocating planned resource capacity to specific initiatives while allowing freedom from scrum masters to manage specific task assignments for team members. Need to measure that actual utilization is not exceeding targeted resource allocations.

A: A major difference with agile approaches from traditional approaches with respect to resource management is that the team is the unit of planning, not an individual resource. We try and put cross functional teams together and keep them together. The team then establishes a velocity - basically the amount of work they can complete in a sprint. Once the teams velocity is established, the team commits to deliver the amount of work that matches their velocity. Agile teams are always looking for ways to optimize their throughput (increase their velocity). This is contrasted with traditional approaches where management attempts to maximize utlization.

Q: Can the Agile On Demand task board be customized to reflect custom statuses ("swim lanes")?"

A: Yes. The swim lanes in the Card Wall are completely configurable.

Q: Who selects scrum team

A: When making a transition from traditional methodologies to SCRUM we think it is a best practice to "invite" members of the team to join the SCRUM. It's important in the early stages to get enthusiastic members on the team who will actively engage and support adoption. Hard to get people like that if they are 'forced' to be on the team.

Q: Since agile / scrum utilizes an Empirical model for managing costs, How can management make an assessment of the total cost of the program in order to fund development.

A: Plan the work as far as the release horizon using coarse grain items. Have the team size the items using rough estimates. Use historical information from the team's activity or make an educated guess about how much work the team can do in a sprint. Determine the cost of a team for a sprint. Calculate how many sprints you need to finish the work in the release. This gives you the estimated cost to complete the work. You'll need to use scope negotiation skills to prevent "scope creep" and decrease the priority of low priority items. You'll also need to recognize that as the software is developed, changes will occur (which is a good thing -- you'll find that what you wanted at the beginning isn't what you need) and trade-offs will have to be made.

Q: I manage a small unit responsible for several projects. Some were started over a year ago and were designed to follow waterfall. Subsequent projects are now developed via Scrum Agile method. With at least a year left on the waterfal project, how or should we switch mid-stream to agile?

A: It is fairly simple to switch to Scrum to manage your work. If your waterfall planning resulted in a pretty gantt chart with tasks, you can use them as the work items in your product backlog. Read up on Scrum and get a certified scrum master if you can.

Q: Have you ever heard of ACTUAL customers (in addition to product owners and PMs) participating in a sprint review? Any ideas how this could be implemented? Thanks!

A: Absolutely. This is done quite frequently in fact. If possible invite the customer to attend in person. Otherwise their are many virtualized meeting tools that can be used quite effectively.

Q: Could you expand some on non-product stories, how they get prioritized since the product owner may not care, how you track those vs. product user stories in metrics, etc?"

A: Pay me now or pay me later. Sometimes the team has to go slow for a few sprints to go faster in future sprints. There are many examples of non-product stories where technical debt is addressed that may not result in direct value to the customer. This is where the team and the Scrum Master must lobby and compel the product owner to include these stories into a sprint. Without this balance, an insurmountable amount of technical debt will build up.

Q: Is Serena overkill for a team just looking for a requirements/project management?

A: No, not at all. Serena's Agile On Demand has the ability to manage many types of backlogs and is ideal for requirements/project management. Internally here, the Marketing department has several teams actively sprinting and using Agile On Demand to manage projects and requirements. Multiple product backlogs, about nine is our case, contain stories for different projects the team is working on, each managed by a separate 'product owner' (usually a Product Marketing Manager). Then there are separate TEAM backlogs that contain the sprints and cross functional teams of marketing types who all contribute to the completion of stories with the project backlogs. The benefits of have a team manage requirements this way include time management, capacity planning for the sprint and minimizing the randomizing effects of fire drills.

Q: How did you convince your managment to use XP?

A: XP is a collection of best practices that increase the likelihood a team will deliver running tested features that meet customer needs in a predictable and repeatable way. Surely management can see the value in that result? ;-) The book "Extreme Programming Applied" has a good discussion of this. Also see the discussion on the c2 wiki: http://c2.com/cgi/wiki?WhyItIsSoHardToSellExtremeProgramming

Q: So, Product Managers can equate to Business Analyst in a traditional organization and Scrum Master is played by Project Managers?

A: The Product Manager's roles and responsibilities align quite well with the Product Owner's. BUT, the approach they take to elaborating requirements and engaging with team varies significantly. A Project Manager can be a Scrum Master, but be cautious. Project Managers are typically in a top-down command and control roll where they plan and dictate to the team how to complete work. A Scrum Master is a passive leader that facilitates the team. The Scrum Master coordinates meetings, interfaces with external teams, and manages the impediments that the team can't resolve themselves.

Q: How do we handle Standup meetings for geographically distributed teams?"

A: Tools are essential. Use planning and collaborations tools like Serena Agile On Demand and Virtual Meeting tools.

Q: What estimation method do you use from sprint planning?

A: We use story points with "unit-less" value. Our teams use planning poker to estimate sizes. There is an interesting video of it's creator talking about it here: http://www.youtube.com/watch?v=1oIaz7aZsno&feature=channel_page and more info on this site: http://www.planningpoker.com/

Q: Would you recommend adding to the backlog a "regression testing" item?"

A: Using stories to manage work that is difficult to complete or track can work well. If the team is having trouble doing regression testing in a predictable way during the course of a sprint, using a story is recommended. If the team does a consistent amount of regression testing every sprint, you could instead consider it "overhead" and choose not to track it.

Q: is serena agile license per user based?

A: Yes. $35 per user per month.

Q: Where would a Business/System Analyst fit into the Scrum team/role

A: Often times we see Business/System Analysts as part of the Scrum team in primarily a Product Owner role.

Q: IS a user story different than a story board. Or is the story board where you post all the user stories for selection?

A: The card wall (story board) is a technique for showing stories and their current state in a single view. It is typically used to manage the state for stories in a sprint.

Q: Where is the current "high-bar" for portfolio management across an enterprise for full Scrum methodology used on across large teams of developers? Team size of 500 + developers? 800+ developers?

A: 1000s of resources.

Innovation Games® workshop

I just finished a two day workshop on Innovation Games® by Luke Hohmann of Enthiosys. In short it was great, stimulating, chaotic, and filled with lots of ideas and games to play. Most of the attendees were independent Scrum Trainers/Coaches with some additional perspectives given by two market researchers. A couple of us were embedded coaches/trainers like myself. It was a stellar group with lots of often expressed opinions! The games we played with are all described in Luke's book, Innovation Games which I certainly recommend.

All good training for agile methods include liberal amounts of experiential exercises. I was first seriously exposed to this approach when I studied with Jerry Weinberg. He called them simulations and from him I learned how precise in terms of teaching objectives they can be. Luke has taken this to even greater levels and he uses the more currently politically correct term, game, to refer to this type of exercise. These days when agile trainers gather together they often share games they are using and practice on each other.

In my view the power of games is that a well designed game will engage more of the our capabilities. Some games get the body involved - somatic involvement. We know that learning for most folks is enhanced when the body is more involved. Some games get both sides of the brain involved - right brain (random, intuitive, synthesis, subjective, wholes) as well as left brain (logical, rational analytic, objective, parts). We need both modes to solve complex problem in new ways. Some games encourage our emotions to play a more active role. Our passions can be expressed and utilized.

As we discussed in the workshop, a current constraint on successful agile projects is the difficulty of writing those stories that are the work to be done. Scrum books assume that they exist. They don't.

Surprise, surprise. For over 40 years from the engineering side I have heard 'If we could just get good requirements we could '. Typically blanks are 'Take less time', 'Have fewer bugs', 'Deliver what is needed', 'Cost less' or 'Get the design right'. This problem has not gone away. And agile has never promised that it could fix this problem. Well executed agile only promises that engineering can reliably deliver what is asked for at a known quality level at a known time.

The short time box aspects of agile allow more in-course correction to be made based on what I think of as reality based feedback. This behavior gives the impression that 'requirements' are better. That is not true. Agile projects can produce bad products faster and more reliably than more traditional techniques.

Innovation Games® as we learned them are a set of tools that can seriously help with the difficult problems of where the product is going. We experienced the ease and FUN that can be experienced as difficult problems are addressed relatively quickly and innovative potential solutions are discovered. High level plans such as roadmaps can be addressed with these techniques as well as prioritization of high level backlogs. Filtering the infinite world of the possible down to the practical world of the doable.

Some of the games are now available online for distributed teams and we got a chance to play one of them. An impressive beginning to a difficult problem.

I was impressed.

Comparing Extreme Programming, Scrum, and Lean Software Development in Agile

I have been talking to a number of people, including some journal editors, who seem to be concerned or even confused about the differences between Extreme Programming (XP), Scrum, and Lean Software Development in Agile. So I want to explain the differences in this post. Bill and I talked it over as I explained these concepts.

Bill: So what are the differences between these three approaches and why is this an issue for some people? What are some of the questions you are hearing?

Jeff: Well, for one, some people see these as contrasting flavors of Agile. They ask which one should we use, for example. These are different aspects of Agile but they do not operate at quite the same level. Asking which one to use is not the really right question.

My view is that these three all relate to Agile but represent different focus points. At the most granular level, Extreme Programming or XP is the one that caught attention first. It is targeted toward individual team practices. It is mostly about what individuals do to improve their development practices such as Test Driven Development, Refactoring, Small Releases, and Simple Design. XP also describes other practices that the team can use to improve its function such as the Planning Game, Daily Stand Ups and Collective Code Ownership.

Since Agile mostly started with individual teams, Extreme Programming was very valuable to them. It taught people a lot of things that are now not seen as radical as they were when they first appeared in the early 90s when XP became visible. So things like test driven development, continuous integration, and refactoring are not seen as very radical … now.

Many people use these three techniques whether they are doing Agile or not since they promote good engineering practice. One of the ways we have seen Agile fail is when it is implemented on top of poor engineering practice. This was not the only thing XP did but it was its major focus.

Scrum predated Extreme Programming in creation but not in terms of popularity. Scrum is agnostic about team development practices and assumes that you have good ones. It is more concerned with work management and team dynamics. XP does have things to say about team dynamics but Scrum is a little stronger on them.

Bill: How is Scrum stronger?

Jeff: It is stronger as it puts in place some behaviors that were not explicitly called out. For example, in retrospection, that we discussed before, the roles are more clearly defined in Scrum than XP.

Bill: Is Scrum a more articulated and defined Extreme Programming? Is it taking it in new directions?

Jeff: No. Scrum, as defined, actually doesn’t say anything about software. Scrum is about work management and team dynamics that can be used in non-software projects. Most people use it in software development and books refer to it in this way but Scrum does not call out any specific software practices. XP, in contrast, is very specifically for software development.

Scrum did not take any of the Extreme Programming practices except the daily standard meeting. It is targeted at a different place, for the management of work and team dynamics. Because of this it has things to say about scaling to larger teams while XP is quiet about this.

When you get even larger in the enterprise then Lean starts to come into play. Lean is a lot like some of the Six Sigma work in that Lean is looking at reducing waste and looking at ways to eliminate unnecessary work, value chain analysis. Scrum teams often use Lean thinking to improve the management their work and that has some interesting implications.

Bill: So if I understand you, it is really a layered situation.

Jeff: That is how I think about it and yet there are things that cross continuously. Lean concepts are useful on any Agile project. So, in fact, I see Agile projects which use all three.

Bill: Extreme Programming started down in the trenches. Scrum is there at the next level at the team leader or supervisor level. Then Lean goes up another level. It goes at the cross team, enterprise level. So all of the higher levels ones can flow back down to the layers underneath.

Jeff: Yes, but the thing I want to emphasize is that those are not cut and dry levels. There is lots of overlap. When XP talks about the practice of pair programming, which is one of the more radical concepts, there is no counterpart in Lean. But the Lean concept of taking smaller bits flows down to XP.

Bill: So it seems clear to me as to why they operate at different levels but you say there is confusion. What is the cause of this confusion? Why do they think they are mutually exclusive?

Jeff: When people hear there are different flavors of Agile, they think they are distinct and one has to choose one or the other. Some of the advocates for the different approaches present their way as the better way over the other approaches. This is what I see going on and it creates unnecessary confusion.

The State of Certification within the Scrum Alliance

Here is a conversation that Bill and I had following my attendance at the Scrum Gathering a few weeks ago. There was a good deal of discussion about the certifications that are available from the Scrum Alliance. There are now five certifications available: Certified Scrum Master (CSM), Certified Scrum Product Owner (CSPO), Certified Scrum Practitioner (CSP), Certified Scrum Trainer (CST), and Certified Scrum Coach (CSC). TLA (Three Letter Acronym) heaven.

The oldest one is the Certified Scrum Master that was started by Ken Schwaber a number of years ago. I first got mine in 2003 and he had been doing them for a little while before then. It basically indicated that you had taken a two day class where the principles and practices of Scrum were taught. It was actually a bit of a joke in the community because it did not really certify anything. It just said you took the class.

But it did work and it got things off the ground. It got people’s attention. I believe the next one to come into existance was the Certified Scrum Trainer. These were people who could give the CSM class – they could certify people. The CST has more meat behind it. You must really ‘prove’ that you know your stuff.

Certified Scrum Practitioner was put together to indicate the step beyond just taking the class. Are you actually practicing Scrum? Successful applicants must be able to document their experience.

More recently, I worked to help put together the Certified Scrum Coach designation. That one was the first certification that was serious about whether you knew what you were doing. Some people have said that it has taken as long as 30 hours to complete the application. You had to have 1500 hours coaching experience and it is pretty serious. Not everyone gets through it.

The bar for these certifications has been going up. At the recent Scrum Gathering the Scrum Alliance announced that they will be giving a test for CSM to determine if they are certified. It appears that this test will be in place in the third quarter of this year (2009). True certifications can only last a certain amount of time before they need to be renewed, usually through a test. Over 65,000 people have become CSMs and they will have two years to take the test or lose their certification.

This is quite a change. All the certifications have gotten more serious. I think this is a good thing. It is part of the maturing of Agile. People are starting to request these certifications and the certifications are starting to have value. It is interesting to watch how this has changed over the years.

Bill: Let me get some more context here. Who administers these certifications?

Jeff: Up until this point for the CSM whenever a Certified Scrum Trainer gave a class he would submit the names of the people who were there to the Scrum Alliance who would issue the certification. The Scrum Alliance administers the certification and they will now administer the test. So as things go forward the certifications will expire and you will have to take another test and demonstrate that you have been practicing Scrum.

Bill: What is the Scrum Alliance? Who are they? Who supports them financially? How is it run?

Jeff: The Scrum Alliance is group of people who are practicing and interested in Scrum. You are invited to join when you become a CSM. It‘s mission to help promote Scrum, handle certifications, and do publicity. They hold the Scrum Gatherings twice a year, one in Europe and one in the North America. This year there is an additional Scrum Gathering in Brazil. They are typically three-day events where people interested in Scrum get together and talk about their experiences.

They have a great web site, http://www.ScrumAlliance.org. You can go there and see a lot of papers and other useful information.

Bill: Is it run as an academic or professional organization? I am curious because in my experience with knowledge management, there were a number of certification efforts that were really designed to be money making efforts for the person running them and very self-promoting. They were trying to create a business for themselves out of this movement or approach. Now this is not necessarily a bad thing if it is appropriate. In the case of knowledge management some times it was not or it was a gray area. So is this a volunteer organization or a business in itself? How does all this come to play?

Jeff: Well, originally it was just a group of people and then there were some money making aspects. The actual formation of the Scrum Alliance as a legal entity only occurred about three years ago even though certifications have been given out by Ken Schwaber for quite a while. It is now a legal non-profit organization. It is not designed to make money. It is not designed to support particular people. It has an independent Board of Directors from various companies. It does have a director and some staff to manage the web site, the events, and the certification process.

Ken Schwaber was the main early advocate of Scrum. He wrote some books and kept pushing it while the rest of us went off and did things. He was involved with Jeff Sutherland very early in the process.

These recent moves have made the Scrum certifications more substantial. There was talk before that it only meant you attended a class. Now the requirements are more involved. The Alliance made a decision not to change the name of the certification (CSM) because of all the people who already had it. They just added the testing requirement.

Bill: Is the Scrum Alliance sponsored by a software firm or firms, or is it independent?

Jeff: There are a number of firms who help sponsor the events but it is independent.

Bill: So it is an organization run by the evangelists rather than sponsored by several software firms.

Jeff: Yes. The Agile Alliance is a similar type of organization. It was formed later. They put on the biggest Agile conference.

Bill: The state of the nation of the Agile Alliance sounds like a good topic for another post.

Report on the Scrum Gathering

I am in Orlando at the biannual Scrum Gathering put on by the Scrum Alliance. The gathering alternates between the US and outside the US and is attended by Scrum Masters, Practitioners, Trainers and Coaches. There are some 260+ folks attending this one.

Jim Cundiff has been running the Scrum Alliance for a bit over a year now and I am glad to report that I am seeing considerable improvement in the professionalism, focus and activity level of the Alliance. The program for this meeting has brought in a number of folks from outside the immediate Scrum community. Ron Jeffries and Chet Hendrickson from XP, Dr. Alistair Cockburn of Crystal, Dr. Mark Paulk of CMM, and Gregory Balestrero from PMI and Jim Coplien of Patterns are all here and adding to a more inclusive feel of the gathering. In particular Mark is beginning some academic research on Scrum and the PMI is 'embracing' Scrum with a large number of PMP folks attending.

The Alliance is showing off the redesigned Scrum Alliance website. I suggest you take a look. It appears to be much more comprehensive and there is a lot of content now easily accessible.

Last night I met with 8 other Certified Scrum Coaches as we considering our next moves. I was involved with the initial definition, construction and roll-out of the Scrum Coach certification program. It is a rigorous certification with a non-trivial application and thoughtful evaluation. One really has to be a Scrum Coach to get certified. Initially there were six of us certified. Now the number is up to 12 and increasing each month. The coaches I met with last night were from Australia, Germany, Denmark, South Africa and the US. I am proud of the work we have done on this program and we are hoping that it grows and becomes recognized as a guide level certification.

With Alliance support we are seeing more and more Scrum User Groups getting started. Now over 50 world wide. And also growing. Some folks have wondered if at over 50,000 certified Scrum Masters if the 'market' was becoming saturated. It appears not to be so. Altho training rates have slowed somewhat, the demand is still reasonably strong. The requests for coaching are increasing as well.

In short, Scrum is alive and well and in fact, thriving.