Agile Advice - Working With Agile Methods (Scrum, XP, Lean)

Christian Gruber pointed out a great blog to me today: Lean Software Engineering.  I’ve read a few of the articles and I like them.  I don’t agree with every detail, but I’m pretty happy with what I see there.  In particular, I really liked the article on using a Priority Filter.  The idea of a priority filter is a great operational technique but it makes it difficult to clearly justify the cost-benefit of doing the work.  Picking the highest priority item from a set of items that all have a negative return is still a bad way of working.  The technique could easily be combined with an analytical assessment of the value of a piece of work relative to the cost of the team (and any capital costs).

One of my favorite books is Good to Great by Jim Collins.  One of my other favorite books is Freakanomics by Steven Levitt.  This article, From Good to Great … to Below Average, takes an interesting look at what has happened to the Good to Great companies in the intervening years.  I found the comments to be particularly valuable to read.

Bob C. Martin (Uncle Bob to you and me) suggested, in his “quintessence” keynote at the Agile2008 conference that he had the perfect metric for code quality. Cyclomatic complexity and others were interesting mostly to those who invented them, etc. His answer was brilliant, and was easily measured during code reviews:

WTFs per minute

I love it. All you need is a counter and a stop-watch. Start code-review and start watch and start clicking anytime you see code that makes you say or think “What the F???”. This dovetails nicely with his original recommendation for a new statement in the agile manifesto: “Craftsmanship over Crap”.

The first day of the Agile 2008 Conference was fabulous!  I’m working with five other great people including my wife, Melanie, my father, Garry, and three good friends: Paul, Travis and Laila.  We have set up a beautiful booth designed by my brother, Alexei Berteig.  Working at the booth has been a tiring yet exhilarating experience.  Before the conference we received Boothmanship Training from The Portables.  The people at The Portables have been excellent and I highly recommend them to anyone embarking on a booth display at a conference.

Today I delivered the first of my two presentations at the conference.  The thirty minute experience report: “Extremely Short Iterations as a Catalyst for Effective Prioritization of Work” was well received.  There were about 25 people in attendance and despite the short time for questions, there were some good ones.  I enjoyed delivering the presentation and it was filmed for later publication on InfoQ.  I also received highly positive feedback after the talk.  On Thursday, I am giving a much more substantial presentation titled “Meta-Agile: Using Agile Methods to Deliver Agile Training“.  This will be a three hour workshop in two parts: a 90 minute presentation, and then a 90 minute participatory patterns workshop.  If you are a trainer or coach, I highly recommend attending this one!

I also ran into many friends… too many to list, in fact.  It is surprising to me to see how many people I know in this community.  There are over 1500 people at this agile conference, and I can’t walk more than 20 steps without running into someone I know.  One other really cool thing is that everyone is commenting on how much they like the notebooks that were included in the conference bag - highly popular.  In fact, three people have commented that it was the only useful thing in the bag… which is actually a bit of an exaggeration since the conference program was also in the bag and I suspect that it might be just a wee bit more useful :-)  Nevertheless, the feedback has been great.

Come check us out at our booth if you are at the conference - it’s unmistakable - strong red backdrop with the Berteig Consulting logo across the top.  We also have a great raffle so be sure to drop off your raffle entry which is found in your notebook pocket.

This post is a little geeky and technical and product-related for AgileAdvice, and is a shameless self-promotion. Nevertheless, since testability, test-driven-development, and incremental design are non-exclusive sub-topics of Agile, I though I’d report this here anyway.

Many developers use the Dependency Injection and Inversion of Control (IoC) patterns through such IoC containers as Spring, Hivemind, Picocontainer, and others. They have all sorts of benefits to testability, flexibility, etc. that I won’t repeat here, but can be read about here, here, and here. A great summary of the history of “IoC” can be found here. J2ME developers, however, especially those on limited devices that use the CLDC configuration of J2ME, cannot use the substantial number of IoC/DI containers out there, because they nearly all rely on reflection. These also often make use of APIs not present in the CLDC - APIs which could not easily be added. Lastly there’s a tendency among developers of “embedded software” to be very suspicious of complexity.

In working out some examples of DI as part of a testability workshop at one of my clients, I whipped up a quick DI container, and being the freak that I am, hardened it until it was suitable for production, because I hate half-finished products. So allow me to introduce the Israfil Micro Container. (That is, the Container from the Israfil Micro project). As I mention in the docs, “FemtoContainer” just was too ridiculous, and this container is smaller than pico-container. The project is BSD licensed, and hosted on googlecode, so source is freely available and there’s an issue/feature tracker, yadda yadda.

Essentially I believe that people working on cellphones and set-top boxes shouldn’t be constrained out of some basic software design approaches - you just have to bend the design approach to fit the environment. So hopefully this is of use to more than one of my clients. It currently supports an auto-wiring registration, delayed object creation (until first need), and forthcoming are some basic lifecycle support, and a few other nicities. It does not use reflection (you use a little adapter for object creation instead), and performs quicker than pico-container. Low, low overhead. It’s also less than 10 classes and interfaces (including the two classes in the util project). It’s built with Maven2, so you can use it in any Maven2-built project with ease, but of course you can always also just download the jar (and the required util jar too). Enjoy…

P.S. There are a few other bits on googlecode that I’m working on in the micro-zone. Some minimalist backports of some of java.lang.concurrency (just the locks), as well as some of the java.util.Collections stuff. Not finished, but also part of the googlecode project.

I recently started writing a book review on Crystal Clear: A Human-Powered Methodology for Small Teams by Alistair Cockburn. Check out the first part of my review. I have read Chapter 1 entitled Explained (View from the Outside). It was a very interesting chapter that set Crystal Clear as the answerer to Alistair Cockburn. It made many aspects of the Crystal family clear in my mind. I enjoyed the questions, and the answers were insightful and helped me to put the ideas into a whole picture.

At the moment I am reading Chapter 2 entitled Applied (The Seven Properties). Frequent Delivery, Reflective Improvement, and Osmotic Communication made sense to me and aligned somewhat to my own beliefs. When I started reading the fourth property, Personal Safety, certain parts seemed fine, while others set off warning bells. I believe that the purpose of any team is to progress. This is achieved through trust, respect and unity.

Cockburn says “Once personal safety and amicability are established, a useful, playful dynamic may emerge. People may wage competition with each other. They may argue loudly, even to the verge of fighting, without taking it personally. In the case where someone does take it personally, they sort it out and set things straight again.” - page 31.

The statements above concern me. Cockburn addresses trust by saying that people will not take it personally. Respect is lost because they “… May argue loudly, even to the verge of fighting”. I would be unable to say that I respect someone if I yell at them or even raise my voice. Now unity is completely destroyed. For some reason our society and many societies around the world not only condone competition, it is seen as a way to judge attributes of excellence in an individual. This is not a good sign for our progress towards unity in human civilization.

I agree that being polite and not stating one’s opinion is harmful for trust. However, it is preferable to use consultation instead of competition. Imagine that a team is encouraged to compete with itself to achieve better results. Would there not be feelings of resentment or heightened levels of stress? Now imagine a team that is encouraged to consult and raise the team together without focusing on individual success. Would not this team feel excited to be around each other? Would they become fast friends and grow as a unit? Would family members of the team be enthusiastic to be included in picnics and socials?

Now the big question:
What is better, individual success or team unity that add value to not only the team but all who interact with them?

I will continue to read this book and post my reviews. I find it interesting that this book has helped to see the confusion that is happening all around the world in terms of progress, success, and human development.

I welcome any comments on my posts.

Check it out: whiteboardwiki.org.  Looks like it’s very flexible.  Has some bugs and lots of basic features yet to be added, but since its open source, we can all contribute!!!  Development is being organized in an agile fashion using a product backlog.  Fun.

I have just started reading Crystal Clear: A Human-Powered Methodology for Small Teams by Alistair Cockburn. I was not too sure what this book would provide for me in the way of relevant learning.

I am intrigued that this work came out of years of experience by Alistair. This quote from the book “Crystal Clear does not aspire to be a “best” methodology; it aspires to be “sufficient,” in order that your team will shape it to itself and then actually use it.” gave me hope. I work on a small team and I wonder about which practices will best suit our situation. I also wonder how our team can use tools and processes then reflect on their usefulness to decide if we will continue their implementation.

I am interested in reading the whole book, but a little concerned that there will be too much techno-words used throughout. I have a background in business, marketing, and the web but not to the degree of the some of the other books that I have read.

What learning have you gained from working on small teams? Have any of you read this book? If so, did you gain any insights that would help my team to develop?

http://pligg.scrum-on.com/ - Just found this through Niall O’Keeffe who is an attendee at my Certified ScrumMaster course.

It seems that Berteig Consulting is blocked from China, so I’ve made a new site called Scrum Beijing to promote the ScrumMaster training I do there.

Also, the price has been reduced on the Scrum Study Guide and the web site now includes screen shots and a bit more useful information.

Berteig Consulting is starting to formally market OpenAgile as a method for helping small businesses manage their work (in any industry).  Heck, we’re using it ourselves.

And of course, a reminder about the Agile 2008 conference happening in Toronto in August - don’t miss it, it’s going to be fantastic!

Mark Levison has written up a great list of Scrum Smells on his blog Notes from a Tool User.  Check them out!  There is a lot of great information there.  He also asks your help: if you know of common problems with Scrum, see if you can add them to the list!

One of the challenges with agile methods is to get a clear perspective on how to measure process improvements. I recently had a brief discussion with a C-level executive at a small organization about this. His concern was that cycle time was meaningless because it depended so much upon the size of the work package. So how do we use cycle time as a meaningful measurement? What else can we use to measure process improvement?

Let’s look at the difference in measuring cycle time in an agile vs. non-agile environment. Then we’ll get to other measurements.

Cycle Time , Waterfall and Agile

First, let’s define cycle time. From iSixSigma we have:

Cycle time is the total time from the beginning to the end of your process, as defined by you and your customer. Cycle time includes process time, during which a unit is acted upon to bring it closer to an output, and delay time, during which a unit of work is spent waiting to take the next action.

This definition is important because it gives us a clue about the potential difference between a waterfall vs. agile method of delivering value. Let’s imagine the typical process used in a waterfall environment. The following are the high-level steps:

1. Customer / User / Stakeholder sees a need, validates it and submits a request to have that need fulfilled. This is when we start the clock on cycle time.
2. The fulfillment organization (IT, Product Development, R&D) puts the request in a queue, backlog or requirements management system.
3. Along with other requests, the fulfillment organization schedules the work on the request, usually by creating a project to fulfill it and other related requests. The project is estimated at a high level, the current status of in-flight projects is noted, and the new project is prioritized relative to other projects.
4. At some point, based on the schedule and the reality of the work on other projects, the project containing our customer’s request is started. Here, “started” means that detailed requirements are gathered.
5. After sufficient requirements are gathered, a detailed technical analysis is done including architecture, high-level design, risk analysis, etc.
6. Development begins. (Note: many people mistakenly start measuring cycle time here.)
7. Developers and testers work to validate the results of development and fix any problems discovered.
8. Final acceptance testing is done.
9. The results of the project are deployed to users, sold to the client, or in some other way passed back to the original requestor. This is when we stop the clock on cycle time.

So from the start of the customer request formally submitted to the time that the fulfillment of that request is made is our true cycle time. There are a few important things to note here. First, there is a queue of work based on requests made but not yet scheduled. There is another queue for work scheduled but not yet started. We know that if we can reduce the size of these queues, we can improve cycle time in a general sense. Second, we know that most organizations of any significant size will have different queues based on the urgency of the request. For example, a high severity bug discovered in the production system of a company’s largest client will be treated differently than a wish list item for a small not-yet-client. These two requests won’t even go in the same queue: the high priority problem will be quickly escalated to a support or development team that can work on it immediately. Third, it is tempting for the development group to measure their local cycle time. This is a Really Bad Idea since it leads to sub-optimizing behaviors. For example, it is easy for the development team to improve their cycle time by sacrificing quality… but this just causes the QA cycle time to increase, and probably the overall cycle time (true cycle time) is affected more than the local improvement in the development group’s cycle time.

Now let’s look at the steps that occur in an ideal agile environment:

1. As before, the Customer / User / Stakeholder sees a need, validates it and submits a request to have that need fulfilled.
2. That request is immediately placed in a ready state for the next iteration (cycle, sprint) of a delivery team. Elapsed time: maximum one month.
3. Team completes the request including all work to actually deliver/deploy and work is delivered to the stakeholder at the end of the iteration. Elapsed time: maximum two months.

So the ideal method of doing agile has a maximum cycle time of two months to deliver from the time a request is made… how many teams are doing this? Not many.

The ideal is extremely difficult to accomplish. Getting to that state requires that the development organization catches up to the business side so that there are zero pending requests at the start of each iteration. It also requires that the business side users and stakeholders are able to articulate their requests so that they are small, and appropriately detailed for the team doing the work.

A realistic agile implementation actually is a lot more messy. Depending on the type of request, the cycle time for a piece of work can vary widely. Some low priority items may take years even in an agile environment. A low priority request is made and approved but then never quite makes it into a project… and then once in a project never quite makes it to the top of the team’s product backlog. This is interesting to look at sometimes, but it points out another important aspect of measuring cycle time: mostly we care about average cycle time (or some other statistically interesting aggregate measure).

The predominant factor in most organizations’ cycle time is the number and size of the queues they use as work is processed. In most organizations there are several queues and most of them contain large numbers of requests or bits of work in process. Queues represent huge amounts of waste. It is easy to see that queue size and cycle time are closely related: the more items in a queue, the longer the cycle time.

This leads to a simple conclusion: regardless of lifecycle approach, reducing the size of an organization’s queues is one of the easiest ways to reduce cycle time. What are some common queues? There are often queues of projects, queues of enhancement requests, queues of defects to be fixed, queues of features, queues of tasks, queues of email (large inboxes), queues of approval requests, queues of production database changes. The number of queues increases the more an organization is oriented around functional groups, and the number of queues decreases the more an organization arranges work to be handled by cross-functional teams.

Cycle Time and Work Package Size

This is where queueing theory and agile methods intersect really well. Cycle time is related to the load on your system, in particular your units of work processing. In most organizations, teams are created to handle work. The more work given to a team simultaneously, the higher their utilization level. Many organizations like high utilization levels because it gives them a guarantee that people are doing valuable work all the time that they are paid to work. This is a completely false benefit and in fact is extremely destructive to overall productivity. From queueing theory we know that the cycle time for a piece of work increases exponentially to the utilization level. We see this whenever we over-load a server… but for some reason we fail to see this when we overload a person or a team or an organization even though it still happens.

Cycle time is also related to the variability in the size of the work packages. Low variability means that the exponential factor related to load is low, and high variability means that the exponential factor is high. In other words, if you have a highway that only allows motorbikes, you ca