If you’ve been around modern product development for any meaningful amount of time, then odds are you’ve come across the idea of vanity metrics.
But in case you haven’t encountered this concept yet, a vanity metric is simply one that looks great on paper, but doesn’t actually provide any true insight into the long-term success of your product. Common examples of these types of metrics include app downloads, user signups, or page views. Each of these metrics makes us feel great when we see them increase, but they are not actionable and none can be directly correlated to the long-term success of your product.
If you’d like to learn more about this concept, which originated from the Lean Startup school of thought, this article from Harvard Business Review that gives a great overview.
While vanity metrics are often associated with your product, that’s not their only use. In fact, vanity metrics crop up in all aspects of product development, including in your product development process. Can you think of a metric that your team often cites but doesn’t actually provide any actionable information?
The first idea that probably comes to mind is velocity. We’ve said it many times before, but velocity is a tragically abused metric for agile teams. We often praise a team with an ever-increasing velocity, but despite the team’s ability to close points at a high rate, those points don’t necessarily correlate to the actual business value the team is delivering.
This makes velocity the ultimate vanity metric for agile teams.
Although velocity might arguably be the epitome of a vanity metric for an agile team, there are other practices that a team may find itself following that exhibit similar attributes. These are behaviors that might allow a team to feel good about their agile adoption, but which aren’t actually yielding actionable information back to the team. As a result, these behaviors are simply providing the team with a false sense of success. Expanding upon our concept of vanity metrics, we can call such behaviors vanity behaviors.
One example a vanity behavior is the daily standup. Although the majority of self-described agile teams conduct a daily standup each morning, the effectiveness of this practice can vary widely across teams.
The main purpose of the daily standup is to give the team the opportunity to plan at the daily level how they will move themselves closer to their goal over the next 24 hours. This is a true collaborative planning session that allows the team to plan at the daily level how they can most effectively move forward as a cohesive unit.
But despite these intentions, too many teams use the daily standup simply as yet another status meeting. These teams squander the opportunity for collaborative planning, instead using the standup to review what each individual accomplished the day before, rehashing the same information that can be found in their backlogs.
Although engaging in this routine day after day may help these teams feel good about how agile they are being, in actuality it’s providing no actionable information to help the team make more informed decisions and keep their project on track.
If any of these metrics or behaviors sound familiar to you, don’t despair. There are steps you can take to break out of these patterns.
The first step is to rethink how your team is measuring success. Is the mark of success for your team how many points you close each iteration or whether you’re checking every box of whatever agile methodology you happen to be following?
Most likely, the true mark of success for your team is whether you’re actually delivering tangible business value each iteration. Another mark may be whether you’re growing in your ability to adapt to unexpected changes in your product’s direction.
If these goals sound like a better description of what your team is trying to achieve, then your next step is to ask yourself whether your current set of metrics accurately reflects those goals. For example, does the number of story points your team is closing each iteration actually correlate to the amount of business value your team is delivering? Or is simply following the mechanics of an agile practice an accurate indicator of how your team is evolving to work more effectively together?
Finally, if the answer to the second question was no, then your next step is to consider what metrics or behaviors could be more actionable—and therefore more informative—of your team’s success.
For example, perhaps you could begin assigning business value points to each story your team tackles to help communicate how impactful the outcome of each story is relative to other stories. Once assigned, you could then begin tracking the sum of business value delivered in each iteration alongside your team’s velocity. This would not only provide better visibility into the amount of business value that your team is delivering each iteration, but would also equip you to make more informed decisions as to whether the expected effort required to deliver certain stories is worth the value they’re expected to yield.
Whatever changes are necessary to put your team on the right track toward actionable information, the first step is to take a hard look at your current set of practices. This will help you better understand what gaps may be present between your actual goals and what you’re measuring.
Jeremy Jarrell is an agile coach who helps teams get better at doing what they love. When he’s not mentoring Scrum Masters or Product Owners, Jeremy loves to write on all things agile. You can read more of his thoughts at www.jeremyjarrell.com, see his videos at Pluralsight, or follow him on Twitter @jeremyjarrell.
Category: Productivity
