Lean Analytics

Rating: 9/10

Best Line #1: Analytics is a necessary counterweight to lying, the yin to the yang of hyperbole.

Best Line #2: Relying on wisdom and experience, rather than rigid analysis, helps us get through our day. You don’t run A/B testing before deciding what pants to put on.

Many years ago, I decided to become more fluent in data-driven methods of management and decision-making. It wasn’t easy. At the time, no one else in my workplace had any qualms about the way things were done. Decisions were made largely on gut intuitions, speculative hunches, past experience, and trust/distrust. Injecting data into that kind of practice was anathema. I was like some kind of propeller-beanie nerd hosting a house party and asking everyone to fill out a satisfaction survey.

My initial efforts were unpopular and ham-fisted. I didn’t have a lot of tools (software specifically) and analysis was ad hoc. But over the course of years, things got better. To a point where you could say we were at least using regular data and KPIs to make real decisions. Yet we never reached the ideal state.

Largely because I had no reasonable sense of what the “ideal” should be. All I had were examples from hospitals, investment firms, a few decent articles, and a lot of glossy stuff written about performance management. Then there were the books.

Searching For Help

I waded through a lot of really boring, crunchy, technical books on data, data science, business analytics, statistics, and probability. There were some real gems on these topics but there was nothing that really helped me bridge the gap from data to management.

I’ve shared one fantastic book that I found along the way: Naked Statistics by Charles Wheelan. The book review is here. It’s the best place to start understanding data. It should be required reading in every classroom. Forget all the dry textbooks. This thing has real vibrance.

Also, if you really want to develop skill in applied methods of analysis, the very best book is John W. Foreman’s Data Smart. In fact, this book is better than most online courses. I deeply admire what he did with this work.

Finally, if you really want to challenge yourself as a data aspirant, give MIT’s The Analytics Edge a try. It’s the first online course that really stretched my brain. Two hours into the experience and I realized it was the absolute real deal and I needed to either get serious or get out. To my great shame and regret, I got out. I need to get back in.  

Why? Because Lean Analytics showed me what I was missing in terms of application. It showed me why the effort to develop these skills was worth every hour of strain. To have that provided through the lens of the Lean methodology made it all the better.

I featured Eric Reis’s book The Lean Startup many months ago. The book review is here. But that book is really just a great introduction. There are deeper, applied extensions of the methodology that make it come alive. Which is where other works in the “Lean library” come into play. There’s the fantastic Running Lean and Lean Customer Development, to name a couple. I mean, it’s a whole franchise really.   

And why not? This stuff works. To try to explain why, I offer the following articles from this week’s review:

Monday: Blind Values, Empty Metrics

Tuesday: Go Ahead. Move The Goalposts.

Wednesday: The Power of Entrepreneurial Empathy

Thursday: Good Measurement, Bad Mimicry

To round out the coverage, I’ll share two final elements that I think really illustrate the core value of this outstanding book. I hope these can persuade you to buy the book yourself. It really is the best resource I’ve found for developing a more data-driven way of managing your work.

The Lean Analytics Model

There are a lot of frameworks and models in the Lean library. From Lean Startup to Lean UX and all points in-between, you’ll find flow charts, diagrams, and sundry other graphics that try to institute specific processes. It is completely overwhelming and a tad absurd.

There needs to be a canon. Because this borders on the edge of comic book multiplicity. Fifteen origin stories for Batman. Twenty alternative universes for Spider Man. You get the idea. As an armchair academic on these matters, I offer the Lean Analytics model as the first entry into the official record:

From O’Reilly media and the author Alistair Croll and Benjamin Yoskovitz

In some ways, this is just a reworking of standard business life cycle stuff. It is presented in the context of a startup effort (like everything else these days) but, thanks to the broader themes of “empathy” and “virality”, it can apply to any initiative without losing its coherence.

The “gates” associated with each stage are where the analytics come in. It gets to one of the more important aspects of the book that helped me a great deal. A structured set of regular, high-signal questions is the most important ingredient in a data-driven practice.

What is the most important question at any given stage of an initiative? What must we ask ourselves about our performance every quarter? What must we ask when exploring any new idea, partnership, or pivot?

I don’t think a manager, leader, founder, or analyst can reliably do their job without those questions formally, routinely defined. This is the “driven” part of a data-driven practice.

These questions must be regular and routine because, otherwise, we simply use data to satiate our confirmation bias. Errors in judgment quickly emerge and the data just reinforces those errors in the same way an accelerator pedal reinforces the speed of travel as a car moves in the wrong direction. I write a lot about this in trade publications. But don’t take my word for it. Here’s a fantastic article from Daniel Kahneman who talks about it better than I ever will.

So the Lean Analytics model is a structured set of regular, high-signal questions. I think it’s great. The soft terms like “empathy”, “stickiness”, and “virality” help conceptualize the kind of metrics you’ll need in each business type. Choosing the actual metrics themselves is just a simple act of pattern-matching.

Which metric helps me prove I found a poorly-met need a’la the Empathy stage?

Which metric shows I’m scalable a’la the Scale stage?

Notice that I use the term “metric” in the singular tense instead of the plural. This gets back to yesterday’s post about The One Metric That Matters. What is that one metric? It depends on the business, the specific operation taking place, and the stage you find yourself. The only constant here is that there must be a single North Star (i.e. a singular goal) to pin your strategy and subsequent tactics. Our authors are very wise in advocating this. Otherwise, again, we find ourselves just awash in too much data. Discipline goes out the window.

The Metrics

All this week, I’ve covered the more theoretical, conceptual aspects of this book and showed very little in actual metrics. This might have been irritating to some. If anyone is like me, they probably saw the title of this book and expected a whole lot of tables featuring various metrics to track. Like some version of Sabermetrics.

Indeed, when I first opened the book, I expected to find instructions. Here is what you measure. Here is the threshold you maintain. Here is the target you need to meet.

Part of me wanted this book to condense our work down to an elegant level of formulaic precision. Just give me the recipe! The authors rightfully avoid the temptation to do such. Startups don’t work that way. They are not a mathematical proof. Most things aren’t.

But some metrics are available and should be considered as part of a broader suite of values that are linked to every business type. Examples are below. I offer these only as a means to illustrate what the authors do to provide comprehensive tables in the way I initially expected:

Sample Metrics by Business and Stage

For a media business in the Stickiness stage, focus on traffic, visits, returns, followers, and click throughs.

For an media business in the Virality stage, focus on SEO and length of time on page.

For SaaS in the Stickiness stage, focus on engagement rate, churn, visitor/user/customer funnels, capacity tiers, feature utilization.

For SaaS in the Virality stage, focus on customer acquisition costs.

You get the idea. It’s fairly obvious if you’re already experienced in those areas. The back half of the book really satisfies my desire for a Moneyball-esque toolkit. It focuses on tech companies but it can give you a broader idea of how to apply the toolkit to other efforts.

Conclusion

It often seems that managers, founders, leaders, and executives have a very distinct relationship with data. They either obsess over it, as Reed Hastings and Jeff Bezos reportedly do, or they completely ignore it. Or worse, they selectively use it in the manner that Kalev Leetaru writes in the fantastically-titled article: Data Science Has Become About Lending False Credibility To Decisions We’ve Already Made.

I honestly wish I had written that article. I have come close many times. Here’s a recent example. It’s just not as pointed. So his is better and it’s worth weathering the pop-up advertising hell that plagues the Forbes website.

It leads to a simple truth. Those who ignore data do so because they can find confirmation bias elsewhere. But I shouldn’t straw-man this anti-data perspective too much. There’s real value in the idea. With moderation.

I think it’s critical for all of us to love our data at a distance. I think it’s vital that we question the information with healthy skepticism. I think it’s important to hear the warning even if we don’t heed them.

If the data shows something bad, does that mean you have to be a slave to it’s suggestions? Of course not. There are perfectly valid moments at which we should all yell “Damn the torpedoes.”

It echoes something that our authors write:

Small lies are essential. They create your reality distortion field. But if you start believing your on hype, you won’t survive. You need to lie to yourself but not to the point of jeopardizing your business. That’s where data comes in.

That’s where data is crucial. Specifically, the data that gives you a measure of the One Metric That Matters. Whatever that happens to be. There are plenty to choose from and the book gives many that relate to tech startups. You can undoubtedly find your own in other fields. Or in your own personal endeavors. The management guru Jim Collins uses an OMTM to rate the quality of his daily experience. It’s a nice idea.

And this is a nice book—the best I’ve found on beginning a data-driven practice. Broad in scope, detailed when appropriate, eloquent (a rarity in this topic), and very smart. I just wish it could have branched out of the tech world a little more. But we can do that ourselves. They show us how. Here’s a link to buy the book.