Entries in experimentation (2)


Experimentation in business vs. prediction

The key theme in Michael Schrage's December article in the MIT Sloan Management Review is that data trumps intuition. He points to evidence that this is true even when experienced senior leaders and subject matter experts are involved, and this can have significant consequences for how companies approach the task of anticipating customer demand.

Organizations may be confident they know their customers, but they’re very likely to be overconfident."

-- Michael Schrage

  • I am particularly sensitive to his call to “embrace our ignorance”, that experts and managers are notoriously bad at predicting the outcome of experiments / decisions, that empathy and experience are very poor replacements for data and careful experiments.
  • Only humble and systematic experimentation is really “customer-centric”.
  • If big data can help us identify promising patterns and correlations, pervasive (and now cheap) experimentation can close the causality loop, creating "virtuous cycles of profitable insight between big data and small experiments”.

This view radically challenges our managerial hero complex (admit it, reader, you, like me, fall prey to it on a regular basis!). Imagine a world where the most valuable skills or personality traits we are looking to hire and retain are humility and curiosity instead of leadership and expertise?

-- Yassine El Ouarzazi


Experimentation in business

In order to understand what works and what doesn't in business, it's best to test and measure--to experiment--and apply what you learn. Thomas Davenport's 2009 HBR article "How To Design Smart Business Experiments" is an excellent primer on some of the challenges as well as the benefits of conducting sound experiments.
Davenport summarizes his findings as follows:
  • Too many business innovations are launched on a wing and a prayer—despite the fact that it’s now reasonable to expect truly valid tests.
  • Companies that equip managers to perform small-scale yet rigorous experiments don’t only save themselves from expensive mistakes—they also make it more likely that great ideas will see the light of day.
  • With a small investment in training, readily available software, and the right encouragement, an organization can build a “test and learn” capability.

To these, I would add the following comments:
  • At the end of the day, little else than controlled, replicated, consistent experimentation can establish causality between two observables. This challenges not only the businesses that do not a have a “test & learn” culture, but also those who attempt to measure what they do, but rely mainly on non-causal ex-post estimations. Rigorous experimentation can tell you when you are wrong, and if you are right, it is a good hygiene factor.
  • The approach of the article is very pragmatic when stating that experimentation mainly serves to test tactical changes / decisions. Based on Mars’ experience with marketing science and “laws of growth,” I would argue that replicable, rigorously established patterns can definitely serve as the foundation for the marketing strategy of a company. While the tactical improvements are the main short term outcome of a good experimentation culture and practice, it can be a critical enabler for scientifically grounded and effective strategic shifts.
  • Last but not least, the idea that "With a small investment in training, readily available software, and the right encouragement, an organization can build a “test and learn” capability” can have far reaching consequences on an organization like Mars. Mars has historically been a decentralized, locally driven, entrepreneurial company. This has arguably been a critical enabler for Mars’ success so far, especially in period of fast economic growth and distribution expansion. Now that growth is expected to be close to flat in mature markets, that population aging and nutrition concerns are effecting a slow decline in the per capita calorie consumption, we are more than ever in need of the creative power of entrepreneurship to sustain our growth. In the increasingly complex, multi-channel, heterogeneous, information-overloaded business ecosystem, it will probably have to be a very well informed entrepreneurship. The opportunity for decentralized learning that is offered by a small scale rigorous experimentation capability is a critical enabler for today’s and tomorrow’s entrepreneurship, probably as important as the freedom of action granted by decentralized decision making. 
And I would recommend this great book about the topic, which is a very easy read and takes us through the disruption from rigorous experimentation to business, manufacturing, sociology, government, education, health, etc :-)
-- Yassine El Ouarzazi