top of page
  • Writer's pictureStephen Plimmer

Small Data in a Big Data world

Updated: Jun 9, 2023

If you enjoy contrarians, then you might well enjoy the idea of “Small Data” proposed by Martin Lindstrom. In a world of Big Data, we argue the case for its Small counterpart not to be forgotten.

First, the Value of Big Data

In the commercial world, the mantra of Big Data - and the Data-Driven Economy - has been sounded since the 1990s: The reason data became “big” being that the amount stored by organisations grew larger than could be contained in traditional software tools, like spreadsheets.

Remember those quotes that drove home the value that data would provide to organisations and wider society?

"Without big data, you are blind and deaf and in the middle of a freeway." - Geoffrey Moore, father of contemporary innovation theory.

"Big data is not a fad. We are just at the beginning of a revolution that will touch every business and every life on this planet." - Tim O'Reilly, Founder of O'Reilly Media, in 2013.

"Big data is changing the face of business. The winners will be those who can make sense of it." - Vivek Ranadivé, CEO of TIBCO Software, in 2014.

Such prophecies from the late 1990s and early 2000s have broadly come true. The amount of global data produced is growing exponentially, multiplying by a factor of about 5 in the past 5 years (Statista).

To further emphasise that ‘big is better’, consultancies and analysts have translated the growth of data into highly impressive estimates of commercial impact:

McKinsey Global Institute in their report "Big Data: The Next Frontier for Innovation, Competition, and Productivity" (2011) estimated that the potential value of big data to the US healthcare system could be more than $300 billion per year. Retailers using big data to improve their operations could increase their operating margins by more than 60 percent.

Accenture’s report "The Value of Data: Unlocking the Power of Customer Insights" (2014) estimated that companies that invest in advanced analytics capabilities could increase their operating margins by more than 60 percent. The value of data to the US economy was cited as being potentially more than $1 trillion/year.

Oxford Economics' report "The Economic Impact of AI and Data" (2019) estimated that data and AI, in combination, could add $13 trillion to global GDP by 2030. Companies using AI and data analytics could increase their revenue growth rates by more than 30%.

Most mid- and larger-organisations have subsequently ridden the wave of Big Data; motivations being to make better decisions, find new products and services, create more targeted marketing to existing customers, and drive efficiencies across operations.

Subsequently, businesses have been ramping up investment in data science and analytics to mine, visualise and model this data, to extract the maximum value. Market research company Polaris expect the data-science platform market to grow by 27% each year over the next few years.

Few question the value of Big Data.

Small Data

Against this clear direction of travel, Lindstrom suggested an alternative narrative: that profound human insights emerged, not from vast datasets, but from making observations of human behaviour.

Small data is the sort of observational data that gets captured in a notebook by a researcher, or perhaps in photographs, or by collecting artefacts that are littered around a place that reveal clues about how people live there. Lindstrom gave intriguing examples to demonstrate the profound impacts that small data had achieved even on sizable, multinational corporations. Take the story of Lego:

Lego was a brand that was in decline by the early 2000s. “Big data” had suggested that further decline was inevitable, as the attention spans of their young users were evaporating in the face of fast and exciting computer games. Lego’s response, of dumbing down kits to try and compete for their users' precious attention-span, wasn’t working either.

However, while working for Lego as a consultant, Lindstrom’s visit to a young American boy’s home found a pair of old battered Adidas trainers on display which the boy heralded as his most valuable possession. Lindstrom probed as to why. It transpired the trainers had accompanied the boy’s journey to becoming an exceptional skateboarder amongst his peers. The trainers reminded the boy of his own skateboarding prowess and the respect and admiration it had earned him from his social group.

This interview offered a thread of insight that Lindstrom pursued, It led him, through further interviews, to discover that many children gained a great deal of social acceptance and confidence from the exceptional skills that they honed (such as skateboarding).

Rather than Lego ‘dumbing down’ to address the trend towards faster and more instantaneous forms of gratification, the insight seeded a new idea: Lego kits should create more difficult and complicated challenges, for it was these that allowed children to demonstrate exceptional skills, and thereby gain self-confidence and social approval. In turn, this would motivate the kind of loyalty to the product and brand that the boy had demonstrated to his old, beaten training shoes.

This was a conclusion that Big Data would have never likely found.

Beyond Lindstrom’s book, the value of small data holds true in many other areas of innovation, from breakthrough companies to successful local café chains.

Steve Jobs famously relied on human observation when creating the Apple iPod in 2001. His reason for going against the wisdom of design at the time came from carefully observing people using clunky MP3 players and the frustrations they brought: the inability for people to carry around the large music collections that they owned, or the effort involved for people to own multiple devices for different forms of music.

The value of such observations transcends research into product design too. Take a successful local coffee chain (or other small business) within any town. It is worth asking how they survive in a market that is notoriously tough:

On a holiday trip to the French city of Aix, I remember the plethora of street eateries in a particularly densely served square beneath the opera house and city walls. In the early evening, most of these restaurants were still very empty, with waiters on edge, waiting to pounce on passers-by; all, that was, except one eaterie that was already well-populated.

Nothing about the menu (pizzas, salads, fries, grilled steaks, and chicken) made it different. Neither did there seem significant differences in the price, the layouts and awnings, and smartly dressed and attentive waiters and waitresses at each venue. The specific location was undifferentiated too.

The difference was spotted on watching the restaurant before the tourists had filtered in the following late afternoon. While all the other restaurants had still been empty, this one had a table of people sitting and enjoying food, front, and centre.

With nothing else to go on to choose between the competing eateries, tourists out looking for a place to eat faced a dilemma akin to trying to choose a space to park in an empty car park. At that moment, of facing too many choices, the fact one place had existing custom offered it social proof.

The further fact that would-be diners got to see what the food looked like - colourful and plentiful - and could then smell the aromas of sizzling steak and chicken - confirmed the food would be good. No tourist wanted to suffer the remorse of consuming upward of a thousand calories without the compensation of enjoying it!

Smaller businesses don’t have the luxury of Big Data. However, even if all of the restaurant owners in that square would have had access to the transaction data from the previous decade, the tens of thousands of meals bought and consumed, or the nationalities and ages of the visitors, then it's not entirely clear that it would have found the insight: the correlation between market share of the restaurants and the variable of having a few staff and their friends stage the first meal of the evening.

Small data vs Big Data

Of course, both small data and big data have value. Big data provides the size of a market, the web traffic of competitors, and how they attract it, and the profiles of consumers, including their interests, values, and financial situations. All of this provides the nature, viability and scale of an opportunity.

When it comes to designing a product or service that gets acceptance, is adopted and goes on to win repeat loyalty, however, there remains a limitation in insight provided by most Big Data.

The difference between product success, mediocrity and failure comes more from attentive and empathic design: a fact well known already by user experience (UX) designers of websites and apps.

In turn, the attentive design comes from human observation and empathic engagement with would-be consumers: Often, it can be the tacit knowledge of the business owner, leaning on years of talking with customers. However, it can also be found from observational research which elicits how people use different objects; respond to existing products and services; or even react to each other in the presence of those products and services.

In her new book, Nuts and Bots, engineer Roma Agrawal talks about the breast pump: a family of inventions with a history of comical ineptitude littered with decades of various ill-fitting, impractical, and ineffectual versions that came before the 1990s, when smaller and portable and usable devices finally appeared.

Quite why a better product did not emerge before then is unclear. However, perhaps it was the taboos of talking about such things that stymied a growing chorus of complaints, allied to the lack of the ability for would-be inventors to observe the regular struggles of users, that made innovation so evidently impossible.

Other medical devices, frequently hidden from public scrutiny, have arguably told a similar story. Insulin injections used by Type 1 and 2 diabetics can be painful and inconvenient as they need to be administered several times a day. Continuous positive airway pressure (CPAP) machines are used to treat sleep apnea but are uncomfortable. Hemodialysis machines filter waste and excess fluids from the blood in people with kidney failure. While treatment is often vital, it can also be time-consuming and uncomfortable for users. Incontinence products may be necessary but also embarrassing for some users if they are bulky or noisy.

However, in other areas, where the suboptimality of products is on clear display, then we see continual innovation: the telephone, the car, and the design of a new house.

Innovators evidently do their work by observing people experiencing problems.

In summary

Lindstrom’s idea was not revolutionary: at his core, he was an advocate of “old-fashioned insight” and the lightbulb moment of seeing patterns in how other people perceive, feel, think and act to resolve their needs.

However, when the ongoing cry still surrounds the value of Big Data as the fuel of new products and services - and we have an annual Big Data conference in London but not a Small Data conference, his ideas are more than a useful reminder to value Small Data too.

8 views0 comments


bottom of page