top of page
  • Writer's pictureStephen Plimmer

The direction of travel for Insight

Technology is having a significant effect on the market and customer insight industry and the practices within organisations. AI is one of the undoubted key trends, but others stem from everything from climate change to societal values. We examine the trends that organisations (and agencies) can expect to be riding over the next few years across eight stages of a value chain for insight creation.


However it is sourced, businesses can’t function for very long without market and customer insight. Knowing, at the very least, what customers want, how customers view a brand and its products, and their experiences in using them, is necessary to sell and achieve a level of customer satisfaction that prevents an exodus.

Small businesses, with one product or a narrow product category, might launch and survive on the industry insight of the founders for a time. However, once competitors respond, new tech emerges, consumers’ priorities change, or new entrants appear, then the shelf-life of such insight can be short-lived.

At some point on their growth curve - as turnover moves from £2m to £10m - an organisation will start to want its own insight function, starting with a team of one or two.

The Insight function usually matures to become responsible for monitoring, understanding and disseminating information about the external environment affecting the current and future business performance. To fulfil their remit, they are likely to run continuous tracking surveys, to measure customer satisfaction and brand metrics, as well as manage ad-hoc projects to address specific challenges, such as informing the development of new products. They will also analyse changes in customer behaviour from the customer database and website, and might additionally work with marketers to test campaign ideas. On the supplier side, research agencies have historically sold services into insight functions.

The sudden emergence of AI, from being a somewhat distant abstraction, thrust into the consciousness of many businesses, has been influencing this Insight eco-system for a few years. However, it is going to have a more substantial or even existential impact, in both organisations and research agencies, over the next 3-5 years.

While AI is undoubtedly disruptive, it is also not the only trend. The value chain for Insight is changing at every step, from the nature of projects to the way data for insight is researched, procured, processed, stored, analysed, visualised, interpreted, managed and ultimately destroyed. Much of this is driven by technology but is also being influenced in different ways by everything from climate change considerations (e.g. the demand to research new sustainable products) to geopolitics (the requirement to protect data from cyber-attacks).

Each year, the US market research consultant, Greenbook, produces a review of global trends as seen by both research and buyers of research. Coupled with trends and developments we've tracked elsewhere, we’ve summarised eight areas of the Insight value chain that we feel are ones that businesses might wish to note as they plan to evolve their Insight capabilities over the coming year.

1. Project selection

Businesses are expected not only to sell but to do it ethically. Medium-sized and larger organisations face pressure from consumers, suppliers, industry bodies and wider society to consider their societal and environmental impact, alongside their customers’ experiences.

As a result, new topics enter the research and analytics programmes: What do consumers expect of us? Which consumers are most receptive to sustainable products and services? What is the willingness to pay more for them? Are customers of different cultural groups finding their experience is comparable?

The imperative to be inclusive is placing more onus on representative sampling from different sections of the population.

New technologies - notably AI, augmented and virtual reality used in marketing and customer service - are also influencing the research agenda. Insight teams have a role to play in prioritising use cases, designing new user- and customer journeys, tracking their performance and benchmarking against competitors.

2. Sources of data and research methods

The days of research programmes comprising surveys and focus groups are long gone. Greenbook’s latest research showed a family of emerging methods, each with different benefits.

Both the fixed and mobile internet have made it easier for researchers to gather deeper insight at scale. For instance, the technique 'mobile ethnography' is growing, using smartphones or tablets to gather rich and real-time data about people's behaviours, experiences, locations and cultural practices in their natural environments.

Speed is often the essence when developing new or adapted products and services. Consequently, tools are becoming more popular that offer online micro-surveys for rapid testing of marketing or proposition concepts.

With consumers somewhat fatigued by long-form surveys, agencies are turning to innovative ways to get insight. One method is to “gamify” research: Research can be conducted with a panel of consumers and the research is done through ‘challenges’, with rewards, badges, leader boards and other incentives for completion. Tasks might be carried out in virtual environments and/or made interactive.

Different data sources and collection techniques are also making direct customer feedback less necessary, particularly of the sort that relies on long surveys. For instance, contact centres can measure the length of time customers wait in a queue and correlate the result directly with the number hanging-up, rather than having to ask customers about their satisfaction with the queue duration in order to set targets on customers' waiting times.

3. Supplier: in-house team vs external agency

ChatGBT means that some of the activities that have traditionally been carried out by an agency can now be done in-house, or much more quickly by an agency, such as writing surveys or interview scripts.

Various providers of digital survey tools also offer the means for non-experts to quickly test marketing campaigns or product ideas - in hours not days - to at least obtain an initial evaluation.

Both examples above are leading to some research tasks, which traditionally sat with agencies, moving in-house to save time and money.

Greenbook explains that agencies are also evolving what they might charge for.

4. Data processing and storage

The appetite for data is unrelenting, which is having a subsequent effect on the data architectures that are being used to store and process it.

Rather than collecting different data in different databases and spreadsheets, contemporary data architectures make use of data warehouses to aggregate information for analytics from databases. Data-lakes, additionally, can hold multiple types of original data such as numeric, text, images, video and audio in an unstructured format. Insight can then be generated by, for instance, connecting machine learning tools with data lakes to find insights across different data types.

Insight generation - which involves cleaning and transforming data - can also be automated for repeatable tasks.

These architectures and tools also provide the means to provide real-time insights, giving greater opportunity for service and marketing teams to respond to feedback.

With many organisations hard-pressed to build, maintain, manage and evolve such systems in-house, particularly given the pace of the industry, cloud-based solutions are more popular, particularly as the price of computer storage continues to fall.

5. Data integration and synthesis

Data was traditionally analysed one-source-at-a-time, such that a research report included the results from a single study. However, thanks to thousands of open data sources and paid-for sources from other data providers, data integration and synthesis is becoming more common.

For instance, data from a customer satisfaction tracker might be synthesised with that from website behaviour, competitor pricing, weather data and the nation’s consumer confidence to give a full picture of what is affecting how customers are feeling.

6. Analysis and interpretation

One of the results of technology has been the ability to carry out more sophisticated analysis, quickly and dynamically. Research platforms that are used to carry out surveys, for instance, usually contain built-in analysis capabilities, including unstructured (text) data as well as numeric survey response data, allowing non-statisticians to rapidly find their own insights.

Newer analytics tools are improving that extract insight from text and the human voice, including the extraction of sentiments as well as topics.

The interpretation of results is also becoming more scientific, rather than relying purely on the human judgement of the researcher or client: One of the rising applications that Greenbook noted, for instance, was that organisations were showing a strong intent to adopt behavioural economics models. Such models allow consumers’ behaviour to be understood better by accounting for the realistic, biased mental calculations that they take in different situations. This, in turn, allows organisations to better influence them.

7. Visualisation

Data visualisation has been turned into an art form by some, using tools like Tableau and Power BI. Aside from their aesthetic qualities, modern data visualisation techniques are themselves an expert discipline, offering ways to imbue readers with complex insights, or the motivation to act, from simple or compelling data representations.

As AI emerges, we would expect platforms to become more intuitive, tell users the story in the data, and provide ever more impactful visuals. Conversational analytics also means that staff (users) can interrogate their data visualisation tools with speech commands using their natural language, rather than with mouse clicks.

Virtual and augmented reality tools are also emerging in data visualisation technology that allows users to visualise data in a 3D space and walk into graphs.

Fundamentally, such tools provide the means for an organisation to “democratise” its data, by sharing it across the entire organisation, and therefore raise the level of knowledge everywhere. Meanwhile, they allow each individual user a customisable way to receive and explore it.

8. Management and governance

Growing concerns about cyber-security and consumer privacy mean that data management is an ever-changing policy landscape. Organisations are increasing their cyber security expenditure on capabilities like encryption, privacy enhancements and user authentication.

Customer profiling techniques, particularly using AI, have come under scrutiny for embedding biases into algorithms that then lead to customers of different groups receiving detrimental treatment. Ethical considerations are becoming more important.

There is even a growing market in data destruction, which analysts MarketWatch expect to grow by 13% CAGR, to become a $20bn global market by 2030 from half that today.


In sum, technology is having a significant effect on the market and customer insight industry and the practices within organisations.

AI is one of the undoubted key trends, but others include the societal influences on organisations that are shaping the research agenda, the democratisation of data, the advantageous business case of cloud-based solutions, and the emergence of new methods (including old methods, such as ethnography or anthropology, that are made more viable by using modern mobile devices).

Organisations that continue to derive a competitive edge from their insight will be addressing most if not all of these trends in their immediate plans. Meanwhile, organisations that are just starting to consider having an insight function have the advantage of being able to bake these trends into their plans from Day 1.

18 views0 comments


bottom of page