Skip to content

Questions?  ·

Big data’s creative and empathetic sibling (and why they struggle to get along)

Joyce Hostyn argues that Better Human Understanding, Not Big Data, Is the Future of Business. Some excerpts (with my emphasis):

Despite the best of intentions, we’re not data driven, we’re hypothesis driven. Our stories (our mental models) are merely hypotheses of how the world works. But we see them as reality and they influence what data we collect, how we collect it and the meaning we glean from it…

In a quest to become data driven, are marketers trapping themselves with outdated mental models of data and analytics? “Big data is being wasted on marketing. The true power of analytics is in revealing cultural dynamics.”

Many echo these concerns about data-driven marketing, and the need to be skeptical and hypothesis-driven. (1) (2) (3)

Hostyn concludes her thoughtful article with a number of questions, including:

  • Can we leverage big data to zoom out and understand patterns and trends, then zoom back in for a dive deep into the hearts and minds of individuals?
  • Are we willing to develop hypotheses with the potential to disrupt our old mental models? Create experiments to test those hypotheses. Prototype to think. Collect feedback. Iterate.

At Primal, we’ve invested years exploring this mode of hypothesis-setting as a lens into big data. It involves a collaboration between humans and machines across the full spectrum of analytical and synthetical thinking.

What follows is a summary of that exploration and what we’ve learned to this point.

Big data analytics can zoom out, not zoom in


To the first question posed above, Can we leverage big data to zoom out and understand patterns and trends, then zoom back in for a dive deep into the hearts and minds of individuals?

The short answer is yes and no. You can use big data to zoom out, but not zoom in.

While it’s true that big data is all about surfacing patterns and trends, the nature of this process is frequently misunderstood. As outlined in this reality check on big data:

The vernacular of big data analysis—trends, correlations, inferences, and frames—betray their bias towards generalization over specification.

When we use statistical methods to analyze data, we reduce that data to statistically significant patterns and trends. This is a process of generalization, of zooming out on more generalized features within the data that we describe as patterns and trends.

Zooming in, unfortunately, is where big data analytics fail most spectacularly. This is precisely the area where Hostyn is encouraging marketers to venture: to find those “extreme users” in the long tail of activities, those insights at the fringes, “to move beyond the comfort zone of the echo chamber“.

To zoom in, you need a set of tools quite different from big data analytics. Obviously, humans possess these tools, described using words like imagination, empathy, and creativity. While these faculties may be uniquely human, tools that support and empower these synthetical activities are emerging.

Just as machined-based analytical tools support the analytical mindset, machine-based synthetical tools support a synthetical mindset, powering our imaginations and our hypothesis-setting.

Small data synthesis: The younger sibling of big data analysis


Primal is uniquely positioned to answer the second question posed by Hostyn, Are we willing to develop hypotheses with the potential to disrupt our old mental models?

Here, the answer is a qualified yes, as innovative businesses lead the way and the old guard clings to purely analytical approaches.

For the past 8 years, we’ve has been developing an artificial intelligence that embraces a consumer-directed approach to data synthesis. Our customers use this technology, via cloud-based data and software services, to understand the interests of individuals and deliver highly targeted content.

In the course of building this technology alongside some tremendously innovative customers, we know that hybrid synthetical-analytical approaches will be embraced, to powerful effect.

One of the most surprising use cases for Primal involves using Primal as a powerful lens into big data. It represents a concrete embodiment of a collaboration between humans and machines across the entire synthetical-analytical spectrum.

A case study analytical-synthetical cooperation


The combination of synthetical and analytical approaches is powerful, particularly when both aspects are supported by machined-based tools.

Here’s an example of how it works:

A political analyst wants to understand how different communities are responding to the Affordable Care Act (“Obamacare”).

He understands it’s a deeply divisive issue and expects to see profoundly differing opinions across the political spectrum.

Using purely analytical approaches, this is a tremendously daunting problem. The amount of social media available for analysis is obviously massive. But more problematic, the landscape of opinion is tremendously fragmented, demanding an assortment of complex algorithms to mine for these insights in the long tail.

Fortunately, we have an insightful person at the head of the process. He begins by outlining key terms of what he expects to see in social media (specifically, topics of interest related to Obamacare).

This is the initial process of hypothesis-setting, discussed above, that is so crucial to success.

Working with Primal, his synthetical mindset is amplified using our synthetical technology. The result is an interest graph, a data structure that encodes his hypothesis in a machine-readable form.

Using Primal’s interest graph as the expression of his hypothesis, the retrieval and analysis of media may be automated. He sets up collections of social media sources, each representing different constituencies of political opinion, and uses his interest graph to analyze them.

After the first cycle through his interest graph, he can see at a glance which topics in his hypothesis that are supported, where reality diverges from his imagination, and the differences across the political constituencies in his study.

Based on this learning, he reformulates his hypothesis, synthesizes a new interest graph to test it, and begins another cycle of analysis.

An intractable problem using purely analytical approaches becomes a dynamic process of machine-human collaboration, and critically, a collaboration that involves machines across the full spectrum of analytical and synthetical thinking.

Getting Started

Check out our developers site to start exploring your own Primal-powered solution.