This article originally appeared in Inside Big Data

In this special guest feature, Jeff McDowell, COO of Primal, discusses the importance of small data and big data analysis when providing timely and accurate AI recommendations based on interest compared to the stale irrelevant recommendations most online channels provide today. Primal is a leading AI research company that builds knowledge graphs in real time, allowing companies to better understand meaning even in small data environments. Jeff has a proven track record of driving strategic growth through a combination of non-linear insight and practical execution. Prior to joining Primal, Jeff is best known for leading the global alliances and enterprise marketing teams at BlackBerry, and the marketing and business development team at Desire2Learn.

There’s no question that companies have benefited from the rise of big data AI solutions, through an exponentially increased understanding of their customers’ behaviors. But is big data always the right approach for gleaning insights about individual customers interests?

It turns out big data approaches have some constraints, not the least of which is that they tend to have a limited ability to understand the fluidity of interests at an individual level. If someone was interested in running a year ago, and playing poker a few months ago, but is now no longer interested in either topic, a statistical approach using big data won’t be able to pick up on these nuanced changes.

As customers begin to demand more individualization of services, companies need to be able to understand them at this specific level and also keep up with how these interests may change over time. The answer lies in small data AI.

Does Your AI Really Understand Your Customers?

Most AI solutions today don’t take small data into account. Let’s take Twitter for example. Twitter attempts to understand its users by analyzing all the data they’ve collected to infer user’s interests and categorize them into static buckets. A year ago someone may have tweeted a few times about knitting (a passing interest of theirs) and Twitter put them in the broad interest category of “knitting”. Now a year later, they’re still included in the knitting interest category and advertisers are sending promotions about yarn. This is annoying for the user and a waste of money for the advertiser.

Try this for yourself by looking at your Twitter settings and click on “Your Twitter Data”. Scroll to the bottom to see a listing of the interest categories that Twitter has placed you in. If you’ve been on Twitter for awhile, this list will certainly include some outdated topics that no longer apply to you. You might also notice some “interests” that are way off. Twitter gives users the ability to remove themselves from any irrelevant or outdated categories, but how many people regularly review this, if ever?

Small Data AI to the Rescue

A real-time small data AI approach would solve Twitter’s ad targeting problem by looking at what individuals are interested in now rather than the categories they may have been placed into months or years ago. Small data analysis can read a very small piece of text (like a tweet) and use AI to understand the meaning and context of the words. Then it can build a knowledge graph around the post to expand upon the understanding of the interest. This provides a richer insight than keyword targeting, which only searches for matches to a very specific keyword or phrase making it much more limited than an AI approach.

As an example, a vendor of fly fishing rods might want to target people who tweet things like: “Catching lots of trout in the Saugeen River!” This tweet doesn’t use the keyword “fishing” but the expanded knowledge graph built by AI would understand that trout is fish, and a method for catching trout in a river is fly fishing.

All of this happens in real-time, so a small data AI solution is able to understand interests as they evolve. In our Twitter example, this approach would mean a user’s current tweets drive the type of advertisements recommended to them, rather than the approach of targeting using static interest categories. This ensures ads are highly relevant to what users are interested in right now – a win for both users and advertisers.

Could Small Data Work For You?

Of course, not every company is Twitter, but there are many other examples of small data – chat sessions with customer service representatives, comments to a blog post, or product reviews – that can be analyzed with AI to gain deeper insight into customers.

Small data isn’t a replacement for big data. An AI solution needs to analyze both if companies want to have a complete view of their customers at both an aggregate and an individual level.

Editor's Note: This blog was originally published on October 31, 2018. It was updated on September 4, 2024.