• Understanding the opinions of customers, prospects, or other groups. This can be based on any combination of documents the user organization controls (email, surveys, warranty reports, call center logs, etc.) — in which case — or public-domain documents such as blogs, forum posts, and tweets. The former is usually called Voice of the Customer (VotC), while the latter is Voice of the Market (VotM).
• Detecting and identifying problems. This can happen across many domains — VotC, VotM, diagnosing equipment malfunctions, identifying bad guys (from terrorists to fraudsters), or even getting early warnings of infectious disease outbreaks.
• Aiding text search, custom publishing, and other electronic document-shuffling use cases, often via document augmentation.

For several years, I’ve been distressed at the lack of progress in text analytics or, as it used to be called, text mining. Yes, the rise of sentiment analysis has been impressive, and higher volumes of text data are being processed than were before. But otherwise, there’s been a lot of the same old, same old. Most actual deployed applications of text analytics or text mining go something like this:

• A bunch of documents are analyzed to ascertain the ideas expressed in them.
• A count is made as to how many times each idea turns up.
• The application user notices any surprisingly large numbers, and as result of noticing pays attention to the corresponding ideas.

Often, it seems desirable to integrate text analytics with business intelligence and/or predictive analytics tools that operate on tabular data is. Even so, such integration is most commonly weak or nonexistent. Apart from the usual reasons for silos of automation, I blame this lack on a mismatch in precision, among other reasons. A 500% increase in mentions of a subject could be simple coincidence, or the result of a single identifiable press article. In comparison, a 5% increase in a conventional business metric might be much more important.

But in fairness, the text analytics innovation picture hasn’t been quite as bleak as what I’ve been painting so far. While standalone, passively-reported text analytics is indeed the baseline, there are some interesting exceptions. For example:

• I once confirmed that SPSS customer Cablecom‘s statistical models for churn and the like absolutely included text data; Cablecom even assigned different weights to the same apparent level of emotion depending on whether the text was in German, French, or Italian. Vertica recently told me of a Vertica/Hadoop customer doing something similar, except for the multilingual aspect. And the end of a 2008 SAS-based paper makes similar claims.
• There long* have been some examples of fact extraction that don’t really fit into my three buckets above. For example, researchers mine collections of articles to try to determine biochemical or biological pathways that would not be apparent from examining single research studies alone.
• It also has long* been the case that some bad-guy-finding applications — especially in the anti-terrorism area — used text analytics to populate state-of-the-art graph-oriented data analysis tools.

*When it comes to text analytics, “long” means “at least for the past several years.”

In more recent examples:

• Greenplum built a document recommender for law firms that does hard-core statistical analysis to determine which .1% of a document set lawyers might actually want to see, and which then learns from users’ feedback after they respond to initial result sets.
• Information extracted from investment news gets included into automated trading algorithms. This was unusual technology a couple of years ago, but is more common today.
• After a series of mergers, Attensity now uses marketing-oriented text analytics in at least three different ways:
  • Attensity text analytics feeds marketing dashboards just as it always did.
  • Attensity text analytics triggers alerts, as I wish dashboards and business intelligence tools more often did, the false positives problem notwithstanding.
  • Attensity text analytics triggers concrete workflows, for example routing specific social media hits for priority response.
  • And in one example that did not actually get into production, a very large social networking company correlated word usage (e.g., choice among different synonyms) against user characteristics such as age and gender.

Finally there are some applications that, while fitting the standard template, just strike me as getting to unusually sophisticated levels of analysis. For example, Vertica told me of another Vertica/Hadoop case where VotM document analysis is carried out to the level of observing which order brand names appear in, and adjusting that for whether or not it was just an alphabetical list.

I suspect text analytics is about to become more interesting again.

Related links

• The enabling technology for text/tabular data integration has existed for years.
• In 2006, I listed major application areas for data mining/predictive analytics. It overlaps pretty closely with the similar list for text mining/text analytics.
• Before being acquired by IBM, SPSS boasted a rather large text mining user base.

That is, of course, hugely more than any of the independents (e.g. Attensity and Clarabridge) have. And it’s focused on marketing-oriented apps — especially Voice of the Customer — just as those vendors are. Even so, they report rarely seeing SPSS, and SPSS agrees with that assessment.

The obvious explanation — which Olivier does not dispute — is that Text Mining For Clementine sales are focused on Clementine data mining users. But that raises an interesting follow-up — how much data mining are these users really doing on text data? Attensity and Clarabridge customers do little true data mining, but Olivier asserts that SPSS customers do quite a bit — predictive modeling, real-time scoring, and the whole enchilada.

By the way, Olivier actually no longer runs SPSS’ text mining business. He’s moved to Chicago as VP of Corporate Development, focused on acquisitions. Coincidentally, he has a glum view of the prospects for independent text analytics companies, and believes the best course for them is to be acquired.

This inspired me to check whether there had been any big changes in text mining capabilities at SAS or SPSS. It turned out there hadn’t. SAS is also still on the bag-of-words level. SPSS, however, does do sentiment analysis (pretty obvious, considering their focus on surveys and the like) and negation.

Thanks go out to Mary Crissey and Olivier Jouve for getting back to me when I asked, along with apologies for taking a while to post what they told me.

But above all, it was “Voice of the Customer.” I know it’s till June, but I think we have our text analytics industry buzzphrase of the year.

I found another example. ClearForest’s strong position in serving the automobile warranty market let them also scarf up customer-satisfaction surveyor J. D. Power as a customer. And it seems that J. D. Power has indeed added several new free-form text entry fields to its surveys.

No report yet, however, on whether any dramatically improved insights have come from these efforts.

So I’m going to try to gather some useful observations on marketing here, filling the gap that the panel left. Key questions I’d love input on include:

1. Which feature-benefit connections do you see customers easily accepting?

2. Which feature-benefit connections is it harder to get them to believe?

3. How are customers defining text analytics market segments?

4. What do they see as the key issues in each segement?

5. Which application areas are showing growth even beyond that of the market overall?

I’m particularly interested in comments from the larger vendors that are selling into multiple parts of the text mining and text analytics market. But everybody else’s input would be warmly appreciated too.

The comment thread to this post is open for business!

So can we apply the same approach to text mining? One place would be surveys. Rather than those annoying, contrived forms demanding we fill in a lot of choices as if we were taking the SATs all over again, maybe users would be more revealing if they could just write whatever they wanted? The obvious firm to ask is SPSS, which is big both in surveys and text mining, not to mention the intersection of the two markets. So I emailed Olivier Jouve, and he shot back an answer from an airport.

Specifically, I asked:

It occurs to me that we could expand the text mining market a lot of we could prove the following claim:

“Customers hate filling out structured surveys. Just let them write what they want, and the response rate will be a lot higher. Then, text mine it, and you’ll discover what they really think.”

A. Is that true???
B. If is is, do you have proof points?

And he responded (lightly edited):

A) True

B) We have 1100 unique organizations using our product called “Text Analysis for Surveys” (launched 15 months ago). You know how strong SPSS is in the market research stuff … This product is a killer for extracting opinions, etc. We have been focusing on this technology for years … You need sophisticated analysis (linguistic dependencies, etc) to extract good results, and to categorize on existing code frames (we propose some clustering algorithms to help building those code frames as well).

However, not all those customers are at the level you describe. They still mix open-ended and structured questions. But we observe more and more open-ended.

Thanks, Olivier! But, uh — what’s a code frame? Something to do with “coding” survey results?