The state of the art in text analytics applications
Text analytics application areas typically fall into one or more of three broad, often overlapping domains:
- Understanding the opinions of customers, prospects, or other groups. This can be based on any combination of documents the user organization controls (email, surveys, warranty reports, call center logs, etc.) — in which case — or public-domain documents such as blogs, forum posts, and tweets. The former is usually called Voice of the Customer (VotC), while the latter is Voice of the Market (VotM).
- Detecting and identifying problems. This can happen across many domains — VotC, VotM, diagnosing equipment malfunctions, identifying bad guys (from terrorists to fraudsters), or even getting early warnings of infectious disease outbreaks.
- Aiding text search, custom publishing, and other electronic document-shuffling use cases, often via document augmentation.
For several years, I’ve been distressed at the lack of progress in text analytics or, as it used to be called, text mining. Yes, the rise of sentiment analysis has been impressive, and higher volumes of text data are being processed than were before. But otherwise, there’s been a lot of the same old, same old. Most actual deployed applications of text analytics or text mining go something like this:
- A bunch of documents are analyzed to ascertain the ideas expressed in them.
- A count is made as to how many times each idea turns up.
- The application user notices any surprisingly large numbers, and as result of noticing pays attention to the corresponding ideas.
Often, it seems desirable to integrate text analytics with business intelligence and/or predictive analytics tools that operate on tabular data is. Even so, such integration is most commonly weak or nonexistent. Apart from the usual reasons for silos of automation, I blame this lack on a mismatch in precision, among other reasons. A 500% increase in mentions of a subject could be simple coincidence, or the result of a single identifiable press article. In comparison, a 5% increase in a conventional business metric might be much more important.
But in fairness, the text analytics innovation picture hasn’t been quite as bleak as what I’ve been painting so far. Read more