• Understanding the opinions of customers, prospects, or other groups. This can be based on any combination of documents the user organization controls (email, surveys, warranty reports, call center logs, etc.) — in which case — or public-domain documents such as blogs, forum posts, and tweets. The former is usually called Voice of the Customer (VotC), while the latter is Voice of the Market (VotM).
• Detecting and identifying problems. This can happen across many domains — VotC, VotM, diagnosing equipment malfunctions, identifying bad guys (from terrorists to fraudsters), or even getting early warnings of infectious disease outbreaks.
• Aiding text search, custom publishing, and other electronic document-shuffling use cases, often via document augmentation.

For several years, I’ve been distressed at the lack of progress in text analytics or, as it used to be called, text mining. Yes, the rise of sentiment analysis has been impressive, and higher volumes of text data are being processed than were before. But otherwise, there’s been a lot of the same old, same old. Most actual deployed applications of text analytics or text mining go something like this:

• A bunch of documents are analyzed to ascertain the ideas expressed in them.
• A count is made as to how many times each idea turns up.
• The application user notices any surprisingly large numbers, and as result of noticing pays attention to the corresponding ideas.

Often, it seems desirable to integrate text analytics with business intelligence and/or predictive analytics tools that operate on tabular data is. Even so, such integration is most commonly weak or nonexistent. Apart from the usual reasons for silos of automation, I blame this lack on a mismatch in precision, among other reasons. A 500% increase in mentions of a subject could be simple coincidence, or the result of a single identifiable press article. In comparison, a 5% increase in a conventional business metric might be much more important.

But in fairness, the text analytics innovation picture hasn’t been quite as bleak as what I’ve been painting so far. While standalone, passively-reported text analytics is indeed the baseline, there are some interesting exceptions. For example:

• I once confirmed that SPSS customer Cablecom‘s statistical models for churn and the like absolutely included text data; Cablecom even assigned different weights to the same apparent level of emotion depending on whether the text was in German, French, or Italian. Vertica recently told me of a Vertica/Hadoop customer doing something similar, except for the multilingual aspect. And the end of a 2008 SAS-based paper makes similar claims.
• There long* have been some examples of fact extraction that don’t really fit into my three buckets above. For example, researchers mine collections of articles to try to determine biochemical or biological pathways that would not be apparent from examining single research studies alone.
• It also has long* been the case that some bad-guy-finding applications — especially in the anti-terrorism area — used text analytics to populate state-of-the-art graph-oriented data analysis tools.

*When it comes to text analytics, “long” means “at least for the past several years.”

In more recent examples:

• Greenplum built a document recommender for law firms that does hard-core statistical analysis to determine which .1% of a document set lawyers might actually want to see, and which then learns from users’ feedback after they respond to initial result sets.
• Information extracted from investment news gets included into automated trading algorithms. This was unusual technology a couple of years ago, but is more common today.
• After a series of mergers, Attensity now uses marketing-oriented text analytics in at least three different ways:
  • Attensity text analytics feeds marketing dashboards just as it always did.
  • Attensity text analytics triggers alerts, as I wish dashboards and business intelligence tools more often did, the false positives problem notwithstanding.
  • Attensity text analytics triggers concrete workflows, for example routing specific social media hits for priority response.
  • And in one example that did not actually get into production, a very large social networking company correlated word usage (e.g., choice among different synonyms) against user characteristics such as age and gender.

Finally there are some applications that, while fitting the standard template, just strike me as getting to unusually sophisticated levels of analysis. For example, Vertica told me of another Vertica/Hadoop case where VotM document analysis is carried out to the level of observing which order brand names appear in, and adjusting that for whether or not it was just an alphabetical list.

I suspect text analytics is about to become more interesting again.

Related links

• The enabling technology for text/tabular data integration has existed for years.
• In 2006, I listed major application areas for data mining/predictive analytics. It overlaps pretty closely with the similar list for text mining/text analytics.
• Before being acquired by IBM, SPSS boasted a rather large text mining user base.

The successful candidate will mine text data from numerous news sources and incorporate the information the proprietary trading systems.

So what else is new? The leading text mining vendors also aren’t active in text search, except to some extent in the custom-publishing vertical, despite the huge reliance of search vendors on text mining technologies. They aren’t getting traction in the archiving/compliance area. There don’t seem to be significant efforts to develop next-generation integrated servers. In fact, there doesn’t seem to be much of anything except:

1. Custom publishing.

2. The same old, same old failure analysis and threat detection.

Jay Henderson, when he was at ClearForest, told me he tracked 80 text mining companies whose aggregate revenue he estimated at under $100 million. He also didn’t think any text mining company would get over $20 million in revenue. Now, I think the latter was a little pessimistic. But directionally, unless the industry seriously gets its act together, he was correct.

Right now, I think the most likely outcome is that the text mining industry pretty much gets merged out of existence in a few years. Consolidation is a pretty safe way to bet in most software sectors these days, and I don’t see the kind of energy in the text mining sector that is required to beat the consolidation trend.

Attivio is having a house party and product rollout in the latter part of January, and details are scarce in the mean time. But here are some highlights.

• Attivio was founded in August. It has 21 people and 1 VC. The VC has invested >$6 million and committed >$12 million total.
• Attivio has ambitious plans for a fully integrated data management/real-time BI stack. It’s currently called the “Active Intelligence Engine.”
• The data management part combines tabular, text, and XML data. The tabular part is some kind of bitmap. The text part is fairly traditional, and based on Lucene.
• One point of this architecture is that one can more or less seamlessly join different kinds of data.
• Another point is surely that — with everything being more or less like a column or bitmap — memory management and administration are manageable issues.
• Despite containing all these wonders, the code is under 10 megs total. At least right now. But then — how much code can one write in a few months?
• Andrew didn’t want me to repeat everything he said about target markets, but clearly Wall Street is one of the top possibilities.

Stay tuned.

• Yep, Voice Of The Customer is hot, in “many markets”; Eric specifically mentioned banking, car, energy, food, and retail. He further sees IBM backing VotC as text’s “killer app.” (Note: Temis has a history of partnering with IBM, most notably via its unusually strong commitment to UIMA.)
• Specifically, THE hot topics in the European market these days are competitive intelligence and sentiment analysis. (Note: I’ve always thought Temis got serious about competitive analysis a little earlier than most other text mining vendors did.)
• Life sciences is an ever growing focus for Temis.
• I confused him a bit with how I phrased my question about custom publishing and Temis’ Mark Logic partnership. But he did express favorable views of the market, specifically in the area of integrating text mining and native XML database management, and even volunteered that nStein appears to be doing well.

• Voice of the Customer is definitely tops.
• Some of the other applications Attensity and Clarabridge mentioned appear as well (e.g., antifraud).
• Business Objects also has a couple of customers looking at text mining as an aid to medical records, e.g. by helping to catch errors in tabular-field coding.
• There are some projects in actual investment research/analysis/trading, e.g. in correlating news announcements and stock price movements.

The Business Objects/Inxight folks also made a couple of interesting general technical points. When I challenged the usefulness of text analytics in dashboards, they pointed out how it can at least be a good drill-down. (Example: You’re getting unusually many of customer complaints in a particular time frame; you drill down into a text mining-based graphic to see which particular areas of complaint have spiked.) Also, when I mentioned exhaustive extraction, Ian Hersey pointed out that in many cases the intermediate results of Inxight tagging and so on happen to be stored in an RDBMS.

Perhaps most important, I got a general feeling that Business Objects is serious about integrating Inxight into its general product offerings, which is not good news for independent text mining vendors such as Attensity or Temis (except insofar as it heats up the acquisition market for same). On the other hand, I’ve gotten nothing but confirmation of my view that Business Objects plans to remain a good OEM partner, even to competitors such as SAS.

This is my first post from the conference. There surely will be more soon on DBMS2 and the Monash Report.

Get great research about text mining, data warehouse appliances, and other hot analytics-related topics! Subscribe to our comprehensive (if not exhaustive) feed, by RSS/Atom or e-mail! We recommend taking the integrated feed for all our blogs, but blog-specific ones are also easily available.

Technorati Tags: Business Objects, Inxight, text analytics

Other highlights of what they said included:

• Voice of the Customer remains hot, hot, hot.
• Closely allied with Voice of the Customer, and also hot, is Voice of the Market and/or more direct competitive intelligence.
• Classical warranty analysis is quiet but not wholly dead. Attensity, historically strong in that application, sees it as merging into Voice of the Customer. Clarabridge, previously not so strong there (if I recall correctly), is getting at least a little of the traditional-style warranty business.
• Human resources (especially Voice of the Employee – I detect a trend in application-naming here) gets mentioned a fair amount. It’s usually not the first text mining application an enterprise deploys, but it’s a common follow-on.
• Antifraud isn’t just for insurance companies. Retailing and money-laundering also got mentioned as areas where text mining helped combat fraud.
• Insurance industry use of text mining for claims analysis, I gather, goes well beyond just fraud detection.
• Intelligence is obviously a huge market for Attensity (not so much for Clarabridge), but I didn’t focus on the classified stuff. That said, I was reminded of Attensity’s awkward phrase link analysis, which has nothing to do with hypertext, but instead is the detection of relationships between entities. This lies at the heart of a non-empty set of civilian law enforcement applications and the like.
• Investment research applications of text mining still seem nascent and experimental, at least if one talks with Clarabridge and Attensity. That said, Factiva is a large subsidiary of Dow Jones now, and ClearForest a smaller one of Reuters, and they’re doing something or other. Apparently, it’s much more document tagging for the sake of readers or search-style filters than it is for use in any kind of business intelligence/statistical mining kind of application.

All this isn’t too different from what I posted back in July, but I think text mining application trends is a subject that bears frequent revisiting.

That’s about all the detail I could muster. When we discussed this, we talked past each other a fair amount. It’s clear they’ve thought about direct analytic integration and also about how text and tabular data could work together side-by-side. But I didn’t get the impression their otherwise top-of-the-line technology-consuming customers were doing everything with text analytics that the technology makes available.

Perhaps a partnership with a tokenization OEM would stand Apama in good stead.

I think a lot of the reason for investment firms not making more use of text analytics has been structural — Factiva, the (relatively speaking) mammoth joint venture of Reuters and Dow Jones, is forbidden by its parent companies from meeting investment firms’ needs. And that’s kind of a pity, as it’s probably the best-positioned firm to do so. It’s good to hear that the little guys are finally filling the gap.