So what else is new? The leading text mining vendors also aren’t active in text search, except to some extent in the custom-publishing vertical, despite the huge reliance of search vendors on text mining technologies. They aren’t getting traction in the archiving/compliance area. There don’t seem to be significant efforts to develop next-generation integrated servers. In fact, there doesn’t seem to be much of anything except:

1. Custom publishing.

2. The same old, same old failure analysis and threat detection.

Jay Henderson, when he was at ClearForest, told me he tracked 80 text mining companies whose aggregate revenue he estimated at under $100 million. He also didn’t think any text mining company would get over $20 million in revenue. Now, I think the latter was a little pessimistic. But directionally, unless the industry seriously gets its act together, he was correct.

Right now, I think the most likely outcome is that the text mining industry pretty much gets merged out of existence in a few years. Consolidation is a pretty safe way to bet in most software sectors these days, and I don’t see the kind of energy in the text mining sector that is required to beat the consolidation trend.

Technorati Tags: Factiva

Other highlights of what they said included:

• Voice of the Customer remains hot, hot, hot.
• Closely allied with Voice of the Customer, and also hot, is Voice of the Market and/or more direct competitive intelligence.
• Classical warranty analysis is quiet but not wholly dead. Attensity, historically strong in that application, sees it as merging into Voice of the Customer. Clarabridge, previously not so strong there (if I recall correctly), is getting at least a little of the traditional-style warranty business.
• Human resources (especially Voice of the Employee – I detect a trend in application-naming here) gets mentioned a fair amount. It’s usually not the first text mining application an enterprise deploys, but it’s a common follow-on.
• Antifraud isn’t just for insurance companies. Retailing and money-laundering also got mentioned as areas where text mining helped combat fraud.
• Insurance industry use of text mining for claims analysis, I gather, goes well beyond just fraud detection.
• Intelligence is obviously a huge market for Attensity (not so much for Clarabridge), but I didn’t focus on the classified stuff. That said, I was reminded of Attensity’s awkward phrase link analysis, which has nothing to do with hypertext, but instead is the detection of relationships between entities. This lies at the heart of a non-empty set of civilian law enforcement applications and the like.
• Investment research applications of text mining still seem nascent and experimental, at least if one talks with Clarabridge and Attensity. That said, Factiva is a large subsidiary of Dow Jones now, and ClearForest a smaller one of Reuters, and they’re doing something or other. Apparently, it’s much more document tagging for the sake of readers or search-style filters than it is for use in any kind of business intelligence/statistical mining kind of application.

All this isn’t too different from what I posted back in July, but I think text mining application trends is a subject that bears frequent revisiting.

*Factiva is the most significant exception. Hint, hint.

If you think about it, text analytics is a “secret ingredient” in search, antispam, and data cleaning,* and this dominates all other uses of the technology. A significant minority of the research effort at companies that do any kind of text filtering is – duh — text analytics. Cold comfort for specialist text analytics vendors, to be sure, but that’s the way it is.

*I.e., part of the “T” in “ETL” (Extract/Transform/Load).

Text-analytics-enhanced custom publishing will surely at some point become a must-have for business and technical publishers. However, it appears that we’re not quite there yet, as large publishers make do with simple-minded search and the like. In what I suspect is a telling market commentary, there’s no headlong rush among vendors to dump text mining for custom publishing, notwithstanding the examples of nStein and (sort of) ClearForest. I don’t want to be overly negative – either my friends at Mark Logic are doing just fine or else they’re putting up a mighty brave front – but I don’t think the nonspecialist publishing market is there yet.

Two business publishers who have made major investments in owning text analytics technology are Dow Jones (now sole owners of Factiva) and Reuters (recent purchaser of ClearForest). Beyond that, however, I don’t yet see a lot of activity in the investor/trading market, although ClearForest reported some activity last year and StreamBase reports that one customer is using them for text filtering, presumably alongside the ticker-munching traders usually use StreamBase for.

Obviously, the intelligence market is what fueled the start of the text analytics business, and still provides the majority of revenue at multiple companies. Certainly it’s still going strong. But it’s tough to gauge the growth potential from here, especially since the details of usage are typically classified.

Similar things could be said about pharmaceutical research. Text analytics is totally accepted in that market, but what’s the growth potential from here? And “here” isn’t actually very big (much smaller than intelligence). The related category of patient records analysis looks very promising, but is basically still at the research-project stage. (In general, an explosion in biological IT can be expected when research methods are adapted for clinical use.)

The warranty analysis market, so promising early on, is not showing a lot of growth and depth. The same thing has happened many times before with innovative technologies sold to manufacturing companies’ engineers. It seems to be happening again now.

Voice of the customer* is pretty much the same thing, but for service industries. And the text analytics market for VotC is evidently stronger right now than that for warranty analysis. This makes sense, because the obvious alternative to text analytics – multiple-choice coded forms – is less appealing, due to two application differences:

• VotC looks for opinion as well as fact.

• VotC looks for input from people under no obligation to share it, and who hence can’t be compelled to play along with a structured form – let alone trained to fill it in accurately.

*Definitional note: Voice of the customer is when customers or prospects communicates with you directly, e.g. via a survey form or an angry email. Reputation management is when you web-scrape and find out what they’re saying to everybody else. At least, I think marketers are still using the terms that way pretty consistently.

Reputation management is surely becoming a standard application for the biggest consumer brands. How deep that market turns out to be, however, remains to be seen.

Text analytics for fraud discovery seems poised to sweep the insurance industry, and then the rest of financial services. Current activity, however, while decent, still seems to consist of more poising than sweeping.

Compliance is a minimum-acceptable-efforts kind of activity in most markets. Accordingly search/clustering seems to be the preferred text-checking approach. Where that’s not the case, the market seems to have gone to specialized products like Assentor (stock brokerage).

Human resources is a good area to sell follow-on applications, at least to enterprises with so many employees that they want to automate the reading of employee feedback. I’m not aware of it being the first-sale app to very many enterprises, however.

SAS used to speak glowingly of text mining used directly for ETL. However, nobody else has talked about this, and even from SAS I get the sense that some of the glow has worn off. As noted above, text analytics is an important ingredient to the transformation part of ETL, but it I think it rarely would be the best option for doing the transformations directly.

If the two main parts of the text mining market are custom publishing and finding warning signs, then both could actually be a good fit with Reuters. The custom publishing part is obvious. As for early warning – well, maybe ClearForest will lose its competitive edge in consumer product warranty analysis or something, but a significant fraction of the early warning market is tied to news articles, web postings, and other things that are a good fit for Reuters.

But the really interesting (at least to me) possibilities arise in the core Reuters and Dow Jones business of supporting investment decisions. Factiva was prevented by partnership agreement from pursuing that market, which made things weird, since the parent firms weren’t doing what they could with text analytics either. Now each has its own text analytics subsidiary, and all possibilities are presumably on the table. And oh by the way – Factiva is arguably the biggest text mining/fact extraction company around, while Clearforest was making progress in the hedge fund market shortly before marketing went radio-silent.

Hmm. I want to think about this a bit before posting a list of possibilities. Or maybe even talk with Factiva, something I’ve – amazingly – almost never done.