This inspired me to check whether there had been any big changes in text mining capabilities at SAS or SPSS. It turned out there hadn’t. SAS is also still on the bag-of-words level. SPSS, however, does do sentiment analysis (pretty obvious, considering their focus on surveys and the like) and negation.

Thanks go out to Mary Crissey and Olivier Jouve for getting back to me when I asked, along with apologies for taking a while to post what they told me.

*Factiva is the most significant exception. Hint, hint.

If you think about it, text analytics is a “secret ingredient” in search, antispam, and data cleaning,* and this dominates all other uses of the technology. A significant minority of the research effort at companies that do any kind of text filtering is – duh — text analytics. Cold comfort for specialist text analytics vendors, to be sure, but that’s the way it is.

*I.e., part of the “T” in “ETL” (Extract/Transform/Load).

Text-analytics-enhanced custom publishing will surely at some point become a must-have for business and technical publishers. However, it appears that we’re not quite there yet, as large publishers make do with simple-minded search and the like. In what I suspect is a telling market commentary, there’s no headlong rush among vendors to dump text mining for custom publishing, notwithstanding the examples of nStein and (sort of) ClearForest. I don’t want to be overly negative – either my friends at Mark Logic are doing just fine or else they’re putting up a mighty brave front – but I don’t think the nonspecialist publishing market is there yet.

Two business publishers who have made major investments in owning text analytics technology are Dow Jones (now sole owners of Factiva) and Reuters (recent purchaser of ClearForest). Beyond that, however, I don’t yet see a lot of activity in the investor/trading market, although ClearForest reported some activity last year and StreamBase reports that one customer is using them for text filtering, presumably alongside the ticker-munching traders usually use StreamBase for.

Obviously, the intelligence market is what fueled the start of the text analytics business, and still provides the majority of revenue at multiple companies. Certainly it’s still going strong. But it’s tough to gauge the growth potential from here, especially since the details of usage are typically classified.

Similar things could be said about pharmaceutical research. Text analytics is totally accepted in that market, but what’s the growth potential from here? And “here” isn’t actually very big (much smaller than intelligence). The related category of patient records analysis looks very promising, but is basically still at the research-project stage. (In general, an explosion in biological IT can be expected when research methods are adapted for clinical use.)

The warranty analysis market, so promising early on, is not showing a lot of growth and depth. The same thing has happened many times before with innovative technologies sold to manufacturing companies’ engineers. It seems to be happening again now.

Voice of the customer* is pretty much the same thing, but for service industries. And the text analytics market for VotC is evidently stronger right now than that for warranty analysis. This makes sense, because the obvious alternative to text analytics – multiple-choice coded forms – is less appealing, due to two application differences:

• VotC looks for opinion as well as fact.

• VotC looks for input from people under no obligation to share it, and who hence can’t be compelled to play along with a structured form – let alone trained to fill it in accurately.

*Definitional note: Voice of the customer is when customers or prospects communicates with you directly, e.g. via a survey form or an angry email. Reputation management is when you web-scrape and find out what they’re saying to everybody else. At least, I think marketers are still using the terms that way pretty consistently.

Reputation management is surely becoming a standard application for the biggest consumer brands. How deep that market turns out to be, however, remains to be seen.

Text analytics for fraud discovery seems poised to sweep the insurance industry, and then the rest of financial services. Current activity, however, while decent, still seems to consist of more poising than sweeping.

Compliance is a minimum-acceptable-efforts kind of activity in most markets. Accordingly search/clustering seems to be the preferred text-checking approach. Where that’s not the case, the market seems to have gone to specialized products like Assentor (stock brokerage).

Human resources is a good area to sell follow-on applications, at least to enterprises with so many employees that they want to automate the reading of employee feedback. I’m not aware of it being the first-sale app to very many enterprises, however.

SAS used to speak glowingly of text mining used directly for ETL. However, nobody else has talked about this, and even from SAS I get the sense that some of the glow has worn off. As noted above, text analytics is an important ingredient to the transformation part of ETL, but it I think it rarely would be the best option for doing the transformations directly.

• Business Objects’ text mining partnerships have been more demo/sales-cycle than actual sales up until now. That said, they have a few deals each with Attensity and Inxight (but not with ClearForest, which pulled in its horns prior to being acquired by Reuters). I still think they’re the leading BI vendor in integrating with text mining, SAS perhaps aside (who if nothing else have a lot of fun using text mining for data cleaning). The working Inxight partnership, by the way, was all about the specific app of email compliance, with the demo being based on the publicly available Enron corpus.
• Inxight’s visualization technology is in the form of an SDK anyway. So integrating it into BOBJ’s product line should be straightforward. Note: Through the Excelsius acquisition, BOBJ has been trying to gain competitive advantage in the cool-visualization area.
• Inxight’s “federation” capability for search is pretty primitive (my term and opinion of course, not theirs). It takes in search result sets from various sources, then clusters and/or refilters them. What it does NOT do is the much harder task of taking actual relevancy rankings from various engines and somehow arbitrating between them. Nor, I’m guessing, does it even assign higher or lower weights to various corpuses or anything like that. Thus, it does not sound terribly competitive with the distributed search capabilities built into any state-of-the-art enterprise search engine.

• Curt Monash, President, Monash Information Services
• Dave Kellogg, CEO, Mark Logic Corporation
• Michelle De Haaff, VP Marketing, Attensity Corporation
• Michel Lemay, VP Marketing, nstein Technologies
• Mary Crissey, SAS Analytics Marketing Manager, SAS Institute

Michelle, Michel, and Mary are all obvious choices, responsible for marketing at leading text mining vendors. In addition, Mary has excelled on the same panel in the past, Michel sent me e-mail with some brilliant thoughts on the panel subject, and Attensity has one of the most interesting strategies in the text analytics market.

As for Dave — he’s simply one of the most astute marketing theorists working in software today. And he runs a very interesting text technology company. And he used to be most senior marketing guy in all of business intelligence, when he was SVP at Business Objects. In his copious free time, he writes a really cool blog.

• Price.
• Ease of installation (real or imagined).
• The familiar Google brand name.
• The familiar Google UI.
• Google OneBox’s ability to search relational records, reports, etc. along with more tradtional record types.

The last point, I think, is the most interesting. Lots of people think text search is and/or should be the dominant UI of the future. Now, I’ve been a big fan of natural language command line interfaces ever since the days of Intellect and Lotus HAL. But judging by the market success of those products — or for that matter of voice command/control — I was in a very small minority. Maybe the even simpler search interface — words jumbled together without grammatical structure — will win out instead.

Who knows? Progress is a funny thing. Maybe the ultimate UI will be one that responds well to grunts, hand gestures, and stick-figure drawings. We could call it NeanderHAL, but that would wrong …

So I’m going to try to gather some useful observations on marketing here, filling the gap that the panel left. Key questions I’d love input on include:

1. Which feature-benefit connections do you see customers easily accepting?

2. Which feature-benefit connections is it harder to get them to believe?

3. How are customers defining text analytics market segments?

4. What do they see as the key issues in each segement?

5. Which application areas are showing growth even beyond that of the market overall?

I’m particularly interested in comments from the larger vendors that are selling into multiple parts of the text mining and text analytics market. But everybody else’s input would be warmly appreciated too.

The comment thread to this post is open for business!