*Or if it does, it’s VERY well-hidden. I looked at the home page and site map alike.

I wanted to refresh my memory as to Mark Logic’s history of working with specific text mining vendors, beyond what’s on the official partner page. No luck. Normally when site search is inadequate, one goes to Google. But that’s problematic too. Marklogic.com pages come up pretty low on Google’s search results, suggesting that:

1. Mark Logic doesn’t put a lot of effort into SEO (or else doesn’t do it very well).
2. One can’t be confident all the site’s significant pages are findable by Google.

Looking to other companies’ sites for clues isn’t conclusive either. E.g., Clearforest lists Mark Logic as a partner, but Mark Logic doesn’t return the compliment. (If memory serves, Mark Logic and Clearforest have worked together both on national security deals and custom publishing deals — but don’t hold me to that.)

When it comes to making its own information conveniently available, Mark Logic is quite the unshod cobbler.

Other highlights of what they said included:

• Voice of the Customer remains hot, hot, hot.
• Closely allied with Voice of the Customer, and also hot, is Voice of the Market and/or more direct competitive intelligence.
• Classical warranty analysis is quiet but not wholly dead. Attensity, historically strong in that application, sees it as merging into Voice of the Customer. Clarabridge, previously not so strong there (if I recall correctly), is getting at least a little of the traditional-style warranty business.
• Human resources (especially Voice of the Employee – I detect a trend in application-naming here) gets mentioned a fair amount. It’s usually not the first text mining application an enterprise deploys, but it’s a common follow-on.
• Antifraud isn’t just for insurance companies. Retailing and money-laundering also got mentioned as areas where text mining helped combat fraud.
• Insurance industry use of text mining for claims analysis, I gather, goes well beyond just fraud detection.
• Intelligence is obviously a huge market for Attensity (not so much for Clarabridge), but I didn’t focus on the classified stuff. That said, I was reminded of Attensity’s awkward phrase link analysis, which has nothing to do with hypertext, but instead is the detection of relationships between entities. This lies at the heart of a non-empty set of civilian law enforcement applications and the like.
• Investment research applications of text mining still seem nascent and experimental, at least if one talks with Clarabridge and Attensity. That said, Factiva is a large subsidiary of Dow Jones now, and ClearForest a smaller one of Reuters, and they’re doing something or other. Apparently, it’s much more document tagging for the sake of readers or search-style filters than it is for use in any kind of business intelligence/statistical mining kind of application.

All this isn’t too different from what I posted back in July, but I think text mining application trends is a subject that bears frequent revisiting.

*Factiva is the most significant exception. Hint, hint.

If you think about it, text analytics is a “secret ingredient” in search, antispam, and data cleaning,* and this dominates all other uses of the technology. A significant minority of the research effort at companies that do any kind of text filtering is – duh — text analytics. Cold comfort for specialist text analytics vendors, to be sure, but that’s the way it is.

*I.e., part of the “T” in “ETL” (Extract/Transform/Load).

Text-analytics-enhanced custom publishing will surely at some point become a must-have for business and technical publishers. However, it appears that we’re not quite there yet, as large publishers make do with simple-minded search and the like. In what I suspect is a telling market commentary, there’s no headlong rush among vendors to dump text mining for custom publishing, notwithstanding the examples of nStein and (sort of) ClearForest. I don’t want to be overly negative – either my friends at Mark Logic are doing just fine or else they’re putting up a mighty brave front – but I don’t think the nonspecialist publishing market is there yet.

Two business publishers who have made major investments in owning text analytics technology are Dow Jones (now sole owners of Factiva) and Reuters (recent purchaser of ClearForest). Beyond that, however, I don’t yet see a lot of activity in the investor/trading market, although ClearForest reported some activity last year and StreamBase reports that one customer is using them for text filtering, presumably alongside the ticker-munching traders usually use StreamBase for.

Obviously, the intelligence market is what fueled the start of the text analytics business, and still provides the majority of revenue at multiple companies. Certainly it’s still going strong. But it’s tough to gauge the growth potential from here, especially since the details of usage are typically classified.

Similar things could be said about pharmaceutical research. Text analytics is totally accepted in that market, but what’s the growth potential from here? And “here” isn’t actually very big (much smaller than intelligence). The related category of patient records analysis looks very promising, but is basically still at the research-project stage. (In general, an explosion in biological IT can be expected when research methods are adapted for clinical use.)

The warranty analysis market, so promising early on, is not showing a lot of growth and depth. The same thing has happened many times before with innovative technologies sold to manufacturing companies’ engineers. It seems to be happening again now.

Voice of the customer* is pretty much the same thing, but for service industries. And the text analytics market for VotC is evidently stronger right now than that for warranty analysis. This makes sense, because the obvious alternative to text analytics – multiple-choice coded forms – is less appealing, due to two application differences:

• VotC looks for opinion as well as fact.

• VotC looks for input from people under no obligation to share it, and who hence can’t be compelled to play along with a structured form – let alone trained to fill it in accurately.

*Definitional note: Voice of the customer is when customers or prospects communicates with you directly, e.g. via a survey form or an angry email. Reputation management is when you web-scrape and find out what they’re saying to everybody else. At least, I think marketers are still using the terms that way pretty consistently.

Reputation management is surely becoming a standard application for the biggest consumer brands. How deep that market turns out to be, however, remains to be seen.

Text analytics for fraud discovery seems poised to sweep the insurance industry, and then the rest of financial services. Current activity, however, while decent, still seems to consist of more poising than sweeping.

Compliance is a minimum-acceptable-efforts kind of activity in most markets. Accordingly search/clustering seems to be the preferred text-checking approach. Where that’s not the case, the market seems to have gone to specialized products like Assentor (stock brokerage).

Human resources is a good area to sell follow-on applications, at least to enterprises with so many employees that they want to automate the reading of employee feedback. I’m not aware of it being the first-sale app to very many enterprises, however.

SAS used to speak glowingly of text mining used directly for ETL. However, nobody else has talked about this, and even from SAS I get the sense that some of the glow has worn off. As noted above, text analytics is an important ingredient to the transformation part of ETL, but it I think it rarely would be the best option for doing the transformations directly.

• Business Objects’ text mining partnerships have been more demo/sales-cycle than actual sales up until now. That said, they have a few deals each with Attensity and Inxight (but not with ClearForest, which pulled in its horns prior to being acquired by Reuters). I still think they’re the leading BI vendor in integrating with text mining, SAS perhaps aside (who if nothing else have a lot of fun using text mining for data cleaning). The working Inxight partnership, by the way, was all about the specific app of email compliance, with the demo being based on the publicly available Enron corpus.
• Inxight’s visualization technology is in the form of an SDK anyway. So integrating it into BOBJ’s product line should be straightforward. Note: Through the Excelsius acquisition, BOBJ has been trying to gain competitive advantage in the cool-visualization area.
• Inxight’s “federation” capability for search is pretty primitive (my term and opinion of course, not theirs). It takes in search result sets from various sources, then clusters and/or refilters them. What it does NOT do is the much harder task of taking actual relevancy rankings from various engines and somehow arbitrating between them. Nor, I’m guessing, does it even assign higher or lower weights to various corpuses or anything like that. Thus, it does not sound terribly competitive with the distributed search capabilities built into any state-of-the-art enterprise search engine.

If the two main parts of the text mining market are custom publishing and finding warning signs, then both could actually be a good fit with Reuters. The custom publishing part is obvious. As for early warning – well, maybe ClearForest will lose its competitive edge in consumer product warranty analysis or something, but a significant fraction of the early warning market is tied to news articles, web postings, and other things that are a good fit for Reuters.

But the really interesting (at least to me) possibilities arise in the core Reuters and Dow Jones business of supporting investment decisions. Factiva was prevented by partnership agreement from pursuing that market, which made things weird, since the parent firms weren’t doing what they could with text analytics either. Now each has its own text analytics subsidiary, and all possibilities are presumably on the table. And oh by the way – Factiva is arguably the biggest text mining/fact extraction company around, while Clearforest was making progress in the hedge fund market shortly before marketing went radio-silent.

Hmm. I want to think about this a bit before posting a list of possibilities. Or maybe even talk with Factiva, something I’ve – amazingly – almost never done.

Does anybody know what’s going on at ClearForest these days? A comparison of the current management team listing and one from last April does suggest there’s been substantial turnover in sales, marketing, and development management.

Technorati Tags: ClearForest

Want to continue getting great research about search, text mining, and other hot text technology topics? Then subscribe to our feed, by RSS/Atom or e-mail! We do recommend taking the integrated feed for all our blogs, so that you learn about other subjects as well, such as database management, business intelligence, and computing appliances. But blog-specific feeds are also easily available.

I think a lot of the reason for investment firms not making more use of text analytics has been structural — Factiva, the (relatively speaking) mammoth joint venture of Reuters and Dow Jones, is forbidden by its parent companies from meeting investment firms’ needs. And that’s kind of a pity, as it’s probably the best-positioned firm to do so. It’s good to hear that the little guys are finally filling the gap.

• Attensity wants to make text analytics very easy to integrate into business intelligence and data mining – at the moment, they’re not too focused on the differences between those two disciplines – and is trying to deliver the best possible fact extraction consistent with that charter.
• ClearForest wants to provide really great information extraction — to the limits of what can be done without excessive knowledge engineering – and is trying to integrate as well as possible with other technologies, the better to serve the customers who need what they offer.

The guy I usually talk with at ClearForest, Jay Henderson, believes that text analytics is a collection of dozens of niche markets. Not coincidentally, a lot of ClearForest’s customers are in the publishing sector (I’ve remarked on ClearForest’s synergy with Mark Logic before). Attensity obviously is trying a broader play. In Jay’s view, Inxight and TEMIS are more analogous to ClearForest than Attensity is, except that Inxight is focused on different markets (e.g. OEM and/or search), and he thinks ClearForest is just better than Temis except in a couple of specific kinds of understanding (e.g., life sciences, sentiment).

That said, both Attensity and ClearForest credibly claim to do large fractions of what the other one does. ClearForest, as the currently nichier player, takes the traditional stance “We do everything they do, and more. Most of our customers are ones who really appreciate the difference.” Attensity conveys the equally traditional attitude “We do most of what they do, and a bunch of other stuff besides. And it’s better-packaged too. As for what they do that we don’t – not a lot of customers really have a need for it.”

Frankly, most enterprises that have a need for this technology should put both Attensity and ClearForest on their short lists. But here’s one technical note that may help predict who you’ll wind up actually selecting: Attensity’s lead strategy for integration is to dump everything into relational tables, for conventional analytics-stack products like Business Objects’ and Teradata’s to manipulate. ClearForest’s lead strategy for integration has more of an SOA/XML flavor, grown out of conventional OO. If one of those sounds like an obviously better fit to your situation than the other, then that’s the vendor you absolutely, positively should not leave out of your evaluation process.

Here, as best I understand, is the story of the custom publishing business. Its core market is publishers of high-cost material sold to people with high-priced time – i.e., scientific/engineering/medical/legal/business/etc. Other markets are general publishing, internal document preparation (e.g., intelligence community), and of course maintenance manuals (maintenance/repair has been a flagship market for just about everything, from expert systems to generic text search to, of course, text mining now as well).

The phrase “custom publishing,” however, obscures the distinction between two different paradigms. One of these is what we might call true custom publishing – assembling paragraphs, articles, chapters whatever from various sources, in an assembly customized for specific reader needs, roles, or preferences. On the revenue side, that’s a fascinating subject. But technically, I’m more interested in the other view: search results plus.

We all know lots of problems with search engines. One of the many is this: Except on rare occasions, getting the benefit of a successful search involves a whole lot of link-clicking and scrolling. But what if the relevant passages were all assembled together for you? Link-clicking would be eliminated, and scrolling might be minimized as well. The potential is huge. But I don’t know what level of precision is needed before the theoretical benefits become real.

The two paradigms can be blended, of course. A publishing product or dashboard or personal web page with a topic filter might get the results in custom-document rather than link-of-lists form. Once again, the problem is conciseness. The more concise the “complete” results can be, the more useful this kind of technology will ultimately prove.

For more on Mark Logic, and more insight about the industry in general, see CEO Dave Kellogg’s blog. For a technical discussion of MarkLogic, see my DBMS2.com write-up.