Text Technologies » Autonomy

How good does e-discovery search need to be?

Curt Monash — Mon, 01 Sep 2008 04:44:58 +0000

Two years ago, CEO Mike Lynch of Autonomy tried to persuade me that Autonomy was and would remain dominant in the e-discovery search market because:

The essence of the buying decision was that enterprises wanted to fulfill obligations to make their information available in a way that would would satisfy the courts.
Autonomy had some high-profile traction (e.g., the Enron case) that made it the default decision, and hence in particular a choice that met the requirement.

Recently, I ran that theory by David Ferris, whose firm Ferris Research has long been a/the leading small analyst firm covering e-mail and related technologies. He wasn’t buying. David believes courts are getting more sophisticated in their understanding of search technology. Even more to the point, David cited several other buying motivations that would lead enterprises to want best-available rather than just-good-enough e-discovery search technology, such as:

Enterprises want to know what information is available to be discovered against them.
Enterprises want to discover the information that will best aid their legal defense.
If they’re archiving the material for one purpose (e-discovery) anyway, enterprises want to get the most possible value out of it for other purposes while they’re at it.

Event stream processors active in text filtering

Curt Monash — Thu, 26 Jul 2007 11:08:35 +0000

OK. I secured permission to actually quote the details on something I’d previously dropped a small hint about — stream processing for text messages. Traditionally, that’s been the province of enterprise search companies. A decade ago, Verity had a kernel group of 6-7 engineers under Phil Nelson. They managed to produce not only a decent search engine, but a search engine “turned on its side” as well. I.e., instead of running one query against a corpus, they could run many queries each against documents as they arrived, one document at a time. Subsequently, the same idea has been implemented by most enterprise search providers, at least those that are serious about the intelligence market.

Well, the event-processing guys are active in that market too. At least StreamBase is. It was an obvious guess to ask if they were, and over the past few months I’ve gotten confirmation (including that they partner w/ Inxight just like almost everybody else does). Here, quoted with permission and lightly edited, is what StreamBase VP Marketing Bill Hobbib has to say on the subject

Regarding text filtering and processing, we’ve done work in financial services for a hedge fund and extensive work with the federal government and intelligence community. The existing StreamBase schema and message/field structure supports a wide array of message types, and text parsing can be done in either the SMTP adapter or the StreamBase engine, where the text processing occurs. Multiplexing onto different streams/schemas is supported.

In terms of text processing, StreamBase can process emails, documents, sentences, individual words or letters using StreamSQL’s existing capabilities, including a variety of built-in standard string operations, or user-defined functions, custom operators, and user-defined aggregates. We can also partner with text-mining companies.

Though you didn’t ask about processing media such as audio or video data, the new BLOB datatype is designed for this purpose. We partner with feature extraction vendors (e.g. speech to text) where necessary, as we don’t process the native audio or video data–just the metadata or the extracted features and converted files.

More on StreamBase can be found over on DBMS2, e.g. in this post or this one.

Analyst reports about enterprise search

Curt Monash — Sat, 29 Jul 2006 12:16:33 +0000

Gartner and Forrester have high opinions of FAST. Not coincidentally, you can download both those firms’ recent search industry survey reports from almost any page of www.fastsearch.com. Of the two, Forrester’s is both better and more recent.

Summarizing brutally, the big firms’ consensus seems to be:

FAST and Autonomy are the clear leaders.
Endeca has great technology and is coming on strong.
Everybody else is a niche player, at least for now.
Convera is in deep yogurt.

Forrester is particularly harsh on Convera. Presumably this has much to do with the fact that Convera did not cooperate well with the survey process. I shall not speculate as to which way the causality runs there – but I should note that Convera was quite cooperative with my research last week.

Autonomy on text mining

Curt Monash — Sun, 23 Jul 2006 05:19:22 +0000

I asked Mike Lynch (Autonomy CEO) about text mining. He responded with an example:

A very well-known company “mines” its incoming emails for signs of trouble, not via any linguistics-driven approach, but just by clustering them. If a cluster changes size anomalously over time, it bears close investigation.

Update: Autonomy/Verity merger

Curt Monash — Sun, 23 Jul 2006 05:12:32 +0000

I had a couple of very interesting calls with Autonomy last week. One message I got was that they do not want to be pigeonholed in search, which they think on the whole is a primitive way of dealing with “unstructured information.” Nonetheless, my first post based on those calls will indeed focus on text indexing and search. You see, I wrote quite skeptically about the Autonomy/Verity merger when it was announced, and I’d like to amend that with an updated opinion. Autonomy’s claims can be summarized in part by the following:

1. Long before acquiring Verity, Autonomy supposedly offered full plug-compatibility with Verity’s text indexing product. (At varous times the Verity product was called “Topic” and “K2″.) This claim is almost credible. After all, there was no “secret sauce” in getting the result set; it was all tokenization and boolean operations and proximity and so on. The reason I’m a little skeptical about the full claim is that the same can’t be said for ranking the results. There, each vendor indeed had/has some secrets which aren’t so easy to reverse-engineer.

2. At the time of the acquisition and for a while before, Autonomy’s underlying engine (IDOL) was simply better than Verity’s. At least, that’s the way the kernels stacked up. When I was closest to Verity, which was for a couple of years including 1997, the kernel group consisted of about 6 guys, led by Phil Nelson. All except one left, and the remaining one wasn’t Phil. Verity never recovered. I bet Dennis McEvoy would have fixed things given enough time, but if memory serves he was there less than a year. (In general, Verity seemed to suffer from brutal turnover.)

The most specific and obvious point is that IDOL is multithreaded, while K2 Classic is singlethreaded. Beyond that, IDOL and K2 have similar architectures (inverted index on words, etc.), although IDOL overall tracks more info (all the Bayesian stuff, and maybe it also was more advanced in recognizing parts of documents, headers, etc. — this is another point where I’d be curious to get clarification.)

3. IDOL is now the upgrade path for K2 users. Well, in light of the above — duh. Of course it is. And even if plug-compatibility wasn’t perfect when the companies were independent, it surely was easy to achieve once Autonomy could learn the ranking functions used in Verity’s product.