Coveo, Exalead, ISYS, Recommind, Vivisimo, and X1 are a few of a select group that are marking a mark in their respective niches, as products ready for action with a short implementation cycle (weeks or months not years).

By way of contrast, Lynda opines:

Autonomy and Endeca continue to bring value to very large projects in large companies but are not plug-and-play solutions, by any means. Oracle, IBM, and Microsoft offer search solutions of a very different type with a heavy vendor or third-party service requirement. Google Search Appliance has a much larger installed base than any of these but needs serious tuning and customization to make it suitable to enterprise needs.

In particular, her views about FAST (now Microsoft) are scathing.

There are two general issues here, based on the fact that Ali and a couple of other key Attivio executives come from FAST. First, they were at a corrupt company — but resigned before the worst (and perhaps all) of the corruption happened. Second, they were at a company that did very well in some respects, but very badly in others, so it’s a mixed-quality resume item.

So far, no biggie. Lots of executives exude overoptimism about their companies products and business prospects. And I haven’t identified anything which suggests to me as a former stock analyst that the controls Ali put in place as CFO/COO were inadequate. (If he’d been long-time CEO, it would have been a different matter, as he would have been more responsible for the general ethical culture of the company — but he wasn’t.)

So the main serious charge is that FAST funneled a lot of sales through small reseller companies owned by its executives, including Ali. Such arrangements could be used either for misappropriation of funds, or to inflate revenue. In the article, Ali denies involvement in any reseller until after he left FAST’s employment, but the reporter purports to have discovered proof to the contrary. I couldn’t quite get Ali to reiterate his denial to me — or, indeed, to talk with me directly about the matter — but did get an emailed statement which reads:

Mr. Riaz categorically denies any wrongdoing during his tenure at FAST or in any relationship with FAST thereafter. He has not been an employee of FAST for almost two years now, and therefore must defer all further comments to Microsoft’s official 2006 and 2007 statements on the matter.

I’ve advised my clients at Attivio that they should be clearer and more specific, but so far I’m not carrying the day. So for now, we’ll go with that.

The gist is that FAST followed a pattern very common in the packaged enterprise software industry:

• It had trouble meeting its growth targets.
• It inflated reported revenue (in the high-margin software industry, inflating license revenue has a huge impact on profits).
• One technique whereby it inflated revenue was to count deals that actually closed after quarter end.
• Another technique was to count deals as closed in which the customer hadn’t actually fully committed to buy.

There’s nothing new here. Back in the 1980s, we used to joke that MSA made 10% of its annual revenue and 100% of its profits between the 32nd and 40th of December.

Often, such problems are associated with difficulties getting product installations to succeed. Stephen Arnold suggests that’s exactly what happened in the case of FAST:

So, Fast Search’s problems began as soon as the company decided to push into the enterprise search market. The adjustments were, as noted in the documents I cited in my previous Fast Search analyses and in the TechCrunch article, small at the outset. Who knew that a customer would not pay his license fee installment? Then more customers groused about slow installs and the up front payments were not followed by any other payments. One Fast Search licensee told me that his Global 1000 company would not pay until Fast Search produced an engineer who could complete the installation per the task order. Well, Fast Search got an engineer to the client, but it was six months after I heard the complaint. Not surprisingly, this big outfit turned to a smaller vendor who got a different system up and running in three weeks.

Related links

The Attivio angle on this story
Edit: The actual article

• Wow.
• Both sides needed it.
• Yahoo wasn’t going anywhere fast on its own.
• Microsoft wasn’t going anywhere fast in search on its own.
• This may be enough critical mass to matter.
• Conference call at 8:30 am

I’ll try to be a bit more analytical than that, but this is still going to be quick. Assuming the deal goes through:

1. Microsoft will recombine both parts of the old FAST/alltheweb.com Therefore, Microsoft will be able to use the same technology for web and enterprise search, to the extent that such commonality makes sense.
2. I’d expect Microsoft to try to differentiate its technology via faceted/structured search. That’s a FAST strength.
3. The old FAST search-as-BI dream might become pretty appealing to Microsoft/Yahoo.
4. In a non-search point, Microsoft is strong in games and Yahoo is strong in fantasy sports. Look for some synergies.
5. There sure would be a whole lot of non-Windows technology inside Microsoft.

Basically, Microsoft is a company that’s a lot more sophisticated in its thinking about user interfaces and experiences than Yahoo is. That’s where the really interesting competitive innovation would be most likely to occur.

• He described enterprise search as consisting mainly of a search box plus faceted searching, with maybe some automated tagging as well.
• He observed that off-page factors such as PageRank don’t work nearly as well in an enterprise as they do on the Web, and that manual tagging by enterprise users falls far short of closing the gap.
• He stumbled a bit compare/constrasting search engines and “structured” DBMS.
• He basically endorsed the worldview of Ali Riaz, late of FAST, now of Attivio.

On the whole, that’s not bad. If this were an easy subject to write about, I’d have explained it a lot more clearly in the past myself. OK. Let me get off my duff and give it a whirl now.

Actually, when writing, I generally stay on my duff. At least, that’s true if I’m guessing correctly what a “duff” is. And this is not just a vaguely humorous digression — it’s also an example of why information retrieval is so hard if you only have the text itself to go by.

With that said, here are some notes on web search, enterprise search, single-site search, and database management.

• Web search has a huge problem that enterprise search doesn’t — adversarial information retrieval. Enterprise document creators generally don’t try to game search results.
• Single-site search sometimes works very well, and sometimes works laughably badly. (I often use regular Google after a site-search engine offers up a long list of bug reports and minor upgrade notes, instead of product overviews.) For an egregious example see Oracle.com, where search seems to have gotten even worse than it was a couple of years ago.
• The difference in almost every case, I think, is whether or not the site owner has done a good job of manual tagging. That would explain why changing a choice of search engine can make a site worse, if you don’t rebuild the tags; I suspect this is what happened in the Oracle case, after some acquisition.
• Full structured search technology can be the difference between “pretty good search” and an e-commerce site that really rocks, but it doesn’t work at all without a lot of manual tagging.
• When you have a revenue-generating e-commerce site, it’s easy to justify the work of manual tagging. When you have a marketing site, it can make sense. But inside an enterprise, the tagging isn’t going to happen much.
• Documents in an enterprise lie in a broad range of disparate corpuses. There are spreadsheets, PowerPoint presentations, structured-field database entries, free-text-field database entries, email, instant messages, and so on. And there also are many different corpuses of traditional text documents. A large-enterprise search engine needs tools to dial up or dial down the relevance of different corpuses, in general and to a specific search.
• Each corpus may also have its own kinds of metadata that are helpful in ranking and summarizing search results.
• As Guy Creese already pointed out in a comment to Eric’s article, security comes into play in enterprise search. I think of that as different users seeing different corpuses.
• Link analysis doesn’t work inside enterprises. Indeed, most of the documents aren’t in HTML and don’t link to each other.
• In web and enterprise search alike, you’re often satisfied by a page that doesn’t help you directly, if it points you at a better resource. But the details of the two cases differ. In enterprises, the better resource is usually a person — i.e., a document author — that you can contact directly. (This is pretty much the main part of knowledge management that actually works.)
• I wrote two years ago about a key missing ingredient to enterprise search technology — an ontology management system. Unfortunately, I’m not aware of significant progress in that direction subsequently.

Finally, I’d like to lay out a few points about the integration of text search and database management.

• Text search became an option in object/relational database management systems a little over a decade ago. The basic paradigm is that a document is stored in a single field, and a full-text index on it is integrated into the RDBMS. SQL syntax was extended to include text operators. Oracle, IBM, and Informix all did this cleanly; Microsoft found a workaround. Although all these vendors were very disappointing in the quality and performance of their search, these engines get a lot of use in application areas where it’s obviously beneficial to integrate search and relational queries.
• Search engines — both standalone and DBMS-integrated — have in recent years gained the capability to query any kind of database field. Why not? They typically can handle 1-200+ other document types as well. Anyhow, business intelligence based on that capability is part of the FAST and now Attivio stories.
• The fundamental technology of full-text indexing is similar to a number of things that go on in the relational/tabular/structured worlds, such as ADABAS’s inverted lists, or anything in the bitmapped and columnar areas. As just one example, SAP’s columnar BI Accelerator is directly built on TREX technology.
• Mark Logic makes a good case that if you’re going to integrate search and DBMS, XML may be the way to go rather than relational.

1. Jeff talked solely about FAST as adding to enterprise search, and rightly contrasted that with web search.
2. However, he deflected questions about web search with “We aren’t talking about that much detail right now” rather than with a firm “Well, we aren’t allowed to use FAST that way.”
3. Specifically, enterprise search is all about integration with SharePoint (portal).
4. Jeff said Microsoft’s current search could handle millions or maybe tens of millions of documents, but thought there was demand for FAST’s ability to handle billions.
5. He positioned FAST as an application development platform, giving an example of structured search (the actual word was “pivot”) in consumer electronics. … Well, at least he’s looking in the right direction.

In September, 2003, a nice woman wrote me to tell me she had joined Microsoft and would personally write the ranking engine for MSN search. That worked out great too.

Now Microsoft has a multi-faceted enterprise search strategy. Guy Creese seems mightily impressed. Should we, for once, be impressed too?

Frankly, yes. So far as I can tell, most traditional text search products have atrophied, including Verity before it was bought by Autonomy. And I’m skeptical about Autonomy’s Bayesian-everything approach. Oracle and Google, in different ways, consistently fail to round out their products. So if FAST’s technology can ever be fleshed out and stabilized, it indeed could be a market leader or even dominator.

But now that I’ve said that, let’s be a little careful about figuring out which enterprise search market(s) Microsoft could realistically lead or dominate, and when anybody would care. The market for enterprise search is medium-sized — $1 billionish per year, I think, depending on how one counts. This is much smaller than the market for relational database management and RDBMS-based business intelligence, for two good reasons:

• Enterprise search, in its current form, isn’t well-suited as a platform for application development.
• Enterprise search doesn’t work so well as an analytic tool either.

Basically, while search certainly saves people some time finding files, and occasionally has other uses as well, enterprise search on the whole isn’t terribly valuable or useful.

So on the one hand, we can say Microsoft is getting very well-positioned in the market for generic enterprise search, knowledge management, and the like. On the other hand, we can say “So what?” Until text analytic technology morphs into a form that can be used as a serious application platform, all text market positions will remain vulnerable and tenuous.

Stay informed! No hassle, no spam — all it takes is an email address or an RSS subscription! Get all our research — on search, text mining, DBMS, BI, and everything else — or just the text analytics part, or even just a very few notifications of our most important news.

Technorati Tags: search, enterprise search, Microsoft

I look forward to hearing why this won’t be a problem.

Technorati Tags: Microsoft, FAST, search

• They’re doing fast queries without using a lot of RAM.
• They’re doing the usual text search thing of indexing across multiple “databases,” only now it’s applied to, well, databases. (Not that there’s much new about that particular aspect. Actually, there seems to be a bit of kludge in that they export the databases to some kind of simple text files.)
• They’re doing some level of concept identification ala the text mining guys. (They don’t call it “entity extraction” because the results aren’t dumped into a database anywhere, but instead are just used on the fly.) Of course, the text mining/search convergence goes both ways.
• They bought a BI/dashboard tool and are using it both to analyze query logs and also to do normal BI/dashboard kinds of things.
• They have big references for this stuff, at least the single-web-site query aspect. Well, actually, the customer names are confidential. Oh well.

And as another example of how this wasn’t the smoothest PR month for FAST, Steve Arnold somehow got the false idea that they were getting out of true text search altogether.

Embargos were a crucial PR tactic when Regis McKenna exploited them for the original rollout of the Macintosh in 1984. But I suspect that in many cases they’ve quite outlived their usefulness. If I wait between the time I’m briefed and the time the embargo is up to write something, my thoughts about it get fuzzy. If I write something at the time and put it on ice, it may be obsolete because of what other people write in the mean time.

More and more, if something is embargoed, I wind up not writing about it at all.

EDIT: Point #4 of my post on the mismatch between relational databases and text search is pretty relevant here.