Notwithstanding the manifest virtues of centralization, there are numerous reasons you might want data marts,  in the tabular and document worlds alike.  For example:

• Price/performance. Your main/central data manager might be too expensive to support additional large specialized databases. Or different databases and applications might have sufficiently different profiles so as to get great price/performance from different kinds of data managers. This is particularly prevalent in the relational world, where each of column stores, sequentially-oriented row stores, and random I/O-oriented row stores have compelling use cases.
• Different SLAs (Service-Level Agreements). Similarly, different applications may have very different requirements for uptime, response time, and the like. (In the relational world, think of operational data stores.)
• Different security requirements. Different subsets of the data may need different levels of security. This is particularly prevalent in the document world, where security problems are not as well-solved as in the tabular arena, and where it’s common for a search engine to index across different corpuses with radically different levels of sensitivity.
• Integrated application and user interfaces. In the relational world, there’s a pretty clean separation between data management and interface logic; most serious business intelligence tools can talk to most DBMS. The document world is quite different. Some search engines bundle, for example, various kinds of faceted or parameterized search interfaces. What’s more, in public-facing search, a major differentiator is the facilities that the product offers for skewing search results.
• Different text applications require different thesauruses or taxonomy management systems. Ideally, those should all be integrated — but the requisite technology still doesn’t exist.

Bottom line: Text data marts, much like relational data marts, are almost surely here to stay.

Related link

• The future of data marts

My opinions about the applicability of semantic technology include:

• The big bucks in web search are for “transactional” web search, and semantics isn’t the issue there. (Slides 3-4)
• When UIs finally go beyond the simple search box — e.g. to clusters/facets or to voice — semantics should have a role to play. (Slide 5)
• Public-facing site search depends — more than any other area of text analytics — on hand-tagging. (Slide 7)
• “Enterprise” search that searches specialized external databases could benefit from semantic technologies. (Slide
• True enterprise search could benefit from semantic technologies in multiple ways, but has other problems as well. (Slides 10-11)
• Semantics — specifically extraction — is central to custom publishing. (Slide 12 — upon review I regret using the word “sophisticated”)
• Semantics is central to text mining. (Slide 18)
• Semantics could play a big role in all sorts of exciting future developments. (Slide 19)

So what would your list be like?

“There are several underlying important developments over the last decade or so:

• Incorporating user feedback to refine search results, usually indirectly rather than explicitly, making results better through machine learning. [Amazon.com is the most-often cited example of this with it’s “if you like A, you’ll also like B.”] [CAM] Technically, that’s not a search example, but the general point is correct even so.
• Assessments based on usage or referral. This is what makes Google so useful and popular. This approach gives higher rankings if other web sites point to a target or if that target gets a lot of hits.
• Various approaches to using taxonomies. The better applications use taxonomies as a navigation guide but don’t force it or require administrators to implement it. Vivisimo.com is an example of interesting, automated clustering approach. [CAM] “Faceted search” seems to be the buzzword here. It’s a big part of what I call “structured search.” But taxonomy use is probably more trivial in search than it is in, say, text mining.
• Better handling of phrases. Google automatically parses phrases and deals with search terms as phrases. This now seems natural but in the AltaVista days, you couldn’t tell a Venetian blind from a blind Venetian [example courtesy of Prof. George Miller, Princeton Univ. - too good not to cite].
• Context-sensitive search is now an emerging trend. Systems track what users have previously searched for and infer interest in the same domain to refine search result. So if you look for “line” and a system knows you’ve just looked for “tacklebox,” then it infers you mean “fishing line.” Or if you search for bagels and the system knows you are in 20009, it tells you that you can buy them at Comet Liquors (which happens to sell bagels). [CAM] That happens a lot with ad serving. But I’m not convinced it hit actual search until Google’s personal search kicked off, and that was quite recent.

“More generally in natural language processing, the statistical and linguistic approaches are converging in a new way: use massive amounts of data (i.e. the Web) to get statistical answers to deep linguistic questions, like “How do we figure out what the most likely referent is for the pronoun ‘they’?” Or “How do we determine the correct sense for ambiguous words?” These things aren’t in search engines yet, but you can expect to see more “intelligent” features coming out of this approach.

“Looking at this list, you can see that the conceptual changes (breakthroughs?), with the exception of better phrase handling, are primarily focused around Web searches. When dealing with one-of-a-kind document collections behind the corporate firewall, many of these developments turn out not to add much to older approaches. So, at least for enterprise search, I too remain partial to some of the older products you mention, though I am disappointed that most of the old-time vendors have not updated their approaches beyond adding taxonomy support.” [CAM] Yep, web search and enterprise search are very different things.

The original blog post did have one error — Sharon’s PhD isn’t in Computational Linguistics, but rather Slavic Linguistics, as I recently noted in my post about text analytics careers for humanities majors.

• Expert System now has 145 employees.
• 2 of the employees are in the US (plus at least one more full-time equivalent on a contract basis). Brooke believes the US operation will eventually be the biggest part of the company.
• Expert System has sold its market intelligence SaaS offering to two global auto manufacturers. Competitors were Nielsen BuzzMetrics, somebody whose name sounded like “flexilytics” (I presume that would be Lexalytics Edit: But see Lexalytics CEO Jeff Catlin’s comment below), and somebody whose named sounded like “Truecast” (I haven’t yet guessed who that is).
• If I understood correctly, Expert System acquired that product by picking up Brooke’s tiny company Acuity Software. Acuity was/is a user of Expert System’s technology, having replaced Coveo’s with it so as to get better semantics.
• Brooke is optimistic about Expert System’s prospects in the intelligence market. New semantic networks in Arabic and English (joining one Expert System already had in Italian) are a big part of the reason. Brooke says the intelligence community is now actively interested in technology that’s been validated by the commercial market, on the theory it’s apt to be more complete than research/government-only products. Expert System is also working on a semantic network in another undisclosed Middle Eastern language; Brooke stoically refrained from confirming the blindingly obvious guess that this would be Farsi.
• Expert System’s third effort in the US market, coming soon, will be a semantic ad platform.

Once again, however, I made it through an Expert System briefing without gaining a real understanding of its technology. I gather they’re proud of their in-memory data structure for their semantic network, but I haven’t a clue (beyond the obvious guesses) as to what that data structure is. Similarly, Brooke said that a distinguishing feature of Expert Systems semantic network is that words have lots of attributes, which are the same thing as categories, and supplied a list of the 11 top-level categories: Objects, animals, plants, people, concepts, places, time, natural phenomena, state, quantity, group. But it’s easy to come up with a lot of things that don’t seem to fit that list very well (especially events, such as numerous different word-senses of “strike”), so absent further elucidation I didn’t find that particularly instructive either.

As of last year, Expert System also has English-language products, and a couple of minor OEM sales in the US (for mobile search and semantic web applications). German- and Arabic-language products are in beta test. The company says that its market focus going forward is national security – surely the reason for the Arabic – and competitive intelligence. It envisions selling through partners such as system integrators, although I think that makes more sense for the government market than it does vis-a-vis civilian companies. In February the company is introducing a market intelligence product focused on sentiment analysis.

Expert System is a bit of a throwback, in that it talks lovingly of the semantic network that informs its products. This semantic net was assembled in the usual way – start with WordNet, add a huge number of proper nouns, license a bunch of domain-specific dictionaries, and handcraft further as individual customers require it. In English the whole thing has 300,000 nodes and 1.2 million relationships.

Expert System insists that there’s a secret sauce in how the semantic net is organized, to optimize performance. But I haven’t gotten the slightest hint of what that magic data structure is — despite having asked more than once – and so have to reserve judgment on that part.

On the search side, Expert System sounds fairly rich in terms of deciding relevancy, going beyond Term Frequency/Inverse Document Frequency + synonyms. Terms are also assigned importance by their grammatical roles, such as whether they’re sentence subjects, sentence objects, in paragraph topic sentences, and so on. Of course, the whole concept of exploiting grammatical structure is a bit old-school. Specifically, it presupposes you’re looking at decently grammatical documents in the first place, which can be a dubious assumption in the era of “im in ur cuzztom3r bazz eatin ur r3venue$z.”

The most interesting application Expert System told me about was one for Pirelli Tire, scanning the web for prices Pirelli products were sold at, to detect gray market activity. Web-crawling for text analytics and price detection are both major activities – e.g., QL2 is active in both – but this is the first I’ve heard of the areas being so tightly integrated.

• Like Attensity, Clarabridge practices exhaustive extraction.* That is, they do linguistics against documents, extract all sorts of entities and relationships among the entities from each document, and dump the results into a relational database.
• Unlike Attensity, which uses a simple normalized relational schema, Clarabridge dumps the extracted data into a star schema. (The Clarabridge folks are from Microstrategy, which – surely not coincidentally – also favors star schemas.)
• For now, the linguistic part of the analysis is within a sentence, or else based on proximity, or (this sounded minor) based on the whole document. But actual anaphora resolution is coming soon.
• The other big thing that goes into Clarabridge’s star schema is a category hierarchy, which has two aspects. One is categories fixed in advance. When I asked how many, CTO Justin Langseth cited an example range of 10-400. I.e., it varies widely. In principle, these are established by line-of-business folks at Clarabridge customers, but I’d venture to guess that professional services play a significant role as well.
• The other kind of categories – subcategories to the first group – are created automagically at data load time via document clustering. Indeed, they’re called “clusters.” These are available for drilldown via business intelligence tools.
• Obviously it is good practice to have dashboards and scheduled reports depend only on the fixed categories, not the clusters.

*I should note that Clarabridge understandably bristles a bit at my use of this Attensity-introduced term to describe what they do too. If Clarabridge wants to start talking about, say, “comprehensive extraction, I’ll consider adopting that term as well. But for now I’m going with what’s most widely used.

Want to continue getting great research about text mining, data warehouse appliances, and other hot analytics-related topics? Then subscribe to our comprehensive (if not exhaustive) feed, by RSS/Atom or e-mail! We recommend taking the integrated feed for all our blogs, but blog-specific ones are also easily available.

Technorati Tags: Clarabridge, text mining, exhaustive extraction

That said, here’s the essence of Baynote’s story, as best I could figure it out.

• Baynote’s secret sauce is a set of 20+ behavioral metrics to identify whether, if somebody clicks on a page, they are SATISFIED with the content.
• Based on that, Baynote provides a “content recommendation” engine. (For now, the distinction between “content” and “web page” is not important, but the concepts are in my opinion diverging over time.) This is manifested in two forms (a typical installation uses both). One is just a list of recommendations. The other is in a search engine – “social search” with an “implicit folksonomy” — and its results pages. Both sit on web pages as boxes/widgets.
• Baynote’s first markets were online support and eMarketing. The company is now rolling out eCommerce as well. I didn’t get clarity about what was different in the nature of the recommendations, if anything, that underlies any small separation between these apps. (Baynote was clear about saying that the differences were indeed small.)
• The whole thing is SaaS, built on a LAMP stack. MySQL 4.something seems to suffice, which makes sense given that Baynote’s system is not handling any significant transactions directly. That said, I didn’t push to understand what it means for a search engine to be built on MySQL. This wasn’t the kind of conversation in which one could elicit substantive detail.
• Baynote claims that a sample size of as few as 7-10 visitors liking a particular piece of content suffices to provide a good basis for predicting who else will like it. I’m not in a position to assess the credibility or, more to the point, limitations of this claim.
• Baynote has the philosophy that they try to watch a user’s behavior on a site and map that to a “context.” I like that approach.
• The company cites tested stats of 20% net lift (revenue increase), with 50% of sales being touched by its recommendations. Those numbers don’t sound terribly impressive, perhaps unless they’re truly additive to those provided by, say, Endeca, which is an announced partner.

In broad outline, Mercado and InQuira each combine three basic search approaches:

• Generic text indexing.
• Augmentation via an ontology.
• A rules engine that helps the site owner determine which results and responses are shown under various circumstances.

Of the two, InQuira seems to have the more sophisticated ontology. Indeed, the not-wholly-absurd claim is that InQuira does natural-language processing (NLP). Both vendors incorporate user information in deciding which search results to show, in ways that may be harbingers of what generic search engines like Google and Yahoo will do down the road.

InQuira has all three standard levels of an ontology – generic, vertical, and customer-specific. They readily admit to being an instantiation of Monash’s Second Law of Commercial Semantics: Where there’s an ontology, there’s consulting. Indeed, professional services are almost 40% of InQuira’s revenue (which was almost $20 million last year). Beyond the ontology, they incorporate surfing and profile evidence to disambiguate users’ interests.

Let’s pause a moment to reflect on structured search and parts of speech. Obviously, when somebody’s shopping, it’s very important to interpret nouns. But adjectives are important too. If a customer expresses interest in a “gold” car, the website had better tell her about which “Metallic Champagne” vehicles are available. And on a retail site it’s rather important to know the difference between a “dress shirt” and a “shirt dress,” a test About.com’s ad-serving software currently fails.

The major differentiating feature of InQuira’s NLP/search technology is to take this further, and also think about verbs. More precisely, the focus is on “intents,” sometimes called “intent categories” instead – i.e., actions the customer is trying to undertake. These are defined in a kind of rules engine, which is separate from the semantic net used to represent the noun/adjective ontology.

Given that they’re defined by rulesets, there are a fair number of these intents. Back in June, 2005, InQuira told me they had packaged the linguistic knowledge for 100 “intents” for cell phone service companies, and were covering 72-72% of total inquiries that way. The most popular intents accounted for 10-12% of inquiries each.

Mercado doesn’t do “intents,” and I don’t think the ontology is as sophisticated either, but otherwise its search story is a lot like InQuira’s. Both companies, for example, offer rules-engine capabilities for displaying various page elements – i.e., portlets — alongside the actual search results (e.g., for upsell). Both also let web site owners tweak search results too, according to what they want to sell, what they think the customer is most likely to buy, or to provide some sort of near matches when the exact search isn’t a good enough match to actual inventory.

Mercado argues that its rules-based technology is particularly powerful, because of a capability they call RBT, for result(s)-based triggers. The idea is that rules can fire based on any characteristics of the search results themselves. Particularly important inputs seem to be the size and estimated precision of the raw result set.

I’ve hinted above that generic search can be beaten by more specialized technologies. I absolutely believe that, with one caveat. Whatever happens under the covers, word-based interaction with computers may well always have a generic interface – search boxes today, voice increasingly in the future. It’s what happens after the initial disambiguation that will be specialized according to – well, according to both the user’s and the server owner’s intent.

But there’s a second problem as well — mapping Wikipedia subjects to DMOZ categories. How’s that supposed to work? For most Wikipedia subjects, there’s no obvious single match in the DMOZ ontology. And it’s more than just a matter of the categories not existing yet; I don’t think they can exist until the DMOZ hierarchy becomes much more interconnected.

I think it would be great if ODP/DMOZ were enhanced to A. Accomodate public input and B. Have a multifaceted ontology. But until there’s a DMOZ 2.0, I don’t see how Joost’s idea could work.

Technorati Tags: Wikipedia, DMOZ