Notwithstanding the manifest virtues of centralization, there are numerous reasons you might want data marts,  in the tabular and document worlds alike.  For example:

• Price/performance. Your main/central data manager might be too expensive to support additional large specialized databases. Or different databases and applications might have sufficiently different profiles so as to get great price/performance from different kinds of data managers. This is particularly prevalent in the relational world, where each of column stores, sequentially-oriented row stores, and random I/O-oriented row stores have compelling use cases.
• Different SLAs (Service-Level Agreements). Similarly, different applications may have very different requirements for uptime, response time, and the like. (In the relational world, think of operational data stores.)
• Different security requirements. Different subsets of the data may need different levels of security. This is particularly prevalent in the document world, where security problems are not as well-solved as in the tabular arena, and where it’s common for a search engine to index across different corpuses with radically different levels of sensitivity.
• Integrated application and user interfaces. In the relational world, there’s a pretty clean separation between data management and interface logic; most serious business intelligence tools can talk to most DBMS. The document world is quite different. Some search engines bundle, for example, various kinds of faceted or parameterized search interfaces. What’s more, in public-facing search, a major differentiator is the facilities that the product offers for skewing search results.
• Different text applications require different thesauruses or taxonomy management systems. Ideally, those should all be integrated — but the requisite technology still doesn’t exist.

Bottom line: Text data marts, much like relational data marts, are almost surely here to stay.

Related link

• The future of data marts

The weakness of Andy’s argument is illustrated by the one and only scenario he posits in support of his crusade:

The result is that a blog post added to a brand new site may well have just broken the story about the capture of Bin Laden (we wish!)–and a link to said post may have been Tweeted and re-tweeted–but Google won’t discover or index that post until it finds a “followed” link. Likely from a trusted site in Google’s index and likely hours, if not days, after it was first shared on Twitter.

Helloooo — if I post something here, it is indexed at least in Google blog search immediately. (As in, within a minute or so.) Ping, crawl, pop — there it is.  The only remotely valid version of Andy’s complaint is that It might take some hours for Google’s main index to update — but even there there’s a News listing at the top.  This simply is not a problem.

Now, I think it would be personally great for me if all the links to my sites from Wikipedia and Twitter and the comment threads of major blogs pointed back with “link juice.” On the other hand, even with NoFollow out there, my sites come up high in Google’s rankings for all sorts of keywords, driving a lot of their readership.  I imagine the same is true for most other sites containing fairly unique content that people find interesting enough to link to.

So other than making it harder to engage in deceptive SEO, I fail to see what problems NoFollow is causing.

My opinions about the applicability of semantic technology include:

• The big bucks in web search are for “transactional” web search, and semantics isn’t the issue there. (Slides 3-4)
• When UIs finally go beyond the simple search box — e.g. to clusters/facets or to voice — semantics should have a role to play. (Slide 5)
• Public-facing site search depends — more than any other area of text analytics — on hand-tagging. (Slide 7)
• “Enterprise” search that searches specialized external databases could benefit from semantic technologies. (Slide
• True enterprise search could benefit from semantic technologies in multiple ways, but has other problems as well. (Slides 10-11)
• Semantics — specifically extraction — is central to custom publishing. (Slide 12 — upon review I regret using the word “sophisticated”)
• Semantics is central to text mining. (Slide 18)
• Semantics could play a big role in all sorts of exciting future developments. (Slide 19)

So what would your list be like?

The theory can’t be denied. What’s more, bad actors have other motives to pollute the Web. For example, if they plant favorable automated comments about their own products or unfavorable about the competition’s, Voice of the Customer/Market applications will naturally be confused. And if automated reputation-checkers get more prominent, there will be a major incentive to game them, just as there has been for Google’s PageRank. So VOTC/VOTM market research tools could polluted as a side effect.

Similarly, if somebody wants to test your e-commerce site by throwing a ton of searches at it, your search logs will lose value.

But disinformation of competitors for the sake of disinformation? Or, as the article suggestions, vandalism/extortion? Off the top of my head, I’m not thinking of a serious near-term threat scenario.

Half a century later, the industry is finally poised to deliver on that vision.

Let me begin by explaining what I mean by some of that list (taken from Slide 2), starting from the bottom.

• Machine translation is a small business, with small specialized vendors. Lernout & Hauspie attempted to combine it with voice recognition in a complex financial play, but that collapsed in a miasma of stock fraud. The remnants turned into what became Nuance Communications.
• Nuance is a roll-up of most of the important independent voice recognition vendors. So far voice recognition has worked best in two areas: “Hands-free” computer use/dictation, and IVR (interactive voice response). While both are important, neither is exactly a mainstream enterprise computer software business. So voice recognition is not closely integrated with the other market segments.
• “Natural language processing” other than voice recognition isn’t much of a business at this time (with apologies to Progress EasyAsk). It doesn’t make the list at all.
• Spam filtering is obviously a major business, whether or not it is getting combined into more general security and/or messaging product suites. Antispam vendors actually perform a lot of machine learning, much like text miners do. But the types of rules they wind up with are quite different. And their hardest problems aren’t linguistic ones, usually, as the spammers have gone beyond text to, e.g., words depicted in graphical images. Besides, even where linguistics are involved, it’s a very different problem to identify words used by bad guys trying to spoof you (and the rest of the world) than it is to understand your particular users.

Why and to what extent I see the other five as separate markets was explained in connection with the subsequent 17 slides.

1. Web search

2. Public-facing site search

3. Enterprise search and knowledge management

4. Custom publishing

5. Text mining and extraction

Three are more standalone:

6. Spam filtering

7. Voice recognition

8. Machine translation

This list comes from a talk I gave Monday at the Text Analytics Summit called The Text Analytics Marketplace: Competitive landscape and trends. In half an hour, I covered the first five areas (in Sue Feldman’s word, at a “gallop”). The slide deck has been uploaded to the link below. I plan to break out the material from the talk into a series of blog posts over the next few (or perhaps not-so-few) weeks.

Slides:

• The Text Analytics Marketplace: Competitive landscape and trends

Other posts based on those slides:

• Three specialized markets for text analytics (based on Slide 2)
• 6 trends that could shake up the text analytics market (based on Slide 19)
• Why search technologies are going to recombine (in A World of Bytes, based on Slide 19)
  

“There are several underlying important developments over the last decade or so:

• Incorporating user feedback to refine search results, usually indirectly rather than explicitly, making results better through machine learning. [Amazon.com is the most-often cited example of this with it’s “if you like A, you’ll also like B.”] [CAM] Technically, that’s not a search example, but the general point is correct even so.
• Assessments based on usage or referral. This is what makes Google so useful and popular. This approach gives higher rankings if other web sites point to a target or if that target gets a lot of hits.
• Various approaches to using taxonomies. The better applications use taxonomies as a navigation guide but don’t force it or require administrators to implement it. Vivisimo.com is an example of interesting, automated clustering approach. [CAM] “Faceted search” seems to be the buzzword here. It’s a big part of what I call “structured search.” But taxonomy use is probably more trivial in search than it is in, say, text mining.
• Better handling of phrases. Google automatically parses phrases and deals with search terms as phrases. This now seems natural but in the AltaVista days, you couldn’t tell a Venetian blind from a blind Venetian [example courtesy of Prof. George Miller, Princeton Univ. - too good not to cite].
• Context-sensitive search is now an emerging trend. Systems track what users have previously searched for and infer interest in the same domain to refine search result. So if you look for “line” and a system knows you’ve just looked for “tacklebox,” then it infers you mean “fishing line.” Or if you search for bagels and the system knows you are in 20009, it tells you that you can buy them at Comet Liquors (which happens to sell bagels). [CAM] That happens a lot with ad serving. But I’m not convinced it hit actual search until Google’s personal search kicked off, and that was quite recent.

“More generally in natural language processing, the statistical and linguistic approaches are converging in a new way: use massive amounts of data (i.e. the Web) to get statistical answers to deep linguistic questions, like “How do we figure out what the most likely referent is for the pronoun ‘they’?” Or “How do we determine the correct sense for ambiguous words?” These things aren’t in search engines yet, but you can expect to see more “intelligent” features coming out of this approach.

“Looking at this list, you can see that the conceptual changes (breakthroughs?), with the exception of better phrase handling, are primarily focused around Web searches. When dealing with one-of-a-kind document collections behind the corporate firewall, many of these developments turn out not to add much to older approaches. So, at least for enterprise search, I too remain partial to some of the older products you mention, though I am disappointed that most of the old-time vendors have not updated their approaches beyond adding taxonomy support.” [CAM] Yep, web search and enterprise search are very different things.

The original blog post did have one error — Sharon’s PhD isn’t in Computational Linguistics, but rather Slavic Linguistics, as I recently noted in my post about text analytics careers for humanities majors.

• Expert System now has 145 employees.
• 2 of the employees are in the US (plus at least one more full-time equivalent on a contract basis). Brooke believes the US operation will eventually be the biggest part of the company.
• Expert System has sold its market intelligence SaaS offering to two global auto manufacturers. Competitors were Nielsen BuzzMetrics, somebody whose name sounded like “flexilytics” (I presume that would be Lexalytics Edit: But see Lexalytics CEO Jeff Catlin’s comment below), and somebody whose named sounded like “Truecast” (I haven’t yet guessed who that is).
• If I understood correctly, Expert System acquired that product by picking up Brooke’s tiny company Acuity Software. Acuity was/is a user of Expert System’s technology, having replaced Coveo’s with it so as to get better semantics.
• Brooke is optimistic about Expert System’s prospects in the intelligence market. New semantic networks in Arabic and English (joining one Expert System already had in Italian) are a big part of the reason. Brooke says the intelligence community is now actively interested in technology that’s been validated by the commercial market, on the theory it’s apt to be more complete than research/government-only products. Expert System is also working on a semantic network in another undisclosed Middle Eastern language; Brooke stoically refrained from confirming the blindingly obvious guess that this would be Farsi.
• Expert System’s third effort in the US market, coming soon, will be a semantic ad platform.

Once again, however, I made it through an Expert System briefing without gaining a real understanding of its technology. I gather they’re proud of their in-memory data structure for their semantic network, but I haven’t a clue (beyond the obvious guesses) as to what that data structure is. Similarly, Brooke said that a distinguishing feature of Expert Systems semantic network is that words have lots of attributes, which are the same thing as categories, and supplied a list of the 11 top-level categories: Objects, animals, plants, people, concepts, places, time, natural phenomena, state, quantity, group. But it’s easy to come up with a lot of things that don’t seem to fit that list very well (especially events, such as numerous different word-senses of “strike”), so absent further elucidation I didn’t find that particularly instructive either.

We’re designing a new blog theme — the current one is just an emergency stopgap — that will (among myriad more important virtues) be more SEO-friendly. I’ll be curious to see whether that makes much actual difference from a search ranking standpoint.