Notwithstanding the manifest virtues of centralization, there are numerous reasons you might want data marts,  in the tabular and document worlds alike.  For example:

• Price/performance. Your main/central data manager might be too expensive to support additional large specialized databases. Or different databases and applications might have sufficiently different profiles so as to get great price/performance from different kinds of data managers. This is particularly prevalent in the relational world, where each of column stores, sequentially-oriented row stores, and random I/O-oriented row stores have compelling use cases.
• Different SLAs (Service-Level Agreements). Similarly, different applications may have very different requirements for uptime, response time, and the like. (In the relational world, think of operational data stores.)
• Different security requirements. Different subsets of the data may need different levels of security. This is particularly prevalent in the document world, where security problems are not as well-solved as in the tabular arena, and where it’s common for a search engine to index across different corpuses with radically different levels of sensitivity.
• Integrated application and user interfaces. In the relational world, there’s a pretty clean separation between data management and interface logic; most serious business intelligence tools can talk to most DBMS. The document world is quite different. Some search engines bundle, for example, various kinds of faceted or parameterized search interfaces. What’s more, in public-facing search, a major differentiator is the facilities that the product offers for skewing search results.
• Different text applications require different thesauruses or taxonomy management systems. Ideally, those should all be integrated — but the requisite technology still doesn’t exist.

Bottom line: Text data marts, much like relational data marts, are almost surely here to stay.

Related link

• The future of data marts

Natural language computer interfaces were introduced commercially about 15 years ago*.  They failed miserably.

*I.e., the early 1980s

For example, Artificial Intelligence Corporation’s Intellect was a natural language DBMS query/reporting/charting tool.  It was actually a pretty good product.  But it’s infamous among industry insiders as the product for which IBM, in one of its first software licensing deals, got about 1700 trial installations — and less than a 1% sales close rate.  Even its successor, Linguistic Technologies’ English Wizard*, doesn’t seem to be attracting many customers, despite consistently good product reviews.

*These days (i.e., in 2009) it’s owned by Progress and called EasyAsk. It still doesn’t seem to be selling well.

Another example was HAL, the natural language command interface to 1-2-3.  HAL is the product that first made Bill Gross (subsequently the founder of Knowledge Adventure and idealab!) and his brother Larry famous.  However, it achieved no success*, and was quickly dropped from Lotus’ product line.

*I loved the product personally. But I was sadly alone.

In retrospect, it’s obvious why natural language interfaces failed. First of all, they offered little advantage over the forms-and-menus paradigm that dominated enterprise computing in both the online-character-based and client-server-GUI eras.  If you couldn’t meet an application need with forms and menus, you couldn’t meet it with natural language either.

Even worse, NL actually had a couple of clear disadvantages versus traditional interfaces.  First of all, it required (ick!) typing, often more typing than the forms and menus did.  Second, forms and menus tell the user exactly what he can do. Natural language, however, lets him give orders the computer doesn’t know how to follow.  This is inefficient, not to mention frustrating.

However, even in 1983, it was obvious that the typing objection would go away some day, because of speech recognition — once desktop computers reached 100 MIPs or so.  (Effective keyboard-replacement speech recognition – as opposed to true natural language understanding — is mainly a matter of processing power.)  15 years later, standard PCs exceed 100 MIPs (assuming that 1 MIPs = a couple of megahertz for these purposes), and speech recognition is indeed getting practical.

In fact, as become increasingly evident recently, speech recognition is now a hot technology.  Bill Gates has been talking it up for a couple of years.  Increasingly, the press has swung to believing him … And my parents just bought a PC with two speech recognition products on it.

That said, speech recognition is as misunderstood (no pun intended) as most artificial intelligence technologies.  Yes, it beats typing, in a number of circumstances:

• On the telephone (duh!)
• “Busy hands” and/or “busy eyes” applications and locales (doctors‘ offices, trading floors, warehouses, etc. – and, some day in the future, your kitchen and car)
• People simply reluctant to type (e.g., anybody with sufficient wrist or back problems, and many males over the age of 45)

But before our computers talk back and forth with us in the voice of Majel Barrett Roddenberry, applications are going to have to add several important elements required for truly functional natural-language  interfaces:

• Intuitively clear names for everything on (or just behind) the screen
• Application-specific disambiguation logic

For most practical purposes, the latter requirement equates to

• A new generation of document selection technology

THE RULE OF NAMES

According to legend, knowing something’s name gives you power over it.  When that “something” is a button or menu choice on a speech-enabled computer, the legend is literally true.  But when a feature doesn’t have an obvious name, you can’t easily invoke it.

When applications consisted mainly of forms and menus, this was rarely a problem.  Everything had a clear role and label.  But web pages are less organized.  Hyperlinks can be scattered all over the place, with little rhyme or reason.

Frankly, I don’t think this is a hard problem to solve.  It wouldn’t take a lot of XML to divide the page into clear regions, so that commands like “Show me article #3″ (on a search results list) could be interpreted in the obvious way.  But it does take at least some discipline; random web pages will not necessarily be easy to “talk” to.

CYBERNETIC LISTENING SKILLS

The bigger challenge is to make sure that the application can respond in some useful way, no matter what command it’s given. This is even more difficult than it was 15 years ago, because of the radical increase in “casual” computer usage.  In the old days, we could assume the user had some clear business reason for using the application, and if necessary that s/he had time to be trained (even if people rarely sat still for as much training as they really needed).  Therefore, we could at least assume that the users had at least a general idea of what the application did, and hence of which commands the computer could obey.  From an NL standpoint, we could assume that what they actually “said” (which in those days meant “typed”) was at least reasonably close to what they were “supposed” to say.

Now, however, some of the most important applications are internet e-commerce and portals, competing and begging for the user’s attention.  The user is there strictly on a voluntary basis, and if he doesn’t get immediate gratification, he‘s gone, history, hasta la bye-bye.  Site-specific training isn’t even a consideration. And even if somebody did actually take a class on “How to use Excite,” the knowledge would be obsolete in six months.  So applications, if they are to have natural language interfaces that please and respond to users, have to be able to respond pretty much to any command.

Ideally, voice-enabled systems would be like the computers on Star Trek, which can return information from vast archives, brew a pot of Earl Grey tea, play three parts of a quartet, create self-aware life forms, or answer questions like “Computer, what is the nature of the universe?”  More realistically, they should be able, for example, to respond to a command like “Tell me about flights to Miami” by automatically giving the user a travel-reservation application or web page, and entering Miami in the appropriate form field.

If one thinks about the complications in such a system, it becomes clear that there are only two possible ways an application system can be designed to respond meaningfully to an enormous range of reasonable possible requests.

1. It can do the equivalent of saying “I’m sorry, I didn’t understand that,” “I’m sorry, I can’t do that,” and so on.

2. It can interpret many commands as text-search strings, and return appropriate results.

The first strategy – application-specific disambiguation logic, clear responses to “errors,” etc. — is absolutely necessary.  No software is perfectly intelligent; the user will have to be asked for disambiguation help from time to time (just as clerks today ask customers to repeat their requests!). I’m not going to go into much detail about how that works because, frankly, it’s a tricky thing to get right.  Users hate unnecessary disambiguation steps. They also hate the incorrect responses that result from ambiguity, and do tolerate being asked for help when it’s truly needed.  In short, whatever you build the first time around will probably be wrong.  So build something fast; then run, don’t walk, to the nearest usability lab, find out how you screwed up, and redo your system until you get it right.

I’m convinced that the second strategy — heavy reliance on text search technology — is a requirement as well. Just try to name a major web site that doesn’t use text search.  True, text search has gotten a bad rap recently, mainly because a whole generation of search engines didn’t really work.  But it will stage a comeback.

Related links

• My December, 2007 survey of speech recognition technology
• Star Trek fun

]]> http://www.texttechnologies.com/2009/05/30/men-are-from-earth-computers-are-from-vulcan/feed/ 2 http://www.texttechnologies.com/2009/05/29/google-wave-finally-a-microsoft-killer/ http://www.texttechnologies.com/2009/05/29/google-wave-finally-a-microsoft-killer/#comments Fri, 29 May 2009 09:49:24 +0000 Curt Monash http://www.texttechnologies.com/?p=330 Google held a superbly-received preview of a new technology called Google Wave, which promises to “reinvent communication.” In simplest terms, Google Wave is a software platform that:

• Offers the possibility to improve upon a broad range of communication, collaboration, and/or text-based product categories, such as:
  • Search
  • Word processing
  • E-mail
  • Instant messaging
  • Microblogging
  • Blogging
  • Mini-portals (Facebook-style)
  • Mini-portals (Sharepoint-style)
• In particular, allows these applications to be both much more integrated and interactive than they now are.
• Will have open developer APIs.
• WIll be open-sourced.

If this all works out, Google Wave could play merry hell with Microsoft Outlook, Microsoft Word, Microsoft Exchange, Microsoft SharePoint, and more.

I suspect it will.

And by the way, there’s a cool “natural language” angle as well.

For starters, here are some basic links:

• Google has naturally set up a home page for the Google Wave project.
• Featured on that page but also separately available is an 80-minute video introducing Google Wave.
• Techcrunch has two highly detailed posts on Google Wave, one summarizing what’s in the main Google Wave video and one reporting on a Google Wave Q&A.

Here are some reasons I think Google Wave could actually live up to its promise:

• The email problem Google Wave purports to solve is real and critical. The email paradigm assumes linear conversations, and what actually happens is that they branch. Google Wave’s message-board-like paradigm is simply better, and more flexible (e.g., not limited to a single enterprise!) than Microsoft Exchange or Lotus Notes.
• The instant messaging problems Google Wave purports to solve are also major. Instant messaging is slow, tedious, disjointed, and ephemeral. Fully integrating IM with email solves most of those problems. And Google Wave’s UI interactivity solves most of the rest.
• Twitter needs to be integrated with other forms of communication. What’s more, Twitter’s functionality needs to be drastically extended. Google Wave is the best hope I know of to meet those needs.  Enterprise Twitter is just a special case of that.
• Workgroups (enterprise or otherwise) need light-weight mini-portals that can be created on the fly by non-technical users, to ease collaboration. Microsoft SharePoint, SAP Rooms, et al. don’t really meet that need.  Google Wave could.
• In particular, collaboration on documents, presentations and so on needs to be more cloud-based and generally easier than is the case in Microsoft Office. Google Wave has the potential to provide that.
• Google + open source is a potentially potent combination, especially versus Microsoft.

One note: Google of course needs to improve the reliability and customer service of its cloud-based offerings to make a huge dent in Microsoft’s market. But even with its flaws Google has already been a good alternative for a while.

As for the “natural language” angle: At the 44:30 mark of the main Google Wave video is a demo of some cool, very grammar-sensitive spell-checking technology. Google’s spell-checking technology is further discussed in a separate, short video.  The basic idea is that Google uses its vast library of web pages — and email and chat? — not just to model intended word usage but also kinds of mis-spelling behavior as well.

Since the early 1980s, I’ve thought that natural language interfaces — spoken or otherwise — would someday win.  While this versatility isn’t natural lanaguage per se, it still in my opinion is evidence in favor of that belief.

• A Google acquisition of Twitter
• Some other kind of relationship built around real-time search

I have three initial thoughts on this:

1. Clearly, in Google’s mission to “organize all the world’s information,” there are several web areas it isn’t yet doing well in, and one of those is microblogs. What’s more, much as in the case of YouTube, it’s hard to see how Google would do that organizing any time soon unless it owned or otherwise was in bed with the leading platform for that kind of content — i.e., Twitter.

2. The YouTube example is apt in another way as well — it’s not clear where the monetization would come from. Google famously doesn’t make much advertising revenue from YouTube. And Twitter is even worse as an advertising platform; sticking ads into the tweetstream would quickly drive users elsewhere, and any other advertising scheme would likely fail because of the broad variety of interfaces — such as various mobile phones — Twitterers use to get at the service.

3. I’ve been suggesting all along that Twitter needs radical user experience enhancements. But when has Google ever made made user experience enhancements to a service? Its core search engine always looks pretty much the same. Ditto GMail. Ditto Blogger. Ditto YouTube.

*Care and feeding of celebrity tweeters. Celebrity tweeting is actually a subject I’ve written and even been interviewed about several times.

As of this writing, the full list is:

Job Title	Location	Department
Operations Engineer	Technical Operations
Operations Engineer – Search	Technical Operations
Product Manager	Product
Product Manager – Platform (Consumer Internet)	Product
Product Manager – Search	Product
UI Designer/Developer	Product
User Experience Designer	Product
Communications Associate (Contract to FTE)	G&A
Founder Associate	G&A
Software Engineer – Business Intelligence	Engineering
Software Engineer – Front-End	Engineering
Software Engineer – Ruby on Rails	Engineering
Software Engineer – Search	Engineering
Software Engineer – Systems	Engineering
Software Engineer – Systems Spam	Engineering
Director, Strategic Partnerships	Business Development
Japan Country Manager	Business Development

Related link

• Twitter’s analytics job opening

The weakness of Andy’s argument is illustrated by the one and only scenario he posits in support of his crusade:

The result is that a blog post added to a brand new site may well have just broken the story about the capture of Bin Laden (we wish!)–and a link to said post may have been Tweeted and re-tweeted–but Google won’t discover or index that post until it finds a “followed” link. Likely from a trusted site in Google’s index and likely hours, if not days, after it was first shared on Twitter.

Helloooo — if I post something here, it is indexed at least in Google blog search immediately. (As in, within a minute or so.) Ping, crawl, pop — there it is.  The only remotely valid version of Andy’s complaint is that It might take some hours for Google’s main index to update — but even there there’s a News listing at the top.  This simply is not a problem.

Now, I think it would be personally great for me if all the links to my sites from Wikipedia and Twitter and the comment threads of major blogs pointed back with “link juice.” On the other hand, even with NoFollow out there, my sites come up high in Google’s rankings for all sorts of keywords, driving a lot of their readership.  I imagine the same is true for most other sites containing fairly unique content that people find interesting enough to link to.

So other than making it harder to engage in deceptive SEO, I fail to see what problems NoFollow is causing.

My opinions about the applicability of semantic technology include:

• The big bucks in web search are for “transactional” web search, and semantics isn’t the issue there. (Slides 3-4)
• When UIs finally go beyond the simple search box — e.g. to clusters/facets or to voice — semantics should have a role to play. (Slide 5)
• Public-facing site search depends — more than any other area of text analytics — on hand-tagging. (Slide 7)
• “Enterprise” search that searches specialized external databases could benefit from semantic technologies. (Slide
• True enterprise search could benefit from semantic technologies in multiple ways, but has other problems as well. (Slides 10-11)
• Semantics — specifically extraction — is central to custom publishing. (Slide 12 — upon review I regret using the word “sophisticated”)
• Semantics is central to text mining. (Slide 18)
• Semantics could play a big role in all sorts of exciting future developments. (Slide 19)

So what would your list be like?

Before you get too excited about hiring opportunities, however — it’s not obvious how many victims are in the core search business in any capacity, and it’s certain not clear whether anybody is being let go in areas like search algorithm research.

*Or if it does, it’s VERY well-hidden. I looked at the home page and site map alike.

I wanted to refresh my memory as to Mark Logic’s history of working with specific text mining vendors, beyond what’s on the official partner page. No luck. Normally when site search is inadequate, one goes to Google. But that’s problematic too. Marklogic.com pages come up pretty low on Google’s search results, suggesting that:

1. Mark Logic doesn’t put a lot of effort into SEO (or else doesn’t do it very well).
2. One can’t be confident all the site’s significant pages are findable by Google.

Looking to other companies’ sites for clues isn’t conclusive either. E.g., Clearforest lists Mark Logic as a partner, but Mark Logic doesn’t return the compliment. (If memory serves, Mark Logic and Clearforest have worked together both on national security deals and custom publishing deals — but don’t hold me to that.)

When it comes to making its own information conveniently available, Mark Logic is quite the unshod cobbler.