Many people think text information is important to analyze, but even so data warehouses don’t seem to wind up holding very much of it.

My working theory explaining this has two parts, both of which purport to show why text data generally doesn’t fit well into BI or data mining systems. One is that it’s just too messy and inconsistently organized. The other is that text corpuses generally don’t contain enough information.

Now, I know that these theories aren’t wholly true, for I know of counterexamples. E.g., while I’ve haven’t written it up yet, I did a call confirming that a recently published SPSS text/tabular integrated data mining story is quite real. Still, it has felt for a while as if truth lies in those directions.

Anyhow, David offered one useful number range:

If you do exhaustive extraction on a text corpus, you wind up with 10-20X as much tabular data as you had in text format in the first place. (Comparing total bytes to total bytes.)

So how big are those corpuses? I think most text mining installations usually have at least 10s of thousands of documents or verbatims to play with. Special cases aside, the upper bound seems to usually be about two orders of magnitude higher. And most text-mined documents probably tend to be short, as they commonly are just people’s reports on a single product/service experience – perhaps 1 KB or so, give or take a factor of 2-3? So we’re probably looking at 10 gigabytes of text at the low end, and a few terabytes at the high end, before applying David’s 10-20X multiplier.

Hmm – that IS enough data for respectable data warehousing …

Obviously, special cases like national intelligence or very broad-scale web surveys could run larger, as per the biggest Marklogic databases. Medline runs larger too.

David’s thoughts as to why the SaaS share isn’t even higher – as it seems to be for Clarabridge* – centered on the point that some customers want to blend internal and external data, and may not want to ship the internal part out to a SaaS provider. Besides, if it’s tabular data, I suspect Attensity isn’t the right place to ship it anyway.

*Speaking of Clarabridge, CEO Sid Banerjee recently posted a thoughtful company update in this comment thread.

When I challenged him on ease of use, David said that Attensity is readying a Microstrategy-based offering, which is obviously meant to compete with Clarabridge and any of its perceived advantages head-on.

Attensity is a text mining vendor with a lot of cool technology. Like other text mining vendors, it’s had mixed market success at best. However, sales activity suggests that Attensity recently put together it’s strongest marketing story ever, specifically in its new Voice of the Customer / Voice of the Market (VotC/VotM) focus.

Attensity Voice of the Market messaging stack

• Know what real consumers think about your products/services, how they react to your marketing, and what stories are being told about you
• The only way to listen in on actual consumer conversations. Humans can’t begin to to do this.
• Mine the Web to find out what’s being said about you; easy SaaS install
• See – here are real, usable results
• Extraction of the essence from any kind of text, as exhibited via proofs-of-concept

That’s a good story. The technology works. Prospects can see that it works. The benefits are self-evident, because the technology gives unique access to highly desirable information. (Obviously, you can’t have employees sit at their screens and try to read the whole Web on your behalf.) The cost, time to installation, and so on are attractive. All is good.

Let’s now compare that to what probably was Attensity’s prior commercial focus, warranty analysis, for products like automobiles, other vehicles, and consumer electronics. In this market, the story was something like:

Attensity warranty messaging stack

• Faster, more accurate warning of product problems
• Human reading of the warranty claims is too slow or costly
• Mine your warranty claims to see why your products break
• See – here are real, usable results
• Extraction of the essence from warranty claims, as exhibited via proofs-of-concept

That worked up to a point, which is a big part of why Attensity remained in business. But in fact, there were relatively few customers for whom the assertion “Human reading of the warranty claims is too slow or costly” was true. So relatively few sales on that basis were ever made.

Now, as a market-success-prediction tool, this kind of analysis may seem like overkill. In essence, all I’ve done is reiterate:

• Text mining has shown slow growth because too few customers had internal corpuses large enough to need it.
• If you’re mining the whole Web, however, your corpus is enormous.

But this analysis has another point. There’s a text mining industry consensus saying, more or less:

The text mining industry used to be too focused on the minutiae of technology and especially semantics, but now we’ve seen the light and are selling straight to business users who don’t really care about how the stuff works.

As with most views held by a broad consensus of smart people, that one contains a lot of truth. But it’s missing a next act. Whether or not Attensity, Clarabridge, and TEMIS get acquired soon – as most industry participants seem to expect – it seems inevitable that there will be large, technology-rich contenders in the text mining market. SAP/Business Objects/Inxight? Oracle/somebody? The enterprise search players? Dow Jones/Factiva? One way or another, there will eventually be big companies in the text mining market. Attensity (and the same goes for Clarabridge) isn’t doing much these days to position itself in advance of such an onslaught.

Anyhow, whatever you think of my market-evolution views, it sure seems as if the layered-messaging template works in this example as well.

]]> http://www.texttechnologies.com/2008/09/08/attensit-layered-messaging-marketing-model/feed/ 3 http://www.texttechnologies.com/2008/08/07/lexalytics-has-merged-with-part-of-infonics/ http://www.texttechnologies.com/2008/08/07/lexalytics-has-merged-with-part-of-infonics/#comments Thu, 07 Aug 2008 19:59:01 +0000 Curt Monash http://www.texttechnologies.com/?p=269 As reported on the Lexalytics blog, sentiment analysis specialist Lexalytics has merged with the text analytics division of Infonic to form Lexalytics Limited. The deal seems to have a screwy financial structure — which Seth Grimes made a valiant effort to decipher (I think from vacation, poor guy) — as is common when companies much too small to be public wind up trading publicly anyway.

Related links

• Lexalytics’ blog
• Introduction to Lexalytics

Actually, he didn’t. But just like every other sentiment analysis vendor I encountered at the Text Analytics Summit or spoke to beforehand, he made it clear that his tool could only handle straightforward, literal expressions of opinion. Idiom, irony, sarcasm, metaphor, et al. are beyond the current reach of the technology.

Aren’t you just thrilled that I shared that earth-shattering news with you?

Text analytics vendors participate in the same trends as other software and technology vendors. For example, relational business intelligence and data warehousing products are increasingly being sold to departmental buyers. Those buyers place particularly high value on ease of installation. And golly gee whiz, both parts of that are also true in text mining.

But beyond such general trends, I’ve identified six developments that I think could radically transform the text analytics market landscape. Indeed, they could invalidate the neat little eight-bucket categorization I laid out in the prior post. Each is highly likely to occur, although in some cases the timing remains greatly in doubt.

These six market-transforming trends are:

1. Web/enterprise/messaging integration
2. BI integration
3. Universal message retention
4. Portable personal profiles
5. Electronic health records
6. Voice command & control

I’ll explain briefly.

1. Google and Microsoft are two of the three leaders in web search. Now that Microsoft has bought FAST, they are also two of the leaders in enterprise search. They are also two of the leaders in hosted email. Ditto instant messaging. So there’s a good chance these various disciplines will converge.

2. There are a number of ways text analytics and traditional analytics can and are being integrated:

• Enterprise search and business intelligence are akin; both involve digging information out of the data you already have.
• Text mining is naturally integrated with business intelligence and/or data mining.
• There’s a trend toward using text search to dig up business intelligence documents such as specific reports, spreadsheets, etc.

To date the latter is focused on reports that already exist, rather than queries that could be run on the fly, but I hope and trust the technology will be extended over time. Natural language queries have merit anyway; I’d like to see the search box be extended in functionality to a true data-retrieval command line.

3. One of the big purchase drivers of storage, search, and clustering technology is mandates to preserve information and make it available to auditors, regulators, and/or people who want to sue you. Email in particular is changing from being ephemeral to becoming part of the permanent record. Well, if the information is being retained anyway, then maybe it’s time to see how to get useful insight from it.

Right now, a company’s overall text archives aren’t being leveraged in the same way data warehouses are. That will change.

4. For over a decade, online companies have fought to exploit the fact that users were registered with their sites or services, but not with others. Huge amounts of investment money were wasted in the dot-com bubble because people thought “registered users” was a significant metric, or that ISP subscribers could be directed to proprietary content. Enormous valuations are being assigned to Facebook and LinkedIn on similar theories today.

But as site owners and other marketers get ever more aggressive about exploiting user-specific information, users will get ever more sophisticated about controlling it. The obvious solution is for each internet user to control a sophisticated database of their contact information, presence information, actions, preferences, and writings, and to be very selective about which online services are allowed to see which portions of the data. I think that will come about some day, but I don’t know when. When it does, text analytics will be affected in a variety of interesting ways.

5. Electronic health records are almost unique in IT. What other enterprise app can you think of for which relational DBMS aren’t the default underpinning? (Intersystems’ object-oriented DBMS Cache’ has huge share in the clinical records market.) Normal tabular data, text, images, sensor output streams – health records have it all. What’s more, the health records area is coming upon some very interesting times in the area of data sharing, at least in the US.

Just as retailing went from being an IT backwater (through the mid-1980s), to a sophisticated user of database technology (1990s), to the leader of the internet revolution (rise of e-commerce), I think health care is due to take a leadership role in IT advances. And when it does, search, text mining, and voice recognition will all play important roles.

6. Most people reading this far have probably watched Star Trek. Well, what is keeping us from being able to command computers in a Star Trek fashion? Not really that much. Sure, there are some big missing pieces. We need a mapping from commands to the specific applications that would carry them out. We also need a more structured kind of analytic middle tier so that there’s something to map questions to. But those are solvable problems. And by the way – when everybody wears headphones, voice commands emanating from the next cubicle are no longer the big annoyance they would be today. Mobile/small devices only add to the business case for voice recognition advances.

When voice becomes a primary mode of human/device communication, “text” analytics will be affected in any number of ways.

Related links:

• The introductory post in this series
• 19 possible Microsoft/Yahoo synergies, many of them related to text technology convergence, e.g. between web search and enterprise search
• The compelling case for letting Google handle your enterprise email
• An old post on why BI vendors flocked to integrate with Google OneBox
• A proposal to refactor social networks
• An old post in which I outlined some of the criteria for Profiles 2.0
• Why text technologies are going to recombine (in A World of Bytes)

]]> http://www.texttechnologies.com/2008/06/19/6-trends-that-could-shake-up-the-text-analytics-market/feed/ 1 http://www.texttechnologies.com/2008/06/19/text-analytics-marketplace-competitive-landscape-trends/ http://www.texttechnologies.com/2008/06/19/text-analytics-marketplace-competitive-landscape-trends/#comments Thu, 19 Jun 2008 07:35:39 +0000 Curt Monash http://www.texttechnologies.com/?p=249 As I see it, there are eight distinct market areas that each depend heavily on linguistic technology. Five are off-shoots of what used to be called “information retrieval”:

1. Web search

2. Public-facing site search

3. Enterprise search and knowledge management

4. Custom publishing

5. Text mining and extraction

Three are more standalone:

6. Spam filtering

7. Voice recognition

8. Machine translation

This list comes from a talk I gave Monday at the Text Analytics Summit called The Text Analytics Marketplace: Competitive landscape and trends. In half an hour, I covered the first five areas (in Sue Feldman’s word, at a “gallop”). The slide deck has been uploaded to the link below. I plan to break out the material from the talk into a series of blog posts over the next few (or perhaps not-so-few) weeks.

Slides:

• The Text Analytics Marketplace: Competitive landscape and trends

Other posts based on those slides:

• Three specialized markets for text analytics (based on Slide 2)
• 6 trends that could shake up the text analytics market (based on Slide 19)
• Why search technologies are going to recombine (in A World of Bytes, based on Slide 19)
  

That is, of course, hugely more than any of the independents (e.g. Attensity and Clarabridge) have. And it’s focused on marketing-oriented apps — especially Voice of the Customer — just as those vendors are. Even so, they report rarely seeing SPSS, and SPSS agrees with that assessment.

The obvious explanation — which Olivier does not dispute — is that Text Mining For Clementine sales are focused on Clementine data mining users. But that raises an interesting follow-up — how much data mining are these users really doing on text data? Attensity and Clarabridge customers do little true data mining, but Olivier asserts that SPSS customers do quite a bit — predictive modeling, real-time scoring, and the whole enchilada.

By the way, Olivier actually no longer runs SPSS’ text mining business. He’s moved to Chicago as VP of Corporate Development, focused on acquisitions. Coincidentally, he has a glum view of the prospects for independent text analytics companies, and believes the best course for them is to be acquired.

Beyond that, Alessandro denies all the rumors that have grown out of TEMIS being hard to reach recently. He reports that pharma is still TEMIS’s big market, but stresses that this covers a range of apps, from research to Voice of the Market. I do get the sense that TEMIS’s sentiment extraction capabilities are less sophisticated than some of the other vendors’ — but the other vendors I’m thinking of are pretty focused on English, SPSS aside. If you need sentiment analysis in non-English languages — e.g., French or Italian — TEMIS should definitely be on your vendor shortlist.