Denis Pombriant obviously attended the same Marc Benioff session I did. Dion Hinchcliffe blogged the whole story in considerable detail.

It’s a cool story, and worthy of attention. But I’d like to step back and remind us that there are numerous different ways to use social technology in the enterprise, which probably shouldn’t be confused with each other. And then I’d like to discuss one area of social technology that’s relatively new to me: integration between social and operational applications.

Suppose we split up social technology use cases by saying it can help you:

• Communicate and collaborate internally …
• … and also with small groups of outsiders, such as your supply chain.
• Observe, listen to, and interact with consumers (and the world at large).

The biggest buzz, of course, is around social technology that reaches out to the buying public or world at large. You can use social technology to:

• Observe and listen to consumers — i.e., classic Voice of the Customer/Voice of the Market text analytics.
• Publish to consumers, influencers, etc., via blogging, broadcast-oriented Twitter, and other social media, or go even further and …
• … communicate with consumers interactively, whether through loosely-structured interaction (e.g. Twitter), or in the more structured ways that Attensity and others provide.

I support all that, and indeed participate ferociously myself. But for now, let’s move on.

On the internal collaboration/communication side, I’d say:

• Any communication tool useful for communicating with the public may be valuable internally as well — portals, blogs, Twitter-imitators, and so on.
• Pure email “push” may not always be the best tool for point-to-point internal communication.
• Text analytics on internal communication can have a variety of uses, e.g:
  • Compliance (yet another privacy intrusion, but sometimes a legitimate one).
  • Internal expert-finding. (In principle, this is the traditional genuine benefit of elaborate “knowledge management” implementations, but without the burdens of traditional knowledge management. In practice, that didn’t work out so great for Tacit Software.)
  • Project management.

That all gives plenty of scope for useful adoption, on both the email-replacement and text-analytic sides. But again, let’s keep going.

The relatively new to me — notwithstanding the “portals” link above — part of the social technology story is integration between social and operational applications. While at Dreamforce, I talked with two manufacturing application SaaS vendors — Kenandy and Rootstock Software. In both cases I asked “So what are you doing that’s an advance over where MRP was 20 years ago?” In both cases the main answer was “Now users can use social technology to track and communicate about particular orders or issues.”

*MRP stood for “Material Requirements Planning” and then “Manufacturing Resources Planning”, and is essentially the  forerunner of ERP. By “Kenandy” I specifically mean Kenandy’s founder — ASK Computer Systems founder and thus MRP legend Sandy Kurtzig.

Good point. Of course, it can be generalized; one can communicate and collaborate around almost any kind of business process. I’ve mentioned this before in analytic contexts; it’s an important concept on the monitoring-oriented side of business intelligence and — if Oliver Ratzesberger is to be believed — in investigative analytics as well. But the operational side may actually be more important.

Some things one does in the business world actually involve using one’s body, from manufacturing products to repairing power stations to standing in a store and serving customers. Most of the rest fits into one or more of three buckets:

• Creating (a product, a marketing plan, a marketing document, a compensation plan, a program for internal use, an analytic insight, …)
• Relating (to an employee, a sales prospect, a reporter, …)
• Participating in a fairly routine business process (data entry, accounting, mortgage approval, parts ordering, …)

And why can’t we just automate those routine business processes away? Because there’s so often a need for manual intervention. And when there’s a need for manual intervention, there’s usually also an element of communicating with other people. This is almost always true in cases of trouble-shooting or exception-handling (an order is late, a system is down, the automated result violates common sense). It may be present in other cases as well (the new account calls for a personal thank you note, the food order needs to be annotated with special requests). General email is commonly an awkward medium for these communications; automated messages are worse. Newer social technologies, however, have the potential to do much better.

So what do you think? Have I drunk too much Kool-Aid, or is this stuff for real?

• Understanding the opinions of customers, prospects, or other groups. This can be based on any combination of documents the user organization controls (email, surveys, warranty reports, call center logs, etc.) — in which case — or public-domain documents such as blogs, forum posts, and tweets. The former is usually called Voice of the Customer (VotC), while the latter is Voice of the Market (VotM).
• Detecting and identifying problems. This can happen across many domains — VotC, VotM, diagnosing equipment malfunctions, identifying bad guys (from terrorists to fraudsters), or even getting early warnings of infectious disease outbreaks.
• Aiding text search, custom publishing, and other electronic document-shuffling use cases, often via document augmentation.

For several years, I’ve been distressed at the lack of progress in text analytics or, as it used to be called, text mining. Yes, the rise of sentiment analysis has been impressive, and higher volumes of text data are being processed than were before. But otherwise, there’s been a lot of the same old, same old. Most actual deployed applications of text analytics or text mining go something like this:

• A bunch of documents are analyzed to ascertain the ideas expressed in them.
• A count is made as to how many times each idea turns up.
• The application user notices any surprisingly large numbers, and as result of noticing pays attention to the corresponding ideas.

Often, it seems desirable to integrate text analytics with business intelligence and/or predictive analytics tools that operate on tabular data is. Even so, such integration is most commonly weak or nonexistent. Apart from the usual reasons for silos of automation, I blame this lack on a mismatch in precision, among other reasons. A 500% increase in mentions of a subject could be simple coincidence, or the result of a single identifiable press article. In comparison, a 5% increase in a conventional business metric might be much more important.

But in fairness, the text analytics innovation picture hasn’t been quite as bleak as what I’ve been painting so far. While standalone, passively-reported text analytics is indeed the baseline, there are some interesting exceptions. For example:

• I once confirmed that SPSS customer Cablecom‘s statistical models for churn and the like absolutely included text data; Cablecom even assigned different weights to the same apparent level of emotion depending on whether the text was in German, French, or Italian. Vertica recently told me of a Vertica/Hadoop customer doing something similar, except for the multilingual aspect. And the end of a 2008 SAS-based paper makes similar claims.
• There long* have been some examples of fact extraction that don’t really fit into my three buckets above. For example, researchers mine collections of articles to try to determine biochemical or biological pathways that would not be apparent from examining single research studies alone.
• It also has long* been the case that some bad-guy-finding applications — especially in the anti-terrorism area — used text analytics to populate state-of-the-art graph-oriented data analysis tools.

*When it comes to text analytics, “long” means “at least for the past several years.”

In more recent examples:

• Greenplum built a document recommender for law firms that does hard-core statistical analysis to determine which .1% of a document set lawyers might actually want to see, and which then learns from users’ feedback after they respond to initial result sets.
• Information extracted from investment news gets included into automated trading algorithms. This was unusual technology a couple of years ago, but is more common today.
• After a series of mergers, Attensity now uses marketing-oriented text analytics in at least three different ways:
  • Attensity text analytics feeds marketing dashboards just as it always did.
  • Attensity text analytics triggers alerts, as I wish dashboards and business intelligence tools more often did, the false positives problem notwithstanding.
  • Attensity text analytics triggers concrete workflows, for example routing specific social media hits for priority response.
  • And in one example that did not actually get into production, a very large social networking company correlated word usage (e.g., choice among different synonyms) against user characteristics such as age and gender.

Finally there are some applications that, while fitting the standard template, just strike me as getting to unusually sophisticated levels of analysis. For example, Vertica told me of another Vertica/Hadoop case where VotM document analysis is carried out to the level of observing which order brand names appear in, and adjusting that for whether or not it was just an alphabetical list.

I suspect text analytics is about to become more interesting again.

Related links

• The enabling technology for text/tabular data integration has existed for years.
• In 2006, I listed major application areas for data mining/predictive analytics. It overlaps pretty closely with the similar list for text mining/text analytics.
• Before being acquired by IBM, SPSS boasted a rather large text mining user base.

*Or if it does, it’s VERY well-hidden. I looked at the home page and site map alike.

I wanted to refresh my memory as to Mark Logic’s history of working with specific text mining vendors, beyond what’s on the official partner page. No luck. Normally when site search is inadequate, one goes to Google. But that’s problematic too. Marklogic.com pages come up pretty low on Google’s search results, suggesting that:

1. Mark Logic doesn’t put a lot of effort into SEO (or else doesn’t do it very well).
2. One can’t be confident all the site’s significant pages are findable by Google.

Looking to other companies’ sites for clues isn’t conclusive either. E.g., Clearforest lists Mark Logic as a partner, but Mark Logic doesn’t return the compliment. (If memory serves, Mark Logic and Clearforest have worked together both on national security deals and custom publishing deals — but don’t hold me to that.)

When it comes to making its own information conveniently available, Mark Logic is quite the unshod cobbler.

The theory can’t be denied. What’s more, bad actors have other motives to pollute the Web. For example, if they plant favorable automated comments about their own products or unfavorable about the competition’s, Voice of the Customer/Market applications will naturally be confused. And if automated reputation-checkers get more prominent, there will be a major incentive to game them, just as there has been for Google’s PageRank. So VOTC/VOTM market research tools could polluted as a side effect.

Similarly, if somebody wants to test your e-commerce site by throwing a ton of searches at it, your search logs will lose value.

But disinformation of competitors for the sake of disinformation? Or, as the article suggestions, vandalism/extortion? Off the top of my head, I’m not thinking of a serious near-term threat scenario.

David’s thoughts as to why the SaaS share isn’t even higher – as it seems to be for Clarabridge* – centered on the point that some customers want to blend internal and external data, and may not want to ship the internal part out to a SaaS provider. Besides, if it’s tabular data, I suspect Attensity isn’t the right place to ship it anyway.

*Speaking of Clarabridge, CEO Sid Banerjee recently posted a thoughtful company update in this comment thread.

When I challenged him on ease of use, David said that Attensity is readying a Microstrategy-based offering, which is obviously meant to compete with Clarabridge and any of its perceived advantages head-on.

In its full conception, the Attivio Intelligence Engine is something like Endeca + RDBMS + search engine + XML store + cool extra features. And all with seamless, lightweight, integrated installation and administration. That’s the goal, anyway. At this point, naturally, each individual piece is far from complete. For example:

• Sufficient SQL support to handle most BI tools is still a matter for future releases — apparently in 2009, although Attivio is one of those agile companies for which pinning down product releases is somewhat difficult.
• The same goes some basic GUI features (such as most non-programmatic search tuning).
• ACID compliance is not a high priority for Attivio. I actually think it should be higher, just because it’s increasingly become an “OK, we don’t have to worry about THAT” checkmark item.

Even in its early days, Attivio has had some nice-sounding customer successes. There are 8 paying Attivio customers, including 2 > $1 million deals, one half-millionish dollar deal, and 1 large OEM. 3 represent actual deployments, with the rest in development. More sales are on the way, as are permissions to disclose customer names that people will actually recognize. Customer application stories Andrew told me about include:

• A web-business parameterized, adjustable-weight search that’s starting with tabular data and only getting to free-text later.
• An enterprise that’s using Attivio for content management, enterprise search, public-facing search, and data warehousing.
• Something big/mysterious/classified, with large document volumes.
• Something to do with compliance, about which Andrew was going to forward a lot more detail that evening (Hint, hint).

Since the major RDBMS (Oracle, Microsoft SQL Server, DB2) all have text search and XML subsystems, they can in principle do everything Attivio does on the back end, and with a lot more features and maturity. The same would go for Marklogic. Performance and overhead might be different matters, however; Andrew certainly believes so.

Except that Lucene is included on the search side, I haven’t actually figured out how Attivio stores data. The fact that SQL features are being added incrementally suggests Attivio is rolling its own relational database capability, but how it’s organized I don’t really know.

Highlights included:

• Information management is central to e-discovery.
• In particular, auditability (my word) is central, if you want electronic documents to hold up as evidence in court.
• Search is good enough, but it’s not the biggest issue in e-discovery.
• E-mail archiving has reached the tipping point, and is increasingly a must-have, largely for its e-discovery benefits.

Attensity is a text mining vendor with a lot of cool technology. Like other text mining vendors, it’s had mixed market success at best. However, sales activity suggests that Attensity recently put together it’s strongest marketing story ever, specifically in its new Voice of the Customer / Voice of the Market (VotC/VotM) focus.

Attensity Voice of the Market messaging stack

• Know what real consumers think about your products/services, how they react to your marketing, and what stories are being told about you
• The only way to listen in on actual consumer conversations. Humans can’t begin to to do this.
• Mine the Web to find out what’s being said about you; easy SaaS install
• See – here are real, usable results
• Extraction of the essence from any kind of text, as exhibited via proofs-of-concept

That’s a good story. The technology works. Prospects can see that it works. The benefits are self-evident, because the technology gives unique access to highly desirable information. (Obviously, you can’t have employees sit at their screens and try to read the whole Web on your behalf.) The cost, time to installation, and so on are attractive. All is good.

Let’s now compare that to what probably was Attensity’s prior commercial focus, warranty analysis, for products like automobiles, other vehicles, and consumer electronics. In this market, the story was something like:

Attensity warranty messaging stack

• Faster, more accurate warning of product problems
• Human reading of the warranty claims is too slow or costly
• Mine your warranty claims to see why your products break
• See – here are real, usable results
• Extraction of the essence from warranty claims, as exhibited via proofs-of-concept

That worked up to a point, which is a big part of why Attensity remained in business. But in fact, there were relatively few customers for whom the assertion “Human reading of the warranty claims is too slow or costly” was true. So relatively few sales on that basis were ever made.

Now, as a market-success-prediction tool, this kind of analysis may seem like overkill. In essence, all I’ve done is reiterate:

• Text mining has shown slow growth because too few customers had internal corpuses large enough to need it.
• If you’re mining the whole Web, however, your corpus is enormous.

But this analysis has another point. There’s a text mining industry consensus saying, more or less:

The text mining industry used to be too focused on the minutiae of technology and especially semantics, but now we’ve seen the light and are selling straight to business users who don’t really care about how the stuff works.

As with most views held by a broad consensus of smart people, that one contains a lot of truth. But it’s missing a next act. Whether or not Attensity, Clarabridge, and TEMIS get acquired soon – as most industry participants seem to expect – it seems inevitable that there will be large, technology-rich contenders in the text mining market. SAP/Business Objects/Inxight? Oracle/somebody? The enterprise search players? Dow Jones/Factiva? One way or another, there will eventually be big companies in the text mining market. Attensity (and the same goes for Clarabridge) isn’t doing much these days to position itself in advance of such an onslaught.

Anyhow, whatever you think of my market-evolution views, it sure seems as if the layered-messaging template works in this example as well.

]]> http://www.texttechnologies.com/2008/09/08/attensit-layered-messaging-marketing-model/feed/ 3 http://www.texttechnologies.com/2008/09/01/how-good-does-e-discovery-search-need-to-be/ http://www.texttechnologies.com/2008/09/01/how-good-does-e-discovery-search-need-to-be/#comments Mon, 01 Sep 2008 04:44:58 +0000 Curt Monash http://www.texttechnologies.com/?p=277 Two years ago, CEO Mike Lynch of Autonomy tried to persuade me that Autonomy was and would remain dominant in the e-discovery search market because:

• The essence of the buying decision was that enterprises wanted to fulfill obligations to make their information available in a way that would would satisfy the courts.
• Autonomy had some high-profile traction (e.g., the Enron case) that made it the default decision, and hence in particular a choice that met the requirement.

Recently, I ran that theory by David Ferris, whose firm Ferris Research has long been a/the leading small analyst firm covering e-mail and related technologies. He wasn’t buying. David believes courts are getting more sophisticated in their understanding of search technology. Even more to the point, David cited several other buying motivations that would lead enterprises to want best-available rather than just-good-enough e-discovery search technology, such as:

• Enterprises want to know what information is available to be discovered against them.
• Enterprises want to discover the information that will best aid their legal defense.
• If they’re archiving the material for one purpose (e-discovery) anyway, enterprises want to get the most possible value out of it for other purposes while they’re at it.