Denis Pombriant obviously attended the same Marc Benioff session I did. Dion Hinchcliffe blogged the whole story in considerable detail.

It’s a cool story, and worthy of attention. But I’d like to step back and remind us that there are numerous different ways to use social technology in the enterprise, which probably shouldn’t be confused with each other. And then I’d like to discuss one area of social technology that’s relatively new to me: integration between social and operational applications.

Suppose we split up social technology use cases by saying it can help you:

• Communicate and collaborate internally …
• … and also with small groups of outsiders, such as your supply chain.
• Observe, listen to, and interact with consumers (and the world at large).

The biggest buzz, of course, is around social technology that reaches out to the buying public or world at large. You can use social technology to:

• Observe and listen to consumers — i.e., classic Voice of the Customer/Voice of the Market text analytics.
• Publish to consumers, influencers, etc., via blogging, broadcast-oriented Twitter, and other social media, or go even further and …
• … communicate with consumers interactively, whether through loosely-structured interaction (e.g. Twitter), or in the more structured ways that Attensity and others provide.

I support all that, and indeed participate ferociously myself. But for now, let’s move on.

On the internal collaboration/communication side, I’d say:

• Any communication tool useful for communicating with the public may be valuable internally as well — portals, blogs, Twitter-imitators, and so on.
• Pure email “push” may not always be the best tool for point-to-point internal communication.
• Text analytics on internal communication can have a variety of uses, e.g:
  • Compliance (yet another privacy intrusion, but sometimes a legitimate one).
  • Internal expert-finding. (In principle, this is the traditional genuine benefit of elaborate “knowledge management” implementations, but without the burdens of traditional knowledge management. In practice, that didn’t work out so great for Tacit Software.)
  • Project management.

That all gives plenty of scope for useful adoption, on both the email-replacement and text-analytic sides. But again, let’s keep going.

The relatively new to me — notwithstanding the “portals” link above — part of the social technology story is integration between social and operational applications. While at Dreamforce, I talked with two manufacturing application SaaS vendors — Kenandy and Rootstock Software. In both cases I asked “So what are you doing that’s an advance over where MRP was 20 years ago?” In both cases the main answer was “Now users can use social technology to track and communicate about particular orders or issues.”

*MRP stood for “Material Requirements Planning” and then “Manufacturing Resources Planning”, and is essentially the  forerunner of ERP. By “Kenandy” I specifically mean Kenandy’s founder — ASK Computer Systems founder and thus MRP legend Sandy Kurtzig.

Good point. Of course, it can be generalized; one can communicate and collaborate around almost any kind of business process. I’ve mentioned this before in analytic contexts; it’s an important concept on the monitoring-oriented side of business intelligence and — if Oliver Ratzesberger is to be believed — in investigative analytics as well. But the operational side may actually be more important.

Some things one does in the business world actually involve using one’s body, from manufacturing products to repairing power stations to standing in a store and serving customers. Most of the rest fits into one or more of three buckets:

• Creating (a product, a marketing plan, a marketing document, a compensation plan, a program for internal use, an analytic insight, …)
• Relating (to an employee, a sales prospect, a reporter, …)
• Participating in a fairly routine business process (data entry, accounting, mortgage approval, parts ordering, …)

And why can’t we just automate those routine business processes away? Because there’s so often a need for manual intervention. And when there’s a need for manual intervention, there’s usually also an element of communicating with other people. This is almost always true in cases of trouble-shooting or exception-handling (an order is late, a system is down, the automated result violates common sense). It may be present in other cases as well (the new account calls for a personal thank you note, the food order needs to be annotated with special requests). General email is commonly an awkward medium for these communications; automated messages are worse. Newer social technologies, however, have the potential to do much better.

So what do you think? Have I drunk too much Kool-Aid, or is this stuff for real?

• Understanding the opinions of customers, prospects, or other groups. This can be based on any combination of documents the user organization controls (email, surveys, warranty reports, call center logs, etc.) — in which case — or public-domain documents such as blogs, forum posts, and tweets. The former is usually called Voice of the Customer (VotC), while the latter is Voice of the Market (VotM).
• Detecting and identifying problems. This can happen across many domains — VotC, VotM, diagnosing equipment malfunctions, identifying bad guys (from terrorists to fraudsters), or even getting early warnings of infectious disease outbreaks.
• Aiding text search, custom publishing, and other electronic document-shuffling use cases, often via document augmentation.

For several years, I’ve been distressed at the lack of progress in text analytics or, as it used to be called, text mining. Yes, the rise of sentiment analysis has been impressive, and higher volumes of text data are being processed than were before. But otherwise, there’s been a lot of the same old, same old. Most actual deployed applications of text analytics or text mining go something like this:

• A bunch of documents are analyzed to ascertain the ideas expressed in them.
• A count is made as to how many times each idea turns up.
• The application user notices any surprisingly large numbers, and as result of noticing pays attention to the corresponding ideas.

Often, it seems desirable to integrate text analytics with business intelligence and/or predictive analytics tools that operate on tabular data is. Even so, such integration is most commonly weak or nonexistent. Apart from the usual reasons for silos of automation, I blame this lack on a mismatch in precision, among other reasons. A 500% increase in mentions of a subject could be simple coincidence, or the result of a single identifiable press article. In comparison, a 5% increase in a conventional business metric might be much more important.

But in fairness, the text analytics innovation picture hasn’t been quite as bleak as what I’ve been painting so far. While standalone, passively-reported text analytics is indeed the baseline, there are some interesting exceptions. For example:

• I once confirmed that SPSS customer Cablecom‘s statistical models for churn and the like absolutely included text data; Cablecom even assigned different weights to the same apparent level of emotion depending on whether the text was in German, French, or Italian. Vertica recently told me of a Vertica/Hadoop customer doing something similar, except for the multilingual aspect. And the end of a 2008 SAS-based paper makes similar claims.
• There long* have been some examples of fact extraction that don’t really fit into my three buckets above. For example, researchers mine collections of articles to try to determine biochemical or biological pathways that would not be apparent from examining single research studies alone.
• It also has long* been the case that some bad-guy-finding applications — especially in the anti-terrorism area — used text analytics to populate state-of-the-art graph-oriented data analysis tools.

*When it comes to text analytics, “long” means “at least for the past several years.”

In more recent examples:

• Greenplum built a document recommender for law firms that does hard-core statistical analysis to determine which .1% of a document set lawyers might actually want to see, and which then learns from users’ feedback after they respond to initial result sets.
• Information extracted from investment news gets included into automated trading algorithms. This was unusual technology a couple of years ago, but is more common today.
• After a series of mergers, Attensity now uses marketing-oriented text analytics in at least three different ways:
  • Attensity text analytics feeds marketing dashboards just as it always did.
  • Attensity text analytics triggers alerts, as I wish dashboards and business intelligence tools more often did, the false positives problem notwithstanding.
  • Attensity text analytics triggers concrete workflows, for example routing specific social media hits for priority response.
  • And in one example that did not actually get into production, a very large social networking company correlated word usage (e.g., choice among different synonyms) against user characteristics such as age and gender.

Finally there are some applications that, while fitting the standard template, just strike me as getting to unusually sophisticated levels of analysis. For example, Vertica told me of another Vertica/Hadoop case where VotM document analysis is carried out to the level of observing which order brand names appear in, and adjusting that for whether or not it was just an alphabetical list.

I suspect text analytics is about to become more interesting again.

Related links

• The enabling technology for text/tabular data integration has existed for years.
• In 2006, I listed major application areas for data mining/predictive analytics. It overlaps pretty closely with the similar list for text mining/text analytics.
• Before being acquired by IBM, SPSS boasted a rather large text mining user base.

The theory can’t be denied. What’s more, bad actors have other motives to pollute the Web. For example, if they plant favorable automated comments about their own products or unfavorable about the competition’s, Voice of the Customer/Market applications will naturally be confused. And if automated reputation-checkers get more prominent, there will be a major incentive to game them, just as there has been for Google’s PageRank. So VOTC/VOTM market research tools could polluted as a side effect.

Similarly, if somebody wants to test your e-commerce site by throwing a ton of searches at it, your search logs will lose value.

But disinformation of competitors for the sake of disinformation? Or, as the article suggestions, vandalism/extortion? Off the top of my head, I’m not thinking of a serious near-term threat scenario.

David’s thoughts as to why the SaaS share isn’t even higher – as it seems to be for Clarabridge* – centered on the point that some customers want to blend internal and external data, and may not want to ship the internal part out to a SaaS provider. Besides, if it’s tabular data, I suspect Attensity isn’t the right place to ship it anyway.

*Speaking of Clarabridge, CEO Sid Banerjee recently posted a thoughtful company update in this comment thread.

When I challenged him on ease of use, David said that Attensity is readying a Microstrategy-based offering, which is obviously meant to compete with Clarabridge and any of its perceived advantages head-on.

Attensity is a text mining vendor with a lot of cool technology. Like other text mining vendors, it’s had mixed market success at best. However, sales activity suggests that Attensity recently put together it’s strongest marketing story ever, specifically in its new Voice of the Customer / Voice of the Market (VotC/VotM) focus.

Attensity Voice of the Market messaging stack

• Know what real consumers think about your products/services, how they react to your marketing, and what stories are being told about you
• The only way to listen in on actual consumer conversations. Humans can’t begin to to do this.
• Mine the Web to find out what’s being said about you; easy SaaS install
• See – here are real, usable results
• Extraction of the essence from any kind of text, as exhibited via proofs-of-concept

That’s a good story. The technology works. Prospects can see that it works. The benefits are self-evident, because the technology gives unique access to highly desirable information. (Obviously, you can’t have employees sit at their screens and try to read the whole Web on your behalf.) The cost, time to installation, and so on are attractive. All is good.

Let’s now compare that to what probably was Attensity’s prior commercial focus, warranty analysis, for products like automobiles, other vehicles, and consumer electronics. In this market, the story was something like:

Attensity warranty messaging stack

• Faster, more accurate warning of product problems
• Human reading of the warranty claims is too slow or costly
• Mine your warranty claims to see why your products break
• See – here are real, usable results
• Extraction of the essence from warranty claims, as exhibited via proofs-of-concept

That worked up to a point, which is a big part of why Attensity remained in business. But in fact, there were relatively few customers for whom the assertion “Human reading of the warranty claims is too slow or costly” was true. So relatively few sales on that basis were ever made.

Now, as a market-success-prediction tool, this kind of analysis may seem like overkill. In essence, all I’ve done is reiterate:

• Text mining has shown slow growth because too few customers had internal corpuses large enough to need it.
• If you’re mining the whole Web, however, your corpus is enormous.

But this analysis has another point. There’s a text mining industry consensus saying, more or less:

The text mining industry used to be too focused on the minutiae of technology and especially semantics, but now we’ve seen the light and are selling straight to business users who don’t really care about how the stuff works.

As with most views held by a broad consensus of smart people, that one contains a lot of truth. But it’s missing a next act. Whether or not Attensity, Clarabridge, and TEMIS get acquired soon – as most industry participants seem to expect – it seems inevitable that there will be large, technology-rich contenders in the text mining market. SAP/Business Objects/Inxight? Oracle/somebody? The enterprise search players? Dow Jones/Factiva? One way or another, there will eventually be big companies in the text mining market. Attensity (and the same goes for Clarabridge) isn’t doing much these days to position itself in advance of such an onslaught.

Anyhow, whatever you think of my market-evolution views, it sure seems as if the layered-messaging template works in this example as well.

]]> http://www.texttechnologies.com/2008/09/08/attensit-layered-messaging-marketing-model/feed/ 3 http://www.texttechnologies.com/2008/06/17/spss-update/ http://www.texttechnologies.com/2008/06/17/spss-update/#comments Tue, 17 Jun 2008 06:51:45 +0000 Curt Monash http://www.texttechnologies.com/?p=245 I emailed a bit with Olivier Jouve last week, and chatted with him at the Text Analytics Summit yesterday. He cited a figure of 2400 SPSS text mining users (unique user organizations). The majority of these are for a low-cost, desktop-based surveys product. But when I pressed him, he eventually gave a 500-1000 figure for actual Text Mining For Clementine users.

That is, of course, hugely more than any of the independents (e.g. Attensity and Clarabridge) have. And it’s focused on marketing-oriented apps — especially Voice of the Customer — just as those vendors are. Even so, they report rarely seeing SPSS, and SPSS agrees with that assessment.

The obvious explanation — which Olivier does not dispute — is that Text Mining For Clementine sales are focused on Clementine data mining users. But that raises an interesting follow-up — how much data mining are these users really doing on text data? Attensity and Clarabridge customers do little true data mining, but Olivier asserts that SPSS customers do quite a bit — predictive modeling, real-time scoring, and the whole enchilada.

By the way, Olivier actually no longer runs SPSS’ text mining business. He’s moved to Chicago as VP of Corporate Development, focused on acquisitions. Coincidentally, he has a glum view of the prospects for independent text analytics companies, and believes the best course for them is to be acquired.

*In reality, the bar opened early, and I took a Sam Adams into the roundtable room.

Now, it’s possible this reflected a certain vendor bias. Most of the show’s attendees are either vendors or users whose attendance the vendors pay for, and many of the rest are prospects the vendors encourage to come. The show’s program is also heavily influenced by what the vendors think is important. Still, this is confirming evidence that the text mining industry’s center of gravity has shifted emphatically to the CRM area.

• Voice of the Customer/Market applications require less linguistic sophistication than other text mining applications.
  
• Hence, Voice of the Customer/Market apps are easier to get running than other text mining applications, which he conjectures is a big part of the reason for burgeoning sales.

I’m guessing most text mining vendors would agree with those views, although they might not agree with his elaborations, which include:

• Attensity’s knowledge extraction technology is more sophisticated than Clarabridge’s or most other competitors’.
• In particular, Clarabridge’s extraction is little more than bag-of-words.
• There’s a good match between companies he thinks have less-sophisticated extraction (e.g., Clarabridge, SAS, SPSS) and companies whose text mining sales are heavily concentrated in Voice of the Customer/Market applications.

So the question arises: Just how much linguistic sophistication is needed in these market-trend-oriented text mining applications?

I actually got onto this subject not just because of what David said, but also via a conversation an hour earlier with Brooke Aker of Expert System, who proposed linguistic sophistication as a key reason for beating the competition (which, however, didn’t include Attensity or Clarabridge) at two accounts. The point Brooke was stressing is that it’s important to be able to extract multiple facts or indicators of sentiment from the same sentence. E.g., “I just had a crummy Chevy, but at least the seats were comfortable” is both a negative indicator about Chevrolet and a positive indicator about Chevrolet’s seats. Attensity captures both of those too, and I think Clarabridge would as well. (If you do comprehensive/ exhaustive extraction, you extract — well, you should extract comprehensively.)

Anyhow, my first-best answer to the question I posed is:

• Sentiment analysis is hard, at least in venues where you have to deal with slang, metaphor, or irony (the real biggie). The more sophisticated, the better.
• Otherwise, the linguistics of customer/marketing applications is pretty straightforward. Just put together the right list of wacky synonyms, and you’re good to go.

But what do you think?

1. Attensity is the older company than Clarabridge, and is good at more things. Is Clarabridge really good at everything you want them to be?

2. In particular, Attensity has more overall sophistication at linguistic extraction. Do any of the differences matter to you?

3. Both companies are working hard on ease of use, for multiple kinds of user (business user tweaking linguistic rules, IT user, etc.). Whose approach and feature set do you like better?

4. Usually, buying one of these products involves some professional services. Whose organization do you like better?

5. Attensity’s default database schema for its exhaustive extraction is pretty flat and normalized, as befits a happy Teradata partner. Clarabridge’s is more of a star schema, as befits a bunch of ex-Microstrategy guys. Either can be straightforwardly translated into the other, so you may not care — but do you?