David’s thoughts as to why the SaaS share isn’t even higher – as it seems to be for Clarabridge* – centered on the point that some customers want to blend internal and external data, and may not want to ship the internal part out to a SaaS provider. Besides, if it’s tabular data, I suspect Attensity isn’t the right place to ship it anyway.

*Speaking of Clarabridge, CEO Sid Banerjee recently posted a thoughtful company update in this comment thread.

When I challenged him on ease of use, David said that Attensity is readying a Microstrategy-based offering, which is obviously meant to compete with Clarabridge and any of its perceived advantages head-on.

1. Attensity is the older company than Clarabridge, and is good at more things. Is Clarabridge really good at everything you want them to be?

2. In particular, Attensity has more overall sophistication at linguistic extraction. Do any of the differences matter to you?

3. Both companies are working hard on ease of use, for multiple kinds of user (business user tweaking linguistic rules, IT user, etc.). Whose approach and feature set do you like better?

4. Usually, buying one of these products involves some professional services. Whose organization do you like better?

5. Attensity’s default database schema for its exhaustive extraction is pretty flat and normalized, as befits a happy Teradata partner. Clarabridge’s is more of a star schema, as befits a bunch of ex-Microstrategy guys. Either can be straightforwardly translated into the other, so you may not care — but do you?

Well, what about text analytics? Much of what I know about natural language processing (NLP) I learned from my friend Sharon Flank, who I met when she was a Slavic Linguistics PhD student at Harvard. My partner in first figuring out search engines — and later in running Elucidate — was my wife Linda Barlow, a 15-times-published novelist who’s also taught English at the college level. And Olivier Jouve’s education is in paleontology, although whether or not that’s a humanity is a sort of borderline definitional issue.

So I ask you all: Is text analytics a fruitful area for humanities majors to find lucrative careers? All insight would be appreciated. If the news is good enough, I’ll do my part in publicizing it to university placement offices and the like.

I’ve started out by asking Attensity (David Bean) and Clarabridge (Sid Banerjee). Attensity turns out to hire humanities students most years, both as full-time employees and interns. Linguistics students are the top priority, but language students and other language-friendly types are of interest as well. David is even involved in trying to set up a computational linguistics certification program at the university where he teaches part-time. And Clarabridge, the much younger company of the two, has over the past year used humanities majors quite successfully as well, for multiple aspects of ontology-building.

• Voice of the Customer/Voice of the Market (Attensity pretty much conflates the two) applications are going really well. David kept repeating that prospects were actually calling Attensity, rather than Attensity having to go out on sales calls and find them. (Of course, that’s what’s supposed to happen when you have good marketing, including but not limited to lead generation.) Phil said that the quarter is going extremely well, including with some big-name customers.
• Attensity’s new favorite buzzword for these applications is “First-person intelligence.”
• About 50% of Attensity’s commercial customers take the technology on a SaaS basis, including some of the biggest ones. I didn’t know that, but thought I’d ask after discovering how important SaaS is to Clarabridge. While Attensity apparently introduced a “new” SaaS offering earlier this year, in fact some of Attensity’s largest customers have always been on SaaS.  Edit:  Actually, that 50% figure is overstated.
• Clarabridge is Attensity’s only significant commercial competitor, at least to the extent David or Phil is aware.
• Government is a strong market for Attensity too. (Recall that Attensity was funded by In-Q-Tel.)
• In particular, Attensity has expanded from intelligence to civilian law enforcement, via something called LEADS (Law Enforcement Analysis Desktop Solution). LEADS extracts relationship information from case notes and the like and dumps it into the I2 link analysis tool (OEMed). I’m aware of only one actual LEADS customer (Chesterfield County, VA). But Attensity is talking with other jurisdictions, plus (an obvious group of prospects) “fusion centers” that combine case information across jurisdictions.

• Clarabridge’s favorite buzzphrase is Customer Experience Management (CEM). That’s how they define their category.
• Clarabridge’s second-favorite buzzphrase is Voice of the Customer.
• Clarabridge is active in a variety of all vertical markets, with the common characteristic being the overall richness of customer interaction (based on some combination of frequency and complexity).
  • Travel/lodging/hospitality continues to be important.
  • Retail has come on strong the past 2-3 quarters.
  • The telecom and internet sectors are going well; neither was big a year ago.
  • High-tech consumer electronics is an active sector.
• Much of the value users derive from Clarabridge is early warning, such as of trends in customer sentiment or loyalty.
• Sometimes, this warning can be same-day, for example of technical malfunctions. Sid told one story of a hotel that was wrongly locking guests of its rooms, and another of a website that didn’t live up at ordering time to the offer it had just promoted. In both cases, the problem was detected and fixed immediately. Sid believes this wouldn’t have happened without Clarabridge technology.
• Examples of applications and benefits not closely tied to early warning include:
  • Scorecarding/benchmarking – all doubts about legitimate statistical significance notwithstanding, some enterprises are including text mining results in their self-measurement.
  • Correlation of trends to specific events – text mining can help figure out whether a specific marketing campaign or seasonal variation is getting results. For example, if you run a marketing campaign calling people’s attention to a specific product feature, and your support calls in connection with that feature spike, chances are that the campaign is working.
• Sid cited one example of a household-name Clarabridge customer using more open-ended survey forms than before. Good idea. Lots more enterprises should follow suit.

• SaaS represents over half of Clarabridge’s revenue. Please note that this surely means well over half of Clarabridge’s new sales are in SaaS, because of how revenue is recognized. And these fractions are still growing.

• Clarabridge is making 10-15 sales transactions/quarter.

• Clarabridge reports 500%+ sales and revenue growth in 2007. (Note that Clarabridge was only recently founded and spun out of Claraview, unlike a number of competitors.)

• Over half of Clarabridge’s customers are already on SaaS. Given the other figures already cited, that’s unsurprising.

• Clarabridge is selling to business departments, in customer-oriented areas (marketing, support, product management, etc.). IT hasn’t embraced text mining technology yet.

• A large customer may get up to 1-3 terabytes of content into Clarabridge over the course of a year. (Obviously, the sample size here is small.)

• Clarabridge says that customers quickly expand their usage to multiple corpuses of data, and multiple business departments. (But then, I’m trying to think of an analytic technology vendor who doesn’t say that, and I’m drawing a blank.)

• In particular, Clarabridge thinks it’s important that its customers can and do fuse internal and external text data. Internal text might be email, survey forms, or call center verbatims. External text might be blog or forum postings.

• Clarabridge has developed a Web services API to bring data back to customer sites, for integration with other data — but few customers care yet. Mainly, they’re operating on a pure outsourced basis.

I’ll post about Clarabridge’s actual applications and customer segments separately.

The most interesting point was text mining SaaS (Software as a Service). When Clarabridge first put out its “We offer SaaS now!” announcement, I yawned. But Sid tells me that about half of Clarabridge’s deals now are actually SaaS. The way the SaaS technology works is pretty simple. The customer gathers together text into a staging database – typically daily or weekly – and it gets sucked into a Clarabridge-managed Clarabridge installation in some high-end SaaS data center. If there’s a desire to join the results of the text analysis with some tabular data from the client’s data warehouse, the needed columns get sent over as well. And then Clarabridge does its thing.

It has always been the case that business intelligence was an IT systems software technology that often wound up being sold on an application basis to end-user departments. Clarabridge very much fits that model. And while it used to be the case that BI adoption was pretty simple, that’s increasingly not the case, which is one reason SaaS is appealing. So this all makes a lot of sense.

Even so, I was surprised to hear that SaaS had so quickly become half of Clarabridge’s business. Wow.

Since Clarabridge touts Cognos as an important partner, and Cognos is being bought by IBM, I also asked Sid about UIMA. He basically responded that UIMA was unlikely to become relevant to Clarabridge any time soon, because the way Clarabridge interfaces with other software is SQL. Up to a point, that makes great sense to me. But if we buy into the comprehensive/exhaustive extraction story — as Clarabridge does — then the day should and will come when serious linguistic processing gets done on text after it is extracted into a relational database. And if that happens, then all of a sudden SQL won’t be the only interface integrating text analytics with BI.

• Like Attensity, Clarabridge practices exhaustive extraction.* That is, they do linguistics against documents, extract all sorts of entities and relationships among the entities from each document, and dump the results into a relational database.
• Unlike Attensity, which uses a simple normalized relational schema, Clarabridge dumps the extracted data into a star schema. (The Clarabridge folks are from Microstrategy, which – surely not coincidentally – also favors star schemas.)
• For now, the linguistic part of the analysis is within a sentence, or else based on proximity, or (this sounded minor) based on the whole document. But actual anaphora resolution is coming soon.
• The other big thing that goes into Clarabridge’s star schema is a category hierarchy, which has two aspects. One is categories fixed in advance. When I asked how many, CTO Justin Langseth cited an example range of 10-400. I.e., it varies widely. In principle, these are established by line-of-business folks at Clarabridge customers, but I’d venture to guess that professional services play a significant role as well.
• The other kind of categories – subcategories to the first group – are created automagically at data load time via document clustering. Indeed, they’re called “clusters.” These are available for drilldown via business intelligence tools.
• Obviously it is good practice to have dashboards and scheduled reports depend only on the fixed categories, not the clusters.

*I should note that Clarabridge understandably bristles a bit at my use of this Attensity-introduced term to describe what they do too. If Clarabridge wants to start talking about, say, “comprehensive extraction, I’ll consider adopting that term as well. But for now I’m going with what’s most widely used.

Want to continue getting great research about text mining, data warehouse appliances, and other hot analytics-related topics? Then subscribe to our comprehensive (if not exhaustive) feed, by RSS/Atom or e-mail! We recommend taking the integrated feed for all our blogs, but blog-specific ones are also easily available.

Technorati Tags: Clarabridge, text mining, exhaustive extraction

Other highlights of what they said included:

• Voice of the Customer remains hot, hot, hot.
• Closely allied with Voice of the Customer, and also hot, is Voice of the Market and/or more direct competitive intelligence.
• Classical warranty analysis is quiet but not wholly dead. Attensity, historically strong in that application, sees it as merging into Voice of the Customer. Clarabridge, previously not so strong there (if I recall correctly), is getting at least a little of the traditional-style warranty business.
• Human resources (especially Voice of the Employee – I detect a trend in application-naming here) gets mentioned a fair amount. It’s usually not the first text mining application an enterprise deploys, but it’s a common follow-on.
• Antifraud isn’t just for insurance companies. Retailing and money-laundering also got mentioned as areas where text mining helped combat fraud.
• Insurance industry use of text mining for claims analysis, I gather, goes well beyond just fraud detection.
• Intelligence is obviously a huge market for Attensity (not so much for Clarabridge), but I didn’t focus on the classified stuff. That said, I was reminded of Attensity’s awkward phrase link analysis, which has nothing to do with hypertext, but instead is the detection of relationships between entities. This lies at the heart of a non-empty set of civilian law enforcement applications and the like.
• Investment research applications of text mining still seem nascent and experimental, at least if one talks with Clarabridge and Attensity. That said, Factiva is a large subsidiary of Dow Jones now, and ClearForest a smaller one of Reuters, and they’re doing something or other. Apparently, it’s much more document tagging for the sake of readers or search-style filters than it is for use in any kind of business intelligence/statistical mining kind of application.

All this isn’t too different from what I posted back in July, but I think text mining application trends is a subject that bears frequent revisiting.

*Actually, the word Clarabridge CTO Justin Langseth used was “verbatim.” But that’s essentially a synonym for document, only with the connotation that these documents will probably be people’s statements (think warranty cards, customer surveys, email, call center notes, etc.), with all that implies for their grammar, structure (or lack thereof), and so on.

I didn’t push Attensity for an answer that clear. What they said was simply that all their capabilities were integrated together, so everybody uses exhaustive extraction. I imagine they’d say something similar, but it seems I should follow up a little bit further …