The embeds of the future will work for the company, and be paid by the company to provide news about the company in a multitude of formats. Print, newsletter, video, blog, podcast, moving billboards, tattoos — whatever it takes. Because the bits and pieces of Corporate America that have a story to tell will still have their stories – just no ready outlets.

How is this different than what you have today? Surely there are corporate PR departments and external agencies already doing these things, right?

No.

What is required is an internal producer who writes in external voice — like the neutral point-of-view so often described by Wikipedia. People can smell marketing and propaganda coming around the corner, and they know when the pitches and puff pieces are missing that edge of neutrality. An accurate and fair piece is accurate and fair, no matter who writes it.

It’s an interesting theory, but it seems to presuppose dual marketing communication efforts, with separate departments of “Straightforwardness” and “Hype”. That may work at some companies, but in most cases I think it will be more practical to try to infuse straightforwardness through multiple parts of the marcom effort.

My more specific quick responses include:

• That sure sounds a lot like Robert Scoble in his Microsoft days.
• It also sounds like “community managers” at MMO game companies. (Both of the MMOs I’ve played have had great ones.) They often only use one or two channels (forums and the associated general website), but otherwise they fit the bill.
• Ike’s views fit very well with mine on the future of the information ecosystem.
• I’m getting ever more sympathetic to the idea that you need people whose main job is external communication of a straightforward kind. Reasons include:
  • Senior executives who write great blogs commonly don’t keep them up. And even when they’re active, the blogging is pretty sparse. E.g., among companies I follow closely, Vertica, Aster Data, and Netezza have all done some outstanding blogging in the past, but do very little of it now. Only Dave Kellogg at Mark Logic really keeps going.
  • It’s not obvious that senior executives are wrong to spend their time at something other than blogging. One of the greatest vendor blogs ever was Jonathan Schwartz’s at Sun.  Umm — how sure are we that he actually did much good for his company with that effort?
  • I frequently tell vendors “If you tell Story X in your own words, I’ll gladly point to it or post it for you.” They usually agree this is a wonderful idea — but then usually don’t free up the rather limited resources that would be required to take me up on it.
• That said, the kinds of people who provide customer support (pre- or post-sales) are often very well suited to fill the role Ike is describing. At least, that’s the case in enterprise tech companies.
• The media mix isn’t really as complex as Ike was suggesting. It basically falls into two groups: Text, and audio/video.
• That said, text/graphics and audio/video media people are increasingly the same. (Just think of sports media, where the newspaper folks make their big bucks on radio or TV. That’s a harbinger of the future. Or think again of Scoble.)
• One flaw of Ike’s idea is that in its pure form it only makes sense for companies large enough to have multi-person PR staffs. Other firms would have to use part-timers, or outsource.  And if you’re going to do that, might it not make more sense to pay part of the cost of sponsoring, you guessed it, an independent blog?
• I know that’s text/graphics-only, or at least text/graphics-mainly, but I happen to think audio/visual business news/PR is minor anyway. People may give enough attention to, for example, listen to audio from a company if it purports to teach them something. But news ABOUT a company? Who’s so interested in that to sit still for audio/video, unless they happen to be employees, or investors in its stock?

Bottom line: I think he’s wrong about some of his detailed views, but Ike Pigott is directionally very right in suggesting that newsmakers will increasingly become content creators for news about themselves.

*Or if it does, it’s VERY well-hidden. I looked at the home page and site map alike.

I wanted to refresh my memory as to Mark Logic’s history of working with specific text mining vendors, beyond what’s on the official partner page. No luck. Normally when site search is inadequate, one goes to Google. But that’s problematic too. Marklogic.com pages come up pretty low on Google’s search results, suggesting that:

1. Mark Logic doesn’t put a lot of effort into SEO (or else doesn’t do it very well).
2. One can’t be confident all the site’s significant pages are findable by Google.

Looking to other companies’ sites for clues isn’t conclusive either. E.g., Clearforest lists Mark Logic as a partner, but Mark Logic doesn’t return the compliment. (If memory serves, Mark Logic and Clearforest have worked together both on national security deals and custom publishing deals — but don’t hold me to that.)

When it comes to making its own information conveniently available, Mark Logic is quite the unshod cobbler.

Airlines put Marklogic to an interesting use: They create “electronic flight bags.” Apparently, flight crews typically carry a whole satchel of documents (flight bags) onto a plane, the precise contents of which frequently vary. Marklogic lets these be automatically generated in electronic form.

Well, in recent news it turns out that a $1.4 billion B-1 bomber crashed because a known prudent take-off/maintenance procedure hadn’t been followed. (Something about heating the components to evaporate water that otherwise destroyed the electronics.) This plane-saving had been discovered, but not propagated to all bases and maintenance crews responsible for the B-1. You think something like Marklogic might have helped?

To quote CNN.com,

However, a technique learned by some two years ago that had gone widely unknown and unadopted probably would have prevented the crash, Carpenter said. The technique essentially heats the sensors and evaporates any moisture before data calibrations.

“This technique was never formalized in a technical order change or captured in ‘lessons learned’ reports. Hence, only some pilots and some maintenance technicians knew of the suggestion,” according to Carpenter’s executive summary of the accident.

The report said, “The human factor of communicating critical information was a contributing factor to this mishap.”

For the most part, MarkLogic is indeed sold (and bought) for the storage, manipulation, and retrieval of text. (One long-confidential exception to this rule is scheduled to be unveiled at the June user conference.) Most applications seem to fit a custom publishing/enhanced search paradigm:

1. Ingest text.

2. Enhance it.

3. Serve it up in chunks, typically via a sophisticated search interface.

Differences vs. conventional search engines include:

• Documents are indexed on the fly, and available for query immediately upon ingestion.

• MarkLogic is a real, ACID-compliant DBMS. So everything else – such as a user tag or comment — is also available for immediate query. Mark Logic says customers are making a lot of use of this feature.

• MarkLogic has a real programming language – specifically XQuery. (Note: XQuery is a much fuller language than, say, standard SQL, with conditional logic, arithmetic, try/catch, and so on.)

• MarkLogic handles fielded information, document chunks, and whole documents in a completely integrated fashion. Truth be told, I don’t know exactly to what extent Autonomy or FAST do or don’t fall short of this standard, but it’s never seemed to be as much of a priority on their part as I’ve felt it should be.

Mark Logic also claims huge advantages in corpus administration. Scalability seems good too; there’s a national-intelligence customer with a 200 terabyte database. And they’re proud of a feature called lexicons, although it seems so obvious to me that I’ve so far failed to muster what they’d probably regard as the proper level of excitement about it. (In SQL terms, it seems to be a combination of SELECT and COUNT DISTINCT, both of which are capabilities I’d think would be in XQuery anyway.)

• Yep, Voice Of The Customer is hot, in “many markets”; Eric specifically mentioned banking, car, energy, food, and retail. He further sees IBM backing VotC as text’s “killer app.” (Note: Temis has a history of partnering with IBM, most notably via its unusually strong commitment to UIMA.)
• Specifically, THE hot topics in the European market these days are competitive intelligence and sentiment analysis. (Note: I’ve always thought Temis got serious about competitive analysis a little earlier than most other text mining vendors did.)
• Life sciences is an ever growing focus for Temis.
• I confused him a bit with how I phrased my question about custom publishing and Temis’ Mark Logic partnership. But he did express favorable views of the market, specifically in the area of integrating text mining and native XML database management, and even volunteered that nStein appears to be doing well.

*Factiva is the most significant exception. Hint, hint.

If you think about it, text analytics is a “secret ingredient” in search, antispam, and data cleaning,* and this dominates all other uses of the technology. A significant minority of the research effort at companies that do any kind of text filtering is – duh — text analytics. Cold comfort for specialist text analytics vendors, to be sure, but that’s the way it is.

*I.e., part of the “T” in “ETL” (Extract/Transform/Load).

Text-analytics-enhanced custom publishing will surely at some point become a must-have for business and technical publishers. However, it appears that we’re not quite there yet, as large publishers make do with simple-minded search and the like. In what I suspect is a telling market commentary, there’s no headlong rush among vendors to dump text mining for custom publishing, notwithstanding the examples of nStein and (sort of) ClearForest. I don’t want to be overly negative – either my friends at Mark Logic are doing just fine or else they’re putting up a mighty brave front – but I don’t think the nonspecialist publishing market is there yet.

Two business publishers who have made major investments in owning text analytics technology are Dow Jones (now sole owners of Factiva) and Reuters (recent purchaser of ClearForest). Beyond that, however, I don’t yet see a lot of activity in the investor/trading market, although ClearForest reported some activity last year and StreamBase reports that one customer is using them for text filtering, presumably alongside the ticker-munching traders usually use StreamBase for.

Obviously, the intelligence market is what fueled the start of the text analytics business, and still provides the majority of revenue at multiple companies. Certainly it’s still going strong. But it’s tough to gauge the growth potential from here, especially since the details of usage are typically classified.

Similar things could be said about pharmaceutical research. Text analytics is totally accepted in that market, but what’s the growth potential from here? And “here” isn’t actually very big (much smaller than intelligence). The related category of patient records analysis looks very promising, but is basically still at the research-project stage. (In general, an explosion in biological IT can be expected when research methods are adapted for clinical use.)

The warranty analysis market, so promising early on, is not showing a lot of growth and depth. The same thing has happened many times before with innovative technologies sold to manufacturing companies’ engineers. It seems to be happening again now.

Voice of the customer* is pretty much the same thing, but for service industries. And the text analytics market for VotC is evidently stronger right now than that for warranty analysis. This makes sense, because the obvious alternative to text analytics – multiple-choice coded forms – is less appealing, due to two application differences:

• VotC looks for opinion as well as fact.

• VotC looks for input from people under no obligation to share it, and who hence can’t be compelled to play along with a structured form – let alone trained to fill it in accurately.

*Definitional note: Voice of the customer is when customers or prospects communicates with you directly, e.g. via a survey form or an angry email. Reputation management is when you web-scrape and find out what they’re saying to everybody else. At least, I think marketers are still using the terms that way pretty consistently.

Reputation management is surely becoming a standard application for the biggest consumer brands. How deep that market turns out to be, however, remains to be seen.

Text analytics for fraud discovery seems poised to sweep the insurance industry, and then the rest of financial services. Current activity, however, while decent, still seems to consist of more poising than sweeping.

Compliance is a minimum-acceptable-efforts kind of activity in most markets. Accordingly search/clustering seems to be the preferred text-checking approach. Where that’s not the case, the market seems to have gone to specialized products like Assentor (stock brokerage).

Human resources is a good area to sell follow-on applications, at least to enterprises with so many employees that they want to automate the reading of employee feedback. I’m not aware of it being the first-sale app to very many enterprises, however.

SAS used to speak glowingly of text mining used directly for ETL. However, nobody else has talked about this, and even from SAS I get the sense that some of the glow has worn off. As noted above, text analytics is an important ingredient to the transformation part of ETL, but it I think it rarely would be the best option for doing the transformations directly.

• Curt Monash, President, Monash Information Services
• Dave Kellogg, CEO, Mark Logic Corporation
• Michelle De Haaff, VP Marketing, Attensity Corporation
• Michel Lemay, VP Marketing, nstein Technologies
• Mary Crissey, SAS Analytics Marketing Manager, SAS Institute

Michelle, Michel, and Mary are all obvious choices, responsible for marketing at leading text mining vendors. In addition, Mary has excelled on the same panel in the past, Michel sent me e-mail with some brilliant thoughts on the panel subject, and Attensity has one of the most interesting strategies in the text analytics market.

As for Dave — he’s simply one of the most astute marketing theorists working in software today. And he runs a very interesting text technology company. And he used to be most senior marketing guy in all of business intelligence, when he was SVP at Business Objects. In his copious free time, he writes a really cool blog.

• Attensity wants to make text analytics very easy to integrate into business intelligence and data mining – at the moment, they’re not too focused on the differences between those two disciplines – and is trying to deliver the best possible fact extraction consistent with that charter.
• ClearForest wants to provide really great information extraction — to the limits of what can be done without excessive knowledge engineering – and is trying to integrate as well as possible with other technologies, the better to serve the customers who need what they offer.

The guy I usually talk with at ClearForest, Jay Henderson, believes that text analytics is a collection of dozens of niche markets. Not coincidentally, a lot of ClearForest’s customers are in the publishing sector (I’ve remarked on ClearForest’s synergy with Mark Logic before). Attensity obviously is trying a broader play. In Jay’s view, Inxight and TEMIS are more analogous to ClearForest than Attensity is, except that Inxight is focused on different markets (e.g. OEM and/or search), and he thinks ClearForest is just better than Temis except in a couple of specific kinds of understanding (e.g., life sciences, sentiment).

That said, both Attensity and ClearForest credibly claim to do large fractions of what the other one does. ClearForest, as the currently nichier player, takes the traditional stance “We do everything they do, and more. Most of our customers are ones who really appreciate the difference.” Attensity conveys the equally traditional attitude “We do most of what they do, and a bunch of other stuff besides. And it’s better-packaged too. As for what they do that we don’t – not a lot of customers really have a need for it.”

Frankly, most enterprises that have a need for this technology should put both Attensity and ClearForest on their short lists. But here’s one technical note that may help predict who you’ll wind up actually selecting: Attensity’s lead strategy for integration is to dump everything into relational tables, for conventional analytics-stack products like Business Objects’ and Teradata’s to manipulate. ClearForest’s lead strategy for integration has more of an SOA/XML flavor, grown out of conventional OO. If one of those sounds like an obviously better fit to your situation than the other, then that’s the vendor you absolutely, positively should not leave out of your evaluation process.

Here, as best I understand, is the story of the custom publishing business. Its core market is publishers of high-cost material sold to people with high-priced time – i.e., scientific/engineering/medical/legal/business/etc. Other markets are general publishing, internal document preparation (e.g., intelligence community), and of course maintenance manuals (maintenance/repair has been a flagship market for just about everything, from expert systems to generic text search to, of course, text mining now as well).

The phrase “custom publishing,” however, obscures the distinction between two different paradigms. One of these is what we might call true custom publishing – assembling paragraphs, articles, chapters whatever from various sources, in an assembly customized for specific reader needs, roles, or preferences. On the revenue side, that’s a fascinating subject. But technically, I’m more interested in the other view: search results plus.

We all know lots of problems with search engines. One of the many is this: Except on rare occasions, getting the benefit of a successful search involves a whole lot of link-clicking and scrolling. But what if the relevant passages were all assembled together for you? Link-clicking would be eliminated, and scrolling might be minimized as well. The potential is huge. But I don’t know what level of precision is needed before the theoretical benefits become real.

The two paradigms can be blended, of course. A publishing product or dashboard or personal web page with a topic filter might get the results in custom-document rather than link-of-lists form. Once again, the problem is conciseness. The more concise the “complete” results can be, the more useful this kind of technology will ultimately prove.

For more on Mark Logic, and more insight about the industry in general, see CEO Dave Kellogg’s blog. For a technical discussion of MarkLogic, see my DBMS2.com write-up.