*Or if it does, it’s VERY well-hidden. I looked at the home page and site map alike.

I wanted to refresh my memory as to Mark Logic’s history of working with specific text mining vendors, beyond what’s on the official partner page. No luck. Normally when site search is inadequate, one goes to Google. But that’s problematic too. Marklogic.com pages come up pretty low on Google’s search results, suggesting that:

1. Mark Logic doesn’t put a lot of effort into SEO (or else doesn’t do it very well).
2. One can’t be confident all the site’s significant pages are findable by Google.

Looking to other companies’ sites for clues isn’t conclusive either. E.g., Clearforest lists Mark Logic as a partner, but Mark Logic doesn’t return the compliment. (If memory serves, Mark Logic and Clearforest have worked together both on national security deals and custom publishing deals — but don’t hold me to that.)

When it comes to making its own information conveniently available, Mark Logic is quite the unshod cobbler.

However, various commentators — even ones as smart as Dan Farber — see something wrong with the use of YouTube for this purpose. I think that’s silly. If YouTube and its competitors are happy to provide the bandwidth for free, I see no reason why the transition team or government should pay for it. Nor should ads be a concern. After all, the president’s weekly radio address has long been provided to ad-supported radio channels.

Of course, I think official videos should be available from multiple sources. No site needs a monopoly. And indeed, with YouTube being banned in some countries, there’s a clear “greater reach” reason for multiple sourcing. (The American President doesn’t just speak to Americans.) But as Dan points out, the Obama video is indeed available through multiple major websites.

Now, I’m not saying that the government shouldn’t have sites of its own where it hosts videos of speeches by the Assistant Secretary of Transportation. But if YouTube or some other firm provides bandwidth in return for being noticed as doing same, I only see one kind of possible harm:

Perhaps the implicit advertisement/endorsement they’re getting is of greater value than the bandwidth being provided.

Fine. There are two ways to deal with that:

1. Require YouTube to remove its logo from the version of the video being put up on official sites (the first link above shows how that’s not happening now).

2. Auction off the right to be the primary video provider, or something like that.

Either way, I fail to see the big deal. YouTube and Google are great American companies. Government does things for companies all the time. Until a competitor comes up with a clear description of how it’s being hurt by this mild favoritism — and I haven’t heard of any yet — there are many bigger problems for Obama’s technology experts to solve.

The theory can’t be denied. What’s more, bad actors have other motives to pollute the Web. For example, if they plant favorable automated comments about their own products or unfavorable about the competition’s, Voice of the Customer/Market applications will naturally be confused. And if automated reputation-checkers get more prominent, there will be a major incentive to game them, just as there has been for Google’s PageRank. So VOTC/VOTM market research tools could polluted as a side effect.

Similarly, if somebody wants to test your e-commerce site by throwing a ton of searches at it, your search logs will lose value.

But disinformation of competitors for the sake of disinformation? Or, as the article suggestions, vandalism/extortion? Off the top of my head, I’m not thinking of a serious near-term threat scenario.

*Which makes perfect sense when you think about how long it takes to actually get a doctor’s appointment — or, all kidding aside, even how long it takes to decide it’s necessary to go to the doctor.

Google, quite credibly, claims that these results are based on aggregated data rather than personally identifiable information. Even so, it heralds a day in which Google observes which groups of users — geographically organized or otherwise — care particularly about certain subjects, and tailors news, ads, or search results accordingly — if that day isn’t already here.

• There are plenty of times when voice is a more convenient form of input than typing.
• Voice recognition was good but far from perfect.
• Editing search strings was annoyingly difficult.
• Search results themselves aren’t 100% perfect.

No big surprises there.

Below are some quotes — fair use!! — from the Authors Guild official statement on the deal (emphasis mine):

Our proposal to Google back in May 2006 was simple: while we don’t approve of your unauthorized scanning of our books and displaying snippets for profit, if you’re willing to do something far more ambitious and useful, and you’re willing to cut authors in for their fair share, then it would be our pleasure to work with you.

…

The payments will flow through the Book Rights Registry, a new independent entity that can be thought of as the writers’ equivalent of ASCAP. Much as ASCAP tracks the uses of songs and collects royalties for songwriters and musicians, the Registry will serve the interests of authors and others who own the rights to books appearing online as a result of this settlement. The Registry will be controlled by a board of authors and publishers; as part of the settlement, Google will pay $34.5 million to get the Registry up and running, notify rightsholders of the settlement, and process claims.

Readers are also big winners under the settlement of Authors Guild v. Google. Readers will be able to browse from their own computers an enormous collection of books. We hope this will encourage some readers to buy full online access to some of the books. Readers wanting to view books online in their entirety for free need only reacquaint themselves with their participating local public library: every public library building is entitled to a free, view-only license to the collection. College students working on term papers will be able to point their computers to resources other than Wikipedia, if they’re so inclined: students at subscribing institutions will be able to read and print out any books in the collection.

This is what writers have been — or at least should have been — awaiting for over a decade, ever since it became clear that the Web would transform media and publishing. With print-on-demand plus an online book registry, authors get complete access (starting in the US, at least) to readers, paying and otherwise.

So let’s review what publishers are good for. In some order, their role is:

• Marketing through branding/imprimatur.
• Marketing through advertising, book tours, and the like.
• Marketing/sales/distribution through the physical supply chain.
• Editing, art, and other actual contributions to the quality of the product.

It’s a publishing industry open secret that advertising and such like are pretty useless, doing more for the egos of all concerned than they do for actual book sales. Amazon is obsoleting most physical book stores, airport locations (for impulse purchases) and the like perhaps excepted.

As for branding/imprimatur: The backing of a major publisher can be worth a few thousand hard copy sales to libraries. But where else does it matter? I was going to suggest that it might in the academic world. But then I looked over at the math books on my shelves, a number of which are bound in familiar Springer Verlag yellow and white. Is there one book there I wouldn’t own if it weren’t a Springer Verlag publication? Probably not. I suspect that your initial reputation boost in academic publishing comes from your institution and peers much more than from an actual academic press.

So for the most part, book publishers and music publishers are left with just one marketing function — getting the ball rolling. Published products ultimately sell through word of mouth, but if you don’t start out with listeners or readers, how can the word of mouth build? The answer, of course, will increasingly be online promotion. If your natural audience is scattered around the country or the world, without being concentrated in one particular geographic location, online marketing is the obvious way to go. And in this world of search engines, YouTube, blogs, and the like, ever more channels for marketing are opening up.

Other than promotion and aggregation (the latter applying more to news/blog publishers than books/music), I do see one other role for publishers — actually creating product. Movies/TV and video games are both far bigger businesses than book publishing, and in both cases products are produced by large teams of people. Music is generally produced by small teams of people. And by the way, books can spin off from other kinds of entertainment (I suspect that a large fraction of all science fiction book sales at this point are Star Wars/Star Trek/etc.). Or vice-versa — some day the economic model for trying an ambitious new comic book project may factor in hoped-for movie and other spin-offs from the getgo.

But traditional publishers aren’t generally set up to do anything that ambitious. As for lesser but still important functions — editing, art direction, etc. — who needs large firms for that? The music industry seems to get by just fine with small recording studios and independent record producers; book producers could spring up just as well. Indeed, they exist already, but suffer from the problem that nobody wants to pay them much because book sales overall are so weak.

If I were a best-selling author, I’d hire away my favorite editor, put her on the payroll directly, and then send her and my agent to squeeze a few extra percentage points out of the big publishers, more than covering the cost of her paycheck. At least, I would if I were a best-selling author and also had my current personality. Now, I’m not, and if I were that kind of author I’d probably be quite fixated on being left alone to write, and might find it easier to leave money on the table than to take that kind of business responsibility myself. That kind of consideration is probably a big reason why traditional publishers are allowed to stay in business.

But the times sure are a’changin’.

Many people think text information is important to analyze, but even so data warehouses don’t seem to wind up holding very much of it.

My working theory explaining this has two parts, both of which purport to show why text data generally doesn’t fit well into BI or data mining systems. One is that it’s just too messy and inconsistently organized. The other is that text corpuses generally don’t contain enough information.

Now, I know that these theories aren’t wholly true, for I know of counterexamples. E.g., while I’ve haven’t written it up yet, I did a call confirming that a recently published SPSS text/tabular integrated data mining story is quite real. Still, it has felt for a while as if truth lies in those directions.

Anyhow, David offered one useful number range:

If you do exhaustive extraction on a text corpus, you wind up with 10-20X as much tabular data as you had in text format in the first place. (Comparing total bytes to total bytes.)

So how big are those corpuses? I think most text mining installations usually have at least 10s of thousands of documents or verbatims to play with. Special cases aside, the upper bound seems to usually be about two orders of magnitude higher. And most text-mined documents probably tend to be short, as they commonly are just people’s reports on a single product/service experience – perhaps 1 KB or so, give or take a factor of 2-3? So we’re probably looking at 10 gigabytes of text at the low end, and a few terabytes at the high end, before applying David’s 10-20X multiplier.

Hmm – that IS enough data for respectable data warehousing …

Obviously, special cases like national intelligence or very broad-scale web surveys could run larger, as per the biggest Marklogic databases. Medline runs larger too.

David’s thoughts as to why the SaaS share isn’t even higher – as it seems to be for Clarabridge* – centered on the point that some customers want to blend internal and external data, and may not want to ship the internal part out to a SaaS provider. Besides, if it’s tabular data, I suspect Attensity isn’t the right place to ship it anyway.

*Speaking of Clarabridge, CEO Sid Banerjee recently posted a thoughtful company update in this comment thread.

When I challenged him on ease of use, David said that Attensity is readying a Microstrategy-based offering, which is obviously meant to compete with Clarabridge and any of its perceived advantages head-on.

Coveo, Exalead, ISYS, Recommind, Vivisimo, and X1 are a few of a select group that are marking a mark in their respective niches, as products ready for action with a short implementation cycle (weeks or months not years).

By way of contrast, Lynda opines:

Autonomy and Endeca continue to bring value to very large projects in large companies but are not plug-and-play solutions, by any means. Oracle, IBM, and Microsoft offer search solutions of a very different type with a heavy vendor or third-party service requirement. Google Search Appliance has a much larger installed base than any of these but needs serious tuning and customization to make it suitable to enterprise needs.

In particular, her views about FAST (now Microsoft) are scathing.

Thanks for your mention to LanguageWare on your blog, albeit a skeptical one I totally understand your scepticism as there is so much talk about text analytics these days and everyone believes they have solved the problem. I guess I can only hope that our approach will indeed prove to be different and offers some new and interesting perspectives.

The key differentiation in our approach is that we have completely decoupled the language model from the code that runs the analysis. This has been generalized to a set of data-driven algorithms that apply across many languages so that you can have an approach that makes the solution hugely and rapidly customizable (without having to change code). It is this flexibility that we believe is core to realizing multi-lingual and multi-domain text analysis applications in a real-word scenario. This customization environment is available for download from Alphaworks, http://www.alphaworks.ibm.com/tech/lrw, and we would love to get feedback from your community.

On your point about performance, we actually consider UIMA one of our greatest performance optimizations and core to our design. The point about one-pass is that we never go back over the same piece of text twice at the same “level” and take a very careful approach when defining our UIMA Annotators. Certain layers of language processing just don’t make sense to split up due to their interconnectedness and therefore we create our UIMA annotators according to where they sit in the overall processing layers. That’s the key point.

Anyway those are my thoughts, and thanks again for the mention. It’s really great to see these topics being discussed in an open and challenging forum.