Attivio is having a house party and product rollout in the latter part of January, and details are scarce in the mean time. But here are some highlights.

• Attivio was founded in August. It has 21 people and 1 VC. The VC has invested >$6 million and committed >$12 million total.
• Attivio has ambitious plans for a fully integrated data management/real-time BI stack. It’s currently called the “Active Intelligence Engine.”
• The data management part combines tabular, text, and XML data. The tabular part is some kind of bitmap. The text part is fairly traditional, and based on Lucene.
• One point of this architecture is that one can more or less seamlessly join different kinds of data.
• Another point is surely that — with everything being more or less like a column or bitmap — memory management and administration are manageable issues.
• Despite containing all these wonders, the code is under 10 megs total. At least right now. But then — how much code can one write in a few months?
• Andrew didn’t want me to repeat everything he said about target markets, but clearly Wall Street is one of the top possibilities.

Stay tuned.

1. Text mining powers search. The biggest text mining outfits in the world, possibly excepting the US intelligence community, are surely Google, Yahoo, and perhaps Microsoft.
2. Search powers text mining. Restricting the corpus of documents to mine, even via a keyword search, makes tons of sense. That’s one of the good ideas in Attensity 4.
3. Text mining and search are powered by the same underlying technologies. For starters, there’s all the tokenization, extraction, etc. that vendors in both areas license from Inxight and its competitors. Beyond that, I think there’s a future play in integrated taxonomy management that will rearrange the text analytics market landscape.

So who does “get it” about the search/text mining connection? The UIMA folks at IBM probably do. Inxight surely does. Attensity seemingly does, and so do most large search engine vendors (FAST and the public guys for sure; I’m not so certain about Autonomy and Convera). A small company whose CEO just called me yesterday does. I think I do.

But I’m not sure that the smaller text mining and search outfits – or the small text-oriented parts of large enterprise software vendors — have gotten the message at all yet …

• A year ago almost no customers cared about UIMA.
• Now UIMA is regularly showing up on government RFPs.
• Private sector interest in UIMA is still very limited.

*And IBM plans to convince me ASAP that even that assessment is too negative, which it well may be. Stay tuned.

So to sum up — 1. We seem to have fair amount of traction with the UIMA framework by communities that are very interested in plug-n-play with components from other providers. This includes the government, life sciences and research communities. 2. The UIMA standard, as opposed to the specific Java Framework implementation, developed under an SDO will broaden the opportunity and strengthen the case of adoption of UIMA as a standard for text and multi-modal analytics that allows interoperability across different frameworks and applications. It would of course be the case that the Java UIMA Framework would comply to the standard.

The complete email follows.

Curt,
Hi. While, we can’t really speak for the vendors, the adoption story is ongoing and to fully appreciate it I think it best to consider it in a bit more depth.

First is adoption of the UIMA Java Framework which we have posted in binary form (as part of an SDK) on the IBM alphaworks site in late 2004 (http://www.alphaworks.ibm.com/tech/uima) and then early this year posted the source on source forge (http://uima-framework.sourceforge.net/). On alphaworks we get a rough average of a couple hundred downloads/month from government, academia and industry. On sourceforge we also get a similar average although it seems to be tampering off of late. What ALL these folks are doing with the framework, we do not know. The forum on alphaworks is moderately active; there hasn’t been as much activity on the source forum so far. We see a lot of use of the UIMA SDK (which includes the Java Framework) by government, universities and research institutions/programs that are not in the business of selling a specific application but rather in the business of creating/customizing their own solutions. From these communities we see more activity on the alphaworks SDK forum, requests for talks, tutorials and white papers and involvement in large collaborative projects using UIMA. This makes sense to us. This is where we expect to see early adoption of the framework. Traditionally these communities do not see their value-add or core competency in developing infrastructure. Rather they want to spend their time on the analytics, task models, integration and solution level stuff. They are also more likely to experiment with 3rd party analytics because they are not focused on competing at the component level, but rather on solving their core problems, often in a collaborative environment. Building adoption here for an interoperability framework, I think, is a good first step.

It appears that text analysis vendors tend to build on their own internal frameworks, which have been in production for a some time and are intimately tied to their applications. Also, they may tend to consider their analytics a corner stone of their competitive advantage and therefore may not be prone to share them or suggest that someone else’s are better. Switching over to a different, externally provided, pluggable framework may not be a top priority. It is reasonable that over time vendors may adopt the UIMA Java framework as part of their internal implementation, but that depends on technical issues surrounding the cost/performance trade-offs relative to maintaining their current implementations and their interest in reusing 3rd party analytics. Vendors may be more immediately motivated to partner with IBM in the creation and/or use of a standard (which I will say more about below). Their hopes are to use the standard to better enable opportunities for strategic partnerships and to find more channels for their technology. The standard enables, for example, network or service-level interoperability across frameworks and applications not necessarily requiring a deeper implementation commitment.

Second is adoption of a UIMA standard for interoperability that accommodates different implementations/applications. We are considering the creation of an external working group under a Standards Development Organization (SDO), to define a standard specification for UIMA. This is independent of the open-source Java Framework implementation and specifies the data representations and abstract interfaces for defining compliant data and for communicating between text (and multi-modal) applications over, for example, a network protocol (e.g., SOAP). It will specify how to encode analysis data, how to publish network services that do annotation etc. We expect that this will be an attractive adoption point for communities that want to interoperate but not necessarily change their internal implementation and development environments. Perhaps the larger portion of text analysis vendor community fall into this camp.

So to sum up — 1. We seem to have fair amount of traction with the UIMA framework by communities that are very interested in plug-n-play with components from other providers. This includes the government, life sciences and research communities. 2. The UIMA standard, as opposed to the specific Java Framework implementation, developed under an SDO will broaden the opportunity and strengthen the case of adoption of UIMA as a standard for text and multi-modal analytics that allows interoperability across different frameworks and applications. It would of course be the case that the Java UIMA Framework would comply to the standard.

Regards,

Dave

The ideal ontology would consist mainly of four aspects:

1. A conceptual part that’s language-independent.
2. A general language-dependent part.
3. A sensitivity to different kinds of text – language is used differently when spoken, for instance, than it is in edited newspaper articles.
4. An enterprise-specific part. For example, a company has product names, it has competitors with product names, those names have abbreviations, and so on.

There are actually two different requirements – the enterprise-independent ontology, and the software to manage and use it in an enterprise-specific way. But while I continue to believe that this dual product category will emerge, my faith has wavered somewhat. The big vendors don’t “get it,” and the little ones lack the resources even if they do see the opportunity.

I discussed this with David Thede of dtSearch last week, and he raised an interesting question: Should this be an open source project? Some initial responses follow; also, I’d be very interested to know what the rest of the community thinks.

Peripheral parts of the software clearly can be drawn from the open source community. IBM has open-sourced UIMA, which seems like a perfectly good framework for modularity, integration, interoperability, and so on. Development tools can probably be based on Eclipse. Etc. Open source has plenty of applicability these days.

The core software needs a profit-motivated vendor. Open-source, to date, has been much more about implementing alternate versions of known technology than it has been about difficult first-time invention. And there’s a lot of invention still to be done here, especially in the area of taxonomy federation. Could there be an open source business model in which the vendor gives away the code and sells services? Sure. If nothing else, Monash’s Second Law of Commercial Semantics states “Where there are ontologies, there is consulting.” But somebody has to own the product, in every sense of “own,” or it never will see the light of day.

An enterprise-independent ontology ideally should be open-sourced. Probably, the leader of this effort should be the supplier of the ontology management software. WordNet is more or less public domain. Various taxonomies of proper nouns and industry jargon can also be found. Individual contributors can naturally provide new small pieces. So there’s a lot of reason to think that public domain/open source is naturally the way to go for an ontology.

That said, a big problem comes to mind – how do you get everyone to agree on what the structure will be? But isn’t that the same kind of problem that open source software development projects solve all the time? I think so.

But unfortunately it’s also the kind of problem that standards committees botch all the time. From that, I conclude that a public-domain ontology will need strong central leadership. Who’s qualified and incentivized to provide that leadership? I can’t think of anybody better suited than whoever emerges to seize the billion-dollar opportunity for ontology management software.