Software Thoughts: Structuring the Unstructured

I know that I’m not supposed to click on my Google ads but sometimes they really look interesting. The title of one ad that just caught my eye is “Unstructured Data / UDM” and I had to click on it. When I was product manager at OnDisplay I had a vision of using the eContent product to build applications on. To turn the server into an application server because the greatest strength that the eContent Agent Server had was its ability to turn unstructured data into structured data. It wasn’t magic although it sure looked like it. Trung Dung, one of the OnDisplay founders, had realized that often unstructured data could become structured data with a little human intervention. Trung came up with the idea of allowing humans to build patterns that allowed software to organize the unstructured data. In would go data that the computer couldn’t understand, a pattern was run against the data, out would come the structured data. Wow.

We called eContent a Content Aggregation Server. Some companies used eContent as the foundation for price comparison websites, aggregating the data from unpartnered websites; Travelocity, Trip.com, Ariba, and Granger were all customers.

What eContent was able to do at that moment in time was not what it could ultimately become. There were some major problems that needed to be overcome. First, the Agent Server architecture was not sound. I had been the lead on the server for a short-time, but I was brought on after the work had been started. So I focused my energy on getting it working, not on redesigning it. Making the IPO was more important then redesigning the server for the long haul. (One of those business decisions that affect the technical decisions.) What needed to happen was the server needed to become a true engine. With the ability to plug-in supported input and output formats. The engine simply getting data from the input plug-in, processing it using the agent (the pattern), and outputting the data using the output plug-in.

With this vision coupled with the agents redesigned as XML instead of Javascript I could see a product that could make programmers and IT folks lives easier. Unfortunately, I was not able to realize my vision. Vignette now owns the code and it’s unlikely that I will be the one to fulfill the vision of structuring the unstructured. I do hope that Vignette does though, it will certainly fill a need.