Thursday, June 23, 2011

Pet project: Metadata Flask

Just a quick announce of a pet project I started a few months ago. Unfortunately I've already slowed down work to a halt as I focus on finishing my MBA dissertation.

The objective of the Metadata Flask project is to provide tools for supporting the process of finding, collaborating and working with metadata on an open and linked-data web.

Nowadays when players of an industry vertical, say: commodities producers and large scale retailers like Walmart, or Oil & Gas equipment producers and Oil companies/operators want to define and evolve data exchange formats they have to build their own infrastructure for documenting, referencing and hosting data exchange definitions.

These players frequently need to exchange a lot of data for describing what is produced, their specs, how they are sold, who wants to buy etc. So they come up with metadata definitions like WSDL contracts, XML schemas, CSV/JSON templates and so on.

I believe their work would be a lot easier if infrastructure for hosting, exchanging, discovering, collaborative editing, moderating and many other metadata related processes was already in place.

The goal is ambitious, the problem not well defined and "tractable" right now, so I started investigating what open data sets are available out there. What I realized is that many of them couldn't be found using general purpose search engines, so I started a first subproject, the Open Data Directory, which is a search engine for open data sets published by governments, private companies and other organizations. It already indexes 360K+ datasets from many sources, and traffic is slowly increasing as search engine bots index its contents.

No comments: