How to build a semantic search engine…
…if you have scores of developers at your disposal.
Let’s imagine you are a senior manager of a large software company. The online advertisement market is huge, and your boss tells you your company deserves at least a quarter of the online revenue pie. The company currently has less than ten percent. So what are you going to do?
Well, first, check out what’s hot on the Web at the moment. Semantic Web! Right, that’s what it has to be. So, you buy a hot semantic search startup - only to figure out later that open domain language-independent natural language technology is just not there yet (who was this Jeeves guy again?).
But fear not, you have enough resources at your disposal - and you were trained in all the relational database magic, the cure to all evil. So: your developers now create “ontologies” which will be the basis for your APIs. In your world, ontology is just the fancy word for a database schema, of course. Ah, chaos ensues, because at first nobody is reusing other people’s things.
To exercise a bit more control, now only ontologists (I really like that word) can create schema, and the chief ontologist will create the Great Ontology. To keep everything tidy, you’ll coerce instance data from the web into your data warehouse (based on, say, AsterBase or DataAllegro). There are plenty of engineers busy writing extraction rules, designing and maintaining schemas for the database, partitioning the databases and creating indices, and dreaming up nifty user interfaces. For each domain.
Mission accomplished! Because you’re using relational technology, your pilots, demos and prototypes seem to work (except keyword search and ranking), and your boss is happy. You are happy, too, because all your people have work and stay busy because of the high amount of manual labour involved. Another great day in paradise.