Collective Intelligence in the Database

I’ve worked for several predictive analytic/data modeling/artificial intelligence companies, and here at inqiri, we’re doing things a little differently.  We are utilizing the core functionality of database systems to efficiently process large sets of data and return meaningful results to our inqiri members.

All of those companies stored data in the database, but that was about the extent of it.  Programs written in C++, C#, and Java would pull the data out of the database, run it through the predictive model and then return the results.  When I started working at the first company using this technology, I was pretty green in the SQL realm and just out of college.  During my time there, I became the go-to guy for T-SQL code problems.  One of the main things I learned there was that database systems were built to process “sets” of data very efficiently.  Yes, you could write a “cursor” or a “while loop” to process records one at a time, but that would not come close to matching the performance of a set-based process.  With that experience, I began to formulate ways that we could take the predictive model out of the coders hands and put it into the database.

In every company I have worked with I always bring the set-based philosophy and spread that throughout the database group.  Most recently, I was able to put my theory to the test while in graduate school.  In my “Portfolio & Securities Analysis” class the professor was very curious about data trends in the market.  I built an SQL Server Integration Services (SSIS) package to pull all of the market data since 1965 from Yahoo and Google and put it into a database.  With this data set I was able to start building the stored procedures (calculating alpha, beta, relative strength, etc.) that would have normally been built by a C# developer, and they were all utilizing the set-based power of the database.  I found that the processes were extremely efficient and able to return results on this extremely large data set quickly and accurately.

Here at inqiri, I’m able to implement these same principles into a system that is going to provide utility for MILLIONS of members and companies.  Our site is utilizing collective intelligence to create an extremely large data set, and I can’t think of a better application for set-based processing.  We will be pulling together user ratings on criteria and options, bundling them as a set, and performing  a sophisticated analysis with our machine learning algorithms, and returning the optimized result to our members.

The collective intelligence aspect of inqiri is vital to our concept and through set-based processing, we’re able to bring that intelligence to the decision-making process in an efficient manner using the true power of a database system.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>