September 3, 2010

kettle history lesson

A VERY TRUE tale about the real origin of kettle a.k.a. Pentaho Data Integration.

It is the year 2000. Most of the IT world is wondering what to do after the Y2K bug (except for counting the $$ wasted or earned). In Europe, the CEE reacts by introducing 'the euro'. The switch of national currencies to unified Euro currency generates again work for the whole of the IT sector. In the US votes are counted and recounted to ensure that Bush Jr can become president. Slobodan Milosovic on the other hand disappears from the stage. In Russia Vladimir Putin is the new man. Hopes grow that maybe the human genome mapping will explain why these people actually come to power? Microsoft is judged a monopoly by District Judge Thomas Penfield Jackson;  he orders  the company to be split up. (Bill Gates and Microsoft will go to the supreme court). In the mean time Bill starts touting the idea that your kitchen will become Microsoft powered. In Belgium a business intelligence specialist wants to snatch that idea away from Bill and starts to hack in his basement in an attempt to out-code Microsoft. In less than no time, he launches a series of products named kitchen, pan, spoon, kettle, menu, waiter ... that will shake the world.

Yes, kettle, now know as Pentaho Data Integration, was originally intended as a kitchen automation system. Or did any one actually believe that those kitchen terms were really intended as a metaphor for data integration technology? Come on, did you seriously buy that story? No no no, the real goal of kettle 0.1.3 - the first version I came across - was to be able to code kitchen recipes using spoon, and have kitchen execute them. A full automated kitchen process, the perfect support for the modern working woman!

For those who find this hard to believe, I have found an original .ktr from that period, know as the cake mix! This shows clearly what the original intentions for kettle were. The screenshot of the .ktr is (as usually is the case with a kettle transformation) completely self-explanatory.

I've opened it using kettle 3.2.3. As amazing as it may seem, backward compatibility for PDI seems to go all the way back to that version 0.1.3. I even tried to add a piece of functionality in the end to invite the very active Pentaho Community Members to the cake party. It integrates perfectly.

An interesting aspect, to which I want do draw your attention is the KQL, Kitchen Query Language. I had completely forgotten about this feature until I rediscovered this piece of code. KQL was a kind of GIS implemenation on house level, or rather on kitchen level, allowing to specify every shelf in your kitchen closets, fridge and yes even basement. Ladies and gentlemen, we are (or were) talking 3 dimensional Kitchen GIS (BTW, project name was KIS) here and that more than 10 years ago. This query language actually unlocked all physical objects in your kitchen through a syntax very similar to SQL. Example screenshot below.

In retrospect, I don't know what most readers think about this, but I find the whole set-up of this project mind-blowing and extremely far ahead of its time. But unfortunately the project was aborted ... 

So what happened to the original plans for kettle? Well seemingly there were a few issues with the business plan for kitchen automation, which made our business intelligence specialist rapidly change his plans. (For your information, Microsoft is also slowly discovering that MS in every kitchen machine is not turning out a huge success.)
  • One major issue with the concept was that by automating kitchen tasks you free up time for the people who do the kitchen work. No statistics are needed to show that (even until today) it is mostly women that do the kitchen work. A consequence of all this free time, was that more time was spent on shopping. This made the investment in kettle - kitchen automation system actually a very expensive adventure instead of an intended saving. Once male investors understood this negative (perverse) effect of the kettle kitchen automation on their household income they massively abandoned the project.
  • Fridge APIs turned out to be a true headache. Most of the electronics producers were investing a lot in the reduction of CFC emission, amongs others through intelligence leak detection systems. Because of the competitive edge such systems could give them, most producers were not willing to open up access to the internals of their systems.  
What happened finally is history. Our guy re-applied his coding to the field he was most familiar with, namely business intelligence and data integration. He open sourced his efforts and Pentaho recognized his genius. Today every data integration specialist is moving over from Datastage, Informatica, BODI, OWB, ... to Pentaho Data Integration.