Skip to main content

kJube

Search This Blog

ETL is IO bound

on February 18, 2010

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

... which doesn't stop Pentaho Data Integration from optimally using all the CPU power it can get out off that old Windows 2000 hag we are developing on :-)

Data Integration Fun and fail Pentaho

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Popular Posts

user defined command line parameters in the kitchen

Hello, Just recently I stumbled upon a way to pass custom command line parameters to PDI jobs that are executed through kitchen. Somehow, this isn't documented in the user documentation , so I thought a few lines were in place. According to the Pentaho documenation, kitchen supports the following series of command line parameters: file: xml file (job) to execute log: log file level: logging level (Nothing, Error, Minimal, Basic, ... Rowlevel) rep: repository name user: user to log in to repository pass: password to log in to repository listdir: list directories in repository listjobs: list repository jobs listrep: list available repositories norep: don't log into repository dir: set repository directory job: repository job to run export: file to export a job/trf too (including all related jobs/trf) These are however all standard parameters. If you want to pass your own parameter into the job you are launching, I thought I was down to postional arguments, ho

Table compare - test automation

Background Some time ago we released a series of kettle plugins as part of the Kettle Francising Factory . The plugins ( kff-plugins3.jar for PDI version 3.2.3 and above and kff-plugins4.jar for PDI 4.0.0 and above) are available on code.kjube.be but sofar not much documentation has been provided. One of the kettle steps that is available in those jar files is the Table Compare . This step does what its name says. It compares the data from two tables (provided they have the same lay-out. It'll find differences between the data in the two tables and log it. We developed this plugin for acceptance tests scenarios in large projects. Consider (as a hypothetical example :-) ) the following use case of an data integration tool migration. Suppose you have a data warehouse which is being loaded using, well, let's say, Oracle Warehouse Builder . Now Oracle has bought Sunopsis in 2006. Since then the development of OWB has somewhat stalled :-). Sometime after Oracle has

KFF & Cookbook

When we ( Matt and myself ) kicked off the KFF project , one of the things we wanted to include was auto-documentation. Matt made a small proof of concept to generate PDF documentation but that didn't reach finalization. At the same time, while Matt , Roland and Jos were busy writing Pentaho Kettle Solutions (Sorry, what were you thinking? You do not own a copy of this book? ), the subject of auto-documentation came up too. Roland picked it up, and turned it into the kettle cookbook , a great auto-documentation tool for kettle . Why did we want auto-documentation in the first place? The reason we wanted auto documentation is that we are lazy . We know this . Actually we have known this since a long time. So we needed a solution that would minimize effort on our side. Also it often turns out that customers do not really want to pay for documentation. They just see that as part of the development process and certainly don't want to pay for any time you put into docum

Powered by Blogger