I target customers who have large data processing needs. These come in various forms, but generically look like this: the customer gets huge data drops in some form or another and must process the data and output results in a very specific time frame. The customer has written some scripts, maybe some code and SQL. They have attempted some optimizations that helped a little, but they're not meeting their timeline. They have opportunities to take on even larger processing jobs, but don't have the capacity. They need help, now!

This is not an uncommon scenario. What to do? And what does this have to do with Java? Good questions. Hold onto the Java question, I'll get to that next. First, there are many products and frameworks for processing large amounts of data (such as relational database management systems or RDBMSes). But the vast majority of data that I see from day to day is not in a database. Data, even very large data, is usually exchanged between businesses in the form of files. Processing this data with single-threaded scripts or other code is just not working anymore. Making many passes through the data with SQL isn't working either.