Friday, October 16, 2015

A paper a day keeps the doctor away: The 8 Requirements of Real-Time Stream Processing

In recent years there has been an explosion of data all around us. The data comes in from a variety of sources, such as financial real-time systems, cell phone networks, sensor networks--RFID and IoT, and GPS. Commensurate with this dramatic increase in data, is a corresponding unquenchable thirst for analysis and insights. The natural question arises: how do we build systems that process and makes sense of this vast amount of data, in as close to real-time as possible? What patterns of software and systems should we look at?

Michael Stonebraker of database fame et al. offer some advice on what to consider in their paper: "The 8 requirements of real-time stream processing" published a decade ago. In the paper, the authors list eight guiding principles that high-volume low-latency systems should follow to be able to process vast amounts of data in near real-time.

First, the systems have to keep the data moving, and do straight-through processing with minimal to no writes to disk to achieve the low-latency desired. The authors compare passive (polling) systems versus active (event driven systems) and recommend the latter.

Second, the authors recommend supporting a high-level language--dubbed StreamSQL, with built-in extensible stream oriented primitives and operators to process the data instead of writing custom code in languages such as C++ and Java.

Third, the system has to handle stream imperfections such as delayed data, missing data, or out of order data, and have timeouts for potentially blocking data to ensure system liveness.

Fourth, the system has to integrate stored and streaming data, to be able to reprocess data when necessary.

Fifth, the system has to generate predictable outcomes and repeatable results, such as when it needs to reprocess data for recovery, or handling duplicate data.

Sixth, the systems have to guarantee data safety and availability, with uninterrupted fail-over between primary and backup systems ala "Tandem-style" computing.

Seventh, the system has to partition and scale applications automatically, between cores and across machines to be able to seamlessly handle any increase in load.

Finally, the system has to be quick, process and respond instantaneously to streaming data, which requires careful planning and coding to minimize boundary crossing, and maximize the ratio of useful work to computation overhead.

The authors examine common architectures that fulfill parts of the requirements they listed above including databases (DBMS), rule engines that are built on condition/action pairs, and stream processing engines. They present in tabular form where the systems excel at, and where they don't. The table leans toward using stream processing engines instead of DBMS which are not optimized for the task.

Despite being a decade old, the paper is still relevant, and referenced in the modern literature. Moreover, it is well written and a pleasure to read.


Friday, October 2, 2015

Why do you need to warm diesel engines

In the Pacific Northwest, a lot of people use their trucks as everyday commute vehicles, which makes sense, since the climate is wet, and the terrain hilly, and in wet and cold conditions people feel safer in their four wheel drive vehicles. Some of these trucks are the heavy duty ones, with big diesel engines, and lately I have noticed at work a couple idling sans driver for at least 5 minutes. It got me curious about why would you need to idle a diesel engine especially since modern gasoline engines do not require idling before driving off and putting load on the engine. A web search helped piece the answer to this puzzle.

Diesels operate differently than gasoline engines. Instead of relying on spark plugs to light up the air and fuel mixture inside of the engine cylinders, diesels rely on high compression ratios that cause the air and fuel mixture inside of the cylinders to ignite.  Because of the high compression ratios, diesel engines are typically bulkier and more sturdy than their gasoline counterparts. Moreover diesels typically operate with a higher thermal efficiency than gasoline engines, which means less heat is dissipated to heat the engine block, and the lubricating fluids.

Both factors mean that the engines need a bit more time before they operate at their sweet spot. This translates to longer idling time before the engine can sustain load. The modern diesel engines have technological advances that help minimize the idling time, such as engine block heaters, and higher idling RPMs. The EPA website contains some useful information about modern diesel engines

As an aside, it turns out if you own a diesel truck, you have to be good at managing your time, since you cannot just turn on the engine in cold weather and get on with your life when you're running late; you have to wait for a little bit till the engine is ready.