What Is Real-Time Data Streaming—and Why Should You Care?

As IoT deployments scale, your company's ability to analyze data as you collect it will become an increasingly important competitive advantage. Here's what you need to know about real-time data streaming.
By Mary Catherine O'Connor
Nov 10, 2015

No company deploys Internet of Things technology because it just wants to use more sensors. It's the data those sensors capture that matters. Yet, just capturing data is not enough. In order to avoid a business loss—a manufacturing process coming offline, for instance, or a turbine at a power plant failing—sensory data needs to be analyzed in real time, so workers can respond to anomalies that are the harbingers for these types of losses before such losses occur. Simply collecting and storing sensory data, in industrial applications such as these, does not provide any real business value.

"IoT introduces wrinkles [into your operations] because data is arriving at a high rate," explains Steve Ehrlich, senior VP of marketing and product management at Space-Time Insight, a data-analytics service provider. "And it's not just coming in fast, but also very often it's dirty, and most companies are not necessarily geared toward dealing with that. So analytics are often required to clean the data before you even decide what to do with it."

IoT assets are analyzed and visualized for a range of conditions, such as excess speed, driving off-road, potential failure or sitting idle. (Click on the above image to view a larger version.)
Since before the current proliferation of low-cost, networked sensors, companies in the industrial sector have collected and stored time-based data relating to machines or processes in what are termed operational historian databases. Applications would then analyze this stored data to generate metrics.

"A decade ago, customers with big data would use a data-warehousing approach and look to get reuse from dense stores of data—looking at customer transaction data, for example," says Chad Meley, VP for unified data architecture product and services marketing at Teradata, a provider of data-analytics services.

Over time, those applications evolved into what are known as complex event-processing platforms. Then, as the volume of data grew, the enterprises realized they wanted to analyze it as they were collecting it, resulting in the emergence of tools for advanced streaming-data analytics. This type of software performs three important functions: analyzing a wide variety of event- and time-based data types (a system could consume data from a wide range of sensors, detecting anything from temperature to vibration to light), analyzing it as soon after collection as possible, and comparing the data with models that convey what it should be showing in order to then detect anomalies.

"Typically, the events, such as sensor readings, are ganged together and aggregated in a window based on time or on the number of events," explains Fern Halper, advanced analytics research director at market research firm TDWI. Filters are used to find only the relevant data from the stream and pull that information into these windows that are only open for a certain amount of time. And as the data is compiled, the software runs calculations. "In complex event processing, you'd do things like take subtotals or averages of data coming through. You might want to track temperature—say, so once every 10 seconds, the software captures a window of temperature data and calculates the average. And then, the next window comes through. And with each window, the software decides if an alarm needs to be sounded, based on temperature thresholds that the user has set."

The total volume of the data is high, and is streaming in at a fast rate, so what the streaming-data analytics software is most valuable for is finding the few but important clues the data holds. "Ninety-nine point something percent of the data will show that the system is working as it should," Meley says, "but every once in a while, we'll see that something is spinning too fast or getting too hot."

Simply enter a question for our experts.
Sign up for the RFID Journal Newsletter
We will never sell or share your information
RFID Journal LIVE! RFID in Health Care LIVE! LatAm LIVE! Brasil LIVE! Europe RFID Connect Virtual Events RFID Journal Awards Webinars Presentations