How real-time data works & why you need it

Before getting started, there are a few things related to real-time data that need to be well understood such as data velocity, the type of processing option best suited for your needs, and overall, how to handle the inevitable issue of maintaining separate systems producing or using your real-time data.

While there are various solutions and benefits of each processing option available, it’s important to choose a technology solution that makes it easy for real-time data to be converted, transformed, and analyzed in ways that allow for flexibility and reduce both operation and training costs.

What is Real-Time Data?

Real-time data is data that is generated and delivered for immediate processing from sources such as sensors, mobile devices, social media, and applications. Some examples of real-time data include telemetry measurements, the location of moving objects (vehicles, mobile devices, etc.), or notifications of malfunctioning systems.

Real-time data may come from fixed sensors or applications. Some examples include:

• Sensors: river flood gauges, equipment temperature, proximity alerts, or alarm situations such as fire, intruders or equipment malfunction, or system overload
• Applications: IT monitoring systems, stock trading systems, page view events, or status of your web server

One of the most popular and significant producers of real-time data are mobile devices like smartphones. Everyone is now walking around with mobile devices containing a myriad of sensors providing new opportunities for companies to connect with team members and user communities like never before.

In short, real-time data is everywhere and growing fast. It’s becoming more apparent that the opportunities for organizations to maximize the value of real-time data are immense.

Types of Real-Time Data

Real-time data falls into two categories: event data and data streams.

• Event data: self-contained messages that are generated when well-defined conditions arise.
• Stream data: continuous, unbounded data that has no discrete beginning or end. Typically data records are small in size, but volumes can be extremely high.

The main difference is that data from an event is discrete, while data from a stream is continuous.

The table below compares examples of real-time event data vs a data stream from the same source.

What is consistent between both types of real-time data is the desire is to process the data as quickly as possible.

If the time between data generation and processing is short, then the data is “actionable”. If the data is processed later, value will only exist when reporting and analyzing historical data and will not be immediately “actionable”. Still valuable, but not the same as being able to take immediate action.

Benefits of Using Real-Time Data

Processing real-time data quickly is advantageous because it gives organizations the ability to make crucial decisions quickly in an ever increasingly competitive landscape.

Here are just some of the advantages to organizations that leverage real-time data:

1. Closer contact with customers and your team

Today, it’s customary for everyone to have mobile devices. Leveraging this by using real-time data exchanges enables you to keep in closer contact with both of them.

From a customer perspective, this means that you can more readily meet their needs and learn about their challenges.

From a mobile workforce perspective, you can leverage location and other information to dispatch the correct resources more efficiently to help your clients.

2. Faster responses and increased awareness to changes in monitored systems or environments

There are many examples of how real-time data has been leveraged for monitoring such as:

• Earthquakes
• Highways
• Weather events

3. Respond quicker to changes in the operation of key systems

Using real-time data enables organizations to detect and respond to changes in their systems thus improving the efficiency of their operations.

For example, using sensors to monitor devices can detect issues before they fail, thereby identifying when maintenance needs to be performed. Take monitoring water pumps. The sensors measure both the water level and the pump temperature. When the pump temperature rises, it’s an indication that the pump needs to be serviced or replaced.

Data Velocity

You can’t talk about real-time data without talking about data velocity. Data velocity is the speed at which real-time data is generated. It’s the velocity of the data that dictates how the data is to be processed.

If events occur at a low rate or periodically, they can be handled as discrete real-time events which we will refer to here as “real-time event processing”.

If events occur at a higher rate, then a stream architecture is needed to process the data in real-time.

Processing Real-Time Data

There are several ways that organizations can leverage real-time data.

Batch Processing of Stored Event Data

In this approach, real-time data is stored and processed at a specified interval. Depending on the specified interval and the data velocity, batch processing may not be able to meet the real-time requirements of many systems. At best, this approach can be considered “near real-time”.

How long is too long? That depends on the system being monitored. Regardless of the environment, the shorter the interval the better for gaining actionable insights.

While this method can work well for processing stored data such as monitoring daily transactions in a retail store on an hourly or quarter-hourly basis, it can be an inefficient and expensive solution for real-time data. However, in many cases, organizations must work with what they have, and batch processing can be a good first step towards processing real-time data.

Real-Time Event Processing

Here, each event is handled separately with connections between events being stored in persistent storage. This is often referred to as complex event processing.With the FME Platform, this can be done with Automations in FME Server where an incoming event is used as the input data and trigger to deploy a workflow. The processing for the event could be very simple or very complex combining information from the event with additional data assets. You are also able to scale your processing power based on your needs. This is an effective and cost-friendly way to process data because it uses a “pay by CPU time” model.

Data Stream Processing

This type of real-time processing is ideal for high-velocity data streams. It is a method that allows organizations to quickly deliver insights across massive datasets on a continuous basis.

In the case of stream processing, real-time data can be processed in milliseconds. Due to the continuous nature of data streams, stream processing is an ongoing task, whereas real-time event processing is performed at the time that an event occurs.

Stream processing can also be done with the FME Platform using Streams in FME Server. Watch Introduction to Data Stream Processing to learn more.

Things to Consider for a Real-Time Solution

There are a few essential things to be aware of when selecting any real-time processing system.

Fault Tolerance and Redundancy

It is imperative that any system consuming real-time data has both fault-tolerant deployment and an easy way to build in redundancy.

Redundancy is key as you must have the capacity to handle potential errors or failures to process real-time data without missing a beat.

FME supports a full fault-tolerant deployment and we recommend having extra processing capacity, or node engines, available in case of any node processing failures or nodes being temporarily removed from the system.

Scalability

A common question from those wanting to work with real-time data is “How much processing power do I need?”The answer is it depends on both data velocity and how much processing you are wanting to do for each event or data stream. When deciding on a solution, consider how you would expand the processing capacity if needed. Processing capacity can change over time due to an increased stream velocity or if you want to do more processing per event. With Dynamic Engines in FME, you can simply add and remove engines to accommodate changing demands.

Variable Rate Streams

Another common problem is some real-time data and their data velocity varies, and it may be difficult to predict when the velocity is going to change. In this case, you need an architecture that supports the adding and removing of capacity to meet these needs or a system in which you can easily deploy to meet the peak loads.

Maximizing Value

At the end of the day, organizations are looking for the most cost-effective solution that enables them to maximize the value of data. And that includes all data, regardless of how it is delivered or captured.

When it comes to real-time data processing, you often have to choose systems whose strengths are in event processing or data stream processing. Even if these solutions come from the same vendor, it is often the case that they are based on different technology and this can be a challenge. If a low-velocity process later becomes a higher velocity stream, you are looking at a painful and costly reimplementation.

One of the strengths of the FME Platform is that it’s built to handle all kinds of data and all kinds of data delivery. So, whether you want to do batch processing, transactional processing, real-time event-processing, or data stream processing, FME can handle the solution you choose. This means that if workflows are moved from one processing model to another, the effort required to alter the workflows is minimal.

Get Started

The explosive growth of real-time data is truly exciting and opens up a whole new world of opportunities for organizations.

To get started with processing real-time data, download the latest version of FME or learn more with a free trial.

Already have FME? Great. Watch our webinars on Empowering Real-Time Decision Making with Data Streaming or Introduction to Data Stream Processing to see how you can use FME to process your real-time data more effectively.

Go ahead and create your own solutions using FME. Learn, share, and build pilots or proof of concepts with your complimentary licenses.

Please don’t hesitate to reach out to us with questions and be sure to share your experiences. We are always excited to learn from you.