Using Azure services for messaging protocols
How can we use Azure Service for Messaging Protocols? Messaging protocols or services can help to solve many different problems. At a high level, they solve two main problems:
- Message Processing: Support decoupled application architectures by providing a way for the components to communicate in a reliable and scalable way.
- Stream Processing: Support stream processing workflows. Messaging services that support this build on traditional messaging processing frameworks, adding support for complex stream processing operations and the ability to go back in time.
There are many different messaging protocols and brokers, but how do you know which is the right one for your project? In this blog, we are going to focus on the Azure Cloud.
The main messaging brokers and protocols.
Overview of the Main Brokers
Before we look into Azure services, especially Azure messaging services, let’s review the lower level brokers and protocols that Azure supports. Azure supports the following messaging services on its platform: MQTT, Kafka, AMQP 1.0. Let’s take a look at the main use cases for each of these message brokers.
What is MQTT?
MQTT is a lightweight messaging protocol for IoT devices. It is designed for low-powered devices with minimal network bandwidth. An MQTT client (e.g. sensor) device connects to an MQTT broker (e.g. IoT application) for bi-directional communication.
In FME, you can use the MQTTConnector to connect to any MQTT broker directly.
What is Kafka?
Kafka is a framework for stream-processing. It provides a high-throughput and low-latency platform for handling real-time data.
In FME, you can connect to a Kafka broker via the KafkaConnector.
What is AMQP 1.0?
AMQP 1.0 (Advanced Message Queueing Protocol) is a messaging protocol that is typically used for reliable message passing between applications. Being a wire-level protocol, AMQP 1.0 is designed to interop with various operation systems, frameworks, and languages without locking into a specific vendor. In Azure, it is the primary protocol for Azure Service Bus and Azure Event Hubs.
Azure Messaging Services
Now let’s look at the messaging services on Azure and how you can connect to them with FME. Understanding these protocols allows choosing the right Azure Service and corresponding FME connector for your application.
Azure IoT Hub
The Azure IoT Hub enables you to manage your IoT devices—think of it as a message broker for your IoT devices and application. It is optimized for connecting IoT devices to receive messages and events like sensor data which can be time-series or discrete facts. Azure IoT supports the MQTT and AMQP protocols. For low-powered devices, MQTT is often used since it is lightweight. The IoT Hub can also be used to send commands back to IoT devices.
An example of device-to-cloud communication is temperature data from a sensor monitoring a cooling system. The IoT device publishes these events to the IoT Hub which then can process these messages to take an action. A command then is sent from applications to devices, say to turn on a backup system if needed if the temperature gets too high.
With FME, you can use the AzureIoTConnector to connect to Azure IoT Hub with the MQTT protocol.
Azure Queue Storage
The Azure Queue storage service is a typical message processing service. Communication to queue storage is through the HTTP protocol. It is ideal for events that hold commands as a payload. Usually, the message includes a set of instructions to be executed within a time frame that is up to the processing client to execute.
A common scenario is that a message is pulled from a queue by the processing client. Once the client is done it reports back to the queue that the message was processed and that it can be removed from the queue.
An example would be uploading images for resizing. The command or the instructions in the payload of the message include a source destination for the image, how it should be resized and a destination to store the processed image. Azure Queue Storage is useful for simple message queue applications and if a queue’s size might exceed 80 GB.
FME integrates with Azure Service Bus via the AzureQueueStorageConnector.
Azure Service Bus
Azure Service Bus is a messaging platform as a service that supports AMQP 1.0. Use cases for Azure Service are very similar to Azure Queue storage and focus on the command-like events which are pulled off a queue by clients and processed individually. The major difference to Azure Queue Storage is the feature richness of Azure Service Bus, namely the support for Queues and Topic/Subscriptions.
Queues are used for point-to-point delivery where senders and receivers all have the same function. Fanout scenarios use a topic and subscription model to one topic, connected to multiple subscriptions via filters. Receivers can receive messages via subscriptions and make sure they only receive messages they are interested in.
An example for using Azure Service Bus, is an application that receives orders from an online store and sends a message to a topic service bus. Receivers can then subscribe to that topic to then take action on that order. A subscription can then send an email or update inventory. If your application is only interested in orders from a specific store, the receiver can filter to receive messages from that store.
FME integrates with Azure Service Bus via the AzureServiceBusConnector.
Azure Event Hub
Azure Event Hubs supports the Kafka and AMQP 1.0 messaging protocols. Azure Event Hubs is recommended when working with a real-time event stream. Existing Kafka clients can be connected to an Azure Event Hub, providing an alternative to deploying your own Kafka cluster.
Azure Event Hub is optimized to allow a high number of messages per second and is ideal for processing data either in real time or analyzing data received during a certain time frame.
An example is using Event Hubs to track click streams from websites. Using the data the user’s journey can be analysed. Events are sent to Azure Event hub and then are processed in parallel downstream by the use of partitions.
FME integrates with Azure Event Hubs via the AzureEventHubsConnector, which is using the AMQP 1.0 protocol. With the azure pricing structure, it may be cheaper to use the AzureEventHubsConnector to connect to an Event Hub rather than using KafkaConnector. The connector has the additional feature to store checkpoint information of where the receiver is in the stream. That way if a receiver disconnects from a partition the checkpoint can provide a way to recover and start receiving at the same place again.
Try it Yourself
With FME you can seamlessly integrate with the Azure platform and transfer data between your applications. Download a free trial of FME today or check out these resources to learn more about Azure and FME:
