FME Blog

How to Connect to Microsoft Azure in FME

Gerhard Fischl

April 7, 2020•7 min

Learn about the Azure services you can connect to in FME 2020, including Microsoft’s storage, database, notification, analytics, cognitive, and IoT services. Plus, deploy FME in the Azure Marketplace.

Microsoft Azure continues to grow in popularity, especially among enterprise customers. Here at Safe Software, we want to enable all of our customers to choose their preferred cloud service, which is why we have expanded our offerings to support commonly used Azure services.

Below is an overview of the Azure services supported in FME 2020. This is part two of a three-part series on Cloud Services you can connect to using FME. Read last week’s blog about Google Cloud and check back next week for a post about AWS.

Connect to Azure Storage Services

Connect to Azure Storage Services using FME’s Azure Storage Package. This includes several transformers:

Microsoft’s object storage, Azure Storage, allows data to be accessed via an API & SMB.

FME’s AzureBlobStorageConnector allows you to access the Azure Blob Storage service to upload, download, delete files and folders, or list file/folder information. Typical uses:

Manage datasets by uploading, downloading, and deleting files and folders.
Read downloaded data using the FeatureReader, or upload data written by the FeatureWriter.
Retrieve file and folder names, paths, links and other information from Azure Blob Storage to use elsewhere in a workspace.

This transformer also allows you to access an Azure Data Lake Storage Gen2 directory to upload, download, delete files and folders, or list file/folder information. This is typically used to store data for Big Data analytics.

Azure Data Lake Storage Gen2 is supported via the Azure Blob API.

The AzureFileStorageConnector allows you to access the Azure Files service to upload, download, delete files and folders, or list file/folder information via the REST API. You can manage the Azure file share service via the API when the file share can not be mounted.

The AzureQueueStorageConnector allows you to access the Azure Storage Queue service to send, receive, peek at and delete messages in a Azure Storage Queue. Typical uses include:

Receive messages from Azure Storage queue and process the data within FME.
Processing a high number of large messages with multiple clients to enable load balancing.

Azure Table Storage

Microsoft’s table storage, Azure Table Storage, allows data to be accessed via an API.

FME’s Microsoft Azure Storage Table reader/writer allows you to read and write data to Azure Storage tables. The reader also supports a wide range of filtering capabilities by supporting filter strings based on the Open Data Protocol. Typical uses include:

Storing hierarchical data in a NoSQL database.
Efficiently querying unstructured and schema fluid databases.

Connect to Azure Database Services

Microsoft’s fully managed database service includes SQL Server, PostgreSQL, and MySQL/MariaDB.

Using FME’s readers/writers (Azure SQL Server, PostgreSQL, MySQL/MariaDB), you can connect to all of the systems supported by Microsoft’s database service. These readers/writers are spatially enabled, which means you can read and write geometric data too.

If you need to execute SQL queries against one of these databases in the middle of a translation, you can use the SQLExecutor transformer instead of a reader/writer. This transformer can extract database records, execute database joins, perform SQL database table operations (create, drop, modify, truncate), and create/drop indexes or constraints before or after data loading.

Azure Cosmos DB

Microsoft’s fully managed NoSQL database, Azure Cosmos DB, supports automatic scaling and geo-replication.

FME’s Azure Cosmos DB reader/writer allows reading and writing large volumes of unstructured data to Azure Cosmos DB with a predictable low latency. This is typically used for storing JSON items in a highly available NoSQL DB, as well as for efficiently querying schema-free data via SQL with predictable performance.

Connect to Azure Notification Services

FME supports Azure Event Grid as a Trigger and an External Action in FME Server Automations, which enables you to integrate your Azure environment with complex automated workflows.

The Azure Event Grid Event Received trigger provides a webhook that can be used in an Azure Event Grid subscription to receive event notifications and FME Server and trigger Automations.

The Azure Event Grid External Action allows you to trigger an Event Grid topic from within an Automation.

Typical uses include:

Integrate FME Server workflows into an Azure environment.
Trigger FME Server Automations based on events like file uploads to Azure Blob Storage.
Trigger FME Server Automations with Azure Functions.

Connect to Azure Data Analytics

Microsoft’s Azure Event Hubs is a fully managed, real-time data ingestion service with Kafka compatibility.

FME’s KafkaConnector can be used to send and receive data from Azure Event Hubs. This transformer supports both stream and batch mode. If you initiate in stream mode, a persistent connection is created and hundreds of messages per second can be read. Typical uses include:

Geoenrichment of stream data that has been sent to Azure Event Hubs.
Stream data to Azure Event Hubs with FME for analytics.
Aggregation of data from Azure Event Hubs.

Connect to Azure Cognitive Services

Microsoft’s pre-trained Computer Vision & Text Analytics machine learning models allow you to quickly classify images into millions of pre-defined categories & text to be analyzed for the language type and the sentiment.

FME’s AzureComputerVisionConnector allows you to interact with the ComputerVision AI service. You can use this to detect and classify objects in an image, detect faces in images, and read printed and handwritten text.

The AzureTextAnalyticsConnector allows you to interact with the Text Analytics API. Use this to understand the overall opinion, feeling, or attitude sentiment expressed in a block of text, as well as to detect the language of the text.

Connect to Azure Internet of Things

Azure IoT Hub allows you to securely connect, manage and ingest data from millions of IoT devices via MQTT.

FME’s MQTTConnector can be used to send and receive data from Azure IoT Hub. This transformer supports both stream and batch mode. If you initiate in stream mode, a persistent connection is created and hundreds of messages per second can be read. Typical uses include:

Processing data streams that have been sent to Google Pub/Sub.
Geoenrichment of IoT data.
Aggregation of data from Azure IoT Hub.

Deploy FME in the Azure Marketplace

When working with large volumes of data, deploying FME close to the data is critical. To make it as simple as possible to deploy the FME Platform in Azure, we are listing both FME Desktop and FME Server in the Azure Marketplace.

Once you launch your virtual machine with the FME Platform installed, connect your existing FME license(s) by entering your serial number(s) in the FME Desktop Licensing Assistant and/or the “Licensing & Engines” page in FME Server. To request a trial license, get started here.

With the ability to connect to a range of Azure services in FME, plus to deploy FME close to your data, we hope to facilitate your data integration and application integration workflows. Give these formats and transformers a try, and as always, let us know what you think by sharing your thoughts in the FME Knowledge Center!