FME Blog

How to connect to Amazon Web Services in FME

Gerhard Fischl

April 9, 2020•8 min

In part three of our Cloud Services series, learn about the Amazon Web Services you can connect to in FME 2020, like storage, databases, IoT, and more.

Since AWS launched its first service in 2004, it has innovatively led the way in the cloud computing space. For many years, FME users have been able to connect to key AWS database and storage services, as well as deploy their FME Server environment in AWS via FME Cloud. With the addition of a new dedicated team at Safe Software working on cloud desktop components, over the past year, we have focused on improving many of the existing transformers and adding support for several new services.

Below is an overview of the Amazon Web Services supported in FME 2020. This is part three of a three-part series on Cloud Services you can connect to using FME. Check out our other blogs about Google Cloud and Microsoft Azure.

Connect to Amazon Storage Services

Amazon S3 allows data to be accessed via an API.

FME’s S3Connector transformer allows you to access Amazon S3 buckets to upload, download, delete files and folders, or list file/folder information. As of FME 2019.2, this transformer replaces the S3ObjectLister, S3Downloader, S3Uploader & S3Deleter in a single transformer and adds support for any S3-compatible API via the new alternate endpoint and SSL verification parameters. Typical uses include:

Manage datasets by uploading, downloading, and deleting files and folders.
Read downloaded data using the FeatureReader, or upload data written by the FeatureWriter.
Retrieve file and folder names, paths, links and other information from Amazon S3 to use elsewhere in a workspace.

Connect to AWS Relational Database Services (RDS)

Amazon’s fully managed database service includes SQL Server, PostgreSQL and MySQL/MariaDB.

Using FME’s readers/writers (SQL Server, PostgreSQL, MySQL/MariaDB), you can read and write data to all of these services. These readers/writers are spatially enabled, which means you can read and write geometric data too.

If you need to execute SQL queries against one of these databases in the middle of a translation, you can use the SQLExecutor transformer instead of a reader/writer. This transformer can extract database records, execute database joins, perform SQL database table operations (create, drop, modify, truncate), and create/drop indexes or constraints before or after data loading.

DynamoDB

Amazon’s fully managed NoSQL database, Amazon DynamoDB, supports automatic scaling and geo-replication.

FME’s Amazon DynamoDB reader/writer allows reading and writing large volumes of unstructured data to Amazon DynamoDB with a predictable low latency. This is typically used for storing JSON items in a highly available NoSQL DB, as well as for efficiently querying schema-free data via SQL with predictable performance.

Aurora

Amazon’s MySQL and PostgreSQL-compatible relational database for the cloud, Amazon Aurora, supports geo-replication and sub-second data access in any region. Additionally Amazon Aurora offers a serverless option which enables automica scaling.

FME’s Amazon Aurora reader/writer allows reading and writing large volumes of data to Amazon Aurora with a very low latency on a global scale.

Connect to Amazon Notification Services

SQS

FME’s SQSConnector, which we recently reimplemented and improved, allows you to access the Amazon SQS service to send, receive, count and delete messages in an Amazon SQS queue. It replaces the SQSSender, SQSReceiver, SQSMessageCounter & SQSDeleter with a single transformer in FME 2020.0 and adds support for FIFO queues and advanced authentication options. Workspaces using the deprecated transformers can be easily upgraded to use the new functionality. Typical uses include:

Receive messages from Amazon SQS and process the data within FME.
Process a high number of large messages with multiple clients to enable load balancing.

FME also supports Amazon SQS as a Trigger and an External Action in FME Server Automations, enabling you to integrate your AWS environment with complex automated workflows.

The Amazon SQS Event Received trigger allows you to connect to an Amazon SQS Queue to receive and delete notifications with FME Server and trigger Automations.

The Amazon SQS External Action allows you to add messages to an Amazon SQS queue from within an Automation.

Typical uses include:

Integrate FME Server workflows into an AWS environment.
Trigger FME Server Automations based on events like file uploads to Amazon S3.

SNS

FME’s SNSSender allows you to send messages to an Amazon SNS topic. FME also supports Amazon SNS as a Trigger and an External Action in FME Server Automations, enabling you to integrate your AWS environment with complex automated workflows.

The Amazon SNS Event Received trigger allows you to subscribe to an Amazon SNS topic to receive event notifications in FME Server and trigger Automations.

The Amazon SNS External Action allows you to send a message to an Amazon SNS topic from within an Automation.

Typical uses include:

Send & receive message to/from Amazon SNS topics
Trigger FME Server Automations based on events like file uploads to Amazon S3

Connect to Amazon Data Analytics services

Amazon’s fully managed data warehouse, Amazon Redshift, supports automatic scaling and geo-replication and has extensive support for geospatial data.

FME’s Amazon Redshift reader/writer allows reading and writing large volumes of data to Amazon Redshift. As with most of our database/data warehouse formats, you can also use the SQLExecutor transformer instead of a reader/writer if you need to execute SQL queries against Amazon Redshift in the middle of a translation. We’re excited to share that we’re working on adding full support for the new geometry data type added to Amazon Redshift (announced in late 2019) to our future FME release this year. Stay tuned to safe.com as more information and content becomes available.

Amazon Athena, Amazon’s serverless interactive query service, allows users to analyze data in Amazon S3 using standard SQL. Additionally, Amazon Athena supports a long list of spatial functions that can be queried by FME.

FME’s AthenaConnector, new in 2020, can be used to send queries to Amazon Athena, store and reuse saved queries and store results of queries. This is typically used for:

Analyze data in Amazon S3 with standard SQL
Use basic spatial functions to query large dataset and refine the result in FME
Efficiently store query results for future queries

Amazon’s Managed Streaming for Kafka (MSK) is a fully managed, real-time data ingestion service with Kafka compatibility.

FME’s KafkaConnector can be used to send and receive data from Amazon MSK. This transformer supports both stream and batch mode. If you initiate in stream mode, a persistent connection is created and hundreds of messages per second can be read. Typical uses include:

Geoenrichment of stream data that has been sent to Amazon MSK.
Stream data to an Amazon MSK with FME for analytics.
Aggregation of data from Amazon MSK.

Connect to Amazon AI Services

Amazon’s pre-trained Rekognition model allows you to quickly classify images into millions of predefined categories.

FME’s RekognitionConnector transformer, new in 2020, allows you to interact with the Rekognition service. You can use this to detect and classify multiple objects in an image, detect faces in images, and read printed and handwritten text.

Amazon’s pre-trained Comprehend model allows you to analyze text for the language type and the sentiment.

FME’s ComprehendConnector, new in 2020, allows you to interact with the Text Analytics API. You can use this to understand the overall opinion, feeling, or attitude sentiment expressed in a block of text, as well as to detect the language of the text.

Connect to Amazon Internet of Things

AWS IoT Core allows you to securely connect, manage and ingest data from millions of IoT devices via MQTT.

FME’s MQTTConnector transformer, new in 2020, can be used to send and receive data from AWS IoT Core. This transformer supports both stream and batch mode. If you initiate in stream mode, a persistent connection is created and hundreds of messages per second can be read. Typical uses include:

Processing data streams that have been sent to AWS IoT Core topic.
Geoenrichment of IoT data.
Aggregation of data from AWS IoT Core.

Our customers have been using FME to connect to Amazon Web Services for years. Because FME Cloud is hosted on AWS, these services are logical choices for technology professionals looking to take advantage of data gravity. With the recent improvements made by our cloud development team, we hope to continue to remove barriers between your data systems and applications and facilitate your cloud-based workflows.

As always, let us know what you think about these transformers and connectors by sharing your thoughts in the FME Knowledge Center.