FME Blog

How to Connect to Google Cloud in FME

Stewart Harper

April 2, 2020•6 min

Learn about the Google Cloud services you can connect to in FME 2020, like storage, databases, analytics, and more, plus deploy FME in the Google Cloud Platform Marketplace.

Over the past few years, Google has established itself as a global leader in the cloud computing space. The numbers speak for themselves, with Google Cloud generating over $8 billion in revenue in 2019, up 53% year on year. This is why we at Safe Software have focused on improving your ability to connect to Google Cloud services in FME.

Below is an overview of the Google Cloud services supported in FME 2020. This blog is part of a 3-part series on how to connect to cloud services with FME. In the coming weeks, we’ll be publishing blogs about Azure and AWS.

Connect to Google Storage Services

Google’s object storage allows data to be accessed via an API.

FME’s GoogleCloudStorageConnector transformer allows you to access the Google Cloud Storage service to upload, download, delete files and folders, or list file/folder information. Typical uses include:

Manage datasets by uploading, downloading, and deleting files and folders.
Read downloaded data using the FeatureReader, or upload data written by the FeatureWriter.
Retrieve file and folder names, paths, links and other information from Google Cloud Storage to use elsewhere in a workspace.

Connect to Google Database Services

Google’s fully managed database service includes MySQL, PostgreSQL and SQL Server.

Using FME’s readers/writers (MySQL, PostgreSQL, SQL Server), you can read and write data to all of these services. These readers/writers are spatially enabled, which means you can read and write geometric data too.

If you need to execute SQL queries against one of these databases in the middle of a translation, you can use the SQLExecutor transformer instead of a reader/writer. This transformer can extract database records, execute database joins, perform SQL database table operations (create, drop, modify, truncate), and create/drop indexes or constraints before or after data loading.

Connect to Google Data Analytics

Cloud Pub/Sub

Google’s real-time messaging services, Cloud Pub/Sub, allow you to send and receive messages between different applications.

FME’s GoogleCloudPubSubConnector transformer allows you to connect to the Google Cloud Pub/Sub service to create topics and subscriptions or send and receive messages. Typical uses include:

Processing data streams that have been sent to Google Pub/Sub. For example, a sensor stream may be streamed to Pub/Sub.
Processing event logs from other Google Cloud services or applications.
Move data from Google Pub/Sub to Big Data their analytics platform
Google Pub/Sub directly integrates with Google Cloud Storage so you can use it to watch for new data arriving there.

BigQuery

Google’s serverless data warehouse service, BigQuery, lets you analyze petabytes of data. There are two ways to interact with Google BigQuery within FME.

FME’s GoogleBigQueryConnector is typically used to load CSV data into new or existing BigQuery tables, as well as to query BigQuery tables and optionally load the result to a specified destination BigQuery table.

FME’s Google BigQuery Writer allows you to write large volumes of data to BigQuery tables. BigQuery also supports spatial data via the Geography data type, which stores data based on a spherical coordinate system. FME supports all BigQuery data types, including Geography.

Connect to Google Artificial Intelligence

Google’s pre-trained computer vision model allows you to quickly classify images into millions of predefined categories.

FME’s GoogleVisionConnector allows you to interact with the Vision AI service. You can use this to detect and classify objects in an image, detect faces in images, and read printed and handwritten text.

Google’s pre-trained NLP model allows text to be analyzed for the language type and the sentiment.

FME’s GoogleLanguageConnector allows you to interact with the Natural Language API. You can use this to understand the overall opinion, feeling, or attitude sentiment expressed in a block of text, as well as to detect the language of the text.

Connect to Google Internet of Things

Google’s Cloud IoT Core allows you to securely connect, manage and ingest data from millions of IoT devices. Both MQTT and HTTP can be used to communicate with Cloud IoT Core.

FME’s MQTTConnector can be used to read and write data to Cloud IoT Core. This transformer supports both stream and batch mode. If you initiate in stream mode, a persistent connection is created and hundreds of messages per second can be read.

Deploy FME in Google Cloud Platform Marketplace

When working with large volumes of data, deploying FME close to the data is critical. To make it as simple as possible to deploy the FME Platform in Google Cloud, we are listing both FME Desktop and FME Server in the Google Cloud Platform Marketplace.

The first offering we have made available is the FME Platform on a Windows virtual machine. With a few clicks, this allows you to launch a virtual machine into your own Google Cloud account that has both FME Server and Desktop pre-installed. To use your existing FME Desktop and/or FME Server license, simply enter your serial number(s) in the FME Desktop Licensing Assistant and/or the “Licensing & Engines” page in FME Server. If you’d like to request a trial license, get started here.

With the ability to connect to a range of Google Cloud services in FME, coupled with the ability to deploy FME close to your data to help you overcome data gravity limitations, your data integration and application integration workflows have never been easier to deploy. Give these formats and transformers a try, and as always, let us know what you think by sharing your thoughts in the FME Knowledge Center!