How to Move Data in the Cloud with APIs

It is now a case of ‘survival of the fittest’ for web services. APIs give you the ability to migrate data and build integrated systems so you can take advantage of services with new functionality or lower costs. This flexibility keeps cloud companies on their toes, and means you can use the best services for your needs.

In Part 3 of this blog series we share an example of how we used APIs to migrate our knowledge content. In Part 4, we share how and why to build your own API.

What are the main challenges when moving data in the cloud?

Two data challenges tend to occur when shifting operations into the cloud. Using APIs is the key to solving both.

1. The Initial Bulk Upload

(How to get data from an on-premise source into the cloud, or move data from an existing cloud service)

Migrating data in bulk, either from on-premises infrastructure or from another service, can take significant effort. And it’s crucial to ensure as much data as possible is mapped from the original data source to the new service. Considerations include:

• Renaming attributes
• Cleaning and validating data
  • Removing duplicates
  • Truncating data
  • Removing special characters
• Merging data from multiple sources into one new schema

2. System Integration

(How to connect new services with existing business processes once the new service is up and running)

Enterprises are leveraging web services to save time and money, but the penalty is a highly fragmented enterprise. It is therefore imperative that you are able to connect these web services. The following are important when integrating services:

• A scheduling tool that enables you to automate your connection workflows
• A cloud based deployment so you don’t have to worry about managing further infrastructure
• Fault tolerance and monitoring

For example, our iPaaS platform FME Cloud leverages over 15 services. Without this combination of services, we wouldn’t be able to deliver the platform as it is. This resulted in fragmentation of our data, but fortunately they each have an API so we were able to build an integrated system.

What kinds of data integration tools are available to work with APIs?

You don’t need to be a hardcore developer to work with APIs. In fact, coding isn’t necessary with data integration tools, and they can save you a lot of time and effort. Two types of tools exist for data integration with APIs: point-to-point and flexible solutions.

Point-to-point Solutions

With the explosion of web services came the explosion of tools to help you move data between services. The majority of these tools provide point-to-point integration, a solution that solves one specific challenge. There are, however, significant limitations with point-to-point solutions.

1. Limited data transformation capabilities
The data transformation component is often baked into the connector with little or no control over transforming the data as it moves between applications. This puts you at risk because if you wish to transform the data in a non-standard way, you will need to do custom development.

2. Short term savings, long-term pain
Once you have spent time training staff in how to use the software, there is no way to reuse that knowledge and apply the logic to other integration workflows.

3. Maintainability
In the short-term, point-to-point offers an enticing promise of lower IT maintenance costs, but as more one-off integrations pile up, complexity and costs rise dramatically.

4. Monitoring
As you add more and more point to point integrations it becomes increasingly hard to monitor and ensure reliability.

Flexible Solutions

Flexible data integration tools are single investments that can accommodate multiple new applications without users having to learn new concepts or build new components. Flexible tools also allow you to transform data as it moves, which means you can use your data exactly how it’s needed. Choosing a tool that provides flexibility, like FME, is a crucial part of delivering long-term data integration architectures.

How does this work in practice?

In the next blog post in this series, we’ll dive into data migration and system integration in the cloud with an example of how we recently moved our knowledge platform from Salesforce/Trello to AnswerHub/Auth0.

*

To learn more about APIs, check out our API webinar where we show how to connect to virtually any web service in the world using FME (no coding required).

If you have comments about this post, please share your thoughts in the comments below.