This work was done in collaboration with Safe Software partner miso.
Watch to discover how you can optimise your data processing, save on cloud costs, and boost your speed with FME.
Managing large datasets in the cloud can be expensive and slow if you’re relying on traditional data processing methods. These often involve transferring data between cloud and on-premises environments for processing.
While this does have some advantages, there are significant drawbacks. Transferring large amounts of data between cloud platforms like AWS (Amazon Web Services) and Microsoft Azure incurs significant transmission costs—not only through direct transfer fees but also due to the necessity for higher-priced internet bandwidth. Latency is another critical factor; you have to wait for the data to be transferred before processing it. Additionally, while cloud providers implement strong security measures, keeping data within a single environment reduces the inherent risks associated with transferring data over the internet.
Using FME remote engines can drastically reduce both time and expense by processing your data right where it lives – on the cloud. In our video, we explore how you can halve your cloud transfer costs and double your processing speed with FME.
Traditional Data Processing
Here’s our scenario, you have 4 million records stored in a PostgreSQL database on AWS, and your FME processing power is running in a different environment – in this case Azure. So, we want to transfer the data from AWS to Azure for processing and then send it back to AWS. Typically, you would transfer data from the cloud to an on-premises location for processing however, it could involve any combination of cloud platforms.
In our tests, this method took about 7 minutes.
Optimizing with FME Remote Engines
Enter FME Remote Engine Services—a solution that allows you to process data within the same cloud environment where it’s stored. By deploying an FME remote engine on a virtual machine in AWS, you eliminate the need to transfer large datasets. Instead, your data processing happens directly within AWS, leveraging the power of FME’s data integration capabilities while still being orchestrated by FME Flow on Azure.
Performance Comparison: Doubling Processing Speed
When we processed the same 4 million records using an FME Remote Engine within AWS, the results were impressive:
- Over 2x Faster: The task completed in just 3 minutes—less than half the time of the traditional method.
- Cost Savings: Eliminated data transfer fees between AWS and Azure, significantly reducing cloud costs.
It also makes this process far more secure by minimizing exposure to external networks by processing data locally.
Conclusion
Using remote engines allows you to optimize your cloud-based data processing workflows by eliminating costly data transfers between platforms like AWS and Azure. By processing data locally, you can half your transfer costs and reduce processing times—enabling your organization to be more efficient and agile.
Learn more about the ways FME can help you by visiting the solutions page. Check out Miso’s partner page to discover more about their journey with FME.
If you want to learn more about what FME can do in the Utilities sector, please check out our utilities solutions page.
