This work was done in collaboration with Safe Software partner Veremes
One of FME’s greatest strengths is its ability to connect to data wherever it lives, whether that’s in the cloud, a remote database, or an external API. But working with remote data efficiently requires more than just connectivity. To get the most out of your workflows, it’s all about how you design them.
Here are five simple and effective ways to streamline remote data processing in FME so your work stays fast, clean, and scalable.
1. Process data where it lives
If you’re working with data stored in a database, one of the best things you can do is let the database handle the heavy lifting.
Rather than pulling entire tables into FME to join or filter them, use SQL queries or the DatabaseJoiner transformer to perform operations directly at the source. This minimizes network traffic and takes advantage of your database’s built-in performance features like indexing.
2. Apply filters at the source
Filtering your data early, before it enters your FME workspace, not only improves performance but also keeps your workflows simpler.
Use a WHERE clause in your Reader when working with attribute data, or go a step further with FeatureReader to combine attribute and spatial filtering. This way, you’re only working with the data that’s truly relevant to your task.
3. Cache files from the web when needed
FME makes it easy to read files from a URL, which is helpful for accessing updated datasets without manual downloads. But when the data doesn’t change often, downloading it every time can slow things down.
By enabling caching in the Reader or FeatureReader, you can avoid repeated downloads and enjoy faster runs. This is especially helpful for scheduled or iterative processes.
4. Take advantage of built-in connectors
FME comes equipped with Connectors for popular cloud platforms and APIs like AWS S3, Azure Blob Storage, and ArcGIS Online. These tools simplify access to remote systems, handle authentication, and reduce setup time.
For less common APIs, you can still build custom integrations using HTTPCaller alongside Transformers like JSONFlattener or XMLFragmenter, giving you full flexibility when working with external data services.
5. Choose the right format for cloud efficiency
When working with large files in the cloud, the format you choose can make a big difference. Some modern formats are designed for partial access, so FME can retrieve only the data it needs without downloading the entire file.
Consider using:
- COG (Cloud Optimized GeoTIFF) for imagery
- COPC for point clouds
- ZARR for scientific or multidimensional arrays
These formats are a great match for remote workflows and help keep things efficient.
Bonus: Keep FME close to the data
If your data is stored in the cloud, running FME in the same cloud region, such as with FME Flow Hosted, can reduce latency, cut down on transfer costs, and boost overall performance.
Learn more about the ways FME can help you by visiting the solutions page. Check out Veremes’s partner page to discover more about their journey with FME
