Guide to Manage Esri Jobs using Queue Control

Many of our users work with ArcGIS formats in FME to handle data validation, transformation, and translation. This all occurs when moving data into and out of their systems. However, ArcGIS has to be installed and licensed on the machine running FME for Esri-based Readers and Writers in FME (like the Enterprise Geodatabase format) to work. This works well on Desktop. Customers will typically have something, like ArcGIS Pro, installed to use locally. This makes it easy to share the license with FME. However, if you’re using FME Server, this can get complicated.

Here, I will cover the basics of how FME Server and its Engines work. I will also touch on how to assign jobs to server queues so that Esri-related jobs can be processed on specific engines.

The Basics of FME Server and Its Engines

If you’re not familiar with it, FME Server is a part of the FME platform that provides you with powerful tools to automate workflows created with FME Desktop. A key component of FME Server are the Engines. One instance of FME Server can have many engines connected to it either installed all on one computer or distributed across several machines.

So, what does this have to do with using ArcGIS formats in FME? If you want to use FME Server to run your Esri-based jobs, you need to ensure the Engines connected to it have access to an installation of ArcGIS (more information here). This is easy if you have all your Engines installed on the same machine. Simply install and license the required Esri Software there. All your Engines will be able to use it.  

Although, what if you have a distributed installation of FME Server with Engines installed on multiple hosts? Do you really want to install Esri software on each of those hosts? Do you have a mix of Esri-based and regular jobs to run on FME Server? You can configure one Engine machine with an ArcGIS license and use FME Server Queues control to handle sending ArcGIS jobs to the correct set of engines.

What is Queue Control in FME Server?

Queue control settings provide flexible ways to control how FME Server processes jobs and uses the available engines. Queue Control is made up of three components. These components work together to ensure jobs are routed to the correct engines for processing: 

• Job Routing Rules
  This determines which queue any submitted job should be sent to.
• Queues
  This holds jobs sent to them from job routing rules and directs them to engines based on engine assignment rules.
• Engine Assignment Rules
  This controls which engine(s) are assigned to each queue.

Job Routing with Queue Control

In this example, you have FME Server set up with four engines. They are distributed with two on one host machine and two on another. 

You have some jobs that use formats that require Esri software to run. You also have many others that do not. You have installed ArcGIS on one engine host but not the other. In this scenario, you need to ensure that any jobs that require ArcGIS software run on the correct engine. 

You can use Queue Control in FME Server to help manage this. Here are the steps:

1. Create a Queue for all your Esri-related jobs. For our purpose, we can create a new queue called ArcGIS
   
2. You need a way of assigning all the Esri jobs to the ArcGIS queue. This can be handled in a few ways.
   (a) If you have specific Repositories set up to store the Esri jobs, you can set up a Job Routing Rule to send all jobs from those repositories to the ArcGIS queue
   (b) Alternatively, the queue can be set when configuring the job. This is useful if there are specific jobs that need to be sent to the ArcGIS queue. For example, if you use the job in an Automation or a Schedule, you can set the Job Queue from the Advanced options for the Workspace.
3. Finally, configure the Engine Assignment Rules to attach the correct engines to the ArcGIS queue. In this case, we can set up the rule to send all ArcGIS jobs to one host name using the Engine properties since that one engine host has the ArcGIS software.

Looking at the Engines page, all engines will have a property with the host name for the engine machine.

We can use this property value when setting up the Engine Assignment Rule.

Now that we have set up these queue rules, we can go to the Engines page to check on our configuration. It should look similar to this:

On that note, we have covered the basics of how FME Server and its Engines work in this blog post. We also went through how to assign jobs to server queues so that Esri-related jobs can be processed on specific engines. Join the FME Community to keep up to date with the latest tips and tricks. Did you have any more questions? We are always here to answer them.

[Further Reading] Using ArcGIS with FME Server

[Further Reading] Top 10 Reasons to Join the FME Community