The power of dynamic engines
In FME 2020.0, we announced a new way to purchase FME Server capacity called “Dynamic Engines”. This blog will explore what Dynamic Engines are, what types of use cases they excel in, and demonstrate Dynamic Engines in action.
FME Server 2020.1 beta license with 100 free hours
Traditional FME Server Licensing
Since its inception, FME Server has been sold on a per-engine basis. Engines sold with the traditional FME Server licensing model are referred to as standard engines. Each engine is able to process a single FME workspace or “job” at a time. The concept is simple. If you have 5 engines then you can run 5 jobs simultaneously.
To maximize ROI with traditional FME Server licensing,
you strive to keep standard FME Engines as busy as possible.
What are Dynamic Engines?
Dynamic Engines deliver FME Server in a whole new way. Rather than purchasing engines, Dynamic Engines allow you to purchase “engine CPU time” as credits. Many vendors refer to this as usage-based pricing, where you only pay for the work that is done.
Since the licensing of Dynamic Engines is based on usage instead of number of engines, this means you are able to launch as many or as few Dynamic Engines as you want, whenever you want.
Dynamic Engines are licensed by CPU time.
1 Dynamic Engine credit = 1 hour of CPU time.
With Dynamic Engines, you only pay when a Dynamic Engine is doing work. This allows you to have as many Dynamic Engines as you want, ready for work at no cost.
With Dynamic Engines you only pay when “work” is done, not when the engine is idle or waiting for data.
Dynamic Engines give you the flexibility to use FME Server to solve a whole new set of challenges.
Can I Use Dynamic Engines and Standard Engines Together?
Yes! Dynamic Engines are designed to work alongside standard FME Engines and we expect many FME Server deployments will consist of both standard and Dynamic Engines.
Dynamic Engine Use Cases
With the flexibility and cost model of Dynamic Engines, organizations can now solve a whole new set of problems with FME Server.
Dynamic Engines are well suited for workflows that have variable demand or one-time big data lifts. Some of the scenarios where we see Dynamics Engines being valuable include:
- Helping to alleviate demand spikes from popular FME Server Apps
- Supporting complex and mission-critical Enterprise Integration workflows
- As a cost-effective solution for low capacity or variable rate IoT streams
- Enabling data-driven parallel processing for tasks requiring 100’s of engines
However, oftentimes organizations don’t know when their server is going to be overloaded. When it does happen, Dynamic Engines will allow administrators to feel reassured because they will know they have extra capacity to handle those peaks.
Dynamic Engines were designed to address all of the above scenarios.
Example: Dynamic Engines in Action
This demo will walk through how an organization can use Dynamic Engines to supplement standard FME Engines. In most but not all cases, you will want to use standard engines first and Dynamic Engines to relieve the peak loads on your FME Server.
The links to the components of this workflow can be found here on FME Hub.
It is easy to assign Dynamic Engines to address peak load workflows using the FME Server REST API.
How to have Dynamic Engines at the Ready
The workflow below provides a framework for auto reassigning Dynamic Engines to queues that reach a specified threshold. The components are as follows:
Monitored Queues
A “watched queue” and High and Low Watermark
Watched Queue – One or more queues being monitored. When any of the queues hit the high watermark then Dynamic Engines are assigned to the queue.
Watermarks – Thresholds at which Dynamic Engines are assigned (High Watermark) and removed from queues (Low Watermark). In this example, the same watermark values are used for all queues but they can easily be changed to different values for different queues.
Dynamic Engines Home Queue
Dynamic Engines, when launched, are assigned to only one queue known as the “home queue”. This Home queue is where Dynamic Engines wait for assignment to a busy queue. The Dynamic Engines are not assigned to the Default queue.
In this example, the home queue is named auto_assign_queue. The queue is given a low priority.
Dynamic Engines waiting to be assigned to busy queues
Metric Endpoint
FME Server has a metric endpoint that when called with an HTTP GET request, returns current load information about FME Server including the number of running and queued jobs. The returned information follows the OpenMetrics standard. The metric endpoint is called:
$(FME_SERVER_WEB_URL)/fmerest/v3/metrics
So for <queue> the values returned will be of the form:
fme_running_jobs {queue=”<queue>”} <num_running_jobs>
fme_queued_jobs {queue=”<queue>”} <num_queued_jobs>
Putting it All Together
Once you have all of the necessary components, the process is quite simple. A simple Automation is used to orchestrate the process as shown below.
Automation designed to reassign engines to busy queues
The autoscaleV2.fmw workspace does the magic and takes the parameters shown below.
The workspace is included in the project below and a video explaining it can be found here.
Initial State – No Load
Initially, all of the Dynamic Engines are assigned to the Home queue called “auto_assign_queue” as shown below.
Initial state with Dynamic Engines ready to be assigned.
Engines Reassignment Workflow
Next, I will trigger an Automation that submits over 900 jobs. The Automation is shown below.
The Automation is pretty straightforward
- 1 ProcessState workspace is triggered and results in 60 jobs being submitted to the County queue (ProcessCounty Workspace)
- The 60 ProcessCounty workspaces run and result in 900 jobs being submitted to the Block queue (ProcessBlock Workspace)
When this is run you will see the engine’s assignment being changed on the fly. First, they will be assigned to the County queue and then the Block queue as the load changes. Once the queues hit the low watermark, the engines are returned to their original state.
Original Queue State => Assigned to County Queue => Assigned to Block Queue
Dynamic Engines and Parallel Processing
Dynamic Engines coupled with Automations makes parallel processing super easy. Take the example above, where parallelism happens. With FME Server Automations, you don’t have to even think about it. Automations makes it happen.
This capability is especially useful if your business needs to complete large data lifts or migration projects. With Dynamic Engines and parallelism you can spin up 100 engines instantly and process jobs in parallel. The benefit of this feature is that you are only charged for the amount of processing done. Therefore it would cost the same to process this job with 1 engine as it would to use 100 engines.
Parallel processing is easy and cost effective with Automations and Dynamic Engines!
Is the old FME Server Pricing Model Going Away?
No. We like to give our customers more options so they are able to use the power of FME to solve more problems in cost effective ways. Every licensing model has its strengths and weaknesses. We encourage you to embrace the one that best suits your business.
Give Dynamic Engines a Try!
We welcome you to give Dynamic Engines a try. If you want to try Dynamic Engines on an evaluation FME Server, then we suggest you download FME Server 2020.1 Beta.
Each FME Server 2020.1 trial license comes with 100 hours of free Dynamic Engine credits.
If you want to try it with FME Server 2020.0 then contact us at sales@safe.com and we will be happy to give you hours to try.
If you are using FME Server 2019 or earlier, unfortunately, this feature is not available.
Have fun and please give us feedback on anything and everything FME. Whether related to Dynamic Engines or not, we love to hear your ideas, stories, and experiences.
