FME 2016 Sneak Peek: Transformer Versions

This post introduces a really significant function new to FME2016: transformer versioning and updating.

For the sake of completeness, I’m going to cover what a transformer version is, and why it is (or isn’t) important. If you already know this – perhaps you read this previous article – then you can skip the first few sections.

Why is updating transformers necessary?

In short, it’s because your “workspace transformer” becomes out of sync with what we might call the “core transformer” – the core definition in FME.

When we implement a new transformer (like the FeatureWriter in FME2016), it starts out as version 0 (zero). When you use this transformer, you place a “workspace transformer”, also version 0 (zero).

If Safe Software makes updates to the “core transformer”, we increase the version number to 1, 2, etc. But, your “workspace transformer” does not get updated. It stays at version 0.

Why is this? Well, we don’t do automatic updates to your workspace transformers* because it could start producing different results! It’s basic backwards compatibility – a workspace should always produce that same result, whichever version of FME you upgrade to.

So, over time your “workspace transformers” start to lag behind updates and fixes to the “core transformer”. However, it’s important to know that it’s not always a big deal…

Why it doesn’t matter!

You’d think transformer updates were a big deal – fixing bugs and the like – but really that’s not the case.

That’s because transformers are just access points into the ‘factories’ and ‘functions’ that make up the lowest-level functionality of FME. If that functionality is updated or fixed, your existing transformer will make use of it, regardless of version.

Think about it: it’s like saying that because a hole in the road got fixed, you have to go out and buy a new car!

Basically we don’t create a new transformer version for low-level fixes. They get applied anyway. Your existing transformer will use the fix just fine.

Your existing car will benefit from road repairs, just as much as a new one would.

We only create a new transformer version when there is new functionality to expose. Perhaps we’ve added a new parameter or setting that you can use to control the transformers behaviour. Then you would need to upgrade to use that capability.

If an electric recharging point was added to your street, then you would have to get a new electric car to use it. But you equally don’t have to. You can choose to stick with what already works for you.

You might feel a little anxiety (transformerschmerz?) that your workspace transformers are out of date. But if you have a workspace that is functioning correctly, and you’re happy with that, there’s no real need to update anything.

Similarly, I feel anxiety that I drive a ten-year-old Dodge minivan, when I should be driving a Triumph TR5 in British racing green. But I don’t really need the speed or acceleration it brings, so perhaps I’m better off sticking with what I know works for me.

Why it does matter!

Of course, sometimes it is important to have up-to-date workspace transformers.

Firstly – and most obviously – it’s because there is new functionality that you can take advantage of. For example, the Sampler transformer now (in 2016) has an option to do randomized sampling; previously you sampled in strict order but will use this because randomization will improve the results. Therefore you need to update your transformers to the latest version.

Or maybe you already built randomization into your workspace, and you can use this new capability instead, replacing 2-3 existing transformers or a Python script.

The second reason for updating is that sometimes the transformer has the same functionality, but has a redesigned GUI that makes it easier to use – recent updates to the SchemaMapper are an example of this. It does the same thing, but in a much more user-friendly way (click to enlarge).

All of which is fine, provided you can identify which transformers need updating. It’s not always that easy, especially in a workspace of several hundred transformers!

The Functionality

So, if you open FME 2016 you’ll see that the Navigator window now shows a section of which transformers are now out of date, and could be updated.

The nice part is that you can see what version the workspace transformer is, and what the latest version of the “core transformer” is – for example, the AttributeFilter used in the workspace is version 1, whereas the latest version available in FME is version 4. I can also see the number of transformers in this state (there are two out-of-date Inspectors).

To upgrade a transformer (here a Counter transformer) I just have to right-click on it on the Workbench canvas:

After a warning (that you can choose to turn off permanently) you are shown a dialog like this:

This is where you are presented with the changes to the transformer parameters dialog, and can make use of any new functionality exposed in the new version. In the above, it looks very similar – as if the only changes are cosmetic and there is no new functionality. However, by clicking a drop-down arrow I can see that the Counter Name parameter now allows either a conditional value or an attribute value – which it never did before.

Oh, and while I remember, you also get any changes you make recorded (and capable of being undone) in the new(ish) history window:

(Yes, the VertexCreator message looks odd – a little rewording is required there!)

The What’s New Window

Did you notice on the above dialog that there was a button labelled “Show Changes”?

That’s where you can find out exactly what has changed between the two versions. For example, the Counter changes dialog looks like this:

…proving that the Counter Name parameter was updated to accept attribute values.

Deprecated Transformers

One of the other transformers in my workspace was a 3DPointReplacer:

But that doesn’t exist any longer, does it? So what does the upgrade option do? It replaces it with the new equivalent – the VertexCreator transformer.

NB: Among our final few changes we’ll probably make the transformer get renamed in this scenario, so there is no confusion.

When there is no real equivalent to a deprecated transformer, then it won’t be upgradable. The AppearanceAdder is the only one I know of so far. It’s newer equivalent – the AppearanceSetter – is so much more advanced that there can be no 1:1 exchange.

Limitations

I think functionality is great; but there are a few limitations.

For example, you can’t just click a button and upgrade every single out-of-date transformer at once. That was a deliberate decision. Because of the changes that a new transformer has (like extra parameters or different input/output ports) we feel you should really upgrade each individually to deal with these changes.

As for custom transformers, this functionality will work on transformers inside a custom transformer definition. But, for the same reasons as above, you won’t be able to just click on a custom transformer (especially a linked one) and expect everything inside it to be updated.

Finally, some transformers use a wizard for their parameters dialog (the FMEServerJobSubmitter springs to mind) and so we can’t show you a single dialog that compares old parameters with new. A few other transformers with specialized dialogs also can’t be shown in this way (the AttributeFilter appears to be one of these)

Your Chance to Try

All of the above screenshots were taken using an old workspace I downloaded from an old incarnation of FMEpedia:

So rather than just reading what I have to say about this functionality, why not download this workspace and open it in an FME2016 beta to see how this all works for yourself.

The first thing you should do is just run the workspace. Notice that it runs just fine and produces the correct result. You don’t need to update it if you are happy with what it is doing.

When you do start to update transformers, you’ll notice the 3DPointReplacer and PointConnector get an extra output port for Rejected features, and the AttributeFilter has extra ports too that you might need to deal with. The 3DPointReplacer is, in fact, replaced by a VertexCreator and the AttributeSetter is replaced by an AttributeCreator.

The only transformer not needing an update is the AttributeCreator – because I added that due to a fault in the workspace logic that I only just noticed!

Conclusion

The message I can’t emphasize enough is that – while you might find out-of-date transformers in your workspace – you don’t have to update them. Really, only if you identify a particular need should you – otherwise you’re risking a change in behaviour for no benefit.

But if you do need to update transformers, then this tool should make it way easier to identify which ones are affected, what has been changed, and then automatically replace them with the newest version. I hope you find it useful,

* NB: Just as a point of interest, although I said transformers don’t upgrade to a new version automatically, there are a couple of exceptions (e.g. the SQLExecutor) that can update themselves when you open the parameters dialog. As you would expect, we’re really, really, really careful to make sure that the behaviour doesn’t change when that happens.