Test clauses remade: Data filtering in FME 2019

When we, at Safe, say that the Tester transformer is our most important, that’s only half the story.

That’s because test clauses for the Tester (#1 most-used transformer) are defined in a dialog that is used in multiple places in FME Desktop. It’s a key part of the TestFilter (#7) dialog, used in the Data Inspector (under the Filter function), and appears in the Edit Breakpoints dialog (setting conditional breakpoints) as well. Finally, it’s used in any transformer that allows conditional parameter values; for me, that’s mostly the AttibuteCreator (#2) and AttributeManager (#3).

So it’s not really the Tester that is most important. The important part is the creation of conditional test clauses in a variety of places. Therefore if we make changes to that dialog – like we’ve done for FME 2019 – then it’s big news indeed…

Multiple Test Clauses

The main part of the dialog that has changed concerns complex tests, with multiple clauses. Let’s take such a scenario and see how it’s changed.

Let’s say I have a dataset of road and weather information, and I’m wondering whether to send out a snowplough (plow). I’ll do that if:

• The estimated snowfall is greater than 100mm (4 inches) AND
• The road is a major (arterial) road AND
• The road has not been cleared in the last four hours

Notice that ALL of the conditions must be true in order to send the plough. In FME2018 and earlier, there was a Pass Criteria parameter, so we would set the individual clauses and then set the Pass Criteria to All Tests (AND):

In FME2019 the dialog has been redesigned, removing the Pass Criteria parameter and adding it as a Logic column to the main table:

So each entry in the Logic column is set to AND. This is the same as the older All Tests (AND) criteria. If each logic column was set to OR, it would be the same as the older Any Test (OR) criteria:

That’s not a huge change, however, the same mechanism can now be used to set up composite tests…

Composite Tests

A composite test consists of a mixture of AND and OR logic (and maybe more) tying together a series of test clauses.
Let’s take the above example, and add more complexity to the logic of dispatching snowploughs. We now dispatch ploughs when…

• The estimated snowfall is greater than 100mm (4 inches) AND
• The road is a major (arterial) road
  • OR
• The estimated snowfall is greater than 150mm (6 inches) AND
• The road is a less important (secondary) road
  • AND (in both of the above)
• The road has not been cleared in the last four hours

That’s a mixture of AND and OR statements, and in FME2018 it needed a composite test like so:

Notice how the Composite Test is constructed from test clause numbers. It works, but there are two drawbacks: firstly there is the issue of conflating numbers to each test, and secondly, it’s a different mechanism to the simpler scenario. That’s where FME2019 comes in:

What we’ve done is removed the need to match each clause to a number; the logic is now alongside each row. Additionally, it’s all a single mechanism now. The logic column allows you to manipulate test clauses in the same way, whether the overall set is simple or complex.

As an added benefit, the Composite test gains exposure. You are more likely to find that composite capability now than before, and that’s a great thing for new users. We’ve been thinking about new users quite a bit for FME2019, and believe that these changes will make it easier for them to understand this dialog.

For example, a user constructing their first workspace should be able to find this capability and grasp the concept of composite tests much more easily. And a user who inherits complex workspaces from a colleague is much more likely to understand what is happening.

But for you existing users, let’s look at the new behaviour in a bit more detail…

Logic Field Essentials

In the old logic fields, I would type in clause numbers interspersed with AND and OR statements. Now logic is set through a column with a drop-down menu:

There are AND and OR operators there, plus the NOT operator, which is a direct replacement for the “negate” column that used to exist.

You might notice that I used a clause with double brackets in its logic, but that they don’t appear on the drop-down list. That’s because the list is just a set of example strings. You can – and this is really important to note – also type in values! (i.e. you can click into a field in the Logic column and type values.)

That’s how I can recreate a composite test such as:

(((1 AND 3) OR (2 AND 4)) AND 5)

…because I can add extra brackets if necessary (and I often add extras just to be certain of my logic).

Logic Order

It might also have occurred to you that the order of the logic is more important in 2019.
For example, if I’m testing for:

(x AND y) OR z

…then x and y must be consecutive test clauses in the dialog. Basically, the old text box was a separate entity and could refer to rows in any order; but now that individual rows make up the composite logic, their order is all-important.

Comparison Mode

Also, notice that the comparison mode setting used to be a separate value per clause, but is now (by default) a single value for all of them:

I say “by default” because it is possible to change that back to give per-clause modes by using “Specify Per Test”.

Usability

At the very start, I mentioned that many different places use this dialog. They are all updated for 2019 and, as far as I can tell, this takes place without any action on your part. You don’t – for example – need to upgrade the Tester transformers in your workspace to get the new interface.

So new FME users will reap the benefits without having to worry about existing workspaces and – like I mentioned – there are other new-user enhancements like hints on the Main canvas:

…and a new “Workbench Essentials” tutorial to get users started more quickly:

But back to test clauses, we’re hoping that these changes benefit existing users too. It’s always nice to simplify things.

Interestingly, Dale says that the dialog is less “clickalicious” than before, whereas I feel it’s a bit more “clicky”. Who’s right? Well, I await your feedback with interest. It feels like a “Marmite Update” to me, where everyone is going to have a strong opinion, so I expect to hear a lot of comments!

Note: Marmite – for those not in the know – is a sticky, black foodstuff, usually spread on toast. The joke is that no-one can possibly have a neutral opinion about it!

Summary

By the time you read this, the official launch of FME 2019 will be just a matter of days away, and the FME World Tour for 2019 will have already started with FMEDays in Germany. The tour continues, around the world, until mid-June.

There’s still plenty of time to sign up and attend, so please click on the graphic above and check out when we’ll be in your vicinity.