FME Evangelism #16: Raster Special Edition

Contents:
1) Raster To Vector Coercing
2) FFS Support for Raster Data
3) Visualizing Rasters
4) Persistent Cache and Raster
5) Transporting and CheckPointing Rasters
6) New Raster Formats
7) Raster Cell Origins
8) Raster Performance
9) Even more raster stuff…

Introduction
Hi FME’ers

This edition of the FME Evangelist is a special raster edition.

I know I wanted to vary topics in each post, to give something for everyone, but there’s so much raster info to go over that I can compile it into a single posting and still have enough left over to sprinkle into future posts. I have to say that our raster support is now very solid in FME, and well worth taking a look if you’ve not used it before. In particular I think being able to combine vector and raster data together, especially in FME Server applications, is going to be a big hit.

Regards,

Raster To Vector Coercing
Vectorizing raster data is a useful function that has just undergone a name change: RasterCellCoercer is the new name for the RasterToPointCoercer transformer. The reason it has been renamed is a very good one: the transformer now supports coercing raster data into polygons as well as points.

In the new ‘polygon’ mode a single polygon is output for each cell in the original raster. This results in a large number of polygon features – potentially useful in itself – but which you could dissolve together (based on the original cell value) to create a more manageable and realistic vector dataset.

In this example on fmepedia (screenshots below) a raster DEM dataset is converted into vector polygons using exactly this method.

FFS Support for Raster Data
I’m pleased to announce that as of FME 2009 beta build 5568 the FFS format is able to store raster features.

Rasters stored to an FFS file will have their data written to a corresponding .frs (FME Raster Store) file. One FRS file may hold the data for multiple raster features. FRS files hold up to 2GB of raster data; if the raster data surpasses the 2GB file size limit the data is automatically split across multiple files.

Above: A CDED raster dataset converted to FFS

Raster FFS datasets are used in FME (e.g. opened in the FME Viewer) by selecting the FFS file [though I tried opening the frs file and it did work – I just had to manually select the FFS format]. If you select an FFS file and the corresponding FRS file is missing then FME will just use the bounding box of the raster.

OK – writing raster to FFS might not seem like your idea of a good night out; however, it does integrate raster better with FME and open up some exciting benefits…

Visualizing Rasters
Did you see the previous FME Evangelist? If so, you may remember that:

“When you send data from Workbench to Viewer using a Visualizer transformer, what happens is that the data is simply written to a temporary FFS format dataset which is subsequently opened in the Viewer.”

In older versions of FME you’ve not been able to send raster data directly from Workbench to the Viewer – all you get is the dataset outline – because in between the data is written to FFS, and FFS doesn’t support raster.

But wait a moment! Hold the front page! FFS does now support raster.

So you are now able to direct raster data through the Visualizer transformer and get the correct view. One of those features a new user might say “yeah, so what?” about, but which an existing user will see as a great improvement.

Persistent Cache and Raster
Ever seen the FME Viewer option View > Options > View Persistent Cache and wondered what it does? Well turning it on causes all the data you open in the Viewer to be saved as FFS data in a temporary directory, so that every time you refresh the view or query the data you’re working on a local copy. That way you’re not having to re-read the source data, which can be a lengthy process if it is a web or database dataset.

Obviously this excludes raster, because raster can’t be saved as FFS. But wait! Hold the… er second page! FFS does now support raster.

So now using the Persistent Cache option in Viewer becomes a valuable time saver when inspecting raster data. The FME Viewer can even be shutdown and started at a later date and retain all the local copies of the data and thus the rapid access to the raster data.

The one word of warning is that this will create large amounts of data in the local temporary directory, for example, “C:/Documents and Settings//Local Settings/Temp/FMEUniversalViewer” – so make sure you have a lot of disk space, and try use your fastest disk.

Transporting and CheckPointing Rasters
Wait! Stop…. actually don’t stop the press on account of these functions; not many users are likely to have heard of these – let alone use them. But the ability to write raster to FFS also:

• Permits support for raster data in the TransporterReceiver and TransporterSender transformers.
• Lets the RasterCheckpointer transformer save the current state in FFS (rather than GeoTIFF) which means that any aspect of raster data expressible in FME can now be accurately stored on disk and restored without data loss.

…and trust me, these are important updates, even if part of slightly less used features.

New Raster Formats

FME 2009 has new support for a couple of raster formats; the first of which may be of interest even to folk who are not generally raster users.

The Shuttle Radar Topography Mission (SRTM) Height format (short name SRTMHGT) is DEM data created by NASA. FME now has a reader for this format. A writer is not planned. The nice thing is that data in this format is freely available at NASA’s repository (though it wasn’t working when I tried it last)

Image Courtesy NASA/JPL-Caltech.

The data comes in two levels: SRTM-1 with data sampled at one arc-second intervals in latitude and longitude, and SRTM-3 sampled at three arc-seconds.

Datasets are divided into individual height files that each contain one by one degree latitude and longitude tiles. Height file names refer to the latitude and longitude of the lower left corner of the tile, for example, N37W105 has its lower left corner at 37 degrees North latitude and 105 degrees West longitude. Height files have the extension ‘.hgt’ and express the height values in meters. These files always have the LL84 coordinate system.

One gotcha is that FME is currently unable to read height files that do not conform to the specified file naming convention, since the file name is the only means of reading the proper referencing for the file.

ER Mapper ERS format (short name ERS) is the internal raster format used by ER Mapper. FME now has a reader AND writer for this format.

The ERS format consists of a header with a .ers extension and a separate raw binary data file that can have any extension. The header file is considered the dataset in FME and contains information about the contents of the binary data file.

Limitations are that the ERS writer currently cannot write nodata values, and that the reader does not support ERS files containing algorithms for image creation and manipulation.

Raster Cell Origins
As you know, a raster is made up of cells, each of which is a single point but which in fact covers a small area. What’s less known (perhaps) is that each cell has an origin point which identifies the representative location of each cell. The origin is an x/y pair between 0.0 and 1.0 inclusively. The default value in most cases is (0.5,0.5) indicating the center of the cell.

Cell origin is particularly important when the cell value is taken from a single position – when the value is the average value of the area the cell covers then cell origin is less important.

FME2009 includes new functionality to set a raster’s cell origin to specified X and Y values. The transformer is the RasterCellOriginSetter and the underlying function is @RasterCellOrigin

Why would you need to do this? Two reasons spring to mind.

Some formats of raster data don’t store cell origin; it is assumed to be a certain location (often upper-left corner, which is counter-intuitive for many map users). So this new tool allows a user to properly adjust data read from those formats, if they so desire.

The second use case is during raster-to-vector, or vector-to-raster, operations. In these scenarios you want to be sure that cells converted to points (or vice versa) end up in the correct location. If you don’t correctly set the cell origin you could find the resulting output is out by up to half the width of a cell.

If you don’t know what the current cell origin is, you can retrieve that information into attributes using the RasterPropertiesExtractor or by inspecting data using the Logger transformer.

Raster Performance
If you’ve not tried an FME2009 beta yet, it might be worth doing just for performance reasons.

Our raster team tells me that several common raster operations take less time to execute in FME2009.

Specifically, raster resampling and rotation should now be faster in all cases. The magnitude of improvement may vary from translation to translation, but these operations should be approximately 30-50% faster than in previous versions.

Raster reprojection benefits primarily when dealing with multi-band rasters. The common case of reprojecting a 3-band raster is now up to 50% faster.

Even more raster stuff…
If you have used FME2009 you may well have noticed the new RasterExpressionEvaluator transformer, and the new Raster clipping capabilities. Since they are so important I’ll talk about them another time: for now check out Dmitri’s RasterExpressionEvaluator examples for some absolutely stunning uses of this transformer, and try for yourself clipping raster features into non-rectangular shapes (very simple and effective).

This Edition of the FME Evangelist…

…was written to the tune of “Only You” by Yazoo.

I’ve been re-watching The Office (UK version) and the great Tim/Dawn romance. Here’s the end of the second series, and here’s the end of the following Christmas special where “Only You” is the background music.

Or, if you only want the music itself, here’s the original.