140 million MOT driving records analyzed with FME

The challenge of processing data on almost every vehicle in Great Britain

Field Dynamics, the sister company of Safe partner miso, created a report that analyzed 140 million Class 4 Ministry of Transport (MOT) driving records to get one of the most accurate views of average vehicle mileage across England, Scotland, and Wales. They wanted to uncover valuable insights about annual mileage per vehicle and the mileage differences between vehicle types and models.

To create the report, the Field Dynamics team needed to process four years’ worth of data for almost every vehicle in Great Britain. To do this, they used MOT tests, a legally required test for cars usually over three years old, to calculate the difference in mileage per vehicle between each test.

Field Dynamics needed a way to handle a massive amount of data, validate it, and load it into a PostgreSQL database quickly and efficiently.

They used FME to validate, sort, filter, and integrate the massive datasets

The Driver and Vehicle Standards Agency (DSVA) publishes MOT data, but since the data is not classed as an ‘official statistic’, the UK Statistics Authority does not assess it for anomalies. Field Dynamics found that some vehicles were inaccurately recorded to have negative mileage, and some were recorded to have driven over a billion miles in a year. FME cleaned the obvious data anomalies and implemented the logic rules needed to load a high-quality dataset into the PostgreSQL database.

With such large datasets, data validation and preparation would have proved a significant challenge. FME’s no-code transformers and high-performance engine made short work of schema standardization, attribute replacement, filtering, data aggregation and more, readying the data for use.

Without FME, Field Dynamics’ analysts would have had to manually validate, segment, and clean and load the data. Field Dynamics used the time saved to create an insightful report that challenges perceptions around how much the average vehicle travels in Great Britain.