FME and Google Gemini AI: The perfect solution for large amounts of unstructured data

Powering millions: UK Power Networks tackles a century of data

How does an electric utility bring over a century of service records into the digital age? By tackling the challenge of modernizing over a million historical documents, including one dating back as far as 1907.

The oldest card from 1907

UK Power Networks (UKPN) delivers electricity to over 8 million homes and businesses across London, the South East, and East of England. As part of its ongoing modernization efforts, UKPN needed to unlock valuable information in a massive archive of over a million physical, historical service record cards. These records, dating back to 1907, documented electrical installations, connections, and maintenance but were stored as paper cards that included handwritten notes, sketches, and early computer printouts. Manually extracting the necessary data would be an impractical, multi-year undertaking.

Automated data extraction using FME and Google Gemini AI – the next evolution in optical character recognition

To overcome this challenge, UK Power Networks collaborated with Avineon-Tensing, a Safe Software partner with data integration and automation expertise. The team developed a solution leveraging the FME platform’s data integration capabilities and Google’s Gemini AI Flash models’ advanced generative AI power.

FME orchestrated the entire process, cataloguing scanned records stored in Azure Blob Storage and placed them in an Azure Queue for processing. FME then encoded the images and generated a JSON prompt for the Gemini Flash model to extract the dates from the service cards. 

The team then used the FME Feature Joiner transformer to compare a list of the original files against a list of processed files so that anything outstanding was placed in the queue for re-processing.

Parallel processing for maximum throughput

Avineon-Tensing implemented a parallel processing approach using FME to achieve high-speed processing, allowing for the simultaneous processing of multiple service record cards and significantly accelerating the data extraction process. Using an Azure queue ensured that each record was only taken by a single FME Engine, with a rate of 20 records being processed per second. The outputs from the Generative AI were delivered in a JSON format, which FME extracted and wrote out to an Azure SQL Database, making the data accessible for analysis and use in other UKPN systems.

As a result, nearly 1.1 million service record cards were processed in just 26 hours, with approximately 700 cards processed per minute. UKPN experienced massive time savings, as the project would have taken 19 years if manual data extraction had been the only option. The project was also cost-effective, with the total cost for the Gemini AI model API being in the range of just a couple hundred pounds, with hundreds of thousands of pounds being saved. UKPN now has a valuable historical dataset in a structured format that is ready for analysis and use in planning and operations. The latest Gemini 2 models also allow for even more accurate extraction from service cards in the future.

For a deeper dive into the story, including demos and workspaces, watch our webinar: All-Data Any-AI Integration: Innovations with FME and Google.