London's War on Duplicate Images in Public Records: How the Capital Stacks Up Against New York, Amsterdam and Tokyo

Transport for London confirmed last month that its digital asset management systems contained more than 340,000 duplicate image files across its infrastructure maintenance records — photographs of bridges, tunnels and station interiors that had been uploaded multiple times by different contractors over a decade of fragmented digital procurement. The admission, made in a published audit summary circulating among public sector IT teams, has thrown a spotlight on a problem that London shares with virtually every major city that rushed to digitise its bureaucracy without standardising how files were stored.

The timing matters. The Starmer government's planning reform agenda, which is pushing councils to process housing applications faster and move more records entirely online by the end of 2027, depends on clean, reliable digital archives. Duplicate images don't just waste storage — they slow down automated document review tools, confuse planning case officers comparing site photographs across years, and in some cases have caused housing applications in boroughs like Hackney and Tower Hamlets to be flagged erroneously by software that detects apparent inconsistencies in submitted files.

What London Is Actually Doing

The Greater London Authority's data services unit began a formal deduplication programme in January 2026, contracting a Shoreditch-based firm to run perceptual hashing algorithms across the planning portal's image repository. Perceptual hashing compares images by their visual content rather than their file names, catching duplicates even when photographs have been resized, re-exported or renamed. The GLA has not yet published completion figures, but the contract notice, published on the Find a Tender service, lists a budget of £480,000 for the initial 18-month phase covering planning and housing records.

Lambeth Council launched its own smaller programme in March, working through the Historic England archive partnership to clean up listed building consent records on Brixton High Street and in the Clapham Old Town conservation area, where duplicate survey photographs had accumulated since 2015. A council procurement document describes roughly 18,000 image pairs identified for review across those two areas alone. Lambeth is running the work in-house rather than contracting out, using open-source software and two dedicated staff members — a cheaper but slower approach that analysts following local government IT say is more typical of London's outer and inner boroughs with tighter budgets.

How London Compares Globally

New York City's Department of City Planning completed a comparable deduplication exercise across its ZOLA land-use portal in 2024, removing approximately 1.2 million redundant files from a system that covers all five boroughs. The project cost the city roughly $2.1 million and took 14 months. Officials there credited the clean-up with cutting average application processing times by around 11 percent, according to a publicly released programme evaluation from the NYC Office of Technology and Innovation.

Amsterdam took a different route. The municipality made deduplication a mandatory step in its procurement rules for all contractors submitting digital files to the city's omgevingsloket — its planning and environment portal — starting in 2023. Rather than cleaning up retrospectively, Amsterdam built the requirement into submission standards, meaning contractors must certify files are unique before upload. The Dutch approach has been studied by planners in Berlin and Copenhagen, both of which are considering similar contractual mandates.

Tokyo's approach is more decentralised, reflecting Japan's layered ward governance structure. Individual ku — wards — manage their own digital records, and deduplication standards vary significantly between, say, Shinjuku-ku and Setagaya-ku. There is no city-wide programme equivalent to what London or New York has attempted.

London sits somewhere between the American retrospective clean-up model and the Dutch preventative mandate. Critics in the local government IT sector argue that spending nearly half a million pounds at the GLA level while dozens of individual boroughs run incompatible systems is a structural mistake — that without a pan-London submission standard similar to Amsterdam's, the duplicate problem will simply regenerate as new contractors upload new files. The GLA's deduplication contract runs until June 2027, which means results should be visible before the government's own planning digitalisation deadline. Whether the boroughs are brought into a unified standard before that point is the question planners and developers with active applications in zones like the Thames Estuary Opportunity Area will be watching closely.