London's Duplicate Image Problem: How the Capital Stacks Up Against New York, Amsterdam and Tokyo

London's public bodies are sitting on millions of duplicate digital images — photographs, scanned documents, architectural records — stored redundantly across servers at a cost that Jisc, the UK's higher education technology body, has previously estimated creates storage inefficiencies running into tens of millions of pounds annually across the public sector. The problem has quietly become one of the more unglamorous headaches in the capital's digital infrastructure, and how London handles it is now being watched by counterparts in New York, Amsterdam, and Tokyo.

The timing is not incidental. The Starmer government's drive to digitise planning and housing records — central to its push to accelerate development approvals under the Planning and Infrastructure Bill — has forced local authorities to confront archives bloated with repeated assets. Duplicate images of the same Southwark streetscape or Tower Hamlets development site filed under different reference numbers slow down searches, inflate storage costs, and, in some cases, cause genuine administrative errors when planners pull what they believe is the most current photograph of a site.

What London Is Actually Doing

The Museum of London Archaeology (MOLA), based at Mortimer Wheeler House in Byng Place, Bloomsbury, has been among the more proactive institutions. It began rolling out perceptual hashing software across its digital asset management system in early 2025, a technique that flags near-identical images even when file sizes or metadata differ. The process identified redundant records across thousands of excavation photographs taken along the Elizabeth line route between 2010 and 2022.

The Greater London Authority's digital team, operating out of City Hall on Kamal Chunchie Way in Newham, has been piloting a deduplication protocol tied to the London Development Database, the GLA's central planning record. The pilot, which began in January 2026, targets image assets submitted by developers as part of planning applications — a category notorious for repeat submissions of near-identical renders and site photographs across multiple application stages.

Transport for London's heritage archive, which holds more than 1.2 million digitised images going back to the early twentieth century, has a longer deduplication backlog. Publicly available TfL digital strategy documents reference image library rationalisation as a medium-term goal, but no completion date has been announced.

How London Compares to New York, Amsterdam and Tokyo

New York City took an earlier, more centralised run at the problem. The NYC Department of Records and Information Services completed a deduplication sweep of its municipal photograph archive in 2023, consolidating holdings across the La Guardia and Wagner Archives and the main repository at 31 Chambers Street. The exercise cut redundant digital storage by a figure the department's annual report placed at roughly 22 percent.

Amsterdam has approached the issue through its Open Data initiative. The Amsterdam City Archives, housed in the Vijzelstraat building, operates a public-facing deduplication tool that allows researchers to flag duplicates in the Beeldbank image database directly — a crowdsourced layer that the city's archivists credit with identifying tens of thousands of redundant entries since 2021.

Tokyo's approach is more centralised still. The Tokyo Metropolitan Archives adopted a mandatory deduplication standard for all submissions from ward offices in April 2024, requiring digital assets to pass automated hash-checking before ingestion. The policy covers images submitted in connection with the city's substantial earthquake preparedness documentation programme.

London, by contrast, has no equivalent citywide standard. Each borough maintains its own digital asset systems, and coordination between bodies like the GLA, Historic England — which holds the National Record of the Historic Environment from its office at Cannon Bridge House near Cannon Street — and individual London boroughs remains ad hoc.

For organisations and individuals submitting planning or heritage documents in London, the practical upshot is straightforward: ensure images submitted to the London Development Database or to borough planning portals carry consistent and unique metadata, including GPS coordinates and a date-stamp embedded in the file, not just the filename. MOLA has published open guidance on its website for community archaeology groups navigating the same problem. The GLA's pilot is expected to report findings to the London Assembly's planning committee before the end of 2026, which may finally push the question of a unified standard onto the formal agenda.