How the FEDORA Catalogue Learned to Read Every Language in Europe
The auto-translation pipeline described here was developed as part of FEDORA Deliverable D3.1 – Integrated Cross-Sectoral Data Space.
Open mobility data in Europe is scattered across hundreds of national and regional portals — and each one speaks its own language. A traffic dataset published by Austria’s data.gv.at portal might describe itself entirely in German; a French or Spanish regional portal will do the same in its own language. For anyone trying to search across all of them at once, that’s a problem: a catalogue that mixes languages becomes fragmented and hard to navigate, and keyword search quietly breaks down.
FEDORA’s answer is a purpose-built translation layer sitting inside its data catalogue, which is built on the open-source CKAN platform. Whenever a dataset is harvested or updated, a custom plugin automatically intercepts the incoming metadata — the title, description and tags — and routes it through a translation pipeline before it is indexed for search.
What makes the approach notable is how it translates. Rather than relying on a commercial cloud translation API, the plugin connects to a self-hosted instance of LibreTranslate, an open-source machine translation engine. Running it locally serves three purposes at once: it keeps sensitive metadata inside FEDORA’s own infrastructure, removes dependence on external services, and avoids hitting rate limits when harvesting large batches of datasets at once — a real risk when pulling in metadata from dozens of sources in a single run.
The translation itself is non-destructive. The English version is written into the catalogue’s main searchable fields, so every dataset becomes discoverable through a single, unified English-language search. But the original text isn’t discarded — it’s preserved alongside the translation, so a dataset originally published in Basque, German or Danish remains fully readable and searchable in its native language too. Users get both: one consistent search experience, and the original context intact underneath it.
The team has also been transparent about the pipeline’s limits. Based on observational checks, the underlying language-detection component performs at around 90% accuracy, with confidence dropping — and translations sometimes being skipped — for very short text strings. Built-in fallback logic ensures the system displays the original text rather than failing outright when something goes wrong, and edge cases are caught through manual review of the catalogue.
Looking ahead, the team is exploring the integration of a locally hosted large language model to strengthen the pipeline further — either to resolve the low-confidence detection cases or to replace the current translation engine altogether — all while keeping the same commitment to data sovereignty that shaped the original design.
It’s a small piece of infrastructure with an outsized effect: it’s what allows a researcher in Vienna, an engineer in Reggio Emilia, and a transport authority in Nicosia to search the same catalogue and actually find each other’s data.
Learn more: Read the full technical description in FEDORA Deliverable D3.1 – Integrated Cross-Sectoral Data Space, Section 4.3.


