← Research

Conditioned in Training, Dispensable at Inference: Typological Conditioning for Multilingual Speech Translation

Yu Pan, Tianjiao Zhang, Yang Hou, Xiongfei Wu, Liang Zhang, Yves Le Traon, Lei Ma, Jianjun Zhao

Multilingual speech translation frontends built on speech large language models typically condition on language identity, as a learned embedding or a prompted language name. Such a signal, however, is learned separately for each language and transfers nothing between typologically related languages. In this paper, we present TypoCast, a frontend that conditions on the typological profile of the source language rather than on its label alone. A typology-informed hierarchical language encoding represents each language by its morphological type, constituent order and family, so that supervision is shared among related languages. A dual-route mixture-of-experts adapter routes inputs through a shared typological route together with a per-language residual, and a typology-aware prompt conveys the same profile to the decoder. Extensive experiments on CVSS-C demonstrate that TypoCast surpasses state-of-the-art speech-to-speech translation systems on French, German and Spanish to English, averaged over the three directions, and outperforms the identity-conditioned S2ST-Omni on every language of a seven-language inventory, with only 228M trainable parameters. Ablation studies further reveal that the conditioning signal acts during training and is dispensable at inference: a model trained without it is consistently worse, whereas removing it from a trained model changes BLEU by less than 0.2, and expert usage is consistent with the signal shaping how adapter capacity is divided among languages.

PDF ↗

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace