Abstract Zero-shot text-to-speech (TTS) can clone a speaker's voice from a short audio prompt, yet most TTS systems still require the audio prompt transcript during inference. This dependency prevents cross-lingual voice cloning when the audio prompt transcript is unavailable, particularly for unseen languages. Cross-Lingual F5-TTS was the first non-autoregressive zero-shot TTS model to remove this dependency and enable transcript-free cross-lingual voice cloning, but it prepares its training data with forced alignment. Forced alignment is sensitive to boundary errors, and its cost grows as more languages are covered. Its speaking rate predictor is also unreliable at estimating duration when the audio prompt begins or ends with silence. In this paper, we present Cross-Lingual F5-TTS 2, a simplified framework for transcript-free cross-lingual voice cloning without forced alignment. Instead of using forced alignment to segment real utterances, we build same-speaker prompt and target pairs using a pretrained F5-TTS model and fine-tune the same model on these constructed pairs. This simplifies data preparation and preserves the acoustic modeling capability of the pretrained model, enabling adaptation with only a short fine-tuning stage. We further make the syllable-level speaking rate predictor robust to leading and trailing silence through silence-aware augmentation. Experiments show that Cross-Lingual F5-TTS 2 reaches higher speaker similarity than F5-TTS and Cross-Lingual F5-TTS while maintaining intelligibility.
Contents
This page is for research demonstration purposes only.
Method Overview
Training Framework
Figure 1: Overview of the proposed SFT pipeline. (Left) Synthetic Data Generation: a pretrained F5-TTS model is used to synthesize speaker-consistent audio pairs. (Right) SFT with Synthetic Prompt: Cross-Lingual F5-TTS 2 is initialized from pretrained F5-TTS and fine-tuned on the constructed real-synthetic pairs, where the audio prompt transcript is replaced by learnable prompt tokens followed by an EOS marker, forcing the model to condition on acoustic prompts without relying on their transcripts.
Silence-Robust Speaking Rate Predictor
Figure 2: Silence-aware augmentation for training the silence-robust speaking rate predictor. With some probability, silence is inserted before, after, or on both sides of the prompt audio, so the predictor sees prompts with leading or trailing silence during training. The speaking rate label is unchanged, because the augmentation only pads the prompt with silence without modifying the speech segment that determines the rate.
Intra-Lingual Zero-shot Generation
Prompt and text from the demo page of Seed-TTS.
| Language | Audio Prompt | Text to Generate | Cross-Lingual F5-TTS | Cross-Lingual F5-TTS 2 |
|---|---|---|---|---|
| EN | I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. | |||
| Perhaps they are driven by the delicious blend of flavors, or it could be the appealing visual presentation. At the end of the day, our choices in food reflect our personal preferences and sometimes, even our lifestyle or belief system. | ||||
| Your safety and the pack's reputation are at stake. Your bravery is admirable, but sometimes bravery is knowing when to retreat. Please, consider returning with me. We can work out a plan, but only if you're willing to listen. | ||||
| ZH | 突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道："我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？" | |||
| 顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我："无论如何，都要冷静下来，重新开始。" | ||||
| 皇上的面色未变，宛如雕塑般静止，他的眼中闪过一丝动人的温度。他深深地看了那位忠心耿耿的臣子一眼，终于开口："诺，我会再考虑考虑的。"他的声音低沉且坚定，留下空气中隐隐的无奈与柔情。 |
Cross-Lingual Zero-shot Generation
Prompt from FLEURS dataset. Each prompt is used to generate speech in both Chinese and English target languages.
| Prompt Language | Audio Prompt | Target Language | Text to Generate | Cross-Lingual F5-TTS | Cross-Lingual F5-TTS 2 |
|---|---|---|---|---|---|
| German | Chinese | 产品经理，其实是随着互联网发展应运而生的。 | |||
| English | They were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound. | ||||
| French | Chinese | 眨眼工夫，几个混混亮明警察身份，一瞬间将其将张某制服。 | |||
| English | If ever he was impelled to cast sin from him and to repent the impulse that moved him was the wish to be her knight. | ||||
| Hindi | Chinese | 真不好意思，从小至今，他还从来没有被哪一位异性朋友亲吻过呢。 | |||
| English | The music came nearer and he recalled the words the words of shelley's fragment upon the moon wandering companionless pale for weariness. | ||||
| Korean | Chinese | 沉寂了好一阵，这时爱丽丝只能偶尔听到几句微弱的语音，如。 | |||
| English | The retreat will begin on wednesday afternoon in honour of saint francis xavier whose feast day is saturday. |