Few-Shot Multilingual Open-Domain QA from Five Examples Open Access

Open-domain QA has demonstrated impressive performance by employing the retrieve-then-read (Figure 1(a)) pipeline (Chen et al., 2017), which is built upon dense retrievers (Karpukhin et al., 2020) and efficient generative readers (Izacard and Grave, 2021). However, this success has been primarily limited to English, leaving the multilingual setting under-explored. This limitation is mainly due to the difficulty and costs of creating high-quality and balanced human-supervised training data for languages other than English. Moreover, multilingual open-domain QA introduces additional challenges with retrieving evidence from multilingual corpora, requiring the underlying retrieval system to be capable of both cross-lingual and monolingual retrieval (Asai et al., 2021b).

More recently, efforts have been made to create multilingual open-domain QA benchmarks from existing multilingual machine reading comprehension tasks (e.g., Xor-TyDi QA [Asai et al., 2021a]) and by translating English datasets (e.g., MKQA [Longpre et al., 2021]). These datasets have enabled various approaches to address multilingual open-domain QA problems, including iterative data augmentation (Asai et al., 2021b) and extensive additional pre-training on Wikipedia texts (Abulkhanov et al., 2023; Jiang et al., 2024). However, these methods still heavily depend on abundant high-quality language-specific data for fine-tuning, making them less effective solutions when language resources are limited. Therefore, a more generalizable approach to multilingual open-domain QA should aim to mitigate this reliance and be capable of facilitating language adaptation with minimally supervised samples.

In this paper, we present FsModQA, a method for Few-ShotMultilingual Open-Domain QA using minimally-sized supervised data (i.e., up to 5 per language).¹ Our approach consists of two core components: a self-supervised pre-training objective on multilingual corpora; and a synthetic data generation pipeline that prompts a large language model (LLM) using few-shot supervised examples. Concretely, we generate question-answer pairs from WikiData triples by leveraging LLMs’ In-Context Learning (ICL) ability. To facilitate ICL prompts, we incorporate ChatBots to generate curated input-output pairs, which serve as examples for prompting LLMs to generate millions of questions from WikiData triples across various languages. After generating these question-answer pairs, we identify the supported Wikipedia passages through answer string matching. We further gather cross-lingual answers and evidence passages through Wikipedia language links to facilitate cross-lingual retrieval. Employing this generated data, we train a multilingual model with a joint objective for retrieval and QA, producing a promising pre-trained model (Figure 1(c)) for subsequent few-shot learning.

In few-shot learning, we employ LLMs for data generation from few-shot examples. For each target language, we feed the few-shot examples to an LLM and prompt it to generate question-answer pairs from a given document. The few-shot examples are assumed to encapsulate the QA style and distribution of the target dataset, enforcing the LLM to generate synthetic data with similar characteristics. With this abundant synthetic data, the pre-trained model can be further fine-tuned to achieve superior results (Figure 1(c)). As an unsupervised alternative, we explore a zero-shot cross-lingual prompting strategy that uses data from other languages as prompts for data generation, and we show this is almost on par with few-shot prompting (Figure 1(c)).

We evaluate FsModQA on various datasets, including cross-lingual and monolingual retrieval, and multilingual open-domain QA. We observe notable improvements over competitive few-shot baselines, with +5.1% gain on retrieval and +8.4% gain on multilingual open-domain QA. To further test FsModQA language adaptation ability, we conduct zero-shot adaptation experiments using our cross-lingual prompting strategy on 15 languages. This adaptation improves performance in both monolingual retrieval and multilingual QA significantly, achieving results that are superior or comparable to strong translation-based methods.²

Figure 2 presents the full pipeline for generating self-supervised pre-training and fine-tuning data.

Our self-supervised training dataset is constructed based on Wikidata (Vrandečić and Krötzsch, 2014), a multilingual knowledge base consisting of fact triplets linked to millions of entities. We manually select 50 common properties (Appendix Table 10) based on English and consider all triples associated with these relations. We then gather fact triplets in the desired target languages through language links.

Given a triplet $T=(s,r,o)$⁠, we aim to write a question q about the head entity s’s property r with the gold answer a being the tail entity o. One can use relation-specific templates to efficiently transform each triple into natural questions (Sciavolino et al., 2021). However, this method lacks diversity, making triples with the same properties generate questions with similar surface forms. Instead, we adopt a generative approach by using a LLM to automatically generate questions with more diverse styles.

Specifically, we first sample five triples for each property and prompt ChatGPT (gpt-3.5-turbo) to generate three questions for each triple. This process yields a curated set of high-quality questions: $K={si,ri,oi,qi}i=0k$⁠.

We additionally generate questions with Yes/No answers from the same set of sampled triples. It is easy to generate Yes questions. For No questions, we need to create false fact triples from existing triples. Specifically, we randomly replace a triple’s head or tail entity with the most similar Wikidata entity, and check the perturbed triple is not a valid fact according to Wikidata. We then generate questions using ChatGPT as before. Examples are included in Appendix Table 11.

Subsequently, these curated questions are used as ICL examples to prompt a smaller LLM to transform all sampled triples into natural questions. We use Gemma-7B (Gemma Team et al., 2024) as the LLM and include the prompts we used in Appendix Table 12.

As shown in Figure 3, for a question q^ja and answer a^ja derived from a triple (s^ja, r, o^ja), we gather all passages from the Wikipedia page $Wja$ linked by s^ja and add passages containing a^ja as positive $Dqja$⁠. If no such passage exists, we use partial match and select the one with the highest lexical overlaps with a^ja as positive. We further include positive passages in other languages to facilitate cross-lingual retrieval. We first translate the triple into target languages (s^L, r, a^L) using language links and identify cross-lingual positives by searching a^L in the Wikipedia page $WL$ linked by s^L as above. This derives monolingual and cross-lingual positive passages $Dq=Dqja∪DqL$⁠. We generate 18.7M $(q,a,Dq)$ triples across 8 languages in total, denoted as MlWikiQA.³

The main idea of FsModQA is to amplify a limited number of annotated examples into a substantially larger volume of synthetic data by prompting LLMs. In this work, we consider Xor-TyDi QA (Asai et al., 2021a) as our target dataset. For each language in Xor-TyDi QA, we randomly sample five triples $K={(qiL,aiL,diL)}i=15$ from the training set as few-shot examples. Each triple contains the question, answer, and the ground truth passage. We ensure that three examples are span answers, while the remaining two are Yes and No answers to align with Xor-TyDi QA distribution.

We populate a hand-engineered template with our few-shot language-specific examples $K$ and use them as the ICL examples to prompt LLM. Given a randomly sampled passage d^L from language L, we append d^L to the template, and the LLM is expected to generate a relevant question q^L and answer a^L in language L. We further constrain the answer a^L to be a span within d^L, a property of the original Xor-TyDi QA dataset.

Many questions classified as unanswerable in Clark et al. (2020) can be answered by referring to English Wikipedia (Asai et al., 2021a). These questions are included as cross-lingual questions in Xor-TyDi QA. To simulate this scenario, we generate synthetic cross-lingual data from English passages. We first use GoogleTranslate to translate the few-shot examples to English: $K′={(qiEn,qiL,aiEn,aiL,diEn)}i=15$⁠. Subsequently, we use these translated few-shot examples $K′$ to fill another template and instruct the LLM to generate QA from a randomly sampled English passage d^En, first in English (q^En, a^En) and then in target language (q^L, a^L). Similarly, we restrict a^En to be a span within d^En. We include the prompts we used in Tables 13 and 14 in Appendix.

We employ a method based on Natural Language Inference (NLI) to enhance the quality of our synthetic data. NLI techniques aim to classify whether a hypothesis text is entailed by, neutral, or contradictory to a given premise text (Bowman et al., 2015). They have been widely used for identifying factual errors in text summarization (Laban et al., 2022) and hallucinations in machine-generated texts (Honovich et al., 2022). In this study, we employ NLI methods for data filtering (Yoran et al., 2024). Given a synthetic example (q, a, d), we consider the source passage d as the premise and the concatenation of the generated question q and answer a as the hypothesis. We retain an example only when the premise entails the hypothesis.

In more detail, we apply a novel local-to-global filtering mechanism. In local filtering, we evaluate whether the originating passage d entails the synthetic QA (q, a) pairs. We take the output probability of the entailment label as the score and keep examples when the entailment score exceeds a threshold $Tl$⁠. In global filtering, we use a pre-trained model (i.e., the self-supervised model in Figure 1(b)) to perform retrieval for the question q and obtain a set of passages $D^q$⁠. We compute an entailment score vector $x∈R∣D^q∣$⁠, with each entry being the entailment score between (q, a) and a retrieved passage $d∈D^q$⁠. We then apply a maximum pooling operation $max(x)$ to derive the final score. The intuition behind this is that a valid (q, a) should be supported by at least one of the retrieved passages, which aligns with open-domain settings. Similarly, we retain only those examples whose scores surpass a predefined threshold $Tg$⁠. In this way, we end up having 1.7M synthetic data in total across 7 languages, denoted as FsMlQA.

Our few-shot setting relies on a few annotated examples to generate synthetic QA pairs in target languages. However, this approach encounters significant challenges when the target language is extremely low-resourced, making it nearly impossible to obtain even a few examples. For this setting, we explore zero-shot prompting, which uses cross-lingual examples to prompt LLMs to generate synthetic QA pairs in target languages.

We consider two zero-shot prompting settings. In English-Prompting setting, we use English QA data to fill up a template and use it as the prompt to ask LLMs to generate QA pairs from passages randomly sampled from the target language. In Multilingual-Prompting setting, we assume access to a handful of examples in a held-out language set. We randomly sample five multilingual examples from this held-out set to populate another template, and prompt LLMs to generate QA pairs in target languages. We include the prompts used in Tables 15 and 16 in Appendix.

Our synthetic dataset, FsMlQA, exhibits a strongly skewed distribution towards shorter answer lengths (often single tokens), whereas the human-annotated answers in Xor-TyDi QA tend to be substantially longer. To address this mismatch, we resample the training data from FsMlQA according to answer length, using a geometric distribution, l ∼Geo(p), to achieve a better balance between short and long answers.⁴

As shown in Figure 4, we employ a single encoder-decoder model to perform both passage retrieval and QA tasks. The first half of the encoder functions as a dual-encoder with shared parameters, which separately encodes the question q and the passage corpus $D$⁠. Additionally, we append an instruction to the question to inform the language of the target answer: “Answer in {lang}”. A LayerNorm operation, followed by average pooling, is applied to compress the inputs into single vectors: E_q and ${Edi∣di∈D}$⁠, which are used for matching via dot products. The top-k most relevant passages to the question are selected: $Dq=argtopkdi∈D(Eq·Edi)$⁠. The embeddings of the question and each top-k passage in $Dq$ are concatenated and fed into the remaining cross-encoder layers. Finally, the cross-encoder embeddings are flattened and incorporated into the decoder through cross-attention to generate the answer a, following the Fusion-in-Decoder approach (Izacard and Grave, 2021).

The entire model is optimized to generate the target answer a_i given q_i and relevant passages $Dqi$⁠. The final loss is: $Le2ei=Lreti+Lansi$⁠, where $Lansi=log∏t=1TP(ati∣a<ti,qi,Dqi)$⁠.

We evaluate on the Xor-TyDi QA dataset (Asai et al., 2021a), with XOR-Retrieve for cross-lingual retrieval and XOR-Full for multilingual open-retrieval QA. We conduct zero-shot evaluations on two benchmarks, MIRACL (Zhang et al., 2023) for monolingual retrieval and MKQA (Longpre et al., 2021) for multilingual open-domain QA. For XOR-Retrieve, we use the February 2019 English Wikipedia dump as the retrieval corpus and the same dumps from 13 languages for XOR-Full and MKQA (Asai et al., 2021a). For MIRACL, we use the monolingual Wikipedia preprocessed by Zhang et al. (2023). Following prior work, we evaluate models at Recall@5kt (top 5000 tokens) on XOR-Retrieve; F1, exact match (EM) and BLEU on XOR-Full; nDCG@10 on MIRACL; and F1 on MKQA.

We evaluate three ranges of representative baselines based on the type of supervised data used: (i) Zero-shot baselines (“-En”) fine-tuned on supervised English-only data (i.e., Natural Questions (Kwiatkowski et al., 2019)). (ii) Supervised baselines that fine-tuned on human-annotated multilingual data (i.e., Xor-TyDi QA). (iii) Few-shot models that improve zero-shot baselines with only a few supervised multilingual instances.

For XOR-Retrieve, we include: (1) Zero-shot retrievers: translate-test methods: DPR+MT (Asai et al., 2021a) and ReATT+MT (Jiang et al., 2022); models pre- trained on multilingual Wikipedia: CLASS-En (Jiang et al., 2024) and LAPCA (Abulkhanov et al., 2023). (2) Supervised retrievers: multilingual dense retrievers: mDPR (Asai et al., 2021a), CORA (Asai et al., 2021b), Sentri (Sorokin et al., 2022), QuiCK (Ren et al., 2022); token-level dense retrievers: DrDecr (Li et al., 2022) pre-trains ColBERT on WikiMatrix (Schwenk et al., 2021). (3) Few-shot retrievers: SWIM-X (Thakur et al., 2024) generates massive synthetic data from LLMs through a summarisation-then-ask technique. CLASS (5-shot) fine-tunes CLASS-En on our 5-shot examples. For MIRACL (Zhang et al., 2023), we include two supervised retrievers: fine-tuned mContriever (Izacard et al., 2022) and Hybrid that combines the results of BM25, mDPR, and mColbert (Khattab and Zaharia, 2020).

(1) Zero-shot baselines: translate-test methods MT+DPR, ReAtt+MT, and GMT+GS generate answers from English retrieved passages with question and answer translations. (2) Supervised baselines: BM25 does in-language retrieval with an extractive multilingual QA model; MT+Mono first applies BM25 and then MT+DPR if no answer was generated. Fusion-in-decoder methods (i.e., CORA, CLASS, Sentri, LAPCA) use retrieval-augmented generation, generating target language answers from multilingual retrieved passages. (3) Few-shot readers: Gemma (5-shot) (Gemma Team et al., 2024) and LLaMa3 (5-shot) (Touvron et al., 2023) prompt LLMs with few-shot examples and retrieved passages using the template in Appendix Table 17; CLASS (5-shot) fine-tunes CLASS-En on few-shot examples. We use the same 5-shot examples for all methods.

With the proposed self-supervised data construction method, we generate 18,735,159 triplets for pre-training across 8 languages, with statistics in Appendix Table 19. We initialize our model from the mT5-large checkpoint (Xue et al., 2021) and pre-train it using the loss function ℒ_ssl for 100K steps with a batch size of 800 on 16 A100 GPUs for 64 hours. We set the learning rate to 5 × 10⁻⁵ with 10% steps of warm-up, and linear decay to 0.

With our few-shot data generation method, we obtain 1,746,156 question-answer pairs across 7 languages included in Xor-TyDi QA after data filtering with $Tl=0.5$ and $Tg=0.8$⁠, with detailed statistics shown in Table 19 in Appendix. For fine-tuning, we first train the pre-trained model using NQ data for 8K steps and then on FsMlQA for 6K–14K steps depending upon the size of the sampled training dataset, with the loss function ℒ_e2e. We set the batch size to 128 and the learning rate to 5 × 10⁻⁵. We apply an asynchronous passage update mechanism, where we periodically refresh the retrieved passages for each training query using the most recent checkpoint every 1K steps.

Table 1 shows that FsModQA, fine-tuned on 100K synthetic data, surpasses the few-shot SWIM-X (7M) by 5.5% at Recall@5kt, despite the latter using substantially more synthetic data generated by a significantly larger proprietary LLM (PaLM2). This indicates our method’s great efficiency in training and data generation. Further scaling up the training data to full size does not improve retrieval accuracy. In addition, we find that fine-tuning CLASS, a sophisticated pre-training method, on the same set of 5-shot examples, lags FsModQA by 3.1 points. This shows our method of amplifying data through LLM prompting is superior to direct fine-tuning.

Table 2 shows that FsModQA surpasses the few-shot retriever SWIM-X by 5.1%, although SWIM-X generates synthetic data on each MIRACL language through 3-shot prompting, whereas FsModQA is exclusively trained on synthetic data generated from 5-shot examples of Xor-TyDi QA and thus, evaluated on a zero-shot manner. We further divide languages into seen and unseen groups based on FsModQA’s training data. It outperforms SWIM-X on all seen languages and 7 out of 10 unseen languages, except on zh, fr, and de. We suspect SWIM-X benefits significantly from large-scale synthetic data generation on these high-resource languages.

In Table 3, we show FsModQA achieves the best results in few-shot settings, outperforming CLASS-En (directly fine-tuning on 5-shot examples) by 8.4% and directly few-shot promoting LLMs for QA by 18%. Compared to supervised readers, FsModQA surpasses CORA and other pipeline methods while achieving results comparable to the rest. It is also noteworthy that in two low-resource languages, FsModQA outperforms comparable supervised baselines in Bengali and achieves a closer match in Telugu, indicating the effectiveness of our method in handling low-resource languages.

In Table 4, FsModQA achieves the best zero-shot results on MKQA in almost all languages, with an improvement of +2.8% compared to supervised CORA and CLASS. This suggests that training on our synthetic data can well generalize to other new languages, indicating that generating synthetic data for each target language may not be necessary for language adaptation.

We perform ablation studies to justify each of our designs, with results shown in Tables 5 and 6.

Excluding cross-lingual synthetic training data enhances performance in answering questions that require only the retrieval of in-language passages. However, the result on questions relying on cross-lingual passage retrieval declines, reducing the overall results. This is further evidenced by retrieval results R^M@100, where the accuracy of finding evidence in any language (e.g., English and in-language) drops, with additional support from the cross-lingual passage retrieval results.

By using the raw synthetic data from LLMs without any quality control, the performance suffers in every examined language except Telugu. We suspect that the NLI model is deficient in this language.

Sampling data according to geometry distribution over answer length leads to a 0.9% gain on average. In languages that contain a significant number of long answers (i.e., ar, fi, ja, ru), geometry sampling shows gains of up to 2.7%. Conversely, in bn, and ko, where short answers dominate, random sampling is usually better.

We observe extremely poor results in all languages without pre-training on our MlWikiQA, primarily due to the model’s low retrieval accuracy in identifying relevant passages. We believe pre-training enables the model to achieve good initial retrieval accuracy, which is essential in the subsequent fine-tuning process.

To investigate the effect of our data scale on models, we train FsModQA on subsets ranging from 0.05M to the entire 1.7M QA pairs, Results on each language and the average performance are shown in Figure 5. As the data size increases, FsModQA shows enhanced average performance up to the 0.6M data scale and gradually decreases afterward. We observe that as data size increases, the proportion of examples with short answers increases (78.4% → 95.3%), and the result on long-answer examples drops from 18.0% to 15.1%, indicating overfitting to short answers.

Our geometric sampling method (§2.2.2) attempts to balance the answers by length, however its use of sampling without replacement means the few long answer instances are quickly exhausted, such that larger sampled datasets become skewed toward shorter answers. To mitigate this issue, we employ sampling with replacement. This method upsamples longer-answer examples such that the length distribution follows the precomputed geometric distribution.⁵ As a result, it effectively increases the number of training epochs for data points with longer answers. As shown in Figure 6, sampling with replacement significantly improves performance on longer answers (≥4 tokens) while maintaining comparable performance on shorter answers relative to the current method.

In Figure 7, we show that directly fine-tuning on the 5-shot examples is beneficial (28.6% → 33.5%) but remains inferior to our few-shot method. When increasing the size of supervised data, both methods achieve consistent improvements although the performance gap narrows. With full-sized training data, FsModQA surpasses CLASS (43.0% v.s. 41.6%), achieving new state-of-the-art results. See Appendix Table 21 for results in each language.

We compare our few-shot prompting strategy with two zero-shot cross-lingual prompting methods in §2.2.1. In English-Prompting, we consider NQ training data and TyDi QA English training data as prompting sources, respectively. In Multilingual-Prompting, we use 5-shot examples from all languages in Xor-TyDi QA (i.e., those used in our few-shot setting) for prompting. When generating synthetic data for each target language, we exclude its 5-shot examples from the prompting source. We compare the success rate of generating valid examples using different prompting strategies in Appendix Table 20, with few-shot prompting achieving the highest rate and English-Prompting with NQ yielding the lowest rate.

Table 7 shows that all three zero-shot prompting variants achieve consistent improvements over FsModQA-En with up to 8.1% gains, highlighting the versatility of our method in zero-shot language adaptation. Prompting with English datasets created with the same guidelines achieves better results (TyDi-En v.s. NQ-En), and using multilingual examples for prompting (i.e., Xor-TyDi-*) is comparable to FsModQA. Specifically, the diversity and QA styles in prompts are more important for fi and te, while for other languages, employing in-language prompts usually leads to the best performance.

We compare three different ways of using TyDi QA English data for zero-shot learning, direct English fine-tuning, fine-tuning on machine-translated data from English, and English Prompting. Table 8 shows the benefits of all three methods, with our English-Prompting approach yielding the best results in all languages. Additionally, combining data from all three methods results in improvements over any of them when used independently, and matches the performance of our few-shot setting.

In §2.2.1, we propose a zero-shot prompting strategy that uses few-shot examples from other languages to generate synthetic data for a distinct target language. The effectiveness of this approach is demonstrated in §3.8. In this section, we evaluate the impact of this strategy in adapting FsModQA to a diverse range of previously unseen languages, using only English labeled data.

We select ten languages unseen by FsModQA from the MIRACL dataset for monolingual retrieval adaptation. We choose ten unseen languages from the MKQA dataset with high, medium, and low resources for multilingual open-domain QA adaptation.

We consider the English NQ training data as the source for prompts. For each target language, we randomly sample five-shot examples from the NQ dataset to prompt the generation of Q&A pairs from selected Wikipedia passages, following the procedure described in §2.2. This approach yields 128,000 training instances for each target language. Additionally, we compare this method to the translate-train baseline (MT), which uses Google Translate to translate the NQ training data into the target languages.

For both methods, we fine-tune FsModQA for 3K steps following the same procedure used in FsMlQA (§3.3). The final checkpoint obtained at the last training step is used for evaluation. Note that separate models are created per language in this experiment.

As shown in the upper part of Table 9, the zero-shot adaptation significantly improves FsModQA’s monolingual retrieval results by an average of 5.4% across ten unseen languages. These improvements are particularly pronounced in low-resource languages (i.e., th, yo, sw), whereas the MT baseline results in notable declines both in these languages (e.g., −36.3% in yo) and overall (−3.3%). Note that MIRACL was created by native speakers from texts in the target languages, which aligns with our data generation process. This explains the consistent gains achieved by our method and shows its superiority to translation-based approaches.

As shown in the bottom of Table 9, the adaptation effectively enhances multilingual open-domain QA performance across seven languages, achieving an average improvement of 11.3%. MT-based approaches yield results comparable to our adaptation, which is expected since MKQA was translated from NQ and the machined-translated data share the same topic distributions (i.e., American-centric). In contrast, our method generates data from Wikipedia texts written in target languages to simulate how native speakers ask questions, which is more common for real-world scenarios.

To assess the overall quality of our synthetic data, we randomly sample 1,000 examples from the silver pre-training data (MlWikiQA) and few-shot synthetic data (FsMlQA). These samples are evaluated using the GPT-4o mini to assess quality based on: 1) Fluency (0–2): assessing whether the query is understandable, readable, and free of spelling or grammatical mistakes; 2) Relevance (0–2): evaluating the alignment between the generated query-answer pair and the passage used for data generation. The prompts employed for quality assessment are included in Appendix Table 18.

Figure 8 illustrates that both types of our generated queries exhibit fluency and strong grounding in the corresponding positive passages. The silver-standard MlWikiQA, derived using heuristics from WikiData (§2.1), consistently achieves higher scores across both metrics in all languages compared to the unfiltered synthetic FsMlQA (w/o Filter columns). However, the quality of FsMlQA improves significantly after applying our tailored filtering mechanism (w/ Filter columns), almost matching the quality and fluency scores for MlWikiQA. This finding underscores the critical role of the filtering procedure in producing a synthetic dataset of quality comparable to the silver-standard dataset.

To examine the distributional differences between our synthetic FsMlQA and the gold-standard data in Xor-TyDi QA, we randomly sample up to 20,000 examples from both datasets and visualize their distributions using t-SNE (van der Maaten and Hinton, 2008), which projects the queries onto a two-dimensional space. Figure 9 highlights several key findings: 1) The synthetic queries exhibit sufficient diversity, as they are scattered across the plot, indicating that our approach is capable of generating queries of various types using only five labeled examples. 2) The synthetic data shows significant overlap with the gold-standard data, demonstrating that it retains the core characteristics of the gold distribution. 3) The gold-standard data exhibits greater diversity than the synthetic data, suggesting that there is still room for improvement in enhancing diversity and variation during the data generation process, which we leave for future work. Similar findings are observed in the other languages (see Appendix Figure 10).

We employ Llama-Guard-2⁶ as the content safety classifier to assess the presence of unsafe content within our synthetic dataset. Our analysis reveals that 98.9% of the 1,746,156 queries in FsMlQA are classified as safe.

Open-domain QA requires retrieving relevant passages and extracting answers from them. This necessity has driven various methods that jointly train retrievers and readers. REALM (Guu et al., 2020), RAG (Lewis et al., 2020), EMDR2 (Sachan et al., 2021), YONO (Lee et al., 2022), ReAtt (Jiang et al., 2022), and Atlas (Izacard et al., 2024) first pre-train retrievers or initialize from pre-trained (Izacard et al., 2022) and fine-tuned retrievers. Subsequently, both components are fine-tuned jointly: the reader is trained using an answer generation loss, and the retriever is trained to promote passages that increase the likelihood of generating correct answers. Recently, this joint training mechanism has been adapted for multilingual open-domain QA (Jiang et al., 2024), where retrievers are initially trained by learning from English teachers using multilingual parallel data, followed by a joint training stage with query-answer pairs generated by LLMs. Our approach follows this joint training paradigm for model pre-training but differs significantly. We use WikiData as a source to generate more informative natural questions and answers. Additionally, our pre-training method is more efficient by eliminating knowledge distillation from English models.

Prompting LLMs to generate synthetic data has been widely adopted to improve the performance of retrieval and QA tasks. UPR (Sachan et al., 2022) and InPars (Bonifacio et al., 2022) use zero-shot or few-shot prompting for passage reranking. Promptagator (Dai et al., 2023) and SWIM-X (Thakur et al., 2024) prompt LLMs with few-shot examples to generate massive synthetic queries, either in English or in multiple languages, for retriever fine-tuning. Gecko (Lee et al., 2024) prompts LLMs to generate synthetic instructions and queries from Web documents and create high-quality labels for retriever fine-tuning. Beyond retrieval, LLMs are employed to generate QA data, where QAmeleon (Agrawal et al., 2023) prompts a 540B LLM to generate multilingual QA pairs from only five examples. Nevertheless, these methods primarily focus on retrieval tasks and the more narrowly defined machine reading comprehension tasks. In our work, we rigorously investigate how LLMs can improve the more challenging multilingual open-domain QA tasks under few-shot settings. In addition, we explore zero-shot prompting, demonstrating that cross-lingual prompting using English data or limited multilingual data from held-out languages can yield results comparable to few-shot prompting, and we show this technique can also be leveraged for effective zero-shot language adaptation.

In this work, we propose FsModQA, a few-shot learning approach for multilingual open-domain retrieval tasks. We present a novel self-supervised pre-training framework that exploits WikiData to effectively initialize both multilingual retrieval and QA capabilities. This process is followed by few-shot synthetic multilingual QA generation from LLMs using only five human-annotated examples. We demonstrate that the resulting model achieves competitive multilingual retrieval and QA performance through fine-tuning on the high-quality synthetic data. We further show that this few-shot approach generalizes to zero-shot settings that only require English-supervised data. This mechanism serves as an effective approach for language adaptation, enabling the adapted model to achieve both boosted retrieval and end-to-end QA performance across fifteen previously unseen languages.

This work uses LLMs for synthetic data generation, which may propagate undesirable biases to generated data. We believe such biases will not be amplified as we sample prompts from Xor-TyDi QA, a dataset annotated with strict guidelines. Our preliminary safety analysis also reveals that only less than 1% data contains potentially harmful queries, as identified by Llama-Guard-2.

We thank the action editor Shay Cohen and anonymous reviewers for their helpful feedback and suggestions. The first author is supported by the Graduate Research Scholarships funded by the University of Melbourne. This work was funded by the Australian Research Council, Discovery grant DP230102775.