TANQ: An Open Domain Dataset of Table Answered Questions Open Access

Understanding and solving problems in real-world scenarios often requires reasoning across multiple documents and data modalities. This includes (i) retrieving information from different sources, (ii) processing and aggregating data to extract insights, and (iii) presenting complex findings in a structured format, for example a table or infographics, to communicate them to readers.

Previous studies show that knowledge workers across domains, e.g., finance, science, and economics, spend around 20% of their time searching and gathering information from different files into one document to extract insights and consequently answer information-seeking questions (Chui et al., 2012). As a result, one of most challenging tasks for workers is to aggregate data and turn it into insights. Tables as a structured representation of data are ubiquitous in real-world sources and are commonly used to communicate complex information. Hence, they can be the perfect modality to answer complex questions.

Large language models (LLMs), often enhanced with external tools, have become a primary method for various application such as answering questions. State-of-the-art question-answering (QA) systems integrate LLMs in various ways, from decomposing complex queries to retrieving documents using external tools or generating context data from knowledge acquired during model training. However, their evaluation is mostly limited to simple datasets, e.g., TabFact (Chen et al., 2020a) or HotpotQA (Yang et al., 2018), whose questions can be answered by reasoning over a single table or text document and generating a short text sequence as answer. This limits the applicability of such systems to perform complex multi-step research explorations. Moreover, it differs from real-world needs where relevant information can be spread across documents and represented in different forms (e.g., text or tables). More often than not, generating a short text as answer is not sufficient for complex information-seeking questions.

In this paper, we investigate LLMs’ capabilities in reasoning over multiple data sources and formats (i.e., text, tables, infoboxes) to answer entity-centric questions and generate table answers as structured artifacts. To address these challenges, we introduce TANQ, an open-domain, multi-hop QA dataset. TANQ requires retrieving and aggregating data from multiple documents to compile and communicate answers as tables. To solve TANQ, models require different skills in addition to data retrieval such as filtering, maths, and name normalization. We create TANQ applying a five-step, automated data collection process. We use QAMPARI (Amouyal et al., 2022) as seed dataset and Wikidata as well as the Wikipedia corpus as data sources. For automated evaluation of different data collection and processing substeps, we use PaLM-2 (Anil et al., 2023b).

We evaluate several state-of-the-art LLMs on TANQ, including close, oracle and open book evaluation settings. Finally, we study model-generated answer tables and discuss common failure cases and challenges related to TANQ. Our evaluation of models across skills can further inform future tools and evaluation setups for LLMs to improve models for complex, information-seeking questions.

Our contributions are as follows:

• We introduce TANQ, the first open-domain question-answering benchmark that requires building answers in form of tables from multiple information sources.

• (b)

  We benchmark state-of-the-art language models in oracle, open, and closed book setups, reaching an overall F1 score of 60.7 with our best-performing (oracle) baseline.

• (c)

  We evaluate model performance across different dataset characteristics, and discuss challenges and common failure types.

Various benchmarks for QA have been released in recent years. Each one addresses different challenges related to the task. Table 1 provides an overview and comparison of benchmarks.

A number of datasets use text and one or multiple tables for QA. While both text and tables have been considered as input modalities, the output of the datasets is mostly limited to short textual answers. HybridQA (Chen et al., 2020b), for example, is a multi-hop QA dataset that requires reasoning over one table and multiple Wikipedia passages related to entities occurring in the table. HybridQA answers are short texts with location names being the most common answer types, followed by numbers, dates, and person names. Moreover, many QA benchmarks for reasoning over text and tabular context concentrate on the finance domain. For example, the MultiHiertt (Zhao et al., 2022) benchmark is created from financial reports. Questions require reasoning over texts and multiple tables. The answers are short numerical values with a focus on numerical reasoning. Other financial QA benchmarks are FinQA (Chen et al., 2021b) and TATQA (Zhu et al., 2021) where the context is one table and minimum two paragraphs related to the table. TATQA answers are short texts consisting of either one or multiple text spans from context paragraphs/tables or are free-form answers. MultimodalQA is a multi-hop, open-domain QA dataset that takes one table and related images and text paragraphs as input with answers similar to previously described datasets.

Most of the earlier described datasets have context provided in form of text and/or tables, whereas open-domain QA datasets first require extracting the relevant context, before answering the given question. The majority of open-domain QA datasets, such as WikiQA (Yang et al., 2015), TriviaQA (Joshi et al., 2017), and RobustQA (Han et al., 2023), are limited to textual context. NQ Tables (Herzig et al., 2021) extends table-QA to an open-domain setting where first top-k tables are retrieved from a given corpus. These tables are processed by a reader component for generating the correct short-text answer. Built on HybridQA, the Open Table-and-Text Question Answering (OTT-QA) benchmark (Chen et al., 2021a) extends this setting by requiring to extract both tables and texts given multi-hop questions.

The work closest to ours with respect to input and output modalities is MultiTabQA (Pal et al., 2023). MultiTabQA seeds on the Spider dataset, a text-to-SQL dataset containing SQL queries, database tables, and natural language translations of the queries. Pal et al. (2023) use table names occurring in SQL queries to extract the input tables and query Spider databases for answer table generation. While the datas et also generates answers as tables, it has certain limitations: (i) the benchmark input is limited to tables; (ii) the input and output tables are highly structured database tables which differ from real-world scenarios where tables occur in documents and websites in various formats; (iii) the questions are limited to SQL-based queries.

TANQ evaluates the capability to answer open domain, multi-hop questions by aggregating data and generating answer tables.

A TANQ dataset instance is a triple (q, t, D), consisting of an entity-centric question q, a table answer t, and a document set D (see Figure 1). To answer the multi-hop question q, first multiple sub-answers are extracted from the document set D. The answer is generated in table form t = {t_{i, j}|i ≤ n, j ≤ m} consisting of n rows (i.e., one per extracted entities) and m columns. The documents in D provide supporting evidence for each cell of the answer table t_{i, j}. D is either provided as input to models (oracle setting) or retrieved from the Wikipedia corpus as D′ (open book) and can consist of texts, tables, and infoboxes.

We use QAMPARI, which is an open-domain QA dataset with lists of entities as answers (Amouyal et al., 2022), as seed dataset. QAMPARI further includes Wikipedia text as supporting evidence for each entry of the answer list. Different from prior QA datasets with short textual answers, they align to natural questions which require a list of answers extracted from multiple sources. The dataset is semi-automatically created with Wikidata and Wikipedia as data sources and evaluated by human annotators.

Following Amouyal et al. (2022), we use Wikidata (WD) as a source for question generation. Wikidata (Erxleben et al., 2014) is a collaborative knowledge graph consisting of triples of entities and relations, i.e., (e₁, r, e₂). Entities e_i are values (e.g., 1990) or items that represent a real-world concept, object, or a topic.² Relations are edges connecting entities, e.g., (DonaldTrump, instanceOf, human), where Donald Trump is the head entity and human the tail entity (Krishna et al., 2022). We follow the notion for formal queries over Wikidata introduced by Amouyal et al. (2022) for QAMPARI. Applying a relation r as query over WD items e_i results in a set of (tail) entities: [[r(e)]] = {e_i|(e_i, r, e) ∈WD}.

We extract supporting evidence in the form of sentences, tables and infoboxes (entity-tables at the top right corner of Wikipedia articles) from Wikipedia (WP).

Figure 2 provides an overview of the TANQ pipeline outlining all steps for data collection, processing, and evaluation. We use QAMPARI, Wikidata, and Wikipedia, as well as PaLM-2 (Anil et al., 2023b) for paraphrasing and validation.³

Starting with the QAMPARI questions and answer lists of entities, we extend each question q with additional WD relations using the answer entities e_a. QAMPARI questions are classified in either simple, composition (e.g., “Who directed movies screen-written by Steven Spielberg?”) or intersection (e.g., the example in Figure 1). We first query the WD knowledge graph to extract additional relations r_ext linked to e_a, i.e., r[e_a] = {e_ext|(e_a, r_ext, e_ext) ∈WD}. Hence, each extension is a WD triple linking the QAMPARI answer e_a (e.g., Hey Ram in Figure 1) through the relation r_ext (i.e., composer) to a new extended entity e_ext, i.e., Ilaiyaraaja for relation composer and answer entity Hey Ram in Figure 1. We only select a relation which fulfils two conditions. First, it is part of a predefined relation set R, i.e., r_ext ∈ R. We manually specify R based on the WD relation used to create QAMPARI questions. Second, the relation exists for all answers e_a of question q. Given n extension relations, which fulfil these conditions, we extend the question q in a template-style fashion: “[q] and what is their [r_ext1],[r_ext2],[...] and [r_extn].” For example, the question in Figure 1, was generated based on the initial question “Which Indian movies were both directed and written by Kamal Haasan?” through extending with the WD relations composer and release year. Additionally, we extract all extension entities e_ext of the extension triple.

Next, we collect for each extension triple (e_a, r_ext, e_ext) supporting evidence using Wikipedia as an evidence source. Hereby, we search for supporting text, tables and infoboxes in the WP articles of e_a and e_ext. We apply simple heuristics and search for mentions of e_a in the e_ext article and vice versa. Moreover, we extend our queries with additional (heuristic-based) query words, for example, considering different formats of how numbers and dates are represented in queries.

To evaluate the correctness of the previously collected evidence texts, tables, and infoboxes, we employ PaLM-2 as an evidence evaluator. We prompt the LLM to evaluate the extracted evidence in a natural language inference setting. For each extension triple (e_a, r_ext, e_ext), we construct template-based sentences s: “$〈ea〉〈rext〉〈eext〉$”, e.g., “Hey Ram composer Ilaiyaraaja” for row 2 in Figure 1. We query the LLM to label the sentence s as “supported”, “refuted”, “not enough information” based on the provided evidence in form of a sentence, table or infobox entry. We then only consider the triples supported by at least one piece of evidence.⁴

Each extended relation corresponds to a column in the answer table (e.g., column “Composer” in Figure 1). Since the question in Figure 1 is extended with two additional relations (i.e., composer and release year), the resulting answer table has three columns. Hence, each cell in the answer table corresponds to a WD triple. For example, (Hey Ram, composer, Ilaiyaraaja) for cell “Ilaiyaraaja”. Some generated answer tables contained multiple entries in a single cell, as seen in the genre column of Table 7. To generate realistic tables, we filtered out samples with more than five entries in any single cell, resulting in a test set of 1,074 TANQ samples for evaluation.

This step increases the naturalness of template-based extension questions generated in Step 1. Similarly to the earlier step, we prompt a PaLM-2 model in a few-shot setting. To ensure the question meaning is preserved during rephrasing, we add structured annotations in parenthesis to questions with the name of each relation (e.g., “Which Indian movies were both directed (directed_by) and written by (written_by) Kamal Haasan?”). We run up to 5 iterations of rephrasing and stop if all relations are present, or discard the question otherwise.⁵

Finally, to generate more challenging questions requiring further reasoning capabilities beyond retrieval, we extend TANQ questions by asking for realistic post-processing steps. Overall, we augment questions with the following additional skills: (i) Filtering of answer table given a numeric, time, or attribute condition (rows 2–4 in Table 2); (ii) Conversion of numbers, dates, locations and corresponding units (rows 5–7 in Table 2); (iii) Calculation and introduction of an additional table column based on time attributes (e.g., “lifespan” in row 8 in Table 2). (iv) Approximating numbers to the nearest ten, hundred, etc. (see last row of Table 2). We use up to three distinct skills for augmenting TANQ questions. Table 4 provides a breakdown of TANQ dataset samples across skills.

Overall, the generated TANQ dataset has 1395 entries, 36.1% of question type simple, 40.9% intersection, and 22.9% composition questions. We further plan to release a TANQ training set with approximately 42k samples. Approximately 72.4% dataset instances (i.e., 1,010) require at least one additional skill to answer the question. See Table 4 for a breakdown of skills and question types in TANQ. TANQ questions have an average a length of 21 tokens and require extracting information about three relations on average. On average, TANQ answer tables have 6.7 rows and 4 columns. See Table 3 for WD entity types used for extending QAMPARI questions.

We manually evaluated 100 TANQ samples to assess if noise is introduced through the dataset’s automated generation pipeline. The focus of the analysis was twofold. First, examining the initial questions sourced from the QAMPARI dataset and, second, evaluating the generation of TANQ questions over the different pipeline steps. We identified five issues which are discussed below.

While occurring only for a small subset of questions, one observation was that some errors present in the initial QAMPARI questions were propagated through the pipeline without being corrected. In 2 out of the 100 samples, the original QAMPARI questions contained issues, one was carried into TANQ while the other was fixed. Examples include: “Robert Benton is the screenwriter and director of which software?” This question confuses a film with software as we see by considering the requested attributes, i.e., screenwriter and director.

While these logical errors were carried forward into TANQ grammatical errors, for example, were fixed during rephrasing with LLMs: “Black Sabbath had who as a member?”—issues phrasing that was later improved to “Who were the members of Black Sabbath [...]?”.

In 3 out of 100 evaluted questions, the question itself partially contained the answer, rendering the query less meaningful: “Eon Productions and Harry Saltzman produced what series and what was their genre, director of photography, duration in second, producer, country of origin, composer, publication date, narrative location, screenwriter, and director?”—in this case, the “producer” is already mentioned in the question.

Minor grammatical issues introduced through the TANQ pipeline were found in 2 out of the 100 samples. In one cases, this was fixed during the rephrasing process, while other remained unresolved. For example, “+89 minute” was rephrased to “89 minutes” in the sentence, “Which film was directed and produced by Mel Brooks and what was their composer, genre, duration? Filter the answer table for duration equal to or larger than +89 minute.” However, “Larger than 89 minutes” remained, but “longer than 89 minutes” would be more appropriate for this sentence.

While our evaluation metrics is not sensitive to the ordering of columns within the answer table, the order of relation appeared confusing for some questions. This occurred for 7 out of the evaluated 100 examples as shown here: “What filmmaker directed a movie written by Val Guest and what was their date of birth, place of death, date of death, and place of birth?”—the order of personal attributes (i.e., date and place of birth, date and place of death) seems unnatural as date of birth would naturally be followed by place of birth.

In 4 out of the 100 samples, certain relations were ambiguous or unclear, causing confusion about what exactly the question was asking for. Such as, “Who directed the movie which had the screenwriting done by Marc Norman and what was their date of birth, language spoken, written or signed, place of birth, sex or gender, country of citizenship, occupation?”—here, the attribute “language spoken, written or signed” is not clearly related to the director in question.

While we observed five different issues during the manual evaluation of the TANQ pipeline, these occurred only in a limited number of samples out of 100 evaluated ones. Thus, while these errors are present, they are not widespread enough to strongly impact the overall quality and utility of the TANQ dataset.

We evaluate TANQ in (i) closed book, (ii) oracle, and (iii) open book setups. We give an overview of the different approaches in Figure 3.

The closed book setup evaluates LLMs’ capabilities in extracting relevant information acquired during training to answer TANQ questions. For all experiments, we use the PaLM-2 Unicorn model (Anil et al., 2023b), GPT4o (Brown et al., 2020), Gemini Pro and Flash (Anil et al., 2023a), as well as Gemma 9B (Rivière et al., 2024). We evaluate all baselines in a few-shot setting with three examples provided in the prompt (see top left prompt in Figure 3).

In the oracle setup, we provide models’ source attribution for every cell of the answer table in form of oracle documents (i.e., text, tables, and infoboxes). Hence, the prompt consists of the TANQ question and multiple evidence sentences, infobox entries and/or tables (see bottom left prompt in Figure 3). To provide further context, we include the Wikipedia page title (for text evidence also sub-/section titles) where the evidence was extracted. Moreover, we add randomly selected oracle documents from other questions with similar attributes. This makes the oracle setting more challenging and requires models to filter the correct evidence from all provided ones first. We exclude models with a context length of less than 4k from the oracle evaluation. The evidence samples require a longer context to be fully displayed. Otherwise, performance may decrease due to the input being cut off.

For the open book baseline, we extend the PaLM-2 Unicorn model with external tools for search and calculation. The Wikipedia-based search tools aim to mimic the human search approach on Wikipedia similar to Yao et al. (2023). The agent model can access Wikipedia information through three tools: (1) WikiSearch(keywords): a keyword based Wikipedia search that returns the Wikipedia article most relevant to the provided keywords; (2) FindEvidence(article,keywords): an article-specific search that returns matched sentences, tables and infobox entries given keywords and a Wikipedia article; (3) GetIntro(article): returns the introduction section of a Wikipedia article. For calculations, we provide the model a Python engine as an external tool, (4) Python(calculation). We design the agent model to decompose and solve the TANQ task in multiple sub-tasks, consisting of (i) retrieving requested entities in an iterative manner (e.g., movies in Figure 1), (ii) searching for entity-related information (e.g., release year), (iii) post-processing information (filtering, calculation, etc.), and finally (iv) aggregating the information in form of a table. For some sub-tasks (e.g., entity retrieval), a separate sub-agent augmented with the required tools (e.g., WikiSearch) is spawned.

We compare model performance against human performance based on 100 answer tables generated by annotators. We provided the annotators the same input prompt as the models in the oracle setting (i.e., TANQ questions and evidence for each answer table cell).

To evaluate answer tables, we adopted a version of the relative mapping similarity (RMS) metrics introduced by Liu et al. (2023). RMS views tables as unordered collections of mappings from row/colum headers to values. Hence, the metric is invariant to transpositions and column/row permutations. It allows small errors between tables keys/values of target and reference tables using the Normalized Levenstein Distance (Biten et al., 2019). The metrics returns both precision and recall scores.

To evaluate the generated answer tables, we first converted table entries into a list of triplets. Each triplet consists of (i) the entity name (given in the first column of the answer table), (ii) the relation names (i.e., column names), and (iii) the content of a table cell. If a table cell contains multiple values, such as in column Genre in Table 7, the cell content is split into multiple triplets with the same entity name and attribute name, but with different values extracted from the cell. For example, given Table 7, the resulting splitted triplets contains multiple triplets for genre: {(Evita, duration, 129 min), (Evita, genre, biographical film) (Evita, genre, musical), …}.

After generating triplet lists for the gold table and the model-generated answer table, we calculate the similarity between them using relative distance for numbers and Normalized Levenshtein Distance for text across each field in the triplet. Each triplet in the target is matched with its closest triplet in the prediction greedily and thus we can compute weighted precision and recall, in the same manner as Liu et al. (2023). The evaluation code will be released.

In this section, we address key research questions: (1) Is TANQ a challenging dataset for state-of-the-art models? (2) How do models perform in a closed book setting compared to using external context (oracle)? (3) How effective are tool-augmented models on TANQ? (4) What challenges arise from different TANQ specifications—question types, reasoning skills, etc.? (5) What are the common failure cases of the evaluated models?

In Table 5, we compare the performance of all oracle, close, and open book baselines. In the oracle setting, we find Gemini Flash consistently outperforming other models with an overall F1 score of 60.7, followed by GPT4o. However, still, a considerable gap remains to the human baseline of 73.0. Both models experience a significant drop in performance in the closed book setting, falling behind the smaller, best-performing PaLM-2 model. Notably, PaLM-2 achieves higher recall scores, indicating that while the other models can generate some rows of the answer tables, they are slightly less accurate than PaLM-2, and their resulting tables are shorter, containing fewer entities. Different to Gemini Flash/Pro and Gemma, GPT4o and PaLM-2 struggle with more complex questions types, i.e., composition and intersection questions. Moreover, the 9B-sized Gemma model demonstrates performance comparable to much larger models in closed book evaluation.

In addition to the evaluations discussed in Section 5, we further assessed the baselines to examine the impact of (i) instruction tuning and (ii) demonstration selection on the models’ performance on TANQ. Table 15 compares the performance of baselines across different numbers of demonstrations in the input prompt (i.e., 1-, 3-, and 5-shot), while Table 16 provides an overview of how different instruction styles affect the performance of Gemini Flash.

For almost all baselines, performance improves slightly when the number of input examples is increased from 1 to 3 in the prompt. However, no further improvements are observed with an increase to five examples, supporting our decision to evaluate models in a 3-shot setting. Except for Gemini Pro and GPT4o, manually selecting input examples does not yield enhanced performance.

When comparing the performance of the top-performing oracle baseline, Gemini Flash, across different instruction styles, we find that the detailed (B), step-by-step (C), and simple (D) instruction styles yield similar performance (see Figure 6). In contrast, using an empty instruction (A)—where we provide only input examples without additional details—results naturally in decreased performance (see Table 16). For our evaluation in Section 5, we used the simple instruction style, as additional details as per instruction B and C did not contribute to observable performance improvements on the evaluated models.

Table 6 presents the performance of models on more challenging questions requiring further skills such as filtering and time conversion.

Overall, questions requiring filtering with numerical conditions pose the biggest challenge for Gemini models resulting in a performance drop compared to other skills. For example, Gemini Flash’s performance drops from 60.7 F1 (overall) to 55.3 in oracle evaluation.

For PaLM-2, we observe that the model struggles particularly with questions requiring numeracy, i.e., approximation of numerical values, quantity conversion, calculations based on datetime attributes, and filtering with numerical conditions. This challenge does not persist in the open book baseline where PaLM-2 is augmented with a calculator tool. Despite these limitations, the PaLM-2 model outperforms all other baselines across all skills in the closed book setting.

For GPT-4o, no clear patterns are observed regarding its limitations in specific reasoning skills. In both the oracle and closed book scenarios, the model struggles with date-to-year conversions and the approximation of numerical values. However, it outperforms other closed book baselines in time calculations.

Table 8 shows baseline performance by the number of skills required to generate the correct answer table. We observe across baselines a stable performance as the number of skills increases and questions become more complex. Except GPT4o, which shows significant performance decreases as the number of skills increases in the close book setting, e.g., from 26.5 (one skill) to 17.6 (two skills).

Table 9 demonstrates baselines’ performance across the number of relations in the given question. For example, in Figure 1 two relations, i.e., composer and release year, are requested for the movies specified in the question. While multiple models show a performance decrease with an increasing number of relations, this gap is particularly significant for the tool augmented model where the F1 score decrease by almost twelve points comparing questions requiring 1 relations vs. 10. Moreover, we observe for most baselines (oracle/close/open book) little performance drops comparing questions with 1 relation to those with 3—indicating that up to a certain limit models can successfully retrieve information for an increasing number of relations. However, the performance gaps increase for GPT4o (close) and the tool-augmented model as the number of relations increases further, i.e., from 3 to 5 and from 5 to 10.

Table 10 gives an overview of model performance across different answer table lengths: short answer tables (up to 3 rows), medium (up to 6 and long tables (7 or more rows). Models that perform well on longer tables are GPT-4o, Gemini Flash, and PaLM-2. As table size increases, so does the number of rows the models can correctly retrieve, resulting in higher recall scores, while precision slightly drops for most oracle baselines models. It is apparent that the model struggles with extracting correct information from the growing list of provided oracle documents as the table size increases. This is obviously not the case for close book evaluation as no oracle documents are given as input.

Hence, we further study common failures cases of oracle baselines (see Table 11). The aim is to understand challenges related to TANQ when the model has access to necessary information, in form of oracle documents.

We manually evaluate a subset of model predictions to identify common failure cases, categorized as follows: 1.) Relation missing: At least one expected relation (column) is missing; 2.) No table header generated; 3.) Filtering: Conditions (attribute, numeric, dates) ignored or incorrectly applied; 4.) Hallucinated relations: Unrequested columns are added; 5.) Other hallucinations; 6.) Missing entity: Expected rows are missing; 7.) Non-table answers: Text or non-table format returned; 8.) Partial answers: Incomplete information for a given entity or attribute; 9.) Wrong answer (cell): Incorrect cell content.

Comparing open and oracle baselines, our observations show that the best performing model, w.r.t least failures, is GPT4o (oracle). The only significant failure category we identify for GPT4o are missing entities in generated answer tables. The most present issue we observe for Gemini Pro (oracle) are missing entities, missing relations, resulting in answer tables with a subset of columns, non-table outputs, and missing headers of generated tables. Similarly, multiple aspects pose a challenge for PaLM-2 (oracle), including hallucinations, missing entities/relations, and applying filtering conditions correctly. For the tool augmented model, missing entities is a frequent issue.

It is noteworthy that the open book, ReAct-based, baseline performed worse than other baselines, also the closed book ones. To better understand the limitations of the open book baseline, we manually evaluated the reasoning chains generated by the model for 50 TANQ samples. These chains outline the verbal reasoning processes and actions taken by the model in an interleaved manner. Our evaluation identified six different areas where the open book model struggles.

The most prevalent issue, found in 19 out of 50 samples, was that the answer list did not contain all entities of the target table. Our review of the ReAct chain logs showed that the model often terminated the search for further entities after finding a list of entities. For example, it might start searching in the introduction section of a Wikipedia article and identify some movies directed by a particular director, but then fail to continue searching the rest of the page, missing additional films.

We also found issues related to answer table formatting in 7 out of 50 samples. For instance, in one case, the name column was incorrectly replicated into a table row (i.e., “Building” below in Table 12).

In six out of fifty samples, the model retrieved entities but failed to find attributes for all entities, resulting in partly empty rows (see Table 14).

A similar issue arises with intersection questions, which ask for entities that fulfill multiple conditions. For example, “What science fiction films were both written and directed by Steven Spielberg? What is their genre, publication date, and duration?” The open book model often produced tables that fulfilled only one of these conditions, such as listing films written by Steven Spielberg without considering if he also directed them. This problem occurred in 5 out of the 50 evaluated examples.

As the name suggests, composition questions require the model to first retrieve a list of intermediate entities, which are then used to compile the final entity list needed for the answer table. For example, consider the question: “Who directed the film that Jules Feiffer wrote? What is their occupation, date of death, date of birth, country of citizenship, place of birth, and award received?” The model should first retrieve the films as intermediate entities to identify the requested directors. However, in four out of the fifty evaluated samples, the model generated answer tables that included intermediate entities only instead of the requested entities. For instance, the answer table for the above question looked as demonstrated in Table 13. The first row correctly shows the directors, but the second and third rows contain the names of movies written by Jules Feiffer rather than their directors.

Finally, the model commonly ignored filtering conditions specified in the input question. We observed this in 10 out of 50 samples. For example, in the question: “Who was a mayor of Saint Paul, MN, and what was their occupation, position held, and date of birth? Filter the answer table for date of birth equal to or before 1829,” the resulting table included mayors born after 1829, failing to apply the requested filter.

Our results on TANQ highlight the need for qualitative benchmarks, metrics, and modeling approaches to address current limitations.

We initially used the RMS metrics from Liu et al. (2023) to evaluate table generation. However, RMS was developed for the chart-to-table conversion task, which involves mostly numerical tables. In contrast, TANQ tables contain significant text content, leading to variations such as expressing “USA” as “United States.” Another difference is the more diverse table structure, where cells often contain lists of values. We adapted the RMS metrics for TANQ (see Section 5). Our experiments indicate that this version of RMS strikes a good balance of providing signal, being simple to explain and implement, and being fast to execute. However, further research on table evaluation is necessary to consider a broader range of formats.

Interestingly, we found that smaller models like Gemma (9B parameters) performed surprisingly well. While Gemma was not among the top models for any specific skill or question type in the closed book setting, it did not fall far behind models that are significantly larger, such as PaLM-2, Gemini Pro, and GPT-4o. This suggests that, despite its smaller size, models like Gemma and Gemini Flash offer promising results, though there is still room for improvement. While the size of Gemini Flash is undisclosed, it is known to be smaller than Gemini Pro, further highlighting that scale alone does not guarantee better performance on complex reasoning tasks. This raises an important question for future research: How can we further enhance the performance of smaller models?

Additionally, while tool-based LMs are popular, our results suggest that they may not be the optimal choice in domains where up-to-date knowledge is not required, especially for answering simple questions. For more complex questions involving intersecting information, tool-augmented LMs perform comparably to other baselines, but for simpler questions, they lag behind both oracle and closed book baselines.

Numeracy remains a challenge for state-of- the-art models. Despite advancements in numerical reasoning with LLMs, many models still struggle with questions requiring simple numerical skills such as filtering the content of a based on a numerical condition. While progress has been made in recent years (Imani et al., 2023; Akhtar et al., 2023; Chen et al., 2022), challenges remain.

Another challenge is that all evaluated models tend to generate incomplete answer tables, often missing some entities (rows) and relations (columns) from the target table. Future work should focus on developing methods to address these limitations and generate answer tables that capture the entire requested information. Addressing these issues and improving table evaluation metrics are important areas for future research.

This paper introduces TANQ, the first open domain question answering dataset where the answers require building tables from information across multiple sources. To create TANQ, we design and apply an automated dataset pipeline using large language models and Wikipedia and Wikidata as knowledge sources. We further release for each cell of the answer tables source attribution in form of text, tabulation, or infobox proofs. We evaluate our dataset on state-of-the-art models in three different setting: oracle documents provided, closed, and open book setting. Our results and analysis suggest that TANQ is a complex task with many challenges ahead.

For our experiments, we use the following GPT4o model available over the OpenAI API: gpt-4o-2024-08-06.⁶ The PaLM-2 model we used for the pipeline and evaluation purposes was the model variant with 340B parameters.

Figures 5 and 6 show the prompts used for evaluating the extracted evidence and rephrasing the TANQ questions to increase the naturalness of template-based questions. Figure 7 shows prompts for the four instruction variants we evaluated: empty, simple, detailed, and step-by-step.