Dolomites: Domain-Specific Long-Form Methodical Tasks

Experts in various fields regularly use writing as a means for planning, organizing, and sharing their work. For instance, a teacher might draft a lesson plan for what they would like to teach in their next class, and a lawyer might draft a patent application for an invention. Experts generally follow a consistent and methodical approach to conduct these writing tasks. In the lesson plan example, a teacher would know the lesson objectives, format, and profile of the class, and would produce a plan with the topics to be covered and activities to improve learning. Importantly, the teacher follows a systematic procedure to write this lesson plan, using their expertise and what they know about the current context (e.g., the class profile).

Across fields, from law to visual arts and engineering, experts accomplish on a regular basis such methodical tasks, i.e., writing tasks which loosely follow a standard template for what is usually given as input and what is required from the output. These tasks often follow a structured and consistent procedure as they are performed regularly and tend to be fairly time-consuming, taking from a few hours to several days (see Figure 2). As large language models (LMs) become more capable and widely accessible to a more sophisticated set of users (Owens, 2023; Mollick and Mollick, 2023; Lee et al., 2023; Birhane et al., 2023; Mollick and Mollick, 2023; Frankenreiter and Nyarko, 2022; Demszky et al., 2023; Wang et al., 2023), they hold great potential for assisting experts with methodical writing tasks, increasing their efficiency and allowing them to focus on complex problem-solving activities (Noy and Zhang, 2023).

Given their potential for assisting experts, it would be beneficial to evaluate language models on a realistic set of methodical writing tasks. However, we currently do not have benchmarks that contain a typology of such tasks. The most natural source for such data would be query logs (Nguyen et al., 2016; Kwiatkowski et al., 2019) or chat histories (Zhao et al., 2023). However, these data sources do not specifically reflect domain-specific use cases and do not allow us to study specific use cases in a controlled manner.

In this work, we bridge this gap by eliciting 519 methodical task descriptions (see a few examples in Figure 1) from 266 experts across 25 different fields (Section 3.1). These writing tasks are formatted in a standard way, with a task procedure, input, and output. Further analysis with an independent group of experts reveals that they are indeed plausible (∼76% of them are likely to be conducted by an expert on a regular basis) and most experts (∼63%) would find it useful if they could use a capable AI model as a writing assistant (Section 3.3). Our tasks serve as the first collection of realistic use cases of experts spanning multiple domains.

To evaluate the ability of existing models to assist experts with these tasks, we collect examples (see Figure 2), where we instantiate each task with plausible inputs and outputs (Section 3.4). Examples are created semi-automatically: We first retrieve Web documents that could potentially serve as samples of the task, and then generate an example using a language model based on the retrieved web documents. These examples are then significantly post-edited by the same experts (who contributed the task) for improving adherence to the task description, factual correctness, and level of detail.

We use our benchmark, called Dolomites (short for Domain-Specific Long-Form Methodical Tasks), to evaluate current models in their ability to generate accurate and detailed outputs (Section 4). We formulate the modeling problem as long-form generation, where models are provided the task description and the example input and asked to generate the example output. Our experiments reveal that there is significant headroom in improving performance on methodical tasks (Section 5) which are inherently difficult (requiring reasoning skills and domain-knowledge), and in terms of improving automatic evaluation of long-form text. In addition to well-known shortcomings (Schluter, 2017; Krishna et al., 2021), conventional metrics are not designed to capture expert knowledge.

We hope that Dolomites can serve as a reference for domain-specific use cases of language models and provide a means for evaluating future models. We release our dataset and code at https://dolomites-benchmark.github.io.

In this section, we first describe the types of writing tasks considered in this work. We refer to these tasks as methodical writing tasks due to two properties that are common to their execution. Firstly, each task requires structured problem-solving, where the task follows a specific order where each step logically flows from the previous ones. For instance, in the Medicine task in Figure 1, the task requires producing an assessment of the patient, then a plan of care and finally a full evaluation of the patient. Secondly, every task usually follows a consistent execution across inputs, where there is a standard specification of the input, the output and the procedure for the task. In the same Medicine task, given a patient’s subjective and objective data, the task structure and procedure would mostly stay consistent across patients.

To elicit descriptions of tasks from experts, we operationalize our definition of a methodical task into a standard template (see Figure 1 for examples). We require that every task contains a brief task objective, a task procedure walking a beginner through how this task is conducted, input and output sections, which include information that is typically given, and information that needs to be generated. Both input and output sections are formatted in the form of section titles and section descriptions. Finally, we collect additional notes about the task, which can include best practices or common mistakes, and missing context that is important when conducting the task.

We further expect our tasks to meet the following criteria: (1) they are purely textual and do not involve other modalities in the input or output; (2) they require domain expertise and can only be completed by an expert; (3) they do not require use of specific equipment or software, with the exception of searching the Web; (4) they are frequent, routinely performed by an expert at least once every few months; and (5) time-consuming, taking a significant but not indefinite amount of time to complete (e.g., from a half hour to a few days, but not several months).

Aside from task descriptions, our dataset contains specific instantiations of methodical tasks (see Figure 2). We create examples by populating descriptions like those shown in Figure 1 with plausible input and output sections (see Section 3.4).

In our data curation process, we first collect a typology of realistic tasks that span multiple fields. These tasks are not meant to be exhaustive, but instead represent realistic use cases across fields.

We recruit 266 participants from the Prolific crowdsourcing platform. We recruit experts from 25 different fields, shown in Table 1, aiming for a broad coverage across disciplines. Participants qualify as experts if they have formal education in the field, and at least 3 years of work experience. Additional details about the participants’ backgrounds are provided in  Appendix A.

We ask each annotator to provide descriptions of two writing tasks they routinely perform in their profession subject to the criteria listed in Section 2. For each task, annotators are asked to fill in predefined fields (task objective, procedure, input and output sections, additional notes), the same way as shown in Figure 1. We ask annotators to give thorough descriptions as if they are teaching a novice how to perform each task. Instructions and interface screenshots are in  Appendix A.

After collecting the initial set of tasks, we filter them manually to ensure they meet the criteria outlined in Section 2, and obtain a total of 519 tasks. We find that there are very few tasks from a field that are highly similar.

Table 1 provides the number of tasks across fields and the task objective of a sample task from each field. Most fields have at least 20 tasks, with some exceptions, where we were not able to recruit as many experts. Across tasks, there are an average of ∼2.78 sections in the input and ∼2.82 sections in the output. Collectively, tasks in Dolomites are cognitively demanding and versatile in the types of reasoning they require. For instance, a diagnostic task in medicine requires inductive reasoning to go from particular symptoms to a general diagnosis. Whereas, in legal analysis, deductive reasoning is required to reason about how laws are interpreted in a specific case and analogical reasoning is needed as lawyers compare current cases with precedents. Similarly, in software application design, abstract reasoning is important while creativity is necessary for certain tasks in the visual arts. While it is hard to describe the type of reasoning required for all tasks, every methodical task essentially involves analyzing the input, making inferences based on the input and domain-specific knowledge, and finally, providing a justification in writing.

We validate our collection of tasks by consulting an independent group of experts. Specifically, we collect Likert ratings for each task from three experts on the following axes (the precise description for each item on the scale is provided in  Appendix A):

• Representativeness: How likely is this task to be conducted by an expert in your field?

• Complexity: How would you rate the complexity of this task?

• Time Required: How much time is typically required to complete this task?

• Usefulness: Would you or other experts find it useful if an AI system were to propose initial outputs for this task (which may be lacking), that can be validated and improved by experts?

The questions above are motivated by prior work showing that AI writing assistants could significantly benefit the productivity of experts (Eloundou et al., 2024; Noy and Zhang, 2023; Dell’Acqua et al., 2023). Beyond productivity, we were additionally interested in expert opinions about the societal implications of using language models as writing assistants. Hence, for each task, we elicit answers to the following questions which require a free-text response in addition to a Likert rating.

• Anonymity Required: Is it important to ensure anonymity of any individuals or organizations if an AI system is used for conducting this task?

• Biased Outcomes: Could relying on automatically generated outputs for this task result in biased or potentially harmful decisions for certain groups of people?

• Ethical Considerations: Are there ethical considerations (e.g., privacy, copyright issues) associated with the use of AI systems for this task?

• Workforce Impact: Could partial automation of this task have an impact on the workforce in the short term?

• Accessibility Requirements: Would the use of AI tools for this task require making exceptional considerations to ensure accessibility?

The main outcomes of our validation study are presented in Figure 3. We find that the tasks collected in Dolomites are ecologically valid, i.e., they are likely (∼76%) to be conducted by field experts. Most of them are of medium or high complexity, requiring moderate (a few years of experience) or substantial (several years of experience) expertise. While they are complex tasks, judgments about time taken reveal that most (∼61%) would take an expert from 1 hour to 7 days to complete. This degree of difficulty suggests that it is conceivable for language models to be useful assistants for these tasks. Finally, an overwhelming majority of experts would be interested (∼62%) or open to trying (∼32%) to use a language model that proposes initial outputs for the task.

With regard to the societal implications of language model use, the need for anonymity emerged as a concern for a significant number of tasks (∼58%). In fields like medicine, psychology, and law, experts emphasized the importance of protecting patient/client confidentiality. Similarly, experts felt strongly that proprietary information and trade secrets should be kept private in fields like business. Experts further thought that using language model responses without careful perusal can result in biased outcomes (∼56% of tasks) which could affect marginalized or underrepresented groups. They also raised various ethical concerns relating to copyright issues, privacy issues and stifling of human creativity due to over-reliance on AI.

Many experts (∼43%) recognized that partial automation of writing tasks is likely to impact the workforce in the short term, potentially leading to changes in job roles or skill requirements. At the same time, they were optimistic that this would improve productivity and bring positive changes to the nature of the work. It is important to ensure that users of all backgrounds and capabilities have equal access to language models. A significant number of tasks (∼40%) were rated as requiring exceptional considerations to be made for ensuring accessibility to all users. Across fields, experts highlighted that while language models as writing assistants can improve productivity, human oversight is important for responsible use of these technologies.

To evaluate language model capabilities in assisting experts with their tasks, we create examples of input and output sections with concrete details. We adopt a human-in-the-loop methodology, where initial examples are generated by a model, which are then post-edited by the same expert who provided the task. We describe this process below.

We believe that samples of methodical tasks are partially available in documents on the Web. Hence, we retrieve relevant documents for each task and generate examples by prompting models with passages from these documents as context. This process is depicted in Figure 4 and explained below.

Given a task description, we first need to find more specific queries that could potentially result in relevant Web documents. For example, for writing a catalog entry for an art exhibition, search queries like “renaissance art exhibition catalog entry” or “picasso guernica catalog entry” are likely to result in documents that contain examples of the task. To generate search queries, we prompt Bard (Manyika and Hsiao, 2023) (with 1 exemplar) with the task objective and instruct it to generate more specific queries that can help find Web documents which contain demonstrations of the task (Table 11 in  Appendix B). We generate 10 search queries for each task and restrict search to reliable and authoritative sources. These are collected by prompting Bard (with 1 exemplar) with the task objective to generate URLs to domain names which will be useful to find real examples for the given task (Table 12 in  Appendix B).

Using each search query, we gather the top-10 documents from Google search restricted to relevant domains with the site: operator in the query. Documents are then split into passages of 4,000 characters with a 100 character sliding window.

Having gathered evidence which may contain task demonstrations, we generate examples by prompting models with this evidence. We explore a multi-document setting, where passages are sampled from multiple documents and a single document setting, where passages are sampled from a single document as we found that the appropriate choice depends on the task.¹ Passages are reranked using an in-house reranker and the top-5 passages are provided as context, along with the task description, to a large language model (Gemini-Ultra (Team, 2023) and Bard in our case), which is asked to generate an example of the task (Table 13 in  Appendix B). Not all information mentioned in the task description is required to be present in the context and the model is allowed to infill content to construct an example. We generate up to 10 examples for each task.

Even though they are based on retrieved Web documents, model-generated examples could have many issues. They may not adhere to the structure specified in the task description, they may contain factual inconsistencies or inaccuracies, lack in depth, or be vague. To remedy these issues, we present the examples to the same experts who wrote the tasks for post-editing. They are asked to choose the most plausible example out of four variants, and use single or multi-document evidence.

Prior to post-editing, experts are asked to label examples according to three criteria on a Likert scale: adherence to task structure, factual correctness, and level of detail. They are also shown 1) the evidence passages for the example and 2) a critique (generated using Gemini-Ultra with the prompt in  Appendix B, Table 14) that may not be comprehensive, to aid them in identifying issues with the example. The critique is provided to make post-editing more efficient. They are required to fix any valid issues they recognize in the example as well as any valid issues identified by the critique.

Expert judgments of automatically generated examples are shown in Figure 5. We find that the majority of examples already follow the task structure (∼85%) and most of them are probably or definitely correct. However, a large number of examples are lacking in depth and detail. We show a histogram of the word-level edit distance between all tokens in the original and edited example in Figure 6a and relevant statistics of the original and post-edited examples in Table 2. The histogram suggests that on average, there are significant changes made to the original examples during post-editing. Since most experts judge that examples are lacking in depth, the edited examples are expectedly much longer on average. The edited examples also adhere better to the task description (on average, 97.67% sections in the task description are found in the edited examples compared to 92.81% in the original examples). We analyzed a random set of 100 examples and labeled each example with the type(s) of edits it contained. We found broadly the following types of edits: fact addition (88%), fact deletion (20%), fact update (65%), stylistic rewrites (76%), and reorganization (23%). These edit types are described further in  Appendix A. Finally, we compute readability scores of the examples as a noisy approximation of the complexity and level of detail in the text, using the Flesch-Kincaid Grade Level test (Kincaid et al., 1975). Higher readability values indicate that a piece of text requires more formal education and expertise to understand. We find that post-edited examples have higher scores, possibly indicating higher level of technical depth.

Examples in Dolomites are created by conditioning on passages from Web documents. We do not require these documents to contain complete examples of the task, and models are allowed to infill information to create an example. However, if these documents are seen by models during large-scale pretraining, it might be difficult to conduct a clean evaluation. We examine whether this is the case by computing the similarity of the post-edited examples to the evidence passages provided during generation. Figure 6b plots the cosine similarity between the n-grams found in the post-edited examples and evidence used as context. As can be seen, similarity to retrieved passages is fairly low in most examples, which suggests a low risk of memorization due to pretraining.

In addition, we check directly if text in our examples can be found in large pretraining corpora. We use an open-source toolkit called WIMBD (Elazar et al., 2024) to measure the presence of text in generated examples in two large pretraining corpora: C4 (Raffel et al., 2020) (364 million documents) and Dolma (Soldaini et al., 2024) (5,245 million documents). These corpora are widely used to train large language models (Raffel et al., 2020; Dodge et al., 2021; Chowdhery et al., 2023; Groeneveld et al., 2024). We follow prior work from Chowdhery et al. (2023) and determine an example as contaminated if 70% of all possible 8-grams in the example were seen at least once in the pretraining corpus. Conducting this process for a random sample of 100 examples in Dolomites, we find that none of the examples are contaminated.

We create a development-test split for Dolomites, with 820 examples in the development (dev) set and 1,037 examples in the test set. There are 172 seen tasks with examples in both dev and test and 99 unseen tasks with examples only in the test set.

For evaluation, we considered multiple performant models from various companies as well as open-source models. In all cases, we favored instruction-tuned variants because of their better performance on other benchmarks. Specifically, we report experiments with Claude-3 Opus (Anthropic, ¹), Command-R-Plus (Cohere, 2024), Gemini-1.5-Pro and Gemini-1.5-0409 (Team, 2024) and Gemini-Pro (Team, 2023), GPT-3.5-Turbo and GPT-4 (OpenAI, 2023), Mixtral-8×7B and Mixtral-8×22B (Jiang et al., 2024) and Mistral-Large (Mistral, 2024), and OLMo-7B-Instruct (Groeneveld et al., 2024). In all cases, we prompt models with the task description, the input sections corresponding to an example, and instruct them to generate the output sections for the example, in a zero-shot manner. Hyperparameters, prompts, and model identifiers are in  Appendix B.

We consider two modes of LM-based evaluation: pairwise evaluation and fine-grained absolute evaluation (illustrated in Figure 7). Language models are being increasingly used as evaluators (Chiang and Lee, 2023), and our primary evaluation also involves using LMs as evaluators. However, recent work points out that LM judgments can be misleading and biased (Shen et al., 2023; Wang et al., 2024; Zheng et al., 2023; Panickssery et al., 2024). We use multiple language models as judges to give preferences for a pair of model outputs. While this does not alleviate the problem of biased LM judgments, we believe it is slightly more reliable since we are not biased by a single model’s judgments. In all comparisons, we use one of the strongest models, GPT-4, as the base comparison model. We sample outputs on the test set from a candidate model and GPT-4 (randomizing their order in the prompt) and ask the evaluator model to judge which output is better and provide a justification. We consider three models as evaluators: GPT-4, Claude-3 Opus, and Gemini-1.5-0409. The win rate is computed by summing up the number of wins for a candidate model plus half the number of ties.

In addition to preference judgments, we evaluate models on five finer grained aspects of response quality: task adherence, factual correctness, depth, completeness, and coherence.² We use GPT-4 to collect absolute ratings (on a scale of 1–5) of individual model responses on each of these aspects.

We also measure the extent to which statements in the model output are consistent with statements in the reference output. We compute 1) forward entailment considering a reference section as the premise and the corresponding model section as the hypothesis and 2) reverse entailment considering a model output section as the premise and the corresponding reference section as the hypothesis. Scores are aggregated over all sections and examples. These metrics loosely capture the notions of precision and recall, we also report the harmonic mean of the two. We use the TRUE model (Honovich et al., 2022) to predict entailment scores (ranging from 0 to 1) and report 95% confidence intervals. Note that this metric has weaknesses, as it assumes that there is a single valid reference for each example, which may not be true for many examples in Dolomites.

Prior work has recognized that conventional metrics for text generation are lacking in various ways (Liu et al., 2016; Novikova et al., 2017; Krishna et al., 2021). Nevertheless, for completeness, we report results with ROUGE-L (Lin, 2004) and BLEURT (Sellam et al., 2020). In addition, we report the average output length and average section presence (i.e., the percentage of output sections specified in the task that are present in the generated output, averaged across all examples) as a measure of instruction-following capabilities. Note that the average length of reference outputs is 341.42 tokens.

To evaluate the reliability of automatic metrics, we measure how well the above automatic evaluation measures correlate with human judgments. Specifically, we sample 200 pairs of model outputs, where each pair comes from two randomly chosen models. We (the authors) then label which model output is better (or if they are tied) according to their task adherence, factual correctness, and depth.³ On these 200 pairs, we also get automatic preference judgments from all the evaluation measures discussed in Section 4.2 (we convert float scores for two outputs into binary judgments). The percentage agreements between human labels (with and without pairs with ties) and all evaluation measures are in Figure 8. In summary, we find that LM-based evaluation measures have the highest correlations with human judgments, followed by the NLI measures and then ROUGE-L and BLEURT. We note that an evaluator that always picks the longer output also has reasonable agreement rates.

We first report overall statistics of model responses and scores using conventional metrics in Table 3. Based on the average section presence, we note that models are largely effective at generating almost all relevant output sections from the task description. A few models (GPT-4 and Gemini-1.5-0409) excel more at following this instruction. Based on the NLI scores, we note that the nli (reverse) scores are on average lower than nli (forward), which suggests that generated outputs contain statements not entailed by the reference, e.g., because they are inaccurate or irrelevant. We observe that Gemini-1.5-0409 and GPT-4 produce more information that is factually consistent with the reference, while Claude-3 and Command-R-Plus are also performant. Once again, we note that reference-based metrics generally penalize outputs which are valid but different from the reference, and there might not just be a single reference for some of these examples, especially when the task is more subjective.

We show the win rates according to different LM evaluators in Table 4. Based on these win rates, we note that a few models such as Claude-3 Opus, Gemini-1.5-Pro, and Gemini-1.5-0409 prove to be comparable to GPT-4. We also report win rates with a length penalty for longer outputs in Table 7 and the overall rankings do not change.

Finally, we show ratings of models according to finer-grained aspects of response quality in Table 5. The overall conclusions are roughly similar, i.e., Gemini-1.5-0409, Claude-3 Opus, and GPT-4 have the highest ratings across axes. Across the axes considered in our rubric, models struggle most with the level of technical depth of the generated text.

Figure 9 illustrates how model performance fluctuates across fields; we show average ratings based on the fine-grained LM evaluation for Claude-3 Opus, Gemini-1.5-0409, and Command-R-Plus. Across models, we find that a few fields have significantly lower average ratings: Education, Sociology, and Chemistry. Tasks from a subset of these fields are sometimes subjective (e.g., Create a lesson plan to teach STEM educators, Write a summary of findings about the conclusions of a sociology book), which can make them hard to evaluate. In other cases, outputs can be lacking technical depth for tasks which require more domain expertise (e.g., Drafting a experimental protocol for a chemical synthesis procedure.). On the other hand, tasks in fields such as Literature, History, and Journalism have higher average ratings, some of which focus on factual reporting and narratives that may be easier to reason about.

Next, we evaluate whether output length is correlated with performance. Specifically, we show average ratings based on the fine-grained LM evaluation for Claude-3 Opus split into bins by length of the reference outputs. Scores stratified by length bins are shown in Figure 10. We find that examples which require longer outputs are significantly harder for models, supported by the fact that examples with longer outputs have lower average scores.

We analyze generated outputs from 3 high-performing models (Gemini-1.5-0409, GPT-4, and Claude-3 Opus) for examples where average ratings of responses are lowest. Broadly, we observe the following patterns:

• Lacking depth (Table 8): Writing technical documents requires depth and focus, which was sometimes found to be lacking. For example, concrete statistical results and method details were absent from a task on writing up a report on an experimental study in clinical psychology.

• Verbosity (Table 9): A common characteristic of some model outputs was their verbosity. Common patterns included defining jargon when not necessary, and generating many filler statements that do not introduce new information.

• Missing information (Table 10): There were a few cases where a single output section required multiple pieces of information, but the model entirely missed producing a subset of them.

The use of language technologies in domain-specific scenarios has the potential to help experts. Prior work has evaluated models through domain-specific benchmarks for standard tasks like QA (Hendrycks et al., 2021; Malaviya et al., 2024) and summarization (Hayashi et al., 2021). Many benchmarks have been proposed for specific fields (Rein et al., 2024; Xia et al., 2024), including law (Shen et al., 2022; Niklaus et al., 2023; Guha et al., 2024) and medicine (Tsatsaronis et al., 2015; Pampari et al., 2018; Jin et al., 2019, 2021; Fleming et al., 2024).

A notable difference between this line of work and Dolomites is the task formulation (i.e., QA vs methodical tasks). QA involves addressing a specific information need in response to a query while conducting methodical tasks requires following a structured and consistent procedure, involving multiple steps, to complete a goal-oriented task.

Evaluation that is grounded in realistic use cases, is important for reliable benchmarking (Rolnick et al., 2024). Prior efforts on creating NLP benchmarks which are representative of real user needs use query logs (Nguyen et al., 2016; Kwiatkowski et al., 2019) and chat histories (Lin et al., 2024). While these benchmarks are useful for evaluating responses to generic user queries, they do not allow us to study their abilities in assisting with domain-specific tasks in an isolated manner. For instance, Ouyang et al. (2023) find that user requests in chat histories often involve “planning” and “design”, but these are largely ignored in benchmarks. Our work fills this gap by presenting a typology of domain-specific tasks grounded in realistic scenarios.

Recent work has investigated the potential of language models to act as writing assistants for domain experts (Calderwood et al., 2020; Lee et al., 2022; Gero et al., 2022; Li et al., 2024). While there is favorable evidence that they can improve productivity of experts (Noy and Zhang, 2023), their usage has broader societal consequences, including potential impact on the workforce (Eloundou et al., 2024). We analyze a subset of these societal implications relevant for our methodical tasks in Section 3.

As part of Dolomites, we present various data artifacts that can be used to study the abilities of language models in assisting domain experts with writing tasks. We outline these artifacts and their intended use below.

We present a collection of 519 writing tasks spanning 25 fields that is representative of work undertaken by domain experts on a regular basis. We believe this is the first collection of tasks built with input from domain experts about scenarios in which language models can be useful to them. These tasks can be used to identify applications of LMs in various fields and to cater model development to ecologically valid tasks.

We conduct an independent assessment of the validity of the tasks and the societal implications of using LMs as writing assistants for these tasks. We believe that these validation labels can be useful to study the practical benefits of using LMs as writing assistants for these tasks and to select tasks that are representative of real-world use. Labels concerning societal implications can enable researchers to take into account important considerations such as anonymity of user data, bias in decision making, accessibility requirements, etc, when considering deployment of language models for assisting experts.

Finally, we present examples of tasks that are semi-automatically generated by seeking input from the same experts who provided the tasks. The examples of tasks are meant to represent concrete and plausible instances of the task, so that models can be evaluated on the tasks. To conduct such an evaluation, models are required to generate the output of an example given the task description and the example input. In Section 4.2.1, we present results on Dolomites using LM-based evaluators as well as other metrics. We find that LM-based evaluators correlate best with human judgments and propose future work to consider the following modes of evaluation on Dolomites:

1. Pairwise preference judgments for overall response quality: Pairwise evaluations from models (especially from Claude-3 Opus) have moderately high agreement with human labels (67% with ties, 77% without ties) that are better than other evaluation metrics. These can be used to get an estimate of overall model abilities.

2. Finer-grained LM evaluation for absolute judgments on given axes: In Section 5, we present fine-grained evaluation on axes that are important for our tasks. We propose future work to conduct finer-grained evaluations on the same axes to gather better insights about the strengths and weaknesses of models.

We believe that as LM-based evaluators improve, we will be able to more accurately evaluate outputs for examples in Dolomites.

We introduce Dolomites, a benchmark that is closely tied to realistic use cases of domain experts. The generalization of these use cases as methodical tasks provides a way to study capabilities of language models across tasks and domains. We consider a scenario where AI systems can act as tools for experts to amplify their problem-solving capabilities (Engelbart, 2023) and perform their tasks more efficiently. We verify that our tasks are representative across fields and that human oversight is necessary if language models propose initial outputs for these tasks. Evaluation of a broad range of contemporary language models suggests that there is a large room for models to improve on generating outputs for our tasks.

Future directions are many and varied. The tasks in Dolomites constitute a mere sample from 25 fields in English language. We hope to further expand the set of tasks to cover a wider range of scenarios and languages. We could also consider tasks that involve modalities other than text in input or output, and multi-turn settings, where models continually improve their outputs through feedback and revision. On the modeling front, we will consider sophisticated generation techniques such as the one proposed by Narayan et al. (2023), that first generate a plan of the output and then fill in different sections, potentially with attributions to sources (Fierro et al., 2024). Our experimental results revealed that automatic evaluation of generated text is particularly challenging. Our data contains a single reference output for an example input and does not model diverse perspectives of experts and the innate subjectivity of tasks (Ganguli et al., 2023). While conventional metrics do not account for this subjectivity, it is unclear if LM-based evaluators innately capture this subjectivity. More research is needed to ensure language model responses are given credit for alternative, but valid responses.

We are grateful to the anonymous TACL reviewers and the action editor Minlie Huang for their helpful feedback. We thank the 266 annotators who participated in our data collection studies and the following people for their help with facilitating data collection: Shereen Ashraf, Michelle Chen Huebscher, and Michael Sharman. We also thank the following people for helpful discussions and feedback: Sumit Asthana, Tuhin Chakrabarty, Michael Collins, Sunipa Dev, Jacob Eisenstein, Sebastian Gehrmann, Kalpesh Krishna, Tom Kwiatkowski, Matthew Lamm, Kenton Lee, Joshua Maynez, Slav Petrov, Hannah Rashkin, David Reitter, Marco Tulio Ribeiro, Elizabeth Sieber, and Kristina Toutanova.

We recruited 266 participants for our study from Prolific. Participants were required to be fluent in English, and came from 25 different countries, across Africa, Europe, and North and South America. In terms of their background, participants were required to have an undergraduate degree and 3 years of work experience in their respective field. These requirements were first enforced through Prolific’s audience filters, followed by a screening where participants were asked to self-report their educational qualifications and work experience. They each provided two tasks, so for each field, we recruited half the number of the participants as the number of tasks reported in Table 1. Lastly, they were required to have at least 50 prior approved submissions and an approval rate of over 99%. Participants were informed that their provided data will be used to evaluate large language models in realistic scenarios. We obtained prior consent from all annotators before recruiting them for all studies.

Annotators were paid $20 per hour for their work. For task collection, we allocated 40 minutes to write two tasks, and for task validation, we allocated 15 minutes per task. For post-editing examples, we allocated 20 minutes per example.

In a random sample of 100 examples, we found the following types of edits were made to the examples:

• Fact Addition (88%): Addition of new statements to the example.

• Fact Deletion (20%): Removing statements from the example.

• Fact Update (65%): Updating existing statements with further elaboration of details, or adding of new numbers or references.

• Stylistic Rewrites (76%): Simplification, paraphrasing, or improving grammar, spelling or tone of text.

• Reorganization (23%): Restructuring of sentences, paragraphs, or sections in the example, which may be done to fit the task description.

We show screenshots of the annotation interfaces presented to annotators for task validation and example post-editing in Figures 11 and 12, respectively.

The specific identifiers for the models evaluated in this work are given in Table 6. Open-source models were obtained from the HuggingFace model hub, while proprietary models were obtained through the organizations’ official APIs.

In all generation tasks, we set the temperature for generation to be 0.1. For both example generation and model evaluation, we sampled a maximum of 4,096 tokens (or the maximum sequence length of the model).

We provide the prompts used for various components of our work. The prompts used for example creation are given in Tables 11–13. The prompt used to generate the critique shown to annotators is shown in Table 14. The prompt used to generate outputs from candidate models is shown in Table 15. Finally, the prompt used for generating LM-based judgments for evaluation are shown in Tables 16 and 17.