Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs

It is crucial for a democratic system to guarantee space for a plurality of ideas and opinions in all kinds of communication situations, be they political, professional, or personal (Balkin, 2017). Over the last few years, one particular communication situation—interactions between chatbots powered by LLMs and their users—has become a commonplace setup for many everyday communication tasks, such as assessing arguments, summarizing texts, or writing emails (Wolf and Maier, 2024). Our understanding of the extent to which such LLM-based scenarios guarantee space for ideas and opinions of various kinds or, conversely, to what extent they are biased (Blodgett et al., 2020), is still unfolding. Continuing work on identifying biases in previous NLP resources and models (Hovy and Prabhumoye, 2021), studies have found biases of numerous types in LLMs, including gender (Kotek et al., 2023), race (Omiye et al., 2023), culture (Arora et al., 2023; Wang et al., 2023b), and political position (Feng et al., 2023). Such biases need to be understood when developing downstream applications to avoid harmful or unpleasant effects on users, such as narrowing one’s view on a topic.

In this paper, we focus on political bias in LLMs. Recent studies claim that the output of LLMs tend to agree more with left-wing political positions (Feng et al., 2023; Motoki et al., 2024). However, the scope and interpretation of these findings is not yet clear: Political positioning is an inherently multidimensional phenomenon, and while political individuals and organizations (e.g., parties) typically exhibit substantial (even if typically imperfect) internal consistency (Moskowitz and Jenkins, 2004; Tavits, 2007), this is not necessarily true for LLMs, which have only a weak notion of consistency (Basmov et al., 2024).

We argue for a distinction between political bias and political worldview. For the former to manifest, it is sufficient that the model shows a distinct preference for a particular policy. This amounts to independent stance taking (Küçük and Can, 2020) with respect to individual target statements. Arguably, this behavior constitutes a form of representation bias (Mehrabi et al., 2021; Suresh and Guttag, 2021), because when the model exhibits a preference, it reflects only one worldview rather than that of a representative sample of the population. The latter, in addition, requires consistency across a set of such policies. This is similar to how political science describes the positioning of human actors in the overall political space spanning multiple policy issues using the term “worldview” (Ecker et al., 2021). The term has also been suggested to apply to LLMs (Bender et al., 2021).

These characterizations suggest that political bias and political worldview can be distinguished with the help of two criteria. If an LLM fits the first, it shows political bias. If it shows both, it shows a political worldview. The first criterion is whether the models show high reliability in assessing political statements¹ —that is, whether they give consistent answers irrespective of the formulation of the prompts. If this is not the case, models merely react to linguistic peculiarities, namely lexical choice, token or (textual) position biases (see Section 2 for details). The second criterion is whether models show consistency in their political worldview: Whether they exhibit a consistent stance towards broad policy issues, with limited variance among statements within these issues or a consistent commitment to a right or left leaning across issues.

To improve our understanding of political bias in current LLMs, we make three contributions:

1. We build ProbVAA, a dataset with statements on policy measures from seven EU countries with the answers from political parties. ProbVAA contains paraphrased, negated, and semantically inverted versions of the statements, and policy issue annotations (§4).

2. We propose a method for evaluating the reliability of the LLMs’ output across variations of statements and prompts (§3). It adheres to psychometric standards and involves expanding the dataset in accordance with these principles. This work is most similar to Shu et al. (2024), but prioritizes a data-centric approach, indicating that the analysis can be conducted on both open- and closed-source models, solely utilizing the responses produced by the LLM.

3. We evaluate a range of SOTA LLMs on the ProbVAA dataset, finding substantial differences among LLMs with regard to reliability (§6). When evaluating stance on reliable statements (§7), we find that LLMs align more with left-leaning parties overall, but lack consistency regarding leanings: They tend to have no preference for some issues (migration, foreign policy) but agree with policies as divergent as pro-environment and law and order.

The characterization of political positions is an important topic in political science, and a considerable number of computational models has shown that positions can be inferred from political texts (e.g., Laver et al., 2003; Slapin and Proksch, 2008; Glavaš et al., 2017). Comparing the positioning of political parties at low dimensional level under pre-defined scales remains an elusive goal in political science (Heywood, 2021). One of the most widely used scales is left-right, arguably distinguishing between progressive position (left), conservative positions (right), and compromise positions (center). Despite concerns about its validity (Kitschelt, 1994; Jahn, 2023), the scale has been validated broadly across countries (Evans et al., 1996; Budge et al., 2001) and also formed the basis for previous analyses of political bias in LLMs (Feng et al., 2023). An alternative to positioning actors on a scale is to carry out a fine-grained analysis at the level of individual policy issues (Iversen, 1994; Ceron et al., 2023). For our consistency analysis in Section 7, we look at both of these levels (left-right scale and positioning within policy issues).

Recent work has examined LLMs’ political ideology using surveys such as Political Compass (Feng et al., 2023; Motoki et al., 2024; Rutinowski et al., 2024), or more country-specific questionnaires such as Pew Research’s ATP, World Values Survey (Santurkar et al., 2023), and voting advice applications (VAAs) (Hartmann et al., 2023).

Different methods have been utilized to capture bias, including integrating the agreement options directly within the prompt, averaging model responses (Rutinowski et al., 2024) and prompt paraphrases (Feng et al., 2023). Another approach stream leveraged the form of multiple-choice questions where the response polarity was determined by extracting log-probabilities of answer options to obtain the model’s opinion distribution (Santurkar et al., 2023), shuffling the option order within the prompt (Durmus et al., 2023) and using response sampling with randomizing question order (Motoki et al., 2024). However, each approach tackled a single aspect of reliability—either the LLM’s prompt sensitivity or the stability of their output.

The assessment of output variability and the quantification of model reliability in recent studies have involved the application of psychometric methods from social psychology. These studies have utilized standardized methodologies (Dayanik et al., 2022) and questionnaires to create controlled environments for extracting reliable “attitudes” from LLMs (Tjuatja et al., 2023; Dominguez-Olmedo et al., 2023; Shu et al., 2024). Such approaches have proven to be instrumental in examining various societal biases in LLMs (Arora et al., 2023; Wang et al., 2023b; Hada et al., 2023; Esiobu et al., 2023; Shu et al., 2024). However, the exploration of psychometric methods to investigate political bias remains limited.

There is a series of studies suggesting that the input to an LLM plays an important role in determining its output. For example, Min et al. (2022) show that swapping out gold labels for random ones only slightly reduces performance—a pattern that remains stable across almost all tested models regardless of the prompt instruction used. Khashabi et al. (2022) observe that continuous prompts manage to solve a task even when presented as an arbitrary instruction, staying surprisingly close (within a 2% range) to the best prompt of the same size designed for that specific task. Finally, the meaning of prompts can be overshadowed by the choice of target words (Webson and Pavlick, 2022) which goes hand-in-hand with observed high result variance caused by recency and common token bias phenomena when the model chooses the most frequent token (Zhao et al., 2021), or position bias when the model prioritizes labels that appear at a specific position (Zheng et al., 2023).

Following up on this motivation, we now present our framework for evaluating the political bias of LLMs which involves two key elements: (1) enrichment of the dataset with prompt variations and policy issue annotations and (2) evaluation of the reliability of answers in terms of stances.

Figure 1 illustrates the workflow for creating model inputs. Overall, given an input which contains a single statement reflecting a particular view on a societal or political issue or a policy proposal, the model is prompted to provide a binary response indicating its support or opposition. In the subsequent discussion, we refer to model response as binarized free-text response with agreement/approval as opposed to disagreement/disapproval towards the given input.

After collecting our target dataset (details in § 4.1) we enrich it with paraphrases, negated and opposed versions of the original policy statements (details in § 4.3) to evaluate whether the model produces coherent responses when confronted with semantically equivalent or logically contradictory inputs in comparison with the responses of the original statement.

As Figure 1 shows, the first step of the method assesses the statement variants (1). In addition to that, the reliability with respect to variations of prompt instructions is evaluated by (2) using two types of instructions (personal and impersonal questions), (3) using synonyms for the response alternatives that the model should select, and (4) swapping the order of the alternatives (§ 5.2).

We argue that, if the answers to a certain statement are reliable under different prompt variations, where the meaning of the original statement is either preserved or logically flipped, there is a high likelihood that this worldview is embedded in a given LLM instead of being the result of a choice in the sampling of the generated tokens caused by frequency or position token bias (§ 2).

To further establish a robust probability for the generated stance with regard to variance induced by decoding 30 responses are generated for each prompt. This allows for an evaluation of the statistical significance of the most-frequent binary response (§ 5.4).

We envisage several points in the workflow as tests which models can pass or fail with regard to a particular statement. As illustrated in Figure 2, the test types are (1) robustness to sampling (with a fixed prompt), (2) robustness to paraphrasing/negation/semantic inversion of the original statement, and (3) robustness to label-order inversion in the prompt instruction. Only statements on which the models pass all tests are used to assess the models’ attitudes. They are considered, in this approach, reliable statements because they have reliably yielded the same stance from the model, and therefore, are worth to be further evaluated. policy issue annotations on the dataset make it possible to make the analysis of the reliable statements more fine-grained (§ 6 and 7).²

To assess the potential political worldviews embedded in LLMs, we collect a set of statements derived from Voting Advice Applications (VAAs). VAAs are tools that provide voters with insights on which parties are best aligned with their own opinions regarding policy issues. Unlike the frequently used Political Compass questionnaire, which categorizes political attitudes into a two-axis system (left/right and authoritarian/libertarian), VAAs offer a nuanced approach that ground political leanings in stances towards practical policies (Palfrey and Poole, 1987; Tavits, 2007). These stances allow for a direct comparison of responses with those from national parties and/or candidates. On the one hand, this offers a more unbiased basis for measuring political leanings, as it does not rely on the questionnaire designer’s external classification to determine if an answer aligns with the “left” or “right” side of the political spectrum. On the other hand, it covers a wide range of policy issues that varies from environmental protection to government expenditures, providing more fine-grained insights on the types of biases.

Concretely, we collect the statements and answers of VAAs of the parliamentary elections (ranging from 2021 to 2023) from seven countries (Poland, Hungary, Italy, Germany, Netherlands, Spain, and Switzerland) in 7 languages. The length of questionnaires varies between 20 and 60 questions (a breakdown of the number of statements per country is shown in Figure 8, Appendix C).

Most of them are in the format of statements, except for the Swiss VAA, which contains questions that we manually convert to statements to align with the other countries. The dataset contains a total of 239 unique statements in the source languages (Switzerland has 60 statements for each language [German, Italian, and French] but only 60 count as unique given that they are the same statements). In order to answer our research questions, we annotate the datasets in a number of ways discussed below.

We have enriched ProbVAA with policy issue annotations based on the pattern of the Swiss VAA, SmartVote.³ It contains annotations that allow for the visualization and deeper understanding of the positioning of parties according to predominant policy issues in the political spectrum. We draw from the documentation provided by SmartVote where eight categories (considered stances on policy issues) are defined: open foreign policy, liberal economic policy, restrictive financial policy, law and order, restrictive migration policy, expanded environmental protection, expanded social welfare state, and liberal society. These categories are based on policy issues identified in the Swiss political spectrum (Hermann and Leuthold, 2001, 2003), but that are generalized across European countries, as evidenced by the similarity with issues analyzed in cross-European studies such as the Chapel Hill Survey (Jolly et al., 2022).

When answering ‘agree’ to a statement emphasizes any of the eight given policies, the statement is marked as a ‘agree’ with that policy issue, while disagreements with a policy are annotated as ‘disagree’. Three annotators with background in traditional or computational political science extended the annotations to the other countries. Table 1 shows that inter-annotator agreement—which is calculated with agreement between ‘agree’, ‘disagree’, and ‘no label’ per statement—is good. The final gold annotations are drawn from the majority votes. Note that some statements do not fall into any category. Therefore, the gold annotations contain 193 statements in total (Tables 9 and 10, Appendix A provide examples and details).

We introduce three variants of each policy statement to test the models’ reliability (cf. statement variants, Figure 1 and robustness to statement variations in Figure 2).

With paraphrasing, we aim to measure how consistently the models (or humans) generate the same stance on semantically similar statements. For every statement (S) in the source language (S_src) and in English (S_en), we generated three paraphrases using ChatGPT4. Native speakers read a sample of 60 paraphrases for 20 Ss in the source language and confirmed that they are syntactically and semantically correct.

These two tests evaluate whether the models (or humans) generate the opposite stance when presented with a negated or semantically inverted version of the original policy statement, i.e., agree for the original and disagree for the opposite and vice versa). Given statement S, its negated opposite, which we denote as Neg(S) is its logical opposite, which is constructed by adding an overt negation marker in the appropriate position in the statement.

The other type, which we call semantic opposite and denote Opp(S), is a statement that takes the semantically opposite sense to the original one while not using an overt negation marker. A minimal number of words is modified to convert the semantic meaning of the sentence.

Each statement in the source language is annotated by a native speaker. Annotators are asked to create Neg(S) by adding a marker corresponding to ‘not’ or ‘don’t’ in the source languages. As for Opp(S), annotators are instructed to first try modifying the head verb in the statement or, if this is not possible, the focal adjective. If neither can be altered, they are asked to apply the minimal change necessary to invert the sentence’s meaning.

Every statement (S) with their respective Neg(S) and Opp(S) has been automatically translated into English with the commercial translation tool DeepL. The quality of the translations has been validated on a subset of the statements by the authors. Altogether, this results in 1434 statements in English and in the source languages. The ProbVAA dataset consists of both English and original-language statements, but we only use the translated statements for this study because the evaluated models have not been instruction fine-tuned in the source languages.

In this section, we describe the models that we use (§ 5.1), our prompting, sampling, and output-mapping strategy (§§ 5.2–5.4).

Given that we formulate our prompts as zero-shot instruction, we opt for the instruction-following model families that vary in parameter size and contain different sets of pretraining data⁴ . Specifically, we focus on all size variants of LLama- 2-Chat (7B, 13B, 70B) (Touvron et al., 2023), the XXL variant of Flan-T5 (11B, Chung et al., 2024), Mistral Instruct (7B, Jiang et al., 2023) for open-sourced models, and GPT-3.5 as a closed-source model. The models form three natural classes: small (<10B parameters, mistral-7b and llama2-7b), medium (between 10 and 19B (flanT5-xxl-11b and llama2-13b), and big (>20B, gpt3.5-20b and llama2-70b). All models utilize the top-p nucleus decoding approach, suggested by Holtzman et al. (2020), when generating responses, aiming to capture the model’s stance distribution through the sampling of the output (cf. Appendix B for information on the implementation).

For clarity, we define a prompt as composed of two parts, the prompt instruction (which contains the instruction given to the model) and the statement (an instance from ProbVAA).

When designing model inputs, we aim at creating templates that reliably elicit responses from models that can be clearly aligned with a defined stance, so either positive or negative.⁵ Considering recent research findings indicating that the meaning of prompts can be overshadowed by the choice of the target words as discussed in § 2, we aim at diversifying the choice for prompt instructions from various angles. We distinguish between personal and impersonal templates (cf. prompt variants as shown in Figure 1). Personal templates formulate the instruction as a direct question (e.g., Do you agree or disagree?, Are you pro or con?), whereas impersonal are framed as objective tasks (Analyze the following statement into the labels “favorable” or “detrimental”..., Classify the following statement as...). The context for evaluating the prompts is specified as Consider the long-term societal impact... Additionally, we vary the wording of the stance (e.g., favorable, detrimental, advantageous, disadvantageous, support, oppose) to explore potential model biases in responding to specific wordings (cf. semantic label order, Figure 1). After a pilot experiment to test which prompts elicit most valid responses, we selected 3 personal and 3 impersonal prompt instructions among 8 impersonal and 6 personal templates (Appendix B.1 details the selection process). Refer to the implemented prompt instructions in Table 7, Appendix A.

In order to test sensitivity of the models to subtle template changes each template is furthermore presented in two versions: the original one and the version where the order of the labels is swapped, e.g., if a template states, Analyze the following statement into the labels “favorable or detrimental”..., the inverted-label version corresponds to “detrimental or favorable”. A reliable model is expected to yield the same response independent of label order (cf. robustness to label-order inversion, Figure 2).

In addition to altering the statements, we modify the templates to investigate if the model maintains consistent stances with semantically equivalent templates. Previous research has demonstrated the impact of template variation on the results (Min et al., 2022; Khashabi et al., 2022). We hypothesize that variations in templates are likely to be an influential factor in shifts in the models’ generated stance.

We automatically map the generated answers of the models to either a positive or negative stance towards the statement using manually designed heuristics. In the best case, the models followed the instructions and just generated one of the two option labels that were asked for in the instructions (each template has exactly one label, in favor or against a certain policy). In case the model outputs some variation of or longer generated output, we search for the first occurrence of one of the option labels so that we can map it to the corresponding stance (Wang et al., 2023a). If the label is negated (e.g., not favorable or don’t agree), we map it to the opposite stance. We manually inspect sample answers across models to check whether the rule-based approach maps all possible responses correctly.

The last component missing is the procedure to determine whether a given prompt is answered reliably by a model. To do so, 30 responses are sampled from the model for each prompt (template + statement) (cf. robustness to sampling, Figure 2). After excluding unclear or ambiguous responses, we calculate the relative frequency of positive and negative stances on the remaining answers. To assess the significance of these proportions, we use a 1000-repetition bootstrap test to estimate 95% confidence intervals for the mean stance. We define a model’s response as reliable if both values 0.55 and 0.45 lie outside the 95% confidence interval. This is a more conservative procedure than checking for the absence of 0.5 to ensure that the model exhibits a clear leaning towards either the positive or the negative stance.

We are now finally equipped to practically identify the precise set of statements for which a model can provide reliable responses.

Within each template, a statement of ProbVAA passes a test when it yields exactly the same stance when comparing with its paraphrased versions and in the inverted label. It passes the test in the negated and semantically opposite versions when it yields the opposite stance. Finally, it passes the significant test when a given stance is statistically significant within the 30 samples. We report the number of statements that a model-template combination has passed for a specific test, and the proportion of statements that passed all tests.⁶

To define an upper bound for the semantic and negation/opposite reliability tests in humans, we conduct an annotation study. We sample 50 different S’s from ProbVAA together with their corresponding Neg(S), Opp(S), and one Para(S), resulting in a total of 200 statements. All statements are in the English translation. This questionnaire is provided to 6 student participants from a survey about political policies (demographics in Table 5, Appendix A) who are asked to answer Agree or Disagree for each statement in line with their personal political positions. As a random baseline, we generate a sample of 30 random answers for each statement variant and evaluate according to (§ 5.4).

Figure 3 shows the percentages of statements that pass different reliability tests for each model. Table 2 reports Cohen’s κ for reliability under paraphrasing, negation and inversion for both models and human annotators. Reliability in general increases with parameter count. Thus, llama2-70b yields a robust probability for more than 80% of the statements while mistral-7b and flanT5-xxl-11b only generate a reliable answer in about 40% of the cases.

All models are substantially reliable for paraphrase and inverted label order, with flanT5-xxl-11b being as reliable as larger models for paraphrases. An outlier for inverted label order is llama2-7b, for which we notice a large variance across templates. This shows that inverting the label order has a significant effect with some templates. Compared to humans, the models still fall short on paraphrase reliability, except for llama2-70b, which is on par with the human annotations set as upper bound.

The models exhibit greater difficulty in maintaining reliability when dealing with negation and inversion. While the lower agreement for humans on these two tests shows that this setting is hard in general, the discrepancy between human performance and model performance is substantial. Notably, llama2-7b and mistral-7b do not even outperform the random baseline on these tests.

Models improve on all reliability tests with increasing parameter count. In the medium-size class, flanT5-xxl-11b often outperforms the larger llama2-13b. gpt3.5-20b though, while notably smaller than llama2-70b, is almost as reliable and shows the best performance on negation and inversion and the lowest variance across templates.

Nevertheless, the gap between models and humans on the three reliability tests targeted in the human annotation study is very large, and that the only case where a model shows comparable performance is llama2-70b on paraphrases.

Table 3 presents the reliability of the models across templates. It shows the agreement in stance for the original template variant across 6 prompt instructions and the number of statements for which the models always predict the same stance. llama2-7b is the least reliable across templates. mistral-7b, flanT5-xxl-11b and llama2-13b, on the other hand, have a moderate agreement, while gpt3.5-20b and llama2-70b are very robust.

This section aims to understand to what extent the models’ answers also exhibit political consistency—i.e., constitute a “worldview” by virtue of taking the same stance on statements related to one another within policy issues, and overall showing a good fit with one political leaning. We only include statements that pass all reliability tests.

In this part of the evaluation, political parties are categorized into left/ center/right-leaning based on the well-established Chapel Hill survey (Jolly et al., 2022) from 2019 (refer to Appendix A.4 for more information about the survey). We then compute the political leaning by counting the number of times the answers of the reliable statements of the models match with the answer of the parties provided to the voting advice applications (cf. Appendix A.2).

We simulate models that always agree and always disagree with the statements of ProbVAA. They are respectively called alwaysAgree and alwaysDISagree. They serve the purpose of disentangling the results of the analysis of the models from the answers of the parties. We use them to ensure that the parties’ tendency to answer “agree” or “disagree” does not affect the analysis of the models’ answers.

Figure 4 illustrates the relative number of answers that match the party’s responses to a given VAA averaged across parties from the same leaning (left, right, and center). The error bars represent the standard deviation of the means across templates. The legend on the right shows the average percentage of reliable statements across templates.

According to Figure 4a, the results of the model alwaysAgree suggest that left- and center-leaning parties tend to agree with the statements, whereas right-leaning parties tend to disagree as shown by the results of alwaysDISagree model. Given this tendency in the answers of the parties and the fact that the models agreed more often within the reliable statements (cf. Figure 6 in Appendix C), we separate our analysis between the agree and disagree answers to ensure that the results are not led by spurious aspects of the dataset. Figure 4b shows that despite the fact that right-leaning parties disagree more often, all models are still more aligned with left-leaning parties. They are also more clearly aligned with left parties than center parties even though there is no discrepancy, as shown in Figure 4a, between left- and center-leaning for agreeing and disagreeing. Among all models, llama2-7b is the one where the gap between center and left is the smallest whereas llama2-70b has the most significant difference with 10% of the alignment with left-leaning parties while only 2.54% with right-leaning and 5.09% with center parties. Similar findings are observed within the set of statements that the models agree with: As Figure 4c shows, the strongest alignment with the left orientation takes place at llama2-70b whereas the weakest alignment is observed in llama2-7b. All models from mid to big sizes have the same alignment with right-leaning parties while the big models align more with center-leaning parties in comparison with the mid-size models.

Figure 5 shows the stance of the models per policy issue with the standard deviation across prompt instructions. Positive values correspond to positive attitudes towards a policy issue, and negative values (visualized in gray) correspond to rejection of a certain policy stance, while values around zero indicate neutrality (or the fact that the model does not have enough reliable statements in that issue). To disentangle these two cases, we mark by dots cases where the models did not consistently answer at least 6 statements per policy issue across all templates.

Dots show that the two small models do not answer a significant number of statements for most policy issues. flanT5-xxl-11b, on the other hand, does not have enough reliable statements relating to restrictive migration and law and order. llama2-13b and the big models, on the other hand, cross the threshold for all policy issues. Nearly all models, except for llama2-13b, have a higher standard deviation in the issue of open foreign policy. It is important to highlight that all models, except for llama2-7b, tend to answer in agreement with the policies within the set of reliable statements (cf. Figure 6 in Appendix C). This explains why llama2-7b is the only model whose answers vary between neutral and negative stance within environment protection, social welfare state, and liberal society.

Across the mid- and big-size models, we observe a strong agreement among models in favor of encouraging the expansion of social welfare state and liberal society while having a moderate positive stance towards liberal economy and restrictive finance. Regarding environmental protection, flanT5-xxl-11b and gpt3.5-20b show a clear positive stance whereas llama2-13b and llama2-70b yield a moderate stance. llama2-13b and the big models, moreover, tend to agree with policies that favor law and order. Lastly, flanT5-xxl-11b is the only model that holds a positive stance towards open foreign policy. Finally, models generally take no clear stance in the issues of restrictive migration policy.

Finally, our results demonstrate that focusing on statements that have passed all reliability tests strengthens the validity of the results. This approach ensures that the findings reflect biases in the models rather than position or token biases given they have been tested across various prompt formulations. The validity can be observed in the variance of the results when comparing statements under different reliability constraints. The standard deviation across templates is lower in all models, except for gpt3.5-20b, in the strongest test (statements that successfully passed all tests) in comparison with fewer constraints. Figure 7 in Appendix C compares the answers of the models under the significant&label inversion&paraphrase tests and no tests, irrespective of reliability. Increasing the number of tests reduces variance across templates, indicating that biases become more consistent within reliable statements and validating the importance of verifying for prompt brittleness.

Compared to human performance, all models fall greatly behind in terms of understanding variations in the semantically opposites or negated statements, showing substantial sensitivity to different prompt formulations. Overall, the higher the number of parameters, the more reliable models are, as shown in previous studies (Shu et al., 2024). Results across reliability tests show that small- to mid-sized models are unreliable in relation to giving consistent answers to the same policy statement while big models are slightly more reliable, but are still prone to generating variable answers, specially in the negated version of statements and prompt instruction variations. Even though previous studies (Feng et al., 2023; Motoki et al., 2024) found that models have a tendency to be more aligned with the left-leaning ideology, this can be reliably claimed only for LLMs with at least 20B parameters count. The results also shed light on the importance of carrying out various tests in order to understand whether a political worldview is really embedded in LLMs due to the training regime or the result of common token bias, lexical or position bias in the sampling of the generated tokens.

Regarding consistency, categories where models hold no or weak stances point to a lack of consistency in the worldview within a given policy issue. This means that even though small models show a left-leaning positioning in the first analysis, they do not take any clear stance towards any issue formulated in the second analysis—showing lack of consistency in supporting any left-leaning agenda. The remaining models show low consistency for a very divisive topic in the political spectrum of left and right scaling such as migration. They exhibit a very moderate take on financial policy (related to expenditures of the government, and tax cuts or increases). In contrast, the analyses reveal a consistent take on issues such as environment protection, liberal society, and social welfare across models. The stronger alignment with left-leaning parties may be expected, given that left-leaning ideological principals tend to be more vocal about these policies (Benoit and Laver, 2006; Budge, 2013). Overall, these findings suggest that models have political biases (cf. § 1), but do not show a consistent worldview in terms of leaning across policy issues. Finally, they reproduce a consistent worldview only at few policy issues.

That said, it is surprising that llama2-13b and big-size models take a positive stance towards law and order (e.g., measures that favor values of discipline and protect public safety) and a moderate stance on liberal economy, which is usually attributed to policies encouraged by right-leaning parties (Budge, 2013). Results thus suggest that mid- and big-size models show a certain degree of inconsistency in terms of political leaning—favoring both left- and right-leaning programs. This emphasizes the need for a thorough evaluation of the stances taken in the answers of LLMs. It is crucial to understand preferences at the fine-grained level in order to better interpret the alignment with one or another overall leaning.

Finally, while understanding where these biases come from is outside the scope of this paper, we believe that there are two main sources. Given that they are relatively similar across models, we hypothesize they may be shaped by the data used for pre-training, which is similar across models incorporating a wide variety of textual sources, such web pages, social media, academic material, books and encyclopedias (Liu et al., 2024; Gao et al., 2020). Our preliminary studies with base models were not reliable, so we cannot investigate whether the reinforcement learning with human feedback has an impact on the biases and worldviews. Further investigation is needed to understand the biases at the different training stages of these models.

In this paper, we proposed a method and dataset for robustly evaluating the political biases in LLMs. Our experiments (1) shed light on the importance of thoroughly evaluating the answers of LLMs under different reliability tests, and (2) provide a more nuanced understanding of the political biases and political worldviews encapsulated within LLMs.

We find that models align best with parties from the left part of the political spectrum, but that even large models lack consistency for at least some salient policy issues, such as migration and foreign policy, and favor policies in the issue of law and order policies that do not correspond to the general left-leaning programs. In this sense, we would advise caution in assigning a leaning to LLMs given that this “worldview” is not consistent across policy issues.

Even though we applied the idea of reliability- aware evaluation to political bias in this paper, we believe that the usefulness of our proposal extends to the analysis other types of biases in generative LLMs. The first step (of generating variants of prompts) should apply straightforwardly to any other bias-related dataset. For the second step (of analyzing variance within broader categories of statements), the experimental materials need to form categories, but this also generally the case.

A crucial question is how to appraise the outcome of our analysis: Are reliable political biases in LLMs good, as long as they align with desirable political values, or would we rather have high-variance models that do not commit to specific political leanings? It is unequivocally clear that we must prevent models from generating responses that exhibit gender bias or racism. However, it is less clear what type of political biases models should embed, given that they align less with common ethical values of society and more with individuals’ values. Therefore, our findings highlight the need (1) to understand where in the process of LLM construction these biases arise, during pre-training, the instruct-fine-tuning, or reinforcement learning stages; and consequently (2) to pressure companies training these models to be more transparent about their training regime so that models can be comprehensively evaluated; (3) to keep developing more robust methods to evaluate LLMs that factor in prompt brittleness (Choshen et al., 2024; Mizrahi et al., 2024), and finally (4) to re-think what type of information these models should embed in real world applications while taking societal implications into account.

Firstly, the simplification of questionnaire responses to agree, disagree and neutral reduce the degree of nuanced perspectives from the parties and the models, as the original questionnaires provide a broader spectrum of response options.⁷ Secondly, by restricting the models’ responses to binary choices without a neutral option, we may have constrained their ability to express more nuanced views. Next, even though the dataset includes a wide range of countries, we only evaluate English translations of the statements given the limitations with prompting LLMs in languages other than English. In addition to that, the dataset is based on data from European countries only. Therefore, some policy issues may include common European issues (such as the use of a common currency and a country’s sovereignty in relation to the European Union) which at times are not representative of the global political spectrum. Finally, given that base models did not yield reliable responses in our setup, it suggests that prompting is not the ideal for identifying biases in base models given that they have not been trained for this purpose. This opens a venue for further investigation concerning the difference of biases between chat and base models, and where biases stem from.

We are thankful for the native speakers of the target languages who volunteered to check the translations, and convert the sentences to their respective negative and opposite versions. We are also grateful for the reviewers and action editor of TACL who provided us valuable and insightful comments, enriching the quality of our study and manuscript. Lastly, we acknowledge funding by Deutsche Forschungsgemeinschaft (DFG) for project MARDY 2 (375875969) within the priority program RATIO.

All survey and annotators were compensated 16 euros per hour for both tasks.

The VAA from Switzerland is provided in German, French, Italian, and English. In order to standardize the VAAs from different countries, we opted for transforming questions into statements whenever necessary. Therefore, all questions from SmartVote (the Swiss VAA) have been slighly modified to become statements. While the original dataset asks “Should cannabis use be legalized?”, in ProbVAA it states “Cannabis use should be legalized.”. Moreover, the original SmartVote is composed of 75 questions. In this study, we only used the questions that could have answered with a stance (agree or disagree), a total of 60 questions.

The original VAA is composed of 40 questions, but we removed 3 questions from the questionnaire because the annotator did not find sentence constructions that were equivalent to the opposite versions.

Some parties in the Polish VAA are new coalitions formed from two or three older parties. Since these coalitions do not have a separate score in the Chapel Hill survey (because they are too recent), we calculate their left-right index score by averaging the scores of the individual parties that make up the coalition.

The statement variants (negations and semantically opposites) are first annotated in the original language by native speakers. Then, all statements are translated to English with the Deepl interface. A sample of 20 statements together with their translations is checked by the same native speakers who annotated for negation and semantically opposites. We asked them to check for the quality (whether they’re semantically and syntactically correct) and do not make any modifications.

Most questionnaires provide five potential response options: ‘agree’, ‘rather agree’, ‘rather disagree’, ‘disagree’, and ‘no opinion’ or ‘neutral’. For the purpose of our experiments, we convert the alternatives into ‘agree’, ‘disagree’, and ‘neutral’.

In the majority of cases, the parties themselves have answered the questionnaires, except for Hungary where experts assigned answers to parties. For Switzerland, where individual candidates answer the questions, we obtain a single answer per party by majority vote. All answers from the parties or candidates compiled in this dataset are publicly available.

More information on the annotations of the policy issues can be found here: https://sv19.cdn.prismic.io/sv19“%2Fc76da00f-6ada-4589-9bdf-ac51d3f5d8c7_methodology_smartspider_de.pdf.

The gold annotations are made available on https://github.com/tceron/eval_political_worldviews/blob/main/data/human_annotations/annotations_spiderweb_gold.csv.

In the survey, expert annotators place parties in a scale from 0 to 10 that indicates how left or right a party is (0 is extreme left and 10 extreme right). Therefore, in our study, parties below 4 are considered left, between 4 and 6 are referred to as center and the remaining ones are right. All countries from ProbVAA are available in the survey, except for Switzerland. In their case, we annotate one of the three leanings for each of their six main parties according to the information available on their Wikipedia page.

Our implementation is based on HuggingFace Transformers 4.34.0 and PyTorch 2.0.1 on CUDA 11.8 and is run on NVIDIA RTX A6000 GPUs. Depending on the size of the model, we occupied from 1 to 8 GPUs in the generation process.

We ran an initial experiment with all open-source models using 14 prompts (8 impersonal, 6 personal) on a subset of the data containing 10 statements per country. We sampled 30 answers for each prompt and each prompt variant and selected the three prompts that resulted in the highest number of reliable responses (i.e., responses that could be clearly mapped to a stance) for each category (personal, impersonal). To lower the costs with experiments on gpt3.5-20b, we manually tested each template with 5 statements and counted the number of reliable responses for each template. We noticed that the personal templates worked less well here so we selected 4 impersonal and 2 personal templates for gpt3.5-20b. The remaining experiments of this study are conducted using the six prompts that were selected in this process.

Each statement from the set described in § 4.3 is inserted into 12 templates (3 personal and 3 impersonal ones and their label-inverted versions), which amounts to a total of 17208 inputs for each model.