Combining methods: a key to understanding complexity in European societies? Open Access

Making sense of complexity in research that crosses national and cultural boundaries is a major issue for social scientists. The growing concentration of human and physical resources on broadly based, multinational integrated projects and networks of excellence, characteristic of the developing European Research Area (ERA), has been driven by the natural sciences. Because they enable the concentration of resources and the sharing of expertise and equipment, in these disciplines large-scale infrastructures and programmes are believed to deliver value for money and a measurable return on investment (COM (2000) 612 final). Although costly infrastructures and sizeable international teams are clearly essential for joint operations such as the European Community Household Panel (ECHP) surveys or the European Social Survey (ESS), and a strong case can be made for developing research infrastructures in the social sciences and humanities as an important resource for researchers,¹ the dominant model in the natural sciences is less readily applicable for most social scientists. It may be especially inappropriate for researchers undertaking cross-national comparisons involving different linguistic and cultural communities. Whereas, for the natural sciences, the language and culture of team members are almost irrelevant, and do not usually constitute an obstacle to the advancement of knowledge, in the social sciences, language is both a medium carrying concepts and an object of scientific observation and discourse (Lisle 1985: 24). Similarly, culture is not only the subject matter under study, but also an agent that has shaped the researcher's mindset and thought processes, raising issues about the ability of scientists from different research and disciplinary cultures to work together in determining the research design and in selecting techniques for the collection and interpretation of data. Notwithstanding Richard Rose's (1991: 447) concept of ‘bounded variability’, whereby he argues that the number of variations, for example in the methods for electing a parliament, are not infinite, the greater the diversity of countries and researchers involved in European social science projects, the greater tends to be the degree of complexity with which they have to contend.

Despite the difficulties inherent in working in multinational and multilingual teams, European funding for collaborative research has stimulated the development of an international social science teamwork model, in which individual researchers frequently serve as native informants, offering insider knowledge about the phenomena under study in their own countries. The lone researcher, accustomed to carrying out library searches or using the safari method to pursue intensive fieldwork case studies in a small number of societies, can thus become part of a larger network interested in the same topic, pooling information and bringing to bear a wider range of cultural and disciplinary perspectives. The development of such a model has intensified the debate about how to ensure that collaborative work produces synergy and helps to make sense of complexity and diversity in a situation where the personal and cultural values of researchers and the research paradigms to which they subscribe are likely to influence not only their choice of methodological approaches but also their interpretations of findings.

During the twentieth century, the competing approaches adopted by quantitative and qualitative cross-national researchers and their epistemological assumptions were encapsulated in the polar extremes of the universalist or positivist traditions, as opposed to culturalist, phenomenological, constructivist or interpretive positions. The two contrasting approaches were associated with different methods of data collection and analysis: positivists opted for social survey techniques, structured interviewing and self-administered questionnaires, while constructivists undertook participant observation, semi- and unstructured interviewing and developed techniques such as focus groups and discourse analysis (Bryman 1992: 58–9). In the first case, as illustrated in political science by the large-scale surveys of electoral systems (Lipset and Rokkan 1967; or more recently Lijpart 1994), researchers were looking for rigorous, reliable and systematic scientific methods, in imitation of the natural sciences, capable of producing trend data with predictive value, enabling generalisations to be made about the patterning of variables, and leading to statements about convergence or divergence. In the second case, the focus was on differentiation and diversity between and within countries, and the complexity of the factors involved, making generalisations more difficult, as exemplified by the work of ethnomethodologists or the biographical case study method (Creswell 1998; Chamberlayne and King 2000).

The debate between advocates of the two positions has been unremitting. Quantitative researchers tended to criticise qualitative approaches for being too context-specific, unrepresentative and focused on differentiation, whereas qualitative analysts identified shortcomings in large-scale surveys due to oversimplification and the propensity to reduce complex phenomena to the lowest common denominator in an attempt to justify generalisations and causal inferences (Preece 1994: 43; Bryman 2004: 78–9, 284–5). Although it is argued that, today, the divide between epistemological positions may not be so clear-cut as is often assumed and has been exaggerated (Bryman 2004; Burnham et al. 2004: 277), the lack of mutual respect and trust between quantitative and qualitative researchers has continued to make dialogue and co-operation difficult, as illustrated in the next section.

By the turn of the century, most disciplines had moved on to develop methods that pay attention to context, thereby helping to bridge the divide. Already in the late 1960s, political scientists were advocating recognition of the embeddedness of measures and their contextual grounding in survey-based comparisons (Verba 1969: 79–85). The value of the concept of path dependency was demonstrated in social policy case studies by analysing the constraining or enabling effect of decisions taken in the early stage of a policy or institution on the future choices available to policy makers (Heclo 1974). The 1980s and 1990s saw a series of methodological breakthroughs, as researchers became more interested in identifying multiple causes for observed phenomena and sought to explain and interpret them with reference to their wider societal settings, an approach developed by Marc Maurice (1989) and the Aix School in the study of business organisations. In political science, proponents of new institutionalism also emphasised the importance of the embeddedness of political behaviour in institutional structures (March and Olsen 1984: 734, 736).

Interest in the case study approach and in the concept of path dependency was stimulated among policy analysts in the 1990s by the acceleration of the European enlargement process and by the need for applicant states to meet the economic, social and political criteria laid down for membership, provoking criticism of narrowly focused evidence, and creating an incentive to extend the knowledge and methodological skills base. The European Commission's Targeted Socio-Economic Research stream under Framework Programme 4 (FP4) and the key action ‘Improving the Socio-economic Knowledge Base’ in Framework Programme 5 (FP5), or the European Science Foundation's European Collaborative Research Projects scheme for social scientists, opened up new opportunities for researchers to co-operate across national and epistemological boundaries. The official launch of the open method of co-ordination by the European Commission in 2000 generated the need for more sensitive social indicators to measure trends in employment, social inclusion, pensions, health and social care. Moreover, context-boundedness has assumed particular relevance in research with a policy focus, requiring researchers to carry out fine-grain analysis if they are to get ‘inside the skin’ of their su bjects and understand how a particular phenomenon has been socially, culturally and politically constructed (Hantrais 2003a: 11). Enlargement has also given governments in East and West a fresh incentive to take advantage of opportunities for cross-border learning and transfer, requiring a discriminating and contextualised analysis of the relationship between macro- and micro-level structures in the exporting and importing countries (Rose 2002).

These developments mean that the frontiers between different methodological approaches have become increasingly porous and blurred. The experience of working in international teams has acted as a spur for theoretical and methodological developments in the social sciences. Whether they are in the habit of ‘working qualitatively or quantitatively’, the pressures on social scientists today are to ‘work qualitatively and quantitatively’ (Brannen 2003: 15; see also Bryman 1988; Brannen 1992; Devine and Heath 1999; Bryman 2004). The two epistemological positions have become less firmly entrenched as awareness has grown that account must be taken in cross-national social science research of a whole range of contextual factors, not least the researchers’ own cultural backgrounds, the language communities to which they belong, the socio-economic and political environments impinging on the phenomena that they are analysing, and the institutional frameworks within which they are operating.

Quantitative researchers are increasingly recognising that qualitative data can help in throwing up hypotheses, in framing and formulating survey questions, in defining, elaborating and exemplifying complex concepts, and in interpreting and contextualising findings from data collected in surveys (Bryman 1988; Harkness 1999). Due to their greater universality and ‘apparent objectivity’ (Preece 1994: 44), quantitative approaches can provide background information for qualitative studies to draw on in identifying broader trends, against which to locate in-depth analysis. Both types of researchers can, therefore, find common ground through their shared interest in understanding the cultural contexts in which the concepts central to their topics of study are socially constructed.

Methods textbooks have long advocated the combination of methods, or triangulation, to enable cross-checking of data (Bryman 1988; Brannen 1992, 2004; Burnham et al. 2004). The findings produced by integrating different methodological approaches may be concordant or complementary: they can generate insights capable of providing a more integrated picture of a phenomenon (Brannen 1992: 14). However, they may also be contradictory or discrepant, indicating that the presumed relationship may not exist or has to be understood differently (Gillham 2000: 29–30). The advantages of combining quantitative and qualitative methods apply not only to within-nation studies but also to cross-national research. Moreover, triangulation is not confined to methods (multiple strategies), datasets and theories, but can also apply to investigators working across disciplines and countries. Robert Burgess (1984): 158–9) has argued that ‘multiple investigators’ can help to overcome the bias often associated with individual researchers in single-nation and single-discipline studies, while warning of the difficulties that can arise due to conflicts of interest and differences in theoretical positions.

The remainder of the paper draws on examples from a cluster of European-funded multinational, multi-methods projects and networks to examine not only the considerable practical and epistemological difficulties that arise in the management of such multifaceted projects but also the advantages of combining methods in cross-national research. It argues that, by integrating different methodological approaches, researchers can gain a deeper understanding of complex social phenomena and produce more complete accounts of social reality (Bryman 1988: 126), but cautions against seeing multi-strategy research as a panacea (Bryman 2004: 464).

The arguments presented in the previous section suggest that the risks of drawing erroneous conclusions can be reduced, if not eliminated, by applying a variety of methods to address research problems. For the purposes of this paper, methodological pluralism does not refer only to the combination of quantitative and qualitative approaches; it also implies the application of a mix of methods within quantitative and qualitative paradigms. The expectation is that each additional approach will contribute further evidence to test research hypotheses by shedding more light on the subject being investigated and by drawing attention to the relative importance of different factors making up the wider picture. Although the value of multiple methods has been amply demonstrated in single-nation and single-discipline studies (Devine and Heath 1999), the paper argues that methodological pluralism is all the more useful and necessary in multinational research and, more especially, in cross-national comparisons if they are to capture the effects of linguistic and cultural diversity on the phenomena under study.

The contribution of mixed methods to cross-national research projects is examined here by reviewing the range of methods used in a cluster of European Commission-funded FP4 and FP5 projects and networks on the common theme of ‘Family and Welfare’.² Although they did not all have an explicit policy focus, the projects and networks were grouped together by the Commission because they were concerned in one way or another with the interactive relationship between family and public policy. The Commission was not, initially, interested in the methods adopted by the researchers, but the cluster provides a useful body of material for an analysis of the many methodological issues that arise in European research programmes. This section of the paper begins by examining the political and policy dimensions of the projects and networks in the cluster and the effects that political influence can have on the conduct of the research for both funders and researchers. It then explores in more detail the methodological issues confronting participants in European research projects.

Like other Directorates General (DGs), following the accusations of lack of transparency and the questions raised about the Commission's legitimacy by the European Parliament in 1999, when all the commissioners were forced to resign, the Directorate-General for Research (DG RESEARCH) is required to demonstrate the accountability and European added value of the activities that it funds. The interest and relevance of FP5 research to the work of the policy DGs have thus become a priority, confirming the importance when analysing European-funded research of knowing who the producers or sponsors are and for what purpose the research was intended (Harrison 2001: 5).

A major objective of the ERA, when it was launched in 2000, was to increase the international impact of the European research effort by strengthening the relationship between research activities and policies. The reason for reinforcing the link between research and policy is clearly articulated in statements from the Commission (COM (2000) 612 final), which express the need to legitimate the use of public funds to support research activities by demonstrating that the findings are of public benefit. The case is made stronger if research can be shown to contribute to the efficacy of public policies or to resolving the problems confronting society, which was a key objective of several of the projects and networks in the Family and Welfare cluster. Another important principle legitimating public funding of research is the added value of European co-operation, which is expected to produce economies of scale by creating critical mass, by bringing together complementary expertise and by underpinning EU priorities and interests. The clustering of projects under the key action for ‘Improving the Socio-economic Knowledge Base’ in FP5 was designed as a way of implementing an integrated approach towards research fields and projects, with the aims of guaranteeing complementarity among projects, maximising European added value within a given field and establishing a critical mass. Such an approach was considered necessary to solve complex multidisciplinary problems effectively (1999 Guide for Proposers, Part I: 10).

In their proposals and final reports, FP researchers are required to identify the contribution that their work will make to policy. With this aim in mind, the content of the calls reflects issues of concern to the policy DGs. Often, however, policy advisers cannot wait for research findings to be produced from projects that routinely span a minimum of four years from the time when the proposal is drafted to the submission of the final report and publication of findings. Among the many explanations posited for the under-utilisation of ‘academic’ research in policy formulation is that the nature of policy making is ‘a fast-breaking, self-serving, influence-driven process incompatible with the methods of social science, which are more time-intensive, intellectual and rational’ (Bogenschneider et al. 2002: 189). The political and scientific objectives of social science research may also be incompatible, and may even be in conflict if the evidence produced by researchers does not chime with political expediency or correctness.

Like national research funding agencies supporting both thematic programmes and responsive-mode grant schemes, DG RESEARCH is often faced with a dilemma when deciding whether to prioritise ‘political’ or scientific objectives. Funding bodies have to choose whether to support only research that meets the highest quality criteria or whether to take account of programmatic needs by bringing other factors into play. Given that the projects and networks funded by DG RESEARCH are selected primarily on grounds of scientific quality, several projects may address the same topic, while other important parts of a call may not be covered. To fill gaps in thematic coverage, the Commission can reshape subsequent calls to exclude certain areas and stimulate interest in under-subscribed or emerging topics, but duplication of effort is more difficult to avoid.

The Family and Welfare cluster affords a useful example of European-funded research for assessing the efficacy of the process in achieving the Commission's political and scientific objectives. In terms of topic coverage, two projects focused on the nature and dynamics of social change, family transitions and social exclusion (see Table 1). Both had originated under the Targeted Socio-economic Research (TSER) strand of FP4, where ‘social exclusion and social integration’ was one of the three major themes. Three projects and one of the networks were interested in the relationships between fertility, employment, parenting and policy, which fell within the overarching themes of ‘social and economic challenges of changing family structures’. One of the projects and the TSER network were particularly concerned with work–life balance, time and flexibility, a topic relevant to the second theme in the call. Issues surrounding caring were addressed in three projects, although the topic was not specifically identified in the first call, illustrating the way in which the research community can draw attention to emerging social questions and help move them up the policy agenda. In the review, the problem of thematic duplication was engaged with by looking for ways of promoting synergy and complementarity across projects on similar topics (Hantrais 2001).

Priority setting inevitably entails political bias, but politics also affect the conduct of European research in less overt ways. The selection of project and network partners and co-ordinators may be driven less by scientific criteria than by the need at EU level to include countries representing different regions in Europe and to avoid over-representation of certain countries or disciplines within a programme. In FP5, the Commission required proposers to involve at least two different member states or one member state and one associated state. It also sought to ensure that less favoured member states would be well represented, and that candidate countries would be integrated into the ERA.³

Selection primarily on the basis of scientific criteria may have resulted in what could be seen as a sub-optimal regional coverage by the Family and Welfare cluster. As shown in Table 1, the projects and networks selected had partners in at least five countries. Only one did not include a Nordic state (MoCho), and only one did not have a Southern European partner (HWF). Germany and the UK were the most frequently occurring combination. Very few projects and networks included Denmark, Finland, Ireland or Portugal. Yet, as demonstrated by those that did, compared with other EU-15 member states, all four countries display a number of interesting features in the area of family and welfare. About a third of the cluster comprised countries from Central and Eastern Europe (CEE), among which Hungary was the most frequent participant, but few of the new EU member states and not all the EU-15 countries were covered by the projects on caring. Even though partners were drawn from a smaller number of countries, some of the projects and networks collated and/or analysed data for all EU-15 member states, where available (CHER, DynSoc, IPROSEC, Men, MoCho). The CHER covered a total of 19 countries, and the ESS incorporated 17 in its first round, excluding three EU-15 member states and including two non-EU countries, namely Norway and Israel. Participants in the NIEPS were drawn from the 21 countries covered by the Fertility and Family Surveys (FFS) sponsored by the United Nations Economic Commission for Europe (UNECE), and carried out between 1988 and 1999.

The justification given by co-ordinators for their choice of participating countries was more often pragmatic than scientific (Hantrais 2001). The scientific reasons presented in the methods textbooks for careful selection of cases to ensure the quality of comparative findings were not generally uppermost in the minds of project and network co-ordinators. In the case of the ESS, countries were essentially self-selected, depending on the availability of nationally funded household surveys. In most instances, the reasons given for the country mix confirmed the importance of established and reliable networks. Although the NIEPS partners were selected from among participants in the FFS, secondary reasons were that most of the institutions chosen had an explicit mission to carry out policy-relevant research and a shared interest in constituting a network incorporating East–West diversity. The DynSoc and FENICs projects chose partners who already had experience of working together and, importantly, of handling the datasets to be analysed. They were not selected on the grounds that they would serve as national informants for their own countries. In the more qualitative projects, the cultural proximity and mutual understanding that had developed between researchers in certain groupings of countries were important factors for team building, although the inclusion of countries representing different welfare regimes or models was frequently mentioned across the cluster as a primary or secondary consideration (Caring, FADSE, HWF, IPROSEC, SocCare, TSFEPS, W&M).

The politics of the selection of participating countries raises two important methodological issues concerning the impact of sampling criteria on the findings: firstly, the question of how representative the countries are of the phenomena under study in cases where partners are acting as native informants, and whether policy-relevant findings can be extrapolated to different socio-economic environments; secondly, how to deal with the bias resulting from the researchers’ own value systems, perceptions and assumptions, personal and political agendas, which affect not only the choice of methods but also commitment to the research. These issues, which are dealt with more fully in the next section, have gained salience with the opening up of funding opportunities to CEE countries that have very different research traditions, and for whom the FPs provided a means of entering the ERA before they became EU members. Prior to enlargement to the East, the different institutional structures for managing research already made joint funding bids between researchers from different regions within the Union problematic. For example, Northern European countries, where research is concentrated in the university sector and supported by research councils, do not operate under the same conditions as Southern European countries, including France, with a strong tradition of state-controlled research centres employing their own ‘permanent’ researchers (Tennom 1995). When the social sciences are left to their own devices and are not obliged to meet European Commission criteria for coverage, countries with similar research cultures are more likely to want to work in partnership, as illustrated by the ERA-NETs, which have tended, in the social sciences, to be initiated by countries that share the same concept of research ethics, practice and management.⁴

Several of the projects and networks had a deliberate and explicit policy dimension, which was an integral component in the research design, and their teams included researchers who were specialists in comparative social policy (FADSE, Genre, IPROSEC, TSFEPS). For others, the interest of researching policy measures lay in recording their presence or absence, their relative generosity, or the extent to which they might be expected, using econometric models and rational choice theory, to have an impact on families (DynSoc, FENICs, MoCho, NIEPS). While the political and, more overtly, the policy dimensions of FP research have been moving up the EU agenda, the Commission has been less concerned about methods, except insofar as the research community is able to demonstrate that it is contributing to methodological advancement and refinement in a way that ensures greater reliability of data. For their part, the policy DGs expect researchers to contribute to the knowledge base underpinning policy decisions, develop robust indicators for measuring the success of member states in reaching EU targets, provide examples of benchmarking and good practice, and build up an evidence base for policy. However, they also commission their own short-term studies and, since the 1970s, they have funded observatories and networks to map and monitor socio-economic and policy developments, and improve social indicators, often involving researchers who are also participating in FP projects and networks (for an example, European Commission 2003).

As most of the projects in the Family and Welfare cluster had not yet produced any policy-relevant findings when the review was contracted, and some had not even started, the Commission agreed that the focus in the first of the planned dialogue workshop between researchers and policy actors would be on methods (Hantrais 2001). The second workshop concentrated on the policy relevance of findings (Hantrais 2003b). The Commission acknowledged the value for the European research community of discussing and confronting different methodological approaches by also agreeing to support a further workshop on qualitative methods and a conference on quantitative methods. The clustering of the FP projects and networks, the dialogue (Hantrais 2001,2003b) and methods workshops (Cameron 2003) and conference (http://www.iser.essex.ac.uk/epag/events/index.php, 02/February/05) enabled researchers working on similar topics to come together to scrutinise methods, exchange and pool information and learn from one another, in line with the Commission's objective of striving for complementarity.⁵

Notwithstanding the many claims noted in the previous section and the widespread interest in methodology among researchers, if not among research funders, it proved difficult to generate productive discussion and exchange of ideas and experience between research teams that considered themselves to be strongly wedded to either quantitative or qualitative methods. While each could see the benefits of taking account of the other's approaches, and most were using a combination of methods, few wanted to enter into detailed discussion of different research paradigms on the grounds that they would have difficulty in understanding the concepts and discourse involved and preferred to devote their limited resources to pooling information on issues of more immediate concern, which also explains why it proved necessary to organise two separate events on methods.

Under FP5, proposers were expected to show that they would generate and use high-quality comparable data, while also improving access to datasets. They were not required to make systematic comparisons of data: DG RESEARCH does fund projects and networks that are confined to data collection and description, in line with the marked preference for quantifiable evidence among policy actors (Davies 2000). The interest of systematic mapping is that it creates datasets that other researchers, including those working for the policy DGs, can then use for the purposes of comparative analysis, by constructing an empirical basis for theory building and refinement or scientific explanations, provided the country coverage is appropriate and the data have been collected using rigorous techniques to ensure their comparability and reliability. Two of the projects in the initial Family and Welfare cluster (CHER and ESS) set out, specifically, to establish databanks that would be a resource for both researchers and policy makers. Most projects and networks have contributed to research infrastructures that support the work of the policy DGs and other research users by developing conceptual and analytical toolkits capable of capturing changing concepts and of serving as a resource for analysis of the policy process.

The projects and networks in the cluster drew on a variety of methods to examine the relationship between family and welfare. In most cases, a dominant methodology could be identified, as shown in Table 1, where the entries in the first column are ordered according to their main research paradigm. At the one end of the methodological spectrum are projects, such as CHER and ESS, that are establishing or developing large-scale databases, covering fertility, demography, the labour force and value systems as an end in themselves. The HWF project, while located at the quantitative end of the spectrum, carried out a survey as a means to an end. Cross-sectional data analysis and mapping exercises of various datasets were used in the FENICS and MoCho projects. The DynSoc and FADSE projects studied the dynamics of social change and the impact of transitions from a longitudinal perspective, using household panel studies. These projects sought to interpret variations between countries with reference to contextual factors. Like the three networks, they re-analysed existing datasets to track socio-demographic and economic trends and identify similarities between countries, particularly with regard to fertility and economic activity patterns, from which to make generalisations and draw out causal explanations. The remaining projects, where qualitative methods are dominant, invariably drew on macro-level data to provide a backcloth for in-depth interviews and other observational techniques and devoted a considerable amount of time and effort to developing interview guidelines and analytical frameworks that could be operationalised in very different national settings (IPROSEC, Genre, TSFEPS, SocCare, Caring).

Researchers who have worked with large-scale datasets, such as those compiled by Eurostat, particularly the now defunct European Community Household Panel (ECHP) surveys and the Labour Force Surveys (LFS), or who have attempted to collect comparable time series data from national sources for candidate countries, are fully aware of the problems of access and of ensuring comparability between data sources and countries. Difficulties in accessing ECHP data, incomplete datasets and variable response rates resulted in project teams that were using the ECHP as their main data source being unable to meet deadlines due to the late release of data (four years from the time of the survey to the release of data to researchers), or prevented them from covering all EU-15 member states for selected variables (DynSoc, FADSE). The CHER was set up to help overcome such problems: the consortium created a comparable longitudinal database by cleaning up and standardising ECHP data, and by preparing a relational data structure to support analysis, thereby obviating the need for researchers to go back to the original data files. The DynSoc project resorted to ‘cloning’ ECHP data to extend the sequence of years available in Germany, The Netherlands and the United Kingdom. These devices could not, however, get round the problem that the ECHP shares with most surveys, namely that the groups of greatest interest to researchers investigating social exclusion or family living arrangements are usually under-represented and difficult to identify. The FENICs project was, for example, interested in comparing ECHP data on childlessness, but was unable to do so because the survey does not distinguish between couples who have never had children and those with children who have left home. The LFS also proved difficult to access, and involved cumbersome procedures that restricted the types of analysis that could be undertaken. For instance, since the LFS identifies individuals rather than family units, they cannot be used for studies of the relationship between family and employment, unless data are reconstructed from household information (FENICs).

Non-availability of comparable data may help to explain why few of the predominantly quantitative projects included CEE countries. On some topics, no time series data are available for many of the variables needed to map trends in the CEE countries, either because they were not collected prior to transition, on unemployment for example, or because definitions vary over time and s pace, as with key variables for fertility rates. The lack of comparable data on flexibility at work meant that the project examining the relationship between households, work and flexibility was obliged to undertake a new survey (HWF). The absence of harmonised datasets at European level meant that other projects and networks extending coverage to CEE countries had to request matching data from national statistical offices, or draw on data sources other than Eurostat, such as Council of Europe, to obtain more complete datasets (IPROSEC). The two sources are not, however, always compatible, despite the fact that, in many cases, they are both starting from the same raw data supplied by national statistical offices. For instance, while data are generally consistent across the two sources for completed fertility rates, significant unexplained discrepancies are found for the 1945–60 birth cohorts for Slovakia, with Eurostat reporting much higher figures. For demographic data, the Council of Europe is considered to be more reliable than Eurostat (Sardon 2002). United Nations datasets were more complete for variables such as the age structure of the EU population. In some cases, however, their figures were found to be higher than those reported by either Council of Europe or Eurostat (Hantrais 2004b: 22, 28).

The best efforts to harmonise data at European level cannot overcome the problems that arise when different concepts and definitions, coding and classification systems are used at national level. The European survey of social values (ESS) had the advantage that it could lay down standards for the data it was requesting from national statistical offices, and could, therefore, anticipate and guard against the problems it would have to face in achieving comparable national samples, attaining high response rates, designing, translating and testing questionnaires for comparability of issues and concepts, and implementing consistent coding systems. The ESS team was determined to resolve these issues at the outset. It did so by combining a central top-down co-ordination structure with strong input from the participating countries, together with advice from independent academics and experts. The very high standards set have enabled the ESS to offer a model of good practice in survey methodology, but its work casts doubt on the reliability of what are ostensibly comparable statistics published by international agencies.

The issue of reaching agreement over concepts is equally, if not more, important in the projects and networks where the dominant method lies towards the qualitative end of the methodological spectrum. Many of these projects used a range of micro-level observational techniques, often in combination, including elite interviews with key policy actors, in-depth interviews with families, ethnographic case studies, focus groups and vignettes. Most of the projects and networks using qualitative methods were following an approach that emphasises differentiation between and within countries to take account of the complexity of the factors involved and of sub-national diversity. These projects (IPROSEC, Genre, TSFEPS, SocCare, Caring) carried out new fieldwork in an attempt to capture the complexity of the intervening factors in parenting decisions, the relationship between working and mothering and the ways in which social care is conceptualised and organised in different countries, thereby complementing and extending the analysis resulting from the projects and networks focusing on large-scale datasets.

Whereas the problems identified by the researchers using the more quantitative approaches were largely concerned with the availability, reliability and comparability of data, many of the difficulties encountered in the projects and networks adopting more qualitative methods were related to subjective bias and differences in understandings of concepts and cultural contexts. It was recognised that many of the key concepts in the research, such as lone parenting, teenage pregnancies, part-time work, leave arrangements and, especially, caring were politically and ideologically charged, leading to lengthy discussions about meanings among team members (see also Hantrais 2004a). Interpretations of events by participants – insider knowledge can become insider bias – were often influenced by national politics and national and disciplinary research cultures, particularly for projects involving CEE countries. A common source of frustration among qualitative researchers was the lack of time available to thrash out issues of conceptual understanding and interpretation in the materials collected. Workshops were the occasion for team members to reflect collectively on central concepts and the most appropriate ways of conducting fieldwork, processing and interpreting findings. As with the quantitative projects, they also played an important role in the training of researchers.

The research teams adopted various strategies to avoid subjective bias. In the Caring project, participants from different countries were involved in the data collection phase of the case studies. Video observation of care workers was followed by recording and analysis of the reactions of partners to the videos. In the SocCare project, each of the partners was responsible for co-ordinating part of the project across all the participating countries. The IPROSEC project gave research assistants the opportunity to participate in cross-border interviews and to comment on reports from other countries. It also organised dialogue meetings with end users to discuss and validate the findings, as did the Genre project. Despite the careful selection of team members, issues of language and cultural understanding often proved to be greater obstacles than had been anticipated, even among seasoned international researchers, highlighting some of the intrinsic problems in the European funding model as applied in FP4 and FP5. Few of the qualitative projects could afford to factor in translation costs, and had to rely on partners producing reports in a common language, which was rarely their own, placing the onus on the project co-ordinator to ensure consistency in reporting and interpretation in co-operation with team members. The Caring project, which set aside 18 per cent of its budget for translation, was an exception, but translation was not a panacea, since it proved impossible to hold constant the conditions in which the interviews were set up and conducted (Cameron 2003: 45–7). Intensive discussion of concepts, scrutiny of interpretations by other teams members, research subjects and users, and the confrontation of findings obtained using a variety of methodological approaches were valuable devices in helping the research teams guard against drawing ill-founded conclusions.

In the past, analysts of multi-methods research strategies in the social sciences took most of their examples from single-case and, consequently, single-nation studies, because relatively few instances of multi-site/multi-methods studies were available to illustrate what has been described as ‘methodological integration’ (Bryman 1988: 129–30). The European FPs and, more especially, the clustering of social science projects and networks around selected topics provide a body of material that lends itself to an analysis of both multi-site and multi-methods research. When the FPs were designed and research proposals were selected for funding, the primary consideration was not, however, to ensure that they would constitute an integrated corpus, except insofar as they were addressing the themes identified in the calls and focused on a problem-solving approach to complex multidisciplinary questions. When the projects and networks in the Family and Welfare cluster were singled out for review, it was because they had in common their – in some cases tenuous – potential policy relevance and interest in changing family structures, and not because they were adopting complimentary methodological approaches.

Moreover, for many of the research teams, methodological advancement was not a primary concern, as confirmed by the relatively modest attention given to methods in many of the final reports submitted to the Commission. The examples cited in this paper provide an indication of the great variety of methodological approaches adopted both within and across the projects and networks in the Family and Welfare cluster. Although almost all of them subscribe to a dominant methodology, none could be said to represent a pure form of either a quantitative or qualitative research paradigm as traditionally portrayed. On a continuum running between the two paradigms, the ESS probably comes closest to meeting the widely recognised criteria for rigorous quantitative research. The efforts to achieve comparability required very detailed qualitative work, which led to the admission that methodological compromise may be necessary to achieve functional equivalence in cross-national surveys. As in other projects involving new survey work (IPROSEC, HWF), where, for example, partners were allowed to decide whether to use postal, telephone or face-to-face interviews, it was found preferable to tolerate variation to attain a common goal. The Caring project could be situated at the other end of the continuum for its use of innovative observational techniques. It was no less rigorous in its efforts to avoid insider bias and sought to interpret the findings with reference to a much broader socio-economic and political context.

While it could be claimed that the combination of methods within projects and networks undoubtedly resulted in the production of more reliable datasets and a more sensitive appreciation of diversity, it is less certain that the post hoc synergy that the Commission was attempting to stimulate through clustering succeeded in producing an integrated approach towards research fields and projects, capable of finding effective solutions to complex multidisciplinary problems. The many projects and networks that undertook the reconstruction and critical secondary analysis of national or international datasets had considerable potential for data sharing and for optimising the research effort. However, the projects (DynSoc, FADSE, FENICs) that might have made use of the standardised ECHP data could not wait for the CHER database to be up and running. Indeed, FADSE was completed before CHER started. The reconstruction of LFS data by FENICs came too late to be used by the other projects (IPROSEC, MoCho) and networks (NIEPS, W&M) in the cluster interested in the relationship between work, family formation and structure. Similarly, the three projects on caring, while all using in-depth interviewing, were examining quite different questions. Together, they do not cover all EU-15 member states, and the two CEE countries that they do include cannot be considered representative of all transition countries. Just as most of the projects and networks using survey data had to start by cleaning up and reconstructing datasets to make them comparable, it would be methodologically unsound to try and draw out any firm conclusions for policy on the basis of their collective findings, since the research questions being investigated, the methods used and the research agendas of team members were different. Even if their findings could be harmonised, it is unlikely that they would produce definitive answers to questions from policy makers, such as whether policy can stem the decline in birth rates or whether the extension of childcare provision will enable the EU to meet its targets for raising female employment rates and making work pay. One consistent finding across the cluster is, however, the growing complexity and diversity not only of the relationship between policy and socio-demographic behaviour but also between research and policy.

Linda Hantrais is Professor of European Social Policy in the Department of Politics, International Relations and European Studies at Loughborough University. Her main research interests are in cross-national comparative research theory, methodology and practice, with particular reference to public policy in the European Union. She has co-ordinated a Framework Programme 5 project and worked as a policy reviewer for the European Commission. Her recent books include: Gendered Policies in Europe: reconciling employment and family life (ed., 2000); Social Policy in the European Union (2000); and Family Policy Matters: responding to family change in Europe (2004).