Implementation of an Open Science Instruction Program for Undergraduates

The scientific, social, and economic advantages that accrue from Open Science (OS) practices—ways of doing research that emphasize reproducibility, transparency, and accessibility at all stages of the research cycle—are now widely recognized in nations around the world and by international bodies such as the United Nations and the Organization for Economic Cooperation and Development. However, program wide or coordinated instruction of undergraduate students in OS practices remains uncommon. At the University of British Columbia in Canada, we have started to develop a comprehensive undergraduate OS program that can be adapted to and woven into diverse subject curricula. We report on the context and planning of the pilot module of the program, “Open Science 101”, its implementation in first-year Biology in Fall 2019, and qualitative results of an attitudinal survey of students following their course.


INTRODUCTION
In the Spring of 2018, the Library and the Department of Biology at the University of British Columbia's (UBC's) Okanagan Campus in Kelowna, Canada, embarked upon an undergraduate Open Science (OS) instructional program-a venture unique not only in Canada, but also in North America, as far as we are aware. This instructional program has since evolved into one wing of a larger Fostering OS initiative spanning all of UBC.

Implementation of an Open Science Instruction Program for Undergraduates
The initiative and growth of the undergraduate instructional program was made possible through the efforts of a number of champions at UBC who succeeded in achieving recognition of OS instruction as a strategic institutional priority. The program has seen planning, aided by relationships between the Library and the Department of Biology, for a multi-module program to be integrated into key courses across all four years of the undergraduate Biology degree. Starting in the fall of 2018, approximately 60 students in two third-year biology classes tested a pilot module to evaluate interest and means of implementing content that supported, without altering, existing curricula and class assignments. After this initial success and the subsequent hiring of a dedicated OS Librarian to support this work in the Spring of 2019, 328 students participated in a voluntary pilot module called Open Science 101 (OS 101) in two first-year Biology classes in Fall 2019.
OS 101 defines OS as a movement to make products of all stages of the research lifecycle (e.g., scientific research, data, and publications) more accessible and provide transparency, openness, integrity, and reproducibility to scientific studies. The module provides an overview of the practical challenges of conducting reproducible research, the societal impact of irreproducible research, and philosophical and ethical issues surrounding OS. Assessment of the module revealed that a major attitudinal shift had occurred among participating students: They were convinced of the importance of a shift toward OS. We will look at the background and context surrounding our OS instruction; the planning, delivery, and evaluation of OS 101; and future plans for the undergraduate program.

BACKGROUND: THE STATE OF OS IN CANADA
For readers to understand the novelty of the UBC Okanagan program fully, we must situate this program within the Canadian context. Although Canada has lagged behind Europe in its promotion of OS, between 2015 and 2020 the Canadian government introduced measures to promote Open Access [1], articulated a strategic plan to foster OS practices among federal government researchers and recipients of federal government funding [2], and participated in a UNESCO recommendation on the development of an inclusive OS policy framework [3]. These developments affected university researchers funded by Canada's TriAgency, three government bodies that support scholarly research in Canada: the National Science and Engineering Council of Canada (NSERC), the Canadian Institutes of Health Research (CIHR, which had implemented an open access policy at an early date), and the Social Sciences and Humanities Research Council of Canada (SSHRC).
These important efforts, preceded by similar initiatives in the United States and Europe, emphasize infrastructure, funding, and other supports targeted largely to the public sector and to practicing researchers or graduate students. But OS requires a culture of change that reaches beyond these domains, one that prepares future citizens and future researchers. It is with this understanding that we undertook our instructional program.
A recent environmental scan confirmed that no other Canadian universities are delivering undergraduate instruction in OS [4]. Open scholarship instruction in Canadian post-secondary institutions appears to concentrate on three areas:

Implementation of an Open Science Instruction Program for Undergraduates
· Open Educational Resources (OER) as learning tools for students; · Open Access databases; · Research Data Management (primarily for graduate students).
Although we found no evidence that any other universities were considering an undergraduate OS program on the scale of UBC's, we know anecdotally that many individual instructors at a variety of universities are integrating aspects of OS into their labs and classrooms. At the very least, this suggests an appetite for a shift in how we teach the scientists of the future.
This landscape, while perhaps more integrated at the institutional policy level, appears very much mirrored in the European Union. While there is little evidence of the degree of undergraduate integration we are attempting at UBC Okanagan, Horizon Europe, the 9th EU framework program for research and innovation, specifically names OS as one of its three pillars [5]. In accordance with the new program, many European universities are beginning to put comprehensive OS policies in place. One example of this is the Delft University of Technology (TU Delft) in the Netherlands. The TU Delft Strategic Plan OS: 2020-2024 outlines plans to implement coordinated OS training modules for instructors, research staff, and students [6]. However, education for students at the undergraduate level is not mentioned.
Given the nascent nature of the OS movement, scant literature on best practices in library OS information literacy instruction exists. Extant publications on this topic (e.g., [7]) tend to focus on information literacy instruction of researchers pursuing graduate studies-that is, EU Level 2 or higher. Some exceptions include Ayris and Ignat [8], who suggested involving undergraduates (EU Level 1 students) in Citizen Science projects; Button [9], who established an undergraduate Psychology consortium in Britain to introduce rigorous research practices to writers of undergraduate theses, and Jekel et al. [10], who incorporated replication projects into Honors thesis requirements.

A SHORT HISTORY OF OS ADVOCACY AT UBC
Efforts to promote best practices in OS at UBC began in earnest in 2015, under the direction of an assistant professor in the Biology Department of UBC Okanagan. Over the first two years efforts were largely focused on (i) identifying and reaching out to the administrative units that would be key to establishing OS as a core tenet of UBC's research agenda, and (ii) building a grassroots network of OS advocates. Key partners in this early work included the Vice-Provost and Association Vice President Academic of UBC Vancouver; UBC Okanagan's Associate Vice-Principal and Associate Vice-President, Research and Innovation; and UBC Okanagan's Chief Librarian and Associate Provost, Learning Services. As well, the movement was supported by UBC's Advanced Research Computing group (ARC), who recognized at the outset how OS best practices aligned with their core mandate of facilitating research through highperformance computing infrastructure and support.
Despite support from upper administrative positions and ARC, one challenge remained: All of this work was happening "off the sides of desks", and so progress was at the mercy of already full work schedules. Nevertheless, between 2017 and 2019, a new Southern Medical Program Librarian, in conjunction with

Implementation of an Open Science Instruction Program for Undergraduates
ARC and the aforementioned Biology professor, succeeded in launching Canada's first instance of OS Framework Institutions at UBC  .
In 2019, the UBC Okanagan OS advocates secured $100K (CAD) in strategic funding for OS over two years. Their proposal included short-and long-term objectives within both the teaching and research realms, the former being described below under Sections 4 and 5. And the funds supported the hiring of a part-time OS Librarian who could focus on building the OS instructional program, and enable team members to broaden the reach of instructional seminars and workshops.

UNDERGRADUATE SCIENCE EDUCATION
Why target undergraduates? Johannes Vogel, director general of the Museum für Naturkunde and Professor of Biodiversity and Public Science at Humboldt University in Berlin, recognizes the power that young people have to mobilize movements and suggests that they can help to open up science. In his article "Scientists need to learn from the young", Johannes Vogel states, [11].

Science must learn to listen, open up and again become part of the community. Such a transformation … will cost time and money, including the restructuring of the incentive systems in science itself
Students-our future scientists, professors, practitioners, and policy makers-have long been catalysts of social and political change. As such, the transformation of academic culture we desire requires cultivating in our undergraduates an appreciation of OS values and familiarity with transparent and reproducible practices.
Introducing OS values and practices as early as possible has numerous benefits, both for the prospective researchers and for the entire undergraduate population. First, by participating in OS training in the first year, students are better equipped to critically evaluate the materials and methods they are exposed to in subsequent years. Our qualitative experience is that students feel empowered by the early OS training, and therefore show greater interest and engagement when presented, for example, with case studies or primary research articles. We are hopeful that this applies more broadly, outside the classroom, where students need to make informed decisions as active members of society.
Second, values and practices that are introduced early can be more fully developed and reinforced throughout the undergraduate curriculum, and this in turn increases the likelihood that the practices become habits. This is especially beneficial to students who aim to pursue science careers. Crucially, we have seen how these habits can propagate up through research labs to principal investigators, and ultimately precipitate wholesale changes to research workflows.
Lastly, by their final year, students will be well equipped to apply OS best practices to research endeavors such as honors research projects and capstone projects. We therefore anticipate substantive improvements in the quality of undergraduate-led research, which is a focus of many undergraduate programs.

Undergraduate Program Vision
Our initiative at the undergraduate level is two-fold. The first is to build a series of learning objects that will allow faculty and lab managers to adapt current curricula and assignments to a model that integrates the fundamental principles and practices of OS in the instruction and evaluation of their existing teaching practices. The goal here is not to change what is being taught, but how undergraduate students are taught and evaluated. The second is to build a test bed of integration working closely with instructors in the Biology Department at UBC's Okanagan campus. This integration allows us to identify where discipline modularity in these objects is best situated and at what levels it makes the most sense to introduce various concepts related to OS. This in turn is allowing us to build better learning objects that can be adapted for different disciplines.

Proposed Learning Objects
At the outset, our undergraduate program sought to give students a grounding in several categories of best practices that facilitate reproducibility, replicability, and inclusiveness in science. These include open access, transparency at all stages of the research cycle, data management, citizen engagement, and applications of technology to OS. The program is designed to scaffold from introductory courses in first and second year to more advanced courses in third year; it will culminate in a fourth-year replication project. We intend the module templates to be flexible enough to integrate into courses in a variety of disciplines.
The plan, and challenge, is to integrate this education smoothly into the existing course content in such a way that it becomes just another part of the experience of completing a science degree, and minimizes the burden to instructors.
Our OS library information literacy program for undergraduates aims to provide templates and resources to enable instructors to weave OS principles and practices into the existing curriculum in such a way that Open practices become as automatic to students as putting on a lab coat.
In creating a comprehensive undergraduate OS program that can be implemented in diverse disciplinary settings, we must consider that professors and instructors have varying levels of comfort with and knowledge about OS and its associated technologies and that OS practices look different in different subject contexts. Therefore, we plan to introduce some initial modules and later, assuming good results, open a dialog with faculty at large about our ideas for a comprehensive program. In the list below, italics indicate a module that has been implemented; as aspects of our work are released, they will be found on our website, https:// openscience.ubc.ca/.

Target Classes
Since the Associate Professor in Biology had cultivated close relationships with his colleagues in Biology and had planted the seeds of change by speaking to them of the benefits of OS practices, this department seemed a natural setting in which to introduce our pilot project. OS 101 was to provide a general introduction to OS concepts and fits naturally into first year. Moreover, the lab manager for first-year Biology showed receptivity to the idea. Thus, the team built a module to be implemented in the Lab components of Biology 116 (for Biology majors) and Biology 122 (for majors in other subjects).

Objectives
Our pilot was designed to give students an overview of the practical challenges of conducting reproducible research, the societal impact of irreproducible research, and philosophical and ethical issues surrounding OS.
Our intent was for students to be able to explain the following on completion of the unit: · The nature of scientific knowledge · The science-society relationship · The replication crisis · Core values of OS · Benefits of OS · Barriers to the realization of OS ideals

Lightening the Load
Both students and instructors face considerable demands on their time. To avoid creating extra work for instructors, we introduced OS 101 in the lab component of the course, rather than the lecture component, and made it a self-contained module with short unit quizzes that were marked automatically. Assessments at the start and end of OS 101 contained some open questions, but these were to be marked by the OS Librarian, not by the lab manager or teaching assistants.
To avoid burdening the students overly, the lab manager and the OS Librarian arranged for the students to receive extra participation marks for attempting the quizzes; students were not penalized if they did not participate in the module. We chose a medium in which the module could be completed online and at the student's own pace: we mounted it on Canvas, UBC's Learning Management System (see Figure 1).

Relatability
We tried to provide examples that the students could relate to based on their own experience: A case in point is this example relating the desirability of transparency in practices to a student's wish to know how and why marks are assigned for an assignment: Transparency and Grading One case where students want and need transparency is in grading. Most of you probably appreciate knowing how marks were given to questions on a midterm exam and what criteria the marker used to score each question. This provides you with both a reason for the assigned grade and a means of comparing your grade with those of your classmates. Ultimately, this specific information can help you to address gaps in your knowledge and perform better on future assessments (such as the final exam). In the same way, transparency in research allows scientists to improve future studies and add to accumulated knowledge [12].
We also introduced dialogs to make the content feel more informal or "low-key".

Accessibility
In making the material accessible to first-year students, it was imperative to consider their limited familiarity with (a) research data collection, management, and sharing and (b) statistics. Note that first-year students, the target of this module, are not yet engaged in collecting or sharing their own data. As such, we did not refer specifically to initiatives such as FAIR data principles [13]. However, data are mentioned as being critical for reproducibility and replicability, and later modules will discuss FAIR data principles.
In a similar vein, we had to consider that most students would not have taken a university course in statistics at this point in their careers. Consequently, when mentioning questionable research practices, we did not refer to topics such as p-values or margins of error: We kept our explanations more general. For instance, we defined replication as "thorough repetition" and skirted the issue of statistical significance in our exploration of publication bias and questionable research practices.
Finally, we could not take for granted that the students coming into first year remembered learning about the nature of scientific knowledge and the scientific method, or had in fact received such instruction in high school. Students come from various educational backgrounds. Therefore, in the first unit, we laid a foundation for the introduction of OS by first examining how scientific knowledge is built.

Assessment and Feedback
At the end of the term, students had the option to answer open-ended questions through an exit survey mounted on Canvas. We posed the following questions: · If you had to sum up your view on OS, what would you say? · In your opinion, what is the most important benefit that OS can bring? · Which aspects of the OS movement discussed this semester resonate the most with you?

Implementation of an Open Science Instruction Program for Undergraduates
As noted in our presentation at the OSC2020 Conference in Berlin in March, the students showed a keen interest in OS principles, and professed, by the end of the course, both positive attitudes toward OS practices and an appreciation of the ways in which OS can improve science and society. The top benefits of OS as cited by the students were scientific progress, transparency, and collaboration.
Seventy-two students responded to the survey, and many of them wrote inspiring and moving words, demonstrating that the module had indeed engendered in them an appreciation of OS as a positive movement for scientific research and for society as a whole [14,15]. The following are two examples of the feedback [14,15].
I think OS principles are very important, and can provide a good foundation for future science projects. I think it is important that as students entering the scientific world, that we learn about OS now so we can carry these principles with us through our careers.
[OS] is something that goes against the mainstream competitive nature of the science community, but it is mutually beneficial for all scientists and society, so I support it fully.
Below is a "wordle" created from the text of students' survey responses ( Figure 2): In the last year and half working on this project, we have been struck by the value of the combined support provided to this initiative by faculty, the Library, the Provost's office, our Advanced Research Computing Department, and the Strategic Initiatives and Planning office at UBC. While the Library has taken the lead on the promotion and development of these undergraduate materials, lacking this support from any one of these bodies would have made our efforts much more challenging.

NEXT STEPS AND CONCLUSION
Although we created detailed content for OS 101 and 102, our role in OS instruction is changing to one of providing templates, resources, and consulting advice to instructors and lab managers interested in introducing OS principles and practices into their classrooms and labs. As such, we've been introducing alternative examples that speak to course areas other than Biology and examples related to COVID-19. Producing templates for second-year modules lies just ahead on the horizon.
The ability to make discrete connections across the research life cycle and to critically identify sources of bias underpins OS principles and practices. Working with undergraduate students and liaising with faculty and lab managers, our OS Librarian introduces students to this landscape of scholarly communication in the context of OS.
In the summer and fall of 2021, we will be working with the Biology Lab Managers to tie OS 101 and 102 more closely to the curriculum and to introduce lab exercises that specifically link to the content. Lab teaching assistants will also receive an OS orientation. OS will make up a larger portion of a student's overall course mark.
At this point, whether the current funding of an OS Librarian position through the President's Excellence fund will be renewed remains unknown. The COVID-19 pandemic has affected our project both by introducing uncertainty into the university's budget forecast and by raising awareness of the boons that OS practices such as data sharing and open-source medical equipment designs can bring. Our qualitative results show that it is possible to foster change in the next generation of scientists with a relatively small investment of human resources, and given the policy changes occurring in the Canadian government and the international context, there seems little doubt that OS will continue to grow in importance.