Session 11: Research assessment, production & systemic reviews; a role for research libraries
Friday 8 July – 9:00 – 10:30
Chair: Charlotte Wien, University of Southern Denmark, Denmark
11.1 How research libraries can contribute to reform the research evaluation system, Maxence Larrieu, Université Paris Cité, France
The intersection between open science and research assessment is not the easiest to clarify. Because it concerns at once national research policies, international rankings, bibliometric tools, for profit organisations, research institutions and management, it can be difficult to get involved in this topic. On the other hand, the open science community is aware of the need to change the research assessment. As Bernard Rentier  used to say in his lectures: ‘we can talk for hours about open science but as long as we did not change the research assessment system, nothing will happen’.
Since the end of 2021 important signals regarding an evolution, or even a reform, of the research assessment system have appeared. For example the european fundraising appeal ‘Services and tools to underpin a research assessment system that incentivises open science practices’  and the creation of a coalition commited to reforming the research assessment system . Furthermore, at a more technical level, the limits of traditional tools (Wos & Scopus) have never been more apparent, and new tools, led by the open science community like Open Alex , are arriving.
This paper comes from a training on bibliometrics and open science given to PhD Students and librarians. At the end of the session, fellow participants wondered ‘what can we, as research librarians, do to help to reform the research assessment system ?’. The purpose of this paper is to begin to answer this critical question in three parts.
The first concerns bibliometrics, that is the statistics that are produced to account for the scientific production of a research institution. The goal is to be collectively aware of the limitations of traditional tools, which have been used for decades in bibliometrics. I will draw a criticism of Wos and Scopus mainly with the term exhaustivity and objectivity, and also with an approach based on cost and services. Finally I will describe some aspects of the new Open Alex tool related to research assessment and open science. The second part will be devoted to the DORA declaration , I will emphasize recommendations that can be disseminated by research libraries. To stay focused on practical aspects, the last part will give examples. I will explain the steps taken by two universities to facilitate open access practices. The first has released an “open access bonus” for the laboratories, and the later has made its institutional repository the sole source of its evaluation system.
11.2 The Role of Academic Libraries in Scientific Production Evaluation – the Experience of University of Zagreb, Croatia, Branka Marijanović, University of Zagreb Faculty of Electrical Engineering and Computing, Croatia, Višnja Novosel, Faculty of Humanities and Social Sciences, University of Zagreb, Croatia, Tatijana Petrić, Central Office for Coordinating Libraries of University of Zagreb, Croatia, Zrinka Udiljak Bugarinovski, Faculty of Economics & Business, University of Zagreb, Croatia
Since internationally visible scientific productivity is a criterion for the state evaluation of Croatian academic and scientific institutions and their scientists, Croatian academic libraries have a key role to play in the quantitative evaluation of scientific productivity using modern methods such as bibliometrics, scientometrics and the like.
In this paper we present the services used in the quantitative assessment of academic productivity in the academic libraries of the University of Zagreb. Consisting of 31 faculties and 3 art academies, the University of Zagreb is the largest university in Croatia. With its tradition since 1669, it is one of the oldest universities in Europe.
The theoretical framework of the paper is based on four areas defined by Christine L. Borgman – the characteristics of the scientific community, the development of that community, the evaluation of scientific ideas and the dissemination of scientific ideas.
The research will use a qualitative methodology that will examine the current situation at the University of Zagreb, the possibility of further developing this type of library service and the necessary knowledge that librarians in academic libraries should have. The research will analyse important national documents that define the assessment system at the University of Zagreb and the current state of such services in the libraries (such as the Ordinance on Scientific Promotion and Advancement Requirements Scientific Titles, draft Standards for Higher Education Libraries in The Republic of Croatia, the Quality Assurance Strategy of the University of Zagreb). The research will use the method of content analysis of these acts and library study programmes, the survey method, and the interview method. The survey method and the interview method will be used to determine the number of libraries at the University of Zagreb that offer services for quantitative evaluation of scientific productivity. It also aims to determine how the libraries are organised in terms of scientific productivity evaluation and what competences the librarians need for this type of service. The sample consists of 39 academic libraries operating at 31 faculties and 3 academies of the University of Zagreb.
The aim of the study is to identify and illustrate the current situation in Croatia and provide recommendations for further development of scientific production evaluation, which could serve as a framework for the systematic introduction of this type of service in all libraries of the University of Zagreb, and wider. In addition, the identification of the required competences of librarians can provide guidelines for further improvement of lifelong education of librarians as well as recommendations for library study programmes in Croatia. The recommendations may also be potentially applicable to the international community of academic librarians.
11.3 Faking It ‘Til We Make It: Libraries Supporting Researchers in Systematic and Scoping Reviews, Anna Marie Johnson, Alyssa Denneler, Amy Minix, Indiana University, United States of America
Medical and health sciences librarians have been participating in supporting their communities of researchers with systematic reviews for over two decades (Boynton, 1998; McKibbon, 1998). Now systematic and related types of literature reviews which attempt to analyze the literature on a particular research question in a repeatable and transparent way are gaining momentum and popularity in other science and social sciences fields. Researchers are using them to ascertain the best and most effective interventions across fields such as public health, biology, education, social work, and political science. In many ways, this type of research is exactly what libraries have been waiting for and it often makes excellent use of library resources; however, supporting this work can also require significant library personnel time, a high level of database searching skills, depth of subject knowledge, and skills with specialized software. The process or the review type itself is sometimes prone to being deeply misunderstood by the researchers most wanting to undertake them. Are academic librarians prepared to provide the necessary and intensive assistance with this type of research? What does this assistance look like and what kinds of professional development is needed?
This session will provide an overview of the current state of systematic and scoping review support among academic libraries in the United States. It will also present the case study of the efforts of a small group of librarians attempting to build their own support community in a large research institution. This case study will examine how librarians can scale this type of research support and offer assistance even when they don’t feel entirely comfortable or equipped to do so.