Measuring the impact of research access in the developing world
How Research4Life is being used to improve health care and economic development
Dr. Sami Hyacinthe is an agronomist for Burkina Faso’s Institute for Environment and Agricultural Research. When his institution first received access up-to-date scientific journals in 2008, he said, it showed him that his research was not as original as he thought. But it was exactly the realization he needed to raise his research – and the work of his institute – to international standards. (Photo courtesy of Dr. Sami Hyacinthe)
As Director of Outreach Programs for the International Association of Scientific, Technical & Medical Publishers (STM), Richard Gedye manages STM’s outreach programs, including Research4Life and patientINFORM.Research4Life is the collective name for four programs in which more than 150 publishers provide access to researchers, academics, students, doctors, agriculturalists, economists and other professionals in the developing world. patientINFORM is a free online service that gives patients and their caregivers access to up-to-date, reliable research about the diagnosis and treatment of diseases, along with help interpreting that information.
Gedye, who lives in Winchester, UK, has worked in academic journals publishing since 1986, initially at Macmillan. In 1991 he joined Oxford University Press, where he held various positions, including director of sales, marketing and research. In 2002 he helped found COUNTER, an international organization that has established a code of practice for vendor-based online usage statistics.
We all know that in many parts of the developing world, there are shortages of food, clean water, adequate housing, health care and other resources necessary for a well-functioning economic and social infrastructure. What is less appreciated is how these countries have historically also lacked access to up-to-date scientific research. Not only does this lack of access limit scientific activity; it deprives policymakers of knowledge that could help them address many of the symptoms of national underdevelopment.
For the past decade, Research4Life has been working to close this knowledge gap by providing developing nations with free or very low-cost access to peer-reviewed journals.
Closing the scientific knowledge gap
Research4Life is a public-private partnership of four UN agencies, Yale and Cornell Universities, the International Association of Scientific, Technical & Medical Publishers (STM) and nearly 200 academic publishers worldwide, both commercial and nonprofit. Elsevier is a founding member and contributes more than 25 percent of the content.
Dr. Nguyen Duc Chinh (center) and colleagues at Viet Duc hospital in Hanoi, Viet Nam perform surgery. Access to Research4Life’s HINARI programme has led to better outcomes for surgery patients. Before gaining access in 2002, they relied on free public tools, such as Google and PubMed, to search for medical literature. (Photo courtesy of Dr. Nguyen Duc Chinh)
With the support of technology partner Microsoft, the program is providing access to research publications in more than 100 developing nations. Researchers in participating institutions have access to a similar level of information as their peers in developed countries, enabling them to contribute to the evolving body of global research. The goal is to help attain six of the United Nation’s Millennium Development Goals by 2015, reducing the scientific knowledge gap between industrialized countries and the developing world. In November, the publishers in the partnership renewed their commitment to Research4life through 2020 to ensure that 18,000 peer reviewed resources continue to reach research communities in low- and middle-income countries.
Research4Life comprises four programs: HINARI for health, AGORA for agriculture, OARE for environment and ARDI for innovation. The oldest and largest, HINARI, uses the resources of the World Health Organization (WHO), Yale University Library and over 150 medical publishers to provide developing world institutions access to some of the world’s most important published medical research.It’s clear from key metrics of HINARI’s growth that the service is having a significant impact. The HINARI online library now makes available over 17,000 scientific publications from more than 150 publishers to over 5,000 organizations, including universities, medical schools and teaching hospitals, research institutions and government offices. A survey for a 2010 commissioned review of the user experience of the Research4Life programs revealed that more respondents (24 percent) cite HINARI as a source for life-science and medical research than any other source, and more respondents (32 percent) cite HINARI as the source they use most frequently.
But what about the practical impacts of the availability and use of HINARI?
In a November 2011 post on The Scholarly Kitchen, the blog of the Society for Scholarly Publishing, Phil Davis looked at the complementary merits of case studies and statistical analysis as tools for measuring impact. Research4Life has adopted a similar twin-pronged approach to examining the impact our programs have had on our users, their research, their communities and their countries’ economic development and welfare.
Research4Life is helping a small charity improve the lives of HIV-infected orphans in Zambia. Here, Dr. Tim Meade treats children at a Tiny Tim & Friends clinic in Lechwe Village, about two hours outside the Zambian capital of Lusaka. (Photo courtesy of Tiny Tim & Friends)
In 2011 we ran a competition which invited our users to tell us how access to Research4Life content had impacted their work. Although we could declare only two winners from the entries we received, we felt that the insights and testimonies contained in these stories deserved a wider circulation, both as a demonstration to our users and potential users of the positive outcomes that can result from access to, and effective use of, the latest research literature, and also as a means of demonstrating to our publisher and other program partners the practical value of their continued support and active participation in our programs. Accordingly we commissioned and co-edited a booklet, Making a difference — Stories from the field, which retells 11 of the original testimonies in a more journalistic style than the original competition entries. The resulting case studies provide a wealth of insights into how access to the results of peer-reviewed research from the Research4Life publisher partners is benefiting the health, well-being and economic and social development of communities in the developing world, as well as contributing to greater environmental health and awareness. For example, access to research content via HINARI has:
- Allowed a practitioner in Ethiopia to successfully treat a patient with adamantanoma of the tibia, a rare form of cancer, and helped his hospital to deliver more effective training to orthopedic physicians.
Enabled a Nepalese pediatrician to save children’s lives through better treatment of diarrheal diseases while developing his hospital’s journal into a scientifically rigorous publication. In fact, a number of entries reported that access to Research4Life content had enabled improvements in local journals. It had been a concern, when HINARI was being developed in 2001, that access to our content could have a negative impact on local publishing; it was reassuring to learn that, at least in some places, HINARI has had the opposite effect.
- Helped a physician improve the lives of HIV-infected children in Zambia. HINARI plays a major role in his work, allowing research that informs the development of policies and medical procedures that provide the best possible treatment.
- Helped an agricultural researcher from Burkina Faso to develop better and more informed scientific writing skills, produce focused research that he can discuss with top researchers worldwide, compete more effectively for research funding, and deliver better teaching programs.
- Allowed a Sudanese policymaker to introduce evidence-based policy development designed to improve the Sudanese people’s health in the long term. Using the published scientific research available through HINARI has made it much easier to convince other policy-makers at federal and state levels about the merits of his agency’s proposals.
- Enabled a midwife to improve maternity care in Zimbabwe and reduce maternal and neonatal mortality rates.
- Helped an Ethiopian physiotherapist find better ways to treat his patients and teach his students (Watch the video below.)
Benefits cited by respondents
As can be seen from these examples, access to HINARI content has resulted in clear improvements in practical patient diagnosis, treatment and care. In some countries, there is flourishing activity in evidence-based health care, policymaking, advocacy (to decision makers and the general public) and problem-based learning, in which students and medical practitioners learn from the experience of actual cases that are written up in the journals. All of this would likely be impossible without HINARI.
Other benefits highlighted by many of the respondents were a new-found ability to develop informed and relevant research agendas, create well-informed and successful research grant applications, conduct research which would command increased respect from peer reviewers and journal editors, and establish credible relationships with the medical research community worldwide.
Another byproduct was that the new knowledge and expertise HINARI access was providing could be shared in a teaching environment, resulting in better prepared and better informed students, graduates and clinicians. There were also indications that HINARI has helped reduce brain drain; some respondents reported that before having local access to HINARI content, it would have been impossible to pursue their particular studies in their own country. Another respondent explained how access to HINARI had freed up more resources to spend on conducting his research whereas previously, a significant portion of his research budget had had to be ring-fenced for the purpose of travelling to Europe to get access to relevant recently published research.
Supporting stories with statistical analysis
Bibliometrics — the science of using statistics and mathematical methods to analyze bibliographic data and references.
Bibliographic data — data about published literature, including the number of publications an author has and the number of citations received by an author, institution, subject area or country.
With the above booklet published, we feel we have developed a convincing body of case study evidence of the impact that the growth and development of the HINARI program has made over the last 10 years.
However, as indicated earlier, case studies are only one way of demonstrating impact. For many stakeholders, such anecdotal evidence is most persuasive when supplemented with a more statistically based analysis. In seeking to develop a statistical analysis that would indicate the level of impact Research4Life has had on its target constituencies, we initially decided to pursue a bibliometric approach that would involve identifying institutions that have had varying degrees of exposure to HINARI content:
- Institutions that have registered for HINARI at least four years ago and have had significant use of its content.
- Institutions that have registered for HINARI very recently (in the last 12 months).
- Institutions that have not registered for HINARI and who do not have formal institutional access to HINARI journals via any other means.
Then, by examining their research output over the last 15 years, we would measure whether, since the introduction of access to HINARI content:
- There has been a significant increase in the production of research articles and the number of cited references in these articles.
- Article reference lists include more citations to HINARI journals
- The median age of article references decreases.
After much discussion and research among our impact working group colleagues, however, we reluctantly concluded that it would be impossible to conduct a bibliometric analysis of the kind we were proposing that would be able to successfully control for a host of potentially confounding variables like faculty distribution, size and funding over time. The information simply isn’t available in any coordinated fashion. Any attempt to measure HINARI impact in the bibliometric manner proposed would require an impossible amount of effort and still result in an analysis with severe and unavoidable methodological flaws. In addition, we decided that establishing that non-registered and recently registered institutions did not have systematic access to HINARI journals via other means – for example, alternate developing country schemes such as the International Network for the Available of Scientific Publications (INASP) and Electronic Information for Libraries (EIFL) or via sister institutions or established personal connections – would be extraordinarily difficult to achieve with a satisfactory degree of accuracy.
Plans for a survey
So after a useful and productive discussion, and inspired by Professor Raymond Wolfinger’s maxim that “the plural of anecdote is data,” our literature impact team decided to develop a survey as the main statistical component of our proposed project. Target groups we will aim to reach with our surveys will include researchers, research funders, research administrators, physicians, trainers and librarians. Survey questions will explore issues such as the frequency of access to HINARI content, which content is accessed and for what purpose and how this access has impacted elements such as:
- Clinical practice and patient care
- The ability to write successful research proposals
- The quality of research undertaken and the efficiency with which it is carried out
- Individual jobs, roles and careers
- Individual departments and institutions
We may also decide to conduct a simpler and more modest bibliometric study that would seek to measure median age of article references over the last 20 years, comparing the figures for developing and developed countries. That would give some indication of the effect of increased access to journal content via all the various initiatives that have been launched over the last 10 years or so.
We look forward to sharing the results of our research in due course.