US funders need to be more proactive in ensuring the quality of their research.
The three top funders of basic research within the United States are under the spotlight as the federal government strives to make the research they fund more reliable. Two of these agencies – those of the US National Institutes of Health (NIH) and the US National Science Foundation (NSF) are putting plans to encourage more transparency and rigor their procedures; NASA in turn has been criticized for not being more active.
The decision by the NIH and NSF is in response to the publication in July 2022 by the US Government Accountability Office (GAO) regarding the credibility of federally funded research and the way in which the NIH, NSF and NASA evaluate the research they support. It concluded that, even though scientists have developed various methods of gathering, sharing and storing research data, the three agencies don’t generally promote these practices when they evaluate research.
According to the report “do not have evidence of a rigorous design or transparency of research findings like study sample size and adherence to plan of research, and the degree to which research results are accessible, searchable, and useful”. That means that they lack the information necessary to improve the manner in which grants are granted or to determine the funding areas that need to be prioritized.
The report provides six recommendations that include two recommendations for each agency on how funders can improve transparency and rigor to the projects they finance. The recommendations focus on gathering information about rigorousness that is the “soundness and accuracy of the study’s design, execution as well as data collection and analysis” as well as on transparency, making sure that the information is recorded and available for sharing.
It is believed that the NIH and NSF are in agreement with the recommendations however, NASA does not agree with one recommendation and is not fully in agreement on the other.
“For me the transparency of research as well as openness are an essential aspect of how we improve the quality of research,” says Marcus Munafo an expert in biological psychology in the University of Bristol, UK.
US Transportation Command’s Top Leader is in the US 5th Fleet
He heads the UK Reproducibility Network, a group of researchers who are aiming to understand the factors that influence high-quality research. It also offers training and disseminates the best practices in this field. “Funder mandates can go a long way towards achieving this, however they have to be enforced in order to truly create change,” he says.
Funders react
In responses to email questions of Nature Index, the NIH and NSF have said they are developing plans for new procedures in response to GAO’s report. GAO report. A NASA spokesperson instead pointed Nature Index to the report and reproduced the June 2022 letter which provides the agency’s response in response to recommendations.
The letter was written by NASA deputy chief scientist David Draper to Candice Wright, director of GAO’s Science, Technology Assessment, and Analytics team. In the letter, Draper argued that NASA doesn’t need to gather information from researchers about scientific rigor.
“NASA considers that the most effective method to guarantee the reliability of research is to use the peer review process that has been the standard of excellence in the field of credibility for scientists,” Draper wrote. “Accordingly, NASA relies on the peer review process within scientists to judge the quality, rigor as well as the transparency and quality of proposals for scientific research that are submitted to NASA as well as journals that publish scientific papers arising of research funded by NASA.”
Wright who co-authored The report’s author, Wright GAO report, has stated that reviewing grant applications does not suffice to weed out non-replicable or unproven research. She believes that she hopes the GAO continues to communicate with NASA regarding the advantages of implementing the recommendations in the report.
Richard de Grijs an Astronomer who works at Macquarie University in Sydney, Australia is in agreement. He claims that, in his opinion, NASA hasn’t adequately responded to the fact that the peer-review system doesn’t provide a guarantee of poor science being published. “It could be beneficial to establish an additional verification,” he says.
In the second recommendation to NASA that recommends looking at and, if needed, reviewing policies regarding transparency in data, Draper wrote that NASA is only in part acquiescent.
It’s because the organization is currently revising its scientific integrity policy and is adopting open-science practices in a larger way as he explained. “NASA’s Science Mission Directorate has launched the Year of Open Science, and the goals of the initiative include, but not restricted to, improving accessibility to NASA-funded research papers by using the FAIR rubric of Findability, Interoperability, Accessibility, and Reuse” the author wrote.
Agency actions
The NIH however, has stated that it is looking into “several different actions” as a response to the GAO report as per a spokesperson from the NIH Office of Extramural Research. Certain are being carried out. One policy, which was implemented on the 25th of January, for instance, requires researchers to determine how their data will be protected and made available for sharing.
In the event that grant applicants are asked to explain their methods to ensure that their research is conducted with rigor for reliable and impartial outcomes in their research The NIH informs applicants that they must ensure that the design of their experiments and methodology, as well as analysis, interpretation, and reporting of findings are all sound and objective The spokesperson states.
A NSF spokesperson explains that the organization is “committed to increasing the credibility of the research it finances” and supports the GAO’s suggestions. According to the spokesperson, NSF is “currently developing strategies for responding to these recommendations” They say.
The GAO intends to meet with the organizations at least once per year to evaluate the progress, Wright says, and will allow the funders the time to implement any modifications in full. The GAO will also be monitoring legislation that is being enacted in the US Congress but for the time being it will focus on what legislation falls in accordance with the recommendations in the report.
Munafo believes that a portion of the budgets of all funders should be allocated to this type of research. “We have to comprehend the factors that affect the quality of research and particularly the impact on research quality that are both intended and unintended – of the policies and mandates of funders,” Munafo says.
Munafo believes it’s appropriate that funders are being challenged to tackle these concerns. “This happens in differing degrees, between funding agencies and countries, but should be an integral component of the way funding agencies operate.”
