A standard journal article contains the results of a single analysis pipeline, with a single set of analysts. Even in the best of circumstances, there is reason to believe that further intelligent analysis may produce different results. For example, in 2020, the UK Scientific Pandemic Influenza Group on Modeling asked nine groups to calculate the R-reproduction number for COVID-19 infections. The groups were selected from a wide range of details (deaths, hospitalizations, screening rates) and modeling methods. Without the clarity of the question, the variance in ratings across the groups was large.
On October 8, 2020, an optimistic estimate suggested that 100 people with COVID-19 would infect 115 others, but probably fewer than 96, the latest figure means the epidemic is being reversed. In contrast, the most unreliable rate was 100 people and COVID-19 infected 166 others, with a high number of 182, indicating rapid spread. Inform future work as the epidemic continues.
A complimentary conclusion
This and other ‘multi-analysis’ projects show that independent statistics have never used the same process2–6. However, in fields ranging from ecology to psychology from medical to material sciences, a single analysis is considered sufficient evidence to publish the findings and present a strong claim.Over the past decade, the concept of P-hacking has made researchers aware of how the ability to apply multiple mathematical processes can tempt scientists to choose the one that leads to the most impressive conclusion. What is not clear is how limiting the analysis to a single process blinds researchers to an important aspect of uncertainty, which makes the results appear more accurate than they actually are.
In mathematician, uncertainty refers to the range of values ​​that can be reasonably taken, i.e., the production number of COVID-19 or the relationship between religion and well-being6, or between cerebral cortical thickness and cognitive power7, any number of mathematical estimates. We argue that the current method of scientific publishing – consistent with a single analysis – focuses on the ‘myopia model’, a limited consideration of mathematical speculation. That leads to overconfidence and bad predictions.To measure the validity of their conclusions, researchers should include data in multiple analyzes; accordingly, these will be done by one or more independent groups. However, we argue that the benefits of broad-spectrum approaches, which are very different from mathematical understanding, can have far-reaching consequences that are so important to consider how the system can be implemented.
2020 study of the Neuroimaging Analysis Replication
For the past 100 years, experts such as the advanced Ronald Fisher and the official methods of examining the hypothesis are now considered important in reaching conclusions with numerical data. (The P value, which is often used to determine ‘mathematical significance’, is the most well-known.) Since then, experimental complexity and methods have been developed to measure indirect uncertainty. But any single analysis pulls in a very limited range of this. We think that, as it is used now, the analysis of uncertainty only reveals the iceberg theme.
Twelve or more official projects by many analysts completed so far indicate that the levels of uncertainty are much higher than those raised by any one group. In the 2020 study of the Neuroimaging Analysis Replication and Prediction Study, 70 groups used the same magnetic resonance imaging (MRI) data to assess 9 hypotheses about brain function in risky decision-making. For example, one theory explored how the brain area is activated when people consider the prospect of greater profits. On average in all hypotheses, about 20% of analyzes comprise a ‘few people report’ with the opposite quality of the majority. Of the three hypothetical hypotheses, about one-third of the groups reported statistically significant results, so publishing work from either of these groups would hide much uncertainty and the spread of possible conclusions. Research coordinators now recommend that multiple analyzes of the same data be performed regularly. All of these projects destroyed two myths about the statistics used. The first myth is that, in any data set, there is a single analysis process, which is uniquely relevant. In fact, even when there are multiple groups and the data is simple, analysts almost never follow the same analysis process.
False alarm?
Indeed, major changes in the way science is made are possible: expectations regarding data sharing are growing. Medical journals now require that clinical trials be registered at launch to publish results. But proposals for change inevitably lead to a critical response. Here are five of them.
Won’t students be confused? Currently, there are no complete standards for how to present and interpret the results of multiple analyzes, and this situation can make it difficult to report results and confuse conclusions. But we argue that potential misunderstandings are an important factor in analyzing multiple groups, not the error. If the conclusions are supported only by a subset of concrete and analytical models, students should be informed. Dealing with uncertainty is always better than sweeping under a blanket.
Aren’t some of the problems that stress? Problems in research science include specialized reporting, lack of transparency about analysis, hypothetical hypotheses and ideas intended to support it, and incorrect data sharing. It is important to make improvements in these areas – indeed, how data is collected and processed, and how variables are defined, will have a major impact on all subsequent analyzes. But the methods of many analysts can still bring understanding. In fact, the projects of many analysts are often at the forefront of data sharing, transparency and theory-driven research. We look at solutions to these problems as consolidation instead of watching a zero game.
Is it really worth the time and effort? Even those who see the benefits of multiple analyzes may not see the need for it at the time of publication. Instead, they may argue that the actual team was encouraged to do more analysis or that shared data could be re-analyzed by other interested researchers after publication. We acknowledge that both can be better than the current situation (sensitivity analysis is a practice that can be used very rarely). However, they will not present the same benefits as multi-group analysis performed at the time of publication.Some skeptics doubt that multilateral analysis will consistently find a sufficient range of results to make the effort worthwhile. We think the results of the existing projects of many analysts contradict that argument, but it would be helpful to gather evidence from many projects. The more methods used by many analysts are made, the clearer it will become and the more important they become.
Will the journals not succeed? One questionable answer to our proposal is that many analysts’ projects will be longer, more complex to present and evaluate, and will require new article formats – problems that will make journals hesitant to embrace the idea. We argue that the review and publication of a paper by many commentators does not require a very different process. Multi-team projects have been published in various journals, and many journals have already published ideas attached to accepted manuscripts. We challenge the editors of journals to give projects to many commentators a chance. For example, planners may test water by editing a special issue that includes model studies. This should make it clear that the additional value of the multidisciplinary approach is worth the extra effort.
Wouldn’t it be a struggle to find analysts? Another answer to our proposal is that a lot of multi-group analysis published so far is a product of demonstration projects wrapped in a single paper. These papers include several analyzes of a long list of authors compiled mainly for reformists; most other researchers may see little benefit from being a small contributor to the paper of many commentators, especially those who are limited to their primary research interests. But we think that enthusiasm has a broad basis. In most of our analysts’ projects, we are known to receive more than 700 subscriptions in about 2 weeks.Yet another option would be to include a lot of analysis in the training programs, which can be useful both in the research community and to open the eyes of mathematicians. Whatever the combination of benefits and formats, if more analytical efforts are used and discussed, it will be much easier. What makes such multi-team efforts effective should be studied and used to develop and enhance practice. As the scientific community learns how multidisciplinary analysis is done and what can be learned, acceptance and enthusiasm will increase.
We argue that rejecting the opinion of many commentators would be like Neo choosing a blue pill in the film The Matrix, thus continuing to dream of something comforting but false. Scientists and the public will be better able to cope with the possible weakening of the reported mathematical results. It is important for researchers and the public to have such an indication of weakness from the time the results are published, especially if these results have real-world results. Recent projects by many analysts suggest that any single analysis will lead to overconfident and presumptuous conclusions. Overall, the benefit of understanding more will outweigh the effort.
Source Journal Reference:Eric-Jan Wagenmakers , Alexandra Sarafoglou&BalazsAczel,One statistical analysis must not rule them all, Nature 605, 423-425 (2022) doi: https://doi.org/10.1038/d41586-022-01332-8
READ ALSO : The Health is real wealth proved during the Covid-19 epidemic: How our lives have been stolen by COVID-19?
