Florian Naudet, MD, PhD

Florian Naudet: This is a presentation where you will need at some point, your phone, just to look at the QR code that that can appear from time to time. In fact, just one is very important so I will tell you. 

So I have a new specific competing interest regarding this talk. I received grants from la foundation Pierre Deniker, it was a long time ago. Rennes university hospital and also from French ministry of health, higher education, and research and now from the institute university of France

I will talk today about zombie trials. So first of all I use the term, Zombie Trial, from this papers that were published in 2020 in Anaesthesia. The first one was published by Professor Carlisle, who is the editor in chief of the journal, and in fact he wrote about trials that he received that that relied on what he called false data and as you can he is the editor in chief and received a lot of randomized control trial. He received 526 trials. Among those he estimated that 73 had false data. That was the red right line here and the red line with the dot points, in fact it was 43 trials that he truly categorized as zombie so what is a zombie. It's a trial that is false beyond repair. So it's obviously very false. He cannot say that it's fraudulent because to say that it's fraudulent you need to know the intention of the person who submitted the trial, but he can say that the trial is false beyond repair, and this is the definition that I will use today.

Obviously, he is a good editor. He knows these trials, and the trial do not appear in the journal, at least he tried to avoid publishing them. Sometimes from time to time, those trials  can be published in the literature and correction of those trials that are published are often too little and also too late. This is the spirit of this trial and in my talk, I will try to give some examples about this.

So first, I need to thank all the team that try to fight against these 2 zombies, and one of the most important person in this team is André Gillibert who identified a lot of the problems in these trials and I will tell you the story ,but this is a group of scientists who found the problems in the trial that I will describe it and they all worked a lot on this trial.

So I will start from here. I will start from the French Parliament, in which they organized a meeting, like today, just to present some things, some data, and a French movie that was called Malaria business, and this was invited by Cedric Villani. He is a great mathematician here in France. He won the fields medal, and he was also a deputy at this French Parliament and invited a group to discuss the movie, malaria business. So I will pitch you the movie, for if you are interested, you can find the movie very easily with the QR code, and you can also find the meeting at the French Parliament. In this movie, the pitch was very simple. Malaria is a big deal, a big problem. Malaria is usually treated with drugs and some of those drugs contain artemisinin, which is a molecule and this molecule, can also be found in plants, in artemisia, and with this plant, if you can make some tea with those plants, artemisinin should be in the tea and then should cure malaria so there is a cheap, inexpensive, treatment for malaria, which is a big public health problem. That it is underused, and this movie says that it's underused because the pharmaceutical industry doesn’t want this remedy to be used because they prefer to use drugs that have of course, as we know, known side effects. So this was the pitch of the movie, and this presented some data that is going to be published or a few that were published. So the group that was presenting the movie was also a conducting trial and here is the first trial. 

This trial is not directly in malaria, it's on schistosomiasis. Because folks involved in this research are believing that artemisia tea infusion, could cure malaria and schistosomiasis and indeed many other diseases that you will see in the presentation. So this is a randomized control trial that's involved 400 patients for praziquantel (PZQ), 200 patients for Artemisia annua and 200 patients for Artemisia afra, so 800 patients for this big randomized controlled trial in artemisia. 

Here I will present you the table about side effects. This table can be found in the web appendix of the paper, and if you have a close look at this table, you will see that there is a something very strange in the data and this pattern is very strange in that many of those numbers are a multiple of 5. So you have 10, 5, 25, 0 ,0 ,5 and so on. This is the artemisia arm so almost no side effects, and almost all in a multiple of 5 and in the prosecutor PZQ arm you have a large number of side effects, and all are a multiple of 5 to which is very unlikely to see this kind of pattern. 

So the trial was bad beyond any repair. Here you can see some Kaplan Mayer curves and you can see that any events occurred at exactly the same time in the group artemisia, and all events occurred also but later in the group PZQ. This neglected a very basic kind of thing which you can see in biology, which is variability. So we are all different. And such kind of homogeneity is not expected in a clinical trial. So this was the triangle about schistosomiasis.

They then performed a second trial, this time in malaria, in treating, Plasmodium falciparum malaria, again in a very large triad with 957 patients who were infected. This was published in the same journal. This journal was called Phyto Medicine. It's a journal that is known for false involvement in  in phyto-therapy. 

it has an impact factor between 4 and 5. Here is a table that is present in the paper, where we can learn about adverse events and you can see exactly the same pattern, multiple of 5 in the events, which is very unlikely. 

So my colleagues wrote a letter about all the problems with those trials and here is a short section from the letter. They computed a p value for there being so many multiples of 5 and you can see this is very unlikely.

In response to this letter, the group in charge of the trial thanked the author soi this was a very kind answer, and then regarding the abnormal multiple of 5 values, said that those were the data that are received okay, and then that these results were very similar to Artemisia trials against malaria, so even in their response, they say that it's exactly the same problem that's in the other study and of course my colleague a response to the second trial, but with such concerns,  with such problems, in fact the editor should try to investigate. At this point the editor has not investigated it. He just published the letters of my colleague and the concerns, as it was quite normal discussion in the scientific process. Also they noticed there was a problem because the number of committee registration was exactly the same in the 2 trials. Then we will see also that the study was conducted before approval, so the trial committee number seemed to be outdated. And then they noted the problem with a multiple of 5 in the tables again with P-values that are very, very low.

Again they responded to the second letter, and they said, “Thank you very much for your in-depth analysis of our study” and they said that the data had a lot of problems when they collected this data. For instance they said that the lead/principal Investigator had great difficulties to implement this Artemisia tea trial against Malaria. So despite having received all appropriate approval, they do not comment on the timing. Several academics and ministers subsequently attempted to obstruct the trial. “There were also attempts to sabotage his work by stealing his laptop”.  This can explain some problems. “There were even efforts to poison him, he almost died.”  So this is kind of a thing that is not very easy to prove, and this is what is written in the response, and it appeared in the peer review literature but no more comment about the validity of the of the data. 

Meanwhile, I discussed with André and said, “you should ask for the data because this is especially the kind of situation, where it may help to have a look at the database and to have a look at the problem in the data.”. So my colleague received the database, and when we opened the database, what he saw was a very problematic pattern in the data with lines that were repeated. So each line was a patient and you had block of 10 patients, who exactly the same values, for instance you had patients who were the same temperature for 5 days and each day the temperature was the same for all patients by block of 10 or 20. So there was clearly a major problem in the database. So at this point because of the 2 letters did not seem sufficient for the editor, and as I said the trials were beyond any repair. So we asked for retraction at this point. But the database was joined to the request for retraction, so the editor could open the database to just have a look at it, but he asked even more detailed information. This was on September 2019, and we had a deadline in October 2019 to give even more detailed information that the database already had and meanwhile my André received legal threats from the sponsor’s lawyers about copyright of the database that was attached for the editor, and he had to delay his submission, because he was requested by his own lawyer to do this, and I cannot show you the database. So it is not possible for me to show the database because I don't have the database and also because I don't want to have a problem with the lawyers. But wait for it, they showed the database in their own movie, so you can look at this QR code with your cell phone and you can go to approximately 25 min and 40 s on the YouTube video, and then you can look a little bit and you will see, the author showing his database to the journalist making this movie, so in fact you can have a look at the database quite easily

https://youtu.be/W6TgP5RlsDQ?t=1536

It's not exactly the same database. We observed exactly the same pattern, so the same kind of pattern. But the numbers we were not able to find exactly the same number in the database received by the author so as you will see the numbers are exactly the same for lines when you read this, so I don't know if you did it now, but hopefully you had the time to do it, and you had the time to see the database, or else you can do it at home just after this talk.

Okay, so for the lawyers. I don't claim that these 2 studies are fraudulent. I cannot say that because I don't know exactly the intent of the authors. I don't know exactly the problems that were in this trial. But there is scientific evidence supporting the fact that the studies are true zombies, meaning that the data is totally wrong and that those studies are dead beyond any repair.

So we wrote for the editor, and we detailed how to find the problems, of course we could not now get through to the database, so we even explained to editor how to open the PDF file, how to open the doc find, so we were a little bit funny because he asked us too much and we wanted to be sure that he would understand everything. 

So we submitted this, and we received no feedback. Then on the 29th of October, the same year, we saw this. Without any information from the editor, we saw it on the Internet, that the letters by my colleagues were removed temporary and this is quite different from a retraction. When something is retracted, and the retraction appears in red in the head on the paper. And this is quite easy to see, what was written in this paper and to see the fact that it was retracted. That with the temporary removal everything disappears except the title. So all that were concerned disappeared from the medical literature all the concern of my colleagues and peers. 

So we have this problem with lawyers so my colleague André started to ask his university for some legal help in order to make sure that everything will be correct for him.  When he asked his university, the University said we won't help you a lot, it's better for you to go through the regional agency of health which is like a regional entity that represents the ministry of health and then, there lawyers can help people who might have problems. But at this time, in Normandy, there was this big program and that's why I used this photo for my slide. There was a big chemical fire in Normandy, a big chemical fire with a lot of problem in terms of public health, so you can imagine that the regional agency of health had many problems to deal with rather than little problems with these trials about malaria so no News from them, not much help. So we also contacted the Agency of Medicine in France, there is a specific address for Whistle-blowers, so we contacted them and at least it was important, because it identified that we are quite in the position of the Whistle-blower. In this case they said, “Thank you for your message”, and not much other help from them. There is also this big entity in France to overseer integrity of research in university. So we contacted them, and they tried to have a look, and said that they can also only investigate for people who are working in French university. Most of those guys are not working in a French university except Professor Perronne who was working in University UVSQ and so we contacted the people in charge of scientific integrity in UVSQ. When we contacted him, he said OKAY, but in fact there is nobody for scientific integrity in UVSQ because it is going to be merged with another university, which was university Paris Saclay to have this big entity, so they tried to merge together. So he said, the reference for integrity should be contacted at Paris-Saclay, so we contacted Paris-Saclay and of course, they said that they are not working for UVSQ yet, so they asked us to contact UVSQ, so no reference for scientific integrity in both universities in this case.

So we started to believe that nothing could be done for those trials because we tried a lot, and we had many problems. So it was very difficult at this point. So I was in this conference the reward equator, perhaps some of you also were in that conference. And there was Ivan Oransky speaking from a retraction watch. And he had this slide in his talk, where the Buck stops, with journal, with publisher, with COPE, with University? Where is the problem: with lawyers, and to be honest the problem was with lawyers for us, so it made a lot of sense for us, a lot of sense. And I tried to discuss with Ivan because he has a good knowledge of retractions and so on. And then he said that he could not really give me advice on this case, but that as a journalist, they often cover this kind of problems

But this was in February 2020, and this was the last days that we lived in the pre-covid world. Then came Covid, so you remember all the streets, you totally avoid everything, and then comes the Covid and come the surge in trials, specifically the 2 trials in New England journal of medicine and Lancet that were retracted because of possibly faked data. Then you had the problems with ivermectin with a lot of trials about ivermectin that were totally zombie. So things were changing, but also the activity for the journalist was focused on Covid, so nobody was really interested in these 2 studies, one about Malaria, and the second one about schistosomiasis. 

So it took time, we had no news, until I saw this which was in June 2020. That's the PI of those 2 trials who was trying to find money to Explore Artemisia effects in COVID-19 so I thought it could be a timely moment to set some light on those studies and I wrote this kind of threat/warning on Twitter. Then the journalist from retraction watch, it was a colleague of Ivan Oransky contacted us to discuss the story, so we discussed the story, and everything now is written in this article by retraction watch.

https://retractionwatch.com/2020/08/05/a-bitter-aftertaste-legal-threats-alleged-poisoning-muddy-the-waters-for-a-trial-of-a-tea-to-treat-malaria/

It details the story, it's very interesting because they also contacted the initial author of the story to ask them, what do they believe about the claim about the trial, and so on. And they also contacted the editor who did not agree to answer to retraction watch, but later asked to discuss directly with the publisher and the publisher, knowing that everything would be public that they will retract the students, and in fact we got retraction of those 2 studies in September 2020 though it was quite long from the beginning.

So those are the 2 examples of zombie trials. Now, how to spot those in literature, it's not so easy. I discuss ivermectin trials, some of those appeared in systematic reviews and meta-analysis, so those tools in ivermectin are not always fit for propose to identify this kind of fake data, it's not so easy. So we have now made a few attempts at a checklist that may help. This checklist is called the reappraised checklist and I can give you some examples of the things that you would like to look at in studies to see if those studies could be zombie or not. So you can look at research governance, you can look at ethics, is there evidence that the work with approved by a specific committee? This is quite important to have a look at this, you can also contact with the community. You can look at things related to authorship, to productivity which is quite important. Is the volume of work reported by research group plausible, including that indicated by concurrent studies from the same group? Is the staffing adequate for the study conducted? Is there something about plagiarism or research conduct? Is a recruitment of participant plausible within the stated timeframe for the research? You have a few things that could be interesting to look at. You also have things about analyses and methods, are things correct and correctly described, you are thinking about statistic and data. If there is some excess of homogeneity that you can find in the data is the data totally implausible? So you can see this in the artemisia trials. Are there errors in the data reported and is there a problem with data duplication and these are the kind of things you can have a look for. 

Now we have 2 first examples, we also have a group. Jack Wilkinson that might be here tonight is developing and iterating a tool for detecting those problematic trials. It means tools we use such as systematic review is not totally fit for purpose to identify this. We don't have a tool for detecting those problematic trials so this work would be very important, and I am looking forward to seeing the results to this

So now we have the tools, and we can say this is a zombie movie, and as any zombie movie you have a sequel so this zombie movie 2: zombie or not zombie. So I will use the same kind of semiology on this trial.  

This new trial is not on artemisia. It is about transdiagnostic treatment of emotional disorders for women with multiple sclerosis. It's a randomized control trial. So I start to have an interest in this study because of a Dorothy Bishop. Dorothy Bishop is a these are the colleague from the UK but maybe a lot of you know, and she found on these studies some problems that I will try to summarize. So first the trial was retrospectively registered, so it can happen that trials are registered retrospectively registered, but it is not of course, something that is a something that is desirable but overall this trial was a complex trial exploring a very complex intervention. It could be expensive and could be time-consuming to conduct. So for instance, you had 64 women with multiple sclerosis randomized to receive treatment as Usual or a new unified protocol which was the intervention of the study. It was 14 weekly 2h group session so it was lots of work lots of logistic to put this in place. But in fact the recruitment of those 64 patients was completed in one month, this is very surprising super-fast between May and June 2019, and the Trial ended on 21st October 2019. That was very good timing for trial it was very sharp. And she said, it's friendly hard to believe and I agree, it’s hard to believe. 

Then there were also problems that you noted from the ethical approval, so you always have to look at this. There were also problems for funding, she said that's a red flag because it seemed that the funding is very low if not 0. It's not possible to do this without any funding and they she found herself some question regarding the data.

I also had a look at the data and first what I saw, it was a very huge effect size, so you have an intervention that have an effect size of -2.1 points for this kind of psychological intervention, compared with treatment as usual, such kind of big effect size is very unexpected, but of course it could have done in likely small trial with no other problem of small trials and these kinds of effects.

Another problem, If you look at the trial report, you can see that you have things that are not really reproducible for instance, in the eligibility criteria, they say high score in difficulties emotion regulation scale, but what is high score? Usually you say it's score superior to 5 to 10 to 8. So this is very difficult to reproduce. Of course, this can be just due to sloppy reporting, but let's keep this in mind because it will be interesting in my next few slides

Then you can look at the table on this paper and you can see that it was just one value, but the P-value for the t-test was not in accordance with the t statistic. So here again, something that could be quite problematic. Then you can also look, these are the data presented in the paper, but on the ISRCTN registry, you can access a group of documents that reports the data of this study, the Patient Consent form, and also the study protocol. So if you use the QR Code, you can screen the registry, and then at the at the end of the page, you will see all of those documents that you can download and compare.

https://www.isrctn.com/ISRCTN95459505?q=cancer%20AND%20(radiotherapy%20OR%20chemotherapy)&filters=&sort=&offset=2&totalResults=1478&page=1&pageSize=10&searchType=basic-search

So for instance, here is a problem, the problem is also here so the problem appears to be on the results published on the registry, but on the registry, you can also see this. For age the P-value for when you compared groups about age was 0.293 and here the p value was 0.61, so we had a difference in P-value statistics, especially for age. Discrepancy between the results on the registries and the results on the publication. Of course it could be a typo, but it starts to look like, of course it could be something because we have a lot of problems now with the study.

Now, if you look at the distribution of P-values here, you can see that there is an excess of small P-values where one would expect to have a uniform distribution of P-values. In general with zombie trials, it's an excess of P-values showing no differences, so not pointing towards a difference. Here we have a lot of P-values that are quite small which isn’t very usual, but at least this is quite strange and then if you look at the manuscript, if you look at the paper, you can see that there are missing data in the paper. So for instance, we have the T statistic here equal, but no value present for the t-statistic. Here you have the Cohen d equal and no value for the Cohen d. No values for the confidence interval. So you have missing pieces even in the paper that was read by the editor and peer reviewers. They missed it here and here to so it was not very right, and not very useful but it can of course always happen. 

This is so you can go in the registry and find the protocol. This is a part of the protocol written in English. So this is a study from Iran and perhaps things were translated in English very badly, but you have something very curious in this version of the protocol. “Finally patient met all inclusion criteria and were randomly assigned to treatment and control groups”. This is not the kind of thing you are used to seeing in a protocol, and even if you have a bad translation, this should not appear in the protocol. “Randomization was carried out using concealed computerized……”, so this uses past tense, which is not good for the protocol? Also they talk about the mean age of the participants that are demonstrated. So beyond the fact that it has no sense, it's again, past tense and there was also a sentence that was started without any end, “given that the analysis was based on the ITT principle” and then no more.

So the protocol was also curious. We also saw differences between the protocol regarding calculation of the number of subjects on the protocol and the paper that was published. 

So here is the address of the research committee and here is the Gmail address, which is quite unusual for a research committee.

What is very interesting with this paper, and what will be one of my last points is that BMC women's health use an open peer review process. We can have a look at reviewer reports and the comments by the authors.

So let's have a look at the first reviewer. The first reviewer to this paper said “It's essentially a well, written paper. I would recommend all the short forms have their expanded versions when you introduce them in the paper, please make these necessary correction”. So I have never received a peer review like that, especially in the case that my paper has missing pieces in it. So this is a very good peer review. We don't know the name of the reviewer because it's not openly available some journals make it available, but not in that case of this one. 

So of course, the author responded,  and they said thank you of course, and also put that all the abbreviation is correct. So you can see that even the answer is longer than the problems as spotted by the review, so it's very interesting. And then you have a second reviewer. They appeared to have read the paper carefully at least better than the first reviewer, and he said that he found some problem, for instance in the inclusion criteria. As I told you those inclusion criteria were ill-defined, any he said there is one criteria that's called moderate to high cognitive impairment or physical disabilities - which screening tool was used for cognitive impairment and how was moderate to high cognitive impairment defined, what was the score? So I told you this for another inclusion criteria, it’s also this for criteria, high cognitive impairment  and the other answer is very interesting because they don't explain why, they don't explain what the problem was, they just say they deleted this item from the selection criteria. So when can you delete an item from the selection criteria during the peer review process without answering any questions? This is not very serious, but it was what they wrote, and their answer and it was accepted by the editor.

So I should say that this review there pointed that there was a first edition of the manuscript, a version that is not available on the internet. That he found some description of the result that did not coincide with the data presented in the table 1, so there were differences between the paper and the table. So perhaps it might explain the inaccuracy that is shown in the table, perhaps things were corrected during the peer review process. I cannot tell you much more because this is not available. At the end of the day both reviewers were very happy. 

Okay, this is a new disclosure for the second paper. I don't claim that this study is fraudulent. But it seemed that at least some parts of the paper are wrong, and it needs investigation from the editor. Is it to zombie? Is it not a zombie? Let’s see. This is ongoing, so I was contacted by the editor, and let's keep in touch you can follow me on Twitter (@NaudetFlorian), and you will have some news soon about these trials. My last point is that open science practices surely for instance with open peer review help, here we can see that there was one reviewer that was very positive, who was this reviewer? We know that peer review can sometimes be rejected so perhaps it was key in this case. And also open material data sharing might help. I would say that in the 2 and Artemisia trials, some of the authors shared the data with us,  and it helped to find that trials were fake beyond repair. 

QUESTIONS

Francois Maignen: So these are fairly recent studies Florian, do you have any idea to what extent these studies are prevalent. Is it a new phenomenon which happens in research, and do you know what the mechanisms behind the conduct of these studies are. What is the motivation of the investigators to conduct some strange results, do you have any ideas on that? 

Florian Naudet: So it's difficult to know. I think this is something that is not so new, so what happened is that during the Covid pandemic, we had a lot of light on this. I think Jack might be here too and he said to me that for instance, he wanted to have funding before the Covid pandemic about this topic and he said it was very hard and with the Covid pandemic, it isn’t because you had a lot of lights on these issues. So you had the ivermectin trial. People got aware of this. But to be honest, everything that happened during the Covid pandemic was here a long time before the covid pandemic, so it's about these practices that were present before, so this is not something that is new. Of course during the pandemic they became very visible and what is the motivation? To be honest regarding the Malaria trial, I would be very interested to know this, but I would say that it was not directly our job as scientists. So our job was to identify the problems in the trial, to discuss those problems, but not to investigate what led the author to do this. Now to know who was responsible for that, these are open questions and it's very interesting, but it's rather the job for journalists. So in case you have some journalists here happy to investigate on this, I would be happy to discuss

David Colquhoun: 2 things occur to me. One is the number of authors: are they all complicit or have some of them simply never read it? I mean it is a quite a long-standing problem of guest authors whose name appears but have made no contribution. I’ve myself published a paper in a case in which a rather senior person attached his name to the paper but clearly never read it which was pretty shocking I thought.

Florian Naudet: I cannot tell you again because I don't know the authors and their motivation, but my impression is that at least some authors share the database with us, so I think those that shared the database were not totally aware of that. I think that in this group there are some professional researchers, including one from a French university, but also a few problems with a certain hydroxychloroquine trial. In the group, there are some people from associations that are not directly very well aware of clinical research, of data quality control, and they don't have a lot of skills in statistics so perhaps they were of good face, but just without any understanding of what they were signing. I think there is a mix of  all of this, but I cannot tell you the exact motivation of everybody of course because this is beyond my expertise.

David Colquhoun: Yes it seems to me that this is an argument for abolishing journals altogether. The argument against that has always been that any old rubbish can get published. The trouble is any old rubbish can get published in a peer reviewed journal which gives it a false appearance of respectability, and it would save a great amount of money if we did that.

Ian Tannock: Reminds me a little bit, first of Gregor Mendel, who of course fiddled his results with the peas because they didn't fit his idea of exact statistics, but I do want to just report a much larger series of zombie papers, and perhaps an insight into why people do it. And this was about 20 years ago where I’m an oncologist. So about 20 years ago, high dose chemotherapy for bone marrow transplantation became widely used, particularly in the United States. The idea was you gave sufficient dose, and then you transplanted bone marrow to protect the women. It was so used that people were actually sued for not offering it to patients, and around that time there were several trials done. There was a couple in the States there was one in Canada there was 2 in Europe, and they were all negative, but at the same time there were 2 trials done by a guy whose name was Bezwoda, from the University of Witwatersrand in South Africa, so a totally unknown man leading these trials from a university that most of us had never heard of, and they were published in high impact journals. The author, Bezwoda was invited to the Large ASCO meeting which is a meeting of now about 40,000 people and presented at the plenary session and then the ASCO organization decided to send a team of 2 or 3 people to South Africa and it took them about half an hour to find out that the results were totally fabricated, and there was some discussion about motivation, and I think here probably what you had was a guy unknown but had this 3 or 4 years in the sun of fame, and then of course has disappeared from the picture and that may have been the balance, that's pure conjecture of course, but as you know this Isn't just sort of papers that appear in relatively minor journals, but as you talked about the New England and Lancet retractions, but can occur and have an incredibly large influence on a field and a lot of women were harmed by that treatment. 

Florian Naudet: I agree, and we proposed it. Me and my colleague André, we wrote a small letter to the Lancet just to suggest that they should have a reproducibility research editor because what I showed was quite easy to spot and to find and you had to have a look at the study, but it's not the usual thing that one looks for one performs peer review so peer review relies on the trust first, it's a good thing that science relies on trust. But peer review relies on trust, methodology criticism, and criticism sometimes of the biological question, but not so much on detection of this kind of fraud or fake/bad data, and we suggested that an editor with specific skills could have a look at each paper and it could be useful, but they did not publish our letter.

David Colquhoun: Did you say that this guy on the first trial you talked about, had been given a grant 2 million.

Florian Naudet: No, he was looking for the grant of 2 million. 

David Colquhoun: Did he get it?

Florian Naudet: I don’t know but he did get appointed to a Research Institute during the Covid research and now he's very closely related with a political power.  I don't know more about his trajectory but perhaps those studies and all his work regarding artemisia helped him

Lydie Meheus: Maybe I missed it, because we also have been trying to get papers retracted from a fully false trial setup regarding Gc-MAF.  You might remember that the Macrophage activating factor and oncology story and, while we could get a retraction from some of the journals that published on that topic, one journal said  it's not them to blame, it's the institutions, and we should reach out to the institutions where the authors are affiliated, is that a pathway you tried did you? 

Florian Naudet: Yeah, so as I told you in France, we have this organization for scientific integrity, and they are representative of scientific integrity in each university, so I try to reach the one in a  UVSQ University where Professor Perronne works and then they said that there was nobody. They asked me to contact another university because the 2 Universities were merging and the one from the other university said “No no, I'm not looking at the studies from the UVSQ University, so he asked me to contact the President of the University. I Contacted him, it was during the Covid Pandemic, and he said “So we have a lot more to do in our university. We have to make sure that our students don't catch the Covid” so they were not so interested in this, so that's what's occurred with universities.

Lydie Meheus: Yeah, because the only thing is we also have the case with Stanford University and there it became really tricky, it's a was a different story about falsification but there it's really became involved with lawyers and then they kind of settled and agreed for the withdrawal but then we should not make too much noise of it, you know how it goes. But I think really, I don't know how helpful it is to go via that pathway, but I think the bigger the universities or the centres maybe you can threaten them in one way or another? I don't know what your ideas are about that.

Florian Naudet: Yeah so, I would say that you contact the president of Stanford because as you know they are actually under investigation of some of his work. But no it's quite difficult. Hopefully, things will change, also because of the COVID-19, people are getting more and more aware of those problems. You know in France we have to tell an institute of research that have that very problematic behaviours regarding publication, and even in a recent journal investigation on the TV show, the new board member of the centre recognised that those studies were a problem so hopefully things will change.

Francois Maignen: No I doubt it.

Maisonneuve H: No. I just wanted to add 2 comments. You have to consider that for artemisia it's not considered as a drug, it's a tea and they were also trying to sell the tea over the net, so they also have a website, and they were selling their tea which is not a drug, so they didn't have to go through drug agencies. It was one very important factor because they wanted to make money, but I assume they didn’t do a lot of money, but they sold quite a lot. 

Florian Naudet: And also the fact that we saw something regarding the communication on the Internet. So at the beginning they were stating to French people who were traveling abroad to use the tea for prevention of malaria and now they don't recommend this because they don't want to have problems with the French authorities. Now they say, put the problem on the countries but not on France, so from the French perspective they are not very interesting to investigate I would say.

Maisonneuve H: Yes, exactly. Just one comment maybe out of the scope. I was a research integrity officer in France for 3 years, no more, and I confirm that the system doesn't really work in France. We have a lot of talks, we have a lot of presentation, but we'd never had a guy that was really condemned, for example, to reimburse money or their grant. We have no sanction at all. We have a full system, but nobody has been accused and condemned. So I am very nasty about this French system, which is officially fine, but they are probably protecting the Institution more than the guy.

David Colquhoun:  Yes, did they say anywhere, how much artemisia is in their tea because if it's a suboptimal dose it just spreads resistance, it’s ghastly, and it probably was.

Florian Naudet: Yeah, this is a problem and actually artemisia is not very soluble in water. So if you put the leaf in the water, there is not much artemisinin in the water. So this is a big problem and that's why the WHO does not recommend the use of this kind of tea infusion but in their narrative, in the movie, they said that the WHO was against because WHO was protecting big pharma, and this is the kind of narrative they add.

David Colquhoun:  Well, phytochemistry is a quack journal anyway, isn't it? Largely.

Francois Maignen: I wanted to, just because you mentioned some possible solutions and you touched on the weaknesses of peer review, and the fact that you can basically publish any paper you want in bad journals with very poor peer review. But I also wanted to raise the point of the issues of health authorities. You know that health authorities were an important weapon to try to prevent or detect inspections. I also wanted to elaborate on what is for me the critical point made by your theory which are sanctions. In the case of a MMR and Andrew Wakefield when he published his fraudulent paper, I think it was in the Lancet and when the fraud was discovered, Wakefield cannot work in England anymore. So what do you believe are the solutions to try to minimize or prevent the accounts of these zombie trials and do you think that health authorities in a sense are weak and guilty of not doing enough to try and prevent this conduct   I thought it is no sense.

Florian Naudet: I think it's difficult to answer on what could help to prevent the problem. Regarding health authorities I think there is something very interesting, but health authorities are responsible for looking at the evidence. Judging if this is sufficient evidence to provide the market approval, but this process is disconnected from the process of publication. So first, it could be very interesting to reconnect them both. So we proposed what we called the registered drug approval, based on the model of registered report. It may help. Not especially regarding this kind of fraud although it could be interesting, but it may help to reduce some research outcomes like outcome switching, like selective outcome reporting, and publication bias. So this could be very interesting to address, but in order to prevent the phenomena, I don't know if it's very preventable so far. At least we can have a more appropriate way to ending the problems. So usually, as I told you, it's very long to have a reaction for correction, so with the things that we saw in the case of artemisia, it should have had an expression of concern within 2 days, and then a deep investigation of the editor, so this could be something. Then we had a lot of problems with this editor in phytomedicine, but the editor is still in place, is still working, even if the publisher disagreed with him, and asked him to retract the paper. So why is he saying it's still working in this journal, this is also the responsibility of the publisher, but I think there is a lot of things to do to correct the program, to prevent it.