Georg Rüschemeyer: I work at Cochrane, Germany, here in Friedberg, and before that I worked as a freelance science journalist for about 18 years nearly 2 decades of freelancing. Today we're meeting here in a small round to discuss the question of sanctions for failing to register or report clinical trial data. We did a strictly randomized selection of what order we go in, and the first one who came up here is Gustav.

Gustav Nilsonne: I'm a neuroscientist and a Meta scientist based in Sweden. Some of the research I do is on following up clinical trials reporting mainly in the Nordic countries. I was thinking about how to frame this issue, and one thing I thought of is a kind of clash between 2 systems or perspectives, and I pulled out of my bookcase this guide. The young man's path to Minerva by Ragnar Granit, one of the Nobel laureates, neuroscientist from the last century. He compares life as a scientist to life as an artist. He describes being a scientist in a way that has very many similarities to the creative process of an artist. You think you come up with new ideas, you test them, you proceed according to your imagination and your inspiration, and you create something new. And this is something. I believe many researchers are happy to experience from time to time and is maybe one of the motivating reasons for choosing an academic career, and this is encapsulated sometimes in the concept of academic freedom. We have the freedom of scientists to choose which questions to investigate, but also to choose how to report our results, in which context, in which forum, in which way, however, there is also, of course, the regulatory perspective, which is extremely important, because, as we know, clinical trials, data have enormous impact on medical practice. It's a matter of literally of life and death for patients that results from clinical trials are accurately and promptly reported. But our systems for conducting science are still in many ways designed for the lone genius who proceeds according to his or her imagination, and the inspiration. Personally, what I favour in order to improve quality in science is to make better incentives for scientists provide better support and increase the institutional responsibility. So, for example, if I run a clinical trial, I believe that my university or hospital should not delegate the entire responsibility for me. There should be mechanisms in place, such that if I die, or if I move to another country, where, if I get a new job in industry, the institution needs to be there and step up and ensure that the results, are properly kept and analysed and reported, and of course they also need to ensure that that happens if I still work there. So in general, I believe we will be well suited by finding ways to improve researchers means to do the right thing, but hopefully, without sanctioning individuals. Sanctioning institutions I would be much more favourable to, and that's my attempt at starting off this discussion.

Birgit Whitman:  I'm head of research governance and integrity, at the University of Birmingham. I've been in this role here in the University for over 4 years, and in Bristol for over 11 years, and before that have worked in the NHS for about 20 years. So I suppose I've been in this game for quite a long time, and that's where I where I wanted to start in as much that I think over the time I've been working with this we've seen a very positive evolution on trial reporting. So when I first started, I think there weren't many researchers and many administrator or professional services. People aware of the reporting needs, and that over time has changed. So we've have seen through awareness, raising and through, supporting, through developing institutional processes that that we can actually get the message out there that this is an important part. So it's not just, you know, you have the idea, you get the funding, you conduct the study, and you're done. But actually that that trial reporting is an essential element, and you need to think about that from the moment you apply for funding to make sure you've got the right funding to actually upload the results at the end and you've got the right statistical support, and so on. So we've seen that changing, and I have seen that changing, and I have been part of that change. Taking an institution from being branded by our lobbyist as totally non-conformant to a hundred percent trial registration, and that has been a really positive journey. It has been a positive journey to foster a really open research culture, and to provide the right support for the researchers, for the team to upscale the teams I've been leading to put the resources into the teams to do that. So for me sanctions wouldn't have achieved this at all. It would have just increased the bureaucracy and a culture that I really don't want to foster in this, and so it is my view that sanction would achieve the opposite from the culture I want to see coming into the community and developing further into community. We're seeing brilliant moves into the open research culture and that's really developing beautifully. It is enhancing reproducibility. We see that through those sorts of movements. And then finally, I think, having the results out there is missing an important point. It's making these results accessible for our participants in the trials. So just because the results are on some registry doesn't actually mean that the participants in our trials will actually understand those. I would much rather put my energy into that. So in the UK we have to health research authority which has a Make it public strategy. And I think that's a really fundamental part of getting our reporting, right? So yes, of course, the trial has to be reported. We have to have the data open in some open research forum. But we need to make sure that our participants are able to understand what we're putting out there, and that they are informed and included in the right way. So for me, sanctions in any shape or form, takes us into a direction where we would put energy and resource into something that will not achieve where I think the research community needs to go.

Georg Rüschemeyer: It's an interesting perspective that's that I haven't probably haven’t put enough thought into the participants’ perspective.

Till Bruckner: I agree with Gustav and with Birgit in 2 important ways. I agree with this Gustav that I'm strongly in favour of sanctions on institutions, but I don't think that sanctions on individuals' are appropriate. Why not? Because if your principal investigator, working at Birgit’s institutions, the results will be made public on time, because Birgit has created those systems and those processes that ensure that results are made public. So chasing off the 100,000 individuals in the medical research communities is far less efficient than you know, making sure that institutions do what they're meant to be doing. I also very much agree with Birgit, I did a study, maybe a year or 2 ago, of people working in preclinical animal research. That is an area where one of the interviewees said to me, If you're doing this kind of work in this particular country, you always have the feeling that you've got one foot in jail already. Because It's a very confrontational and toxic relationship with the regulators in part. It makes it incredibly difficult within the community to openly discuss problems. It creates a sort of spirit that is really contrary to everything that we want to see in research. So I'm in favour of sanctions, but I think that sanctions should be like a last bottom line. A last resort, when all else fails. Why do I think that sanctions are appropriate? First of all, it's normal for an institution to face a sanction if it breaks the rules. So right now you might have a cancer patient parking outside hospital to participate in the clinical trial. If she parks a car in the wrong place, she will get a parking ticket and she has to pay that parking ticket. But if the hospital doesn't make the clinical trial result public, as Gustav said, endangering many, many lives, the hospital faces no consequences whatsoever. Is that fair? No. Is that appropriate? No. So I think for that reason sanctions are appropriate. The second point is that voluntary compliance has a history of failure, of delivering 100%. I think realistically, we can never get to 100% of clinical trials registered and reported, especially reported If there aren't any sanctions in place. In the UK there's been like a really strong efforts to improve clinical trial reporting, they've been hugely successful. Not only positive outlier institutions like the University of Birmingham, but across the board, but we're still far from 100%  and closing that lost 1% gap, I think, will always require sanctions. And third, I think we need to remember that we're working in a community where we're dealing with 99% or more people who are good actors, you know, people go into medical research because they want to help other people. People want to do the right thing. There’s so much good will that I've encountered over the past few years, but there are a few bad actors out there. So if you've got a pharmaceutical company that run a phase 4 trial on a product that is already on the market, and they discovered adverse effects and they discovered safety signals during that trial. They discovered a lack of efficacy, and they refused to make those trials as public. I think there needs to be a legal mechanism to force that company to make the trial public, to keep people safe, to keep patients safe. It should be targeted institutions, not at individuals, and it should be a last resort measure. So there should always be 1, 2 warning emails, warning letters, giving people the chance to please do the right thing before you start sanctioning.

Sarah White:  I'm the executive director of the multi-regional Clinical Trial Centre at Brigham Women's Hospital in Harvard, here in Boston. The MRCT Centre is essentially a think-do tank. A research and policy centre looking at challenges of global clinical trials. That's my day job. I also play this role as the co-chair of a US based task force that is very much focused on academic, university, hospitals, in the US and the people that are responsible for registration and results reporting at that institution. It is a very large group. We are now at about 400 members. We get about a 170 people per month on task force calls. So there is this huge momentum for a need of conversation and help in the academic community all at these institutions for doing the right thing. The group is mostly focused on FDA or the Federally regulations for registration and results reporting MIH, ICMJE, the journal requirements, and then a small sprinkle of veterans’ affairs, Medicare, and Medicaid and a few other kinds of specific requirements. As you know, especially in the US we've got this massive infrastructure of research. A massive number of the trials registered on clinical trials.gov are at academic medical centres, universities, and hospitals. We don't in general have a good reputation for doing a great job. There have been potential compliance actions going way back, but mechanisms really being put in place since 2017, that the FDA could have taken action. The Federal Government could have collected a lot of money based on this, and they haven't to date. As you probably know, there have been some recent tiptoeing in this area with a series of pre notices to specific pharmaceutical companies or investigators that they are in compliance. It has led to their individual compliance, but not to any sort of enforcement. On the NIH grant side there has not been any sort of withholding that we know about publicly of that federal grant. I want to mention very quickly the ICMJE requirements. Their requirements of you must register before the first patient is enrolled, that has been tremendously effective, at least in the US. And I happen to be previously to my position now, I was at Mass general in the Brigham, overseeing the central registration and results reporting there, which you know about 4,000 studies, under kind of my direct report. There was a moment in time where ICMJE said, all right, we're ready and started, really enforcing. And we were getting phone calls from investigators saying they won't even look at my paper and it's like, well you didn't comply and so that at least in the US really enabled compliance. That if an investigator was really serious about doing this, they were going to register. I think when I think about consequences or compliance and enforcement, I do think there's a place for it. I think about the FDA regulated studies, drugs, devices, biologics. These are pivotal studies that are the basis for approval, and then obviously going into the general population, I think the public deserves to see the results of those studies. I think the public deserves to see what are massive National Institute of Health funding is paying for, and I want to go back to Gustav’s mention of honouring the patient. We do deserve to honour the patient by being transparent about what happened in that research. So I definitely see a place for compliance and sanctions. What I feel very strongly about, and this really plays on what I have kind of done through my career is you got the lawmakers, you got the regulators, and then you have the investigators like this Gustav, Birgit, who are doing the work and there's a huge disconnect to you shall do and how do I do? And I think if sanctions and compliance are going to be involved, there has to be a meeting of the minds of okay, this is what we want to do. These are the mechanisms. What are the resources that our Federal Government needs to actually follow through on any sort of compliance? What are the resources at the institutions needed to support the that the individual investigators, you know, for example, when I was at Mass general on the Brigham,  4000 studies, I had to fight for more than one FTE to oversee that, that is not right. When you see that across the board the whole idea of registration and results reporting at these institutions is not well funded and not well recognized, and part of that is because we've got these sanctions and nothing's happening. I guarantee if FDA started actually following through with it, that there would be more funding. So again I think my position is, that enforcement and potential sanctions are important, but I think we have a long way to go at least in the US and I would hope that if Europe moves towards this, there's a lot of critical thought about how do you make them transparent and actionable for the research community.

Georg Rüschemeyer: We are working with Cochrane Germany at the moment on some advocacy action to get some regulation forward in Germany that will make sure that more trial reporting is happening. That's why it is very interesting for me to be in this discussion, even though my background isn't really in this area. In the US you said that the FDA hasn't issue defines that they could be issuing. Is that because of a lack of interest or is it a conscious decisions in the sense of well this is the whip that we could get out if you don't comply with. We are hoping if we give you a few carrots here, there will be more compliance without making that necessary.

Sarah White: That's a fascinating question and I'm not the FDA and Till knows better than me that the FDA is so guarded about what they say and what they don't say. But with that said, Rob Califf, the FDA Commissioner, he's really dedicated to clinical trials transparency. He's got a lot of other fish, the FDA has a lot of fish to fry, and whether this is something that trickles up to the top or not is questionable. We are very lucky on the task force that we have a lot of direct conversation, not only the clinical trials.Gov, but with representatives in the FDA that are responsible for this, and again very guarded about what they say but we understand there are mechanisms there now, but there's resource constraints, and so I think I think it goes back to what I was saying before. There are good intentions. There's legal basis. There are mechanisms now through the USFDA's Bio Research monitoring program that can do this. I think there's a resource issue. I think the other incredibly important thing which this doesn't necessarily kind of excuse anything. What we see publicly on the clinical trials.Gov website is not always necessarily the truth and when the FDA does go into kind of inspect, they look at a study, they collect all the other information in addition to what's on the public website before moving forward, which is pretty time consuming and resource intensive. We don't really know what comes of what they see publicly to the hundreds of thousands of studies that are late for results and kind of what it actually means behind that.

Georg Rüschemeyer: So mostly resources, which is probably often the problem. So let me quickly try to sum up, I think we heard 3 people who were in favour of sanctions, but more on the institutional level, than on the individual level. Was there anyone who wanted sanctions on the individual level I don't think so. Then Birgit was a bit more sceptical about sanctions in general saying that in her experience it from Birmingham. The report was of the scientist was very good, and it was actually possible to get the registration numbers up from basically 0 to close to 100% within just a few years. That was quite recent developments?

Birgit Whitman: I had luckily the experience of another institution of doing it but within a year we manage to get them 100%.

Sarah White: How many central resources do you all have to help you with that?

Birgit Whitman: Very little. So I work in quite a large university, and we have about at any one time 105-110 interventional trials all over the world. So we have them not just in in the UK. We have about 43 countries. We run our interventional trials, it's a very large international trial portfolio, and I have in the governance team 5 people, and that is how I move it forward, and then I have clinical trial units, who we work with, and then obviously, I've got people in the clinical trial units but they're not central governance teams. They're the people who support the researchers.

Sarah White: When you say governance, does that mean 5 people that are responsible for doing the result?

Birgit Whitman: For the whole sponsorship review process, so the studies will come to us. We review for sponsorship; we give comments back before they go to the regulators. Then, when they come back from the regulators, we look at what responses need to be made, and then we follow it through with the conduct of the study. Obviously the CTUs will support the conduct of the studies if they're funded for that. So we haven't got many, and I had to literally when I came here, I had to adjust a job description and upscale a member of the team to actually have that person who could work with the researchers who were struggling to upload to help them upload.

Georg Rüschemeyer: From your experience is it easy enough now to register studies and to register the results of the studies. I've talked to a couple of scientists in Germany who said, yeah, that's all sounds really good in theory. But that when you try to register things, then often it's quite difficult, and the websites don't work the way they're supposed to, it's just a thing you really don't want to do late in the evenings after your regular work time. So I think that's always been an important point, make it easy, so it's easy to comply. So basically that could be the carrot. That will make it easier for people to comply. Is that in the different countries you come from, is it good enough by now? Or I mean I guess you could always make it better.

Birgit Whitman: So from my point of view, I think it's pretty much thanks to Till, and the lobbying that we've seen. We've actually seen a lot of improvement in the way the registers work. They're not ideal yet, and routinely our studies could be registered on 3 registries. Sometimes they're registering on more than one. If it's US funded, for example, or NIH is involved you then have a double registry entry that you have to manage. But clinicaltrials.gov has always been a really good registry to work with. We could tidy up the register, so if I run a report, and I find that a study is not a study which is supported by my institution, I could actually get in touch with US colleagues, and they would actually sort this out for me relatively quickly, but other registries have been really problematic, especially the EU ones, and Till has been fundamental in helping us, through lobbying, to get the right information on how we should manage it and best practice approaches where we all could pull our resources to get this sorted that has helped. For us what of course, is problematic is Brexit. So from the UK, where we thought we were actually getting there, right now we are having to manage the Brexit element within that setting. So at the moment we actually are running dual reporting in just about everything. That's added another layer of complexity to the trial reporting. So I think we are still a long way away. What I would love is one entry to a registry, and then that populates wherever it needs to populate. That would be fantastic. I think we are very long way away from it. ISRCTN, a UK registry are making really good progress in the UK on this, and they're becoming the registry of choice, but we still have a little bit of a cost to it so we're permanently battling with studies that are not funded to a high level. Are they going to go on, clinictrials.gov, or are they going to go on ISRCTN? I discourage clinicaltrials.gov but of course there's a registration fee so there's a lot of a lot of discussion of where's the right registration and then who's going to maintain that registration if they do that, so the registries are getting easier but we're not there yet.

Gustav Nilsonne: So I do agree that there's a scope for improvement on the technical side to make things easier, to have better interfaces and so on, but perhaps more importantly, I believe that we need to make improvements to the organization of how the reporting is conducted. I don't believe that it should be delegated all the way down to the individual researcher who is responsible for a trial. I believe we need specialist support staff whose job it is to make sure that trials are properly registered. You can imagine different ways this could happen. It could be a centralized service within a university where the researcher can send in their protocol and materials and the support person who knows intimately how the registry works can then translate that into a registration, or perhaps even better, it could be delegated to some kind of external body at the national level, or at least above the university, whose job it would be not merely to register the trials, but also to be an independent party following up different sponsors’ registration and reporting, and to provide overall statistics. And this, ties back a bit, too the question of sanctions. One step that I do believe should be taken before sanctions is to publish openly data on reporting by the different trial sponsors. I'm sure you're aware of the dashboard built by the Quest Centre that compares university medical centres in Germany and their trial reporting and 1 point I’d like to make is that that ought to be done routinely by a responsible agency and not as a research project. I hope I didn't take us too far out on the tangent now.

Sarah White: Gustav, can I ask a question about what you were saying? When you talk about the institutional support to put the data into the registry, and also the statistics, who pays for that? Does the investigator pay for it per study or does the institution fund those resources in a perfect world?

Gustav Nilsonne: At the moment I can speak to what we have at my university. We now have approximately one full time equivalent of a person who is funded by general university overhead costs, so at no direct costs to the investigator. They're supporting new registrations and they're also trying to work through the backlog of studies that are not reported. That was not very easy to get, and I can join the chorus and say that it was largely thanks to Till's lobbying, the data he put out that we managed to get that.

Sarah White: We have a variety of models of the academic institutions. In some cases the investigators are completely responsible. The institution delegates the responsibility. There are central resources that can help advise, but not responsible for registration or results reporting data entry. Then there's some other models where there is a central resource that can help some investigators with data entering results reporting and then there's a third model where it's actually pay per service and I think a lot of that depends on how the institution spends their institutional funds, and I think it's an interesting model. But getting back to the original question. Clinical trials.gov is ever so slightly different than some of the other registries where you are basically creating rows and columns, tables to really capture the study design and the results reporting. So it's very, very much for the academic institutions a manual process. Big sponsors. They’ve got some API which is coming from their clinical trial management system, plugging it into there. My experience with academic institutions is that it's not that, it's purely manual. I think over my time doing it, being in the database and helping investigators, pretty easy. You get used to it, but if you're doing it once a year, it is definitely not easy, and there are some references that talk about registration taking about 20h and results reporting taking about 40 h. That’s a lot of time. I think it raises the question of is 40 h easy or not. It's someone probably doing it at midnight.

Till Bruckner: I just wanted to sort of curve back onto sanctions here. The huge amount of time lost because registries are inefficient because the system is really unhelpful. I think we can all agree on that. If we're talking about 40 h to report a clinical trial or something, I mean, that is far too much but then I think you also have to ask well there's 40 h, how do they compare to all the other time that went into the clinical trial? I mean from developing the protocol to the taxpayer regulated time of reviewing that, to the institutional managed time, to the patient's time, because we shouldn't forget you know, people might drive 1, 2 h in the US, probably 4 h to get to clinical trial site. So it's all that time that has gone into there and we're really talking about a fraction of that. So I think if you're looking at it from an institutional perspective, it's absolutely horrible. If you're looking at it from my comfortable, I don't have to deal with this far away perspective, it's only a tiny fraction of the resources. To bring this back to sanctions I just want to give Birgit a hard time, by just throwing you 2 examples of where I think that sanctions would be appropriate in a perfect world. These are both 2 examples that I've sort of been involved with. So the first one was the MBAnderson cancer centre in the US which have got brilliant record on legal compliance. On putting the results up there when it's legally required. But 16 years ago they run a trial involving patients with brain cancer. They never made the results public. The German health Technology Assessment Agency requested the results of those trials off them so that it could make an assessment. Anderson said, No, we got the data, we prepared it for publication, but we couldn't find a journal to publish it, and we're not giving you the data because we think it would be irresponsible to share non peer-reviewed data which is a weird thing to say to health technology assessment. So I tried to put a bit of pressure on MDanderson, and I said just upload the results on clinicaltrials.gov, that's exactly what it's there for, that's the function of the trial registry, you don't have to wait for the journal. They didn’t do that. I filed an ethics complaint. Didn’t hear from them for a long time. 2 years later, I asked what happened with the ethics complaint. I got back this email saying we looked at the case and we concluded that no rules were broken, and everything is on it already essentially and blog forthcoming, and at that point, the results are still not up on clinicaltrials.gov. The German health technology assessment still cannot assess this treatment so German patients with brain cancer might be getting suboptimal treatment because they can't assess that. I think at this point, 16 years later, yes, sanctions would be appropriate a second example is a medical device company called Cutera based in the US who've actually been breaking the law by not uploading trial results. I highlighted that with them several times email twitter on blogs. For about 2 or 3 years we saw absolutely no movement by Cutera to upload any clinical trial results, and only recently maybe 6 months ago they started to upload some results. So I think right now 3 of their 22 clinical trials have results up now. I've got no idea what happened internally there. Maybe they got an FDA letter, a pre notice of noncompliance, and that sort of focused their minds. Maybe they had to change in staff and just decided that it was a time to do this. After 3, 4 years of raising the issue very publicly, you know that company still hasn't put all its results up. That is the kind of hard case that I'm talking about so I just like to have Birgit’s perspective on, if we don't have sanctions in context, you know what is the alternative here.

Birgit Whitman: For me it's a culture thing, it's a culture that we want to foster. That too is especially those people who dig in. Now it's very difficult for me to comment. I don't know what's happened there, but if I have people who don't engage, then I need to find a way off of engaging them. So when I first came here there were some trials which just couldn't upload results, because there had been an inspection, It was found that the data should not be used, and we cleared that with the regulator. Now it might be something like this. There is a there is a story behind the scenes that cannot go in the public domain, and therefore the results are not uploaded. So there are some circumstances sometimes, where results are not uploaded, because it wouldn't be appropriate to upload results that have come into question through some shape or form. It might do that. If it is a resource thing clearly that should come out. Obviously, we have those discussions so when we couldn't upload results because we just couldn't get it sorted on the registry, if it's that, then obviously we need to iron these out. So there will be some hurdle, so what I would usually expect is that a researcher would want to get the results out there, so if it's one of our researchers, clearly they are assessed by the research output, and it would be very unusual for researcher not to publish the results. In a Higher education institution that would be very unusual. I don't know if this was in a hospital or not. If it's in a hospital and take our NHS hospital, for example. At the moment we have a totally overloaded workforce post Covid, and I expect that's the same maybe in other places, so has there just not been the time. So I would just want to understand what is behind this scene happening which you would normally expect, somebody should be proud of that study, and if they're not proud of that study, then there's a reason why they're not. There's either regulatory reason, or they just don’t have enough time to write it up or to too much pressure or not enough support, not enough technical knowledge. So if you think this is 16 years ago, I think that the stuff is outdated anyway so we might as well start again, because things, no doubt, would have changed in treatment, and whatever we're doing, so I think 16 years later you might as well forget it. So this has to happen. Contemporaneously, I think, because research moves on so fast that we certainly wouldn’t want some outdated stuff out there which was set out 16 years ago. For the company I feel the same. I haven't worked with companies, I've never worked in the commercial sector, so I’m not sure but the company normally would be in it for money I would have thought, to make money out of out of those things, so have they not put it into regulatory approval processes to put the product on the market? So that would be my question to them of how did they get licensed, or whatever for the stuff if they haven't put the results somewhere?

Till Bruckner: Well it could be, for example, post approval studies. So in many cases there are studies done where there's absolutely no regulatory obligation to run the study and I think from a commercial perspective, it's completely understandable that if the results don't show what you hope that they were going to show then you're not going to post them all over the internet. I think that's a very clear commercial incentive, and I'm not even going to go and call companies evil for doing that, because I think that's just the market logic behind these things. But I do think that there should be some regulatory safeguards and a red line that's on point where it's just like, where actually its, sorry, you involve patients in this. Patients volunteer to participate in this. Whatever intervention you're testing is actually on you were hoping to generate some scientific benefit otherwise this wouldn't have gotten ethics approval, so I do think it's appropriate to have a red line there, but for individual clinical trial, sometimes the results have to be written off. I've seen that on the European registry, and I agree with that.  I just submitted a pre-print yesterday that argued the same thing. Sometimes there needs to be a pathway to write off individual studies. I just think that at some point as a very last result you just need that as Sarah said. Just for the regulator to have that resort, can really help to convince institutions to invest in building those kinds of systems. For me the question is when I'm looking at this the US model seems to be like “okay we can fine you up to $13,000 per day while the trial is overdue”. Now we gasp and say, that's a lot of money. If I was running a pharmaceutical company that was putting out a blockbuster drug with a turnover of, say, 6 billion dollars a year for that single drug, and there was like a single unflattering trial results I’d definitely take the $13,000 a day for the results not to be made public. So I think that might not be the right way. Then you're looking like at the European approach where they're saying,  “we're going to give people like a small one off fine”. I think that's good at concentrating minds in non-commercial sponsors. But again commercial sponsors that doesn't really work. But the Europeans have the back fall of yes, we can actually put them into prison in extreme cases, and then you have got the UK one where they say, well we're just not going to approve new trials, which I think is a great approach If the results are missing, because you can say, you've got the results, you put them up there, and everything's fine again, and life can go on as normal. But if somebody failed to register clinical trial, in a way the damage is already done. So how are you going to hold approvals? How long are you going to hold them for? It's this really for the benefit of patients. So I think there needs to be a lot of thinking about how to do it but I do think there should be a red line.

Georg Rüschemeyer: So what's your personal favourite in terms of what kind of sanctions would be the most appropriate? It just sounded like, maybe the stopping any further research until you've finished up the last one is that it?

Till Bruckner: For reporting results I think that's a good approach, unless of course you are dealing with a company that's just got one product on the market and that doesn't care because they're not starting up anymore clinical trials. What I'm really in favour for is having great discussions like we're having now before somebody goes and writes an ill-informed law as we've seen all over North America. I really think there's benefit in having these discussions and we need a lot more discussions about this.

Gustav Nilsonne: I just wanted to respond briefly to catch on to one of the Birgit said. I do agree very much with the general idea Birgit of fostering a culture and building normative expectations among academic researchers. But I thought your statement was maybe more charitable towards us academic researchers than we deserve. Don't you think there's a lot of publication bias going on where researchers just don't get the results that they hope for and then it ends up in the file drawer, and we move on to something else.

Birgit Whitman: I'm aware that happens of course, but that is coming back to what culture are we fostering? Are we actually fostering a culture where we say, whatever you do, good, positive, or negative result. It's going to be valuable and you're going to be recognized for that, even if you're  amazing study suddenly doesn't show that what you had hoped for or you, for whatever reason, didn't recruit enough people, and you can't show this. But If I had a culture which says to these people, Look wherever you get it, you have to get it out there, wherever you're going to put it. So I came through the hospital ranks, worked in surgery, and I was really lucky to work with a surgeon who whatever we did, and honestly my stuff really wasn't shattering, he made me publish. He said whatever it is, and if we don't get it into this journal, we tried probably 10 others. Eventually we got it in some non-descript journal, but it was out there, and I was working with someone who basically said, you're on a learning curve. This is earth-shattering, but we need to put it out. You need to publish, and that is what we should foster as a culture. It doesn't have to be absolutely Nobel Prize stuff for it to be published. It should just be out there in the public domain, and we should help people do that. In the UK obviously we have the research assessment via the publications. As you know, all our researchers are assessed every 5 years and graded into 4 Star 5 Star papers, their writing and stuff. So that is fostering a very specific culture. And we have to, as a nation, ask ourselves, is that really what we want to do? Is that really what's going to bring out the best ideas? And are these people then publishing? Because they will put it in the draw and publish something else which is probably going to be higher rated.  If we do the research, we should get it out there, because it will help us, just to start somewhere different. If mine didn't work out, then actually, someone else will be able to say, okay, well, that didn't work so I'll try something else. Providing it is reliable, and we have an open approach to our research so now we have a very open research agenda. We are encouraging people to share their results, even though they're published, they can still put the results out even in an institutional registry. So the stuff is out there. I just think we need to come away from this sort of cut throat approach and actually get people involved and hopefully enjoy the research they do and have that learning curve.

Sarah White: Birgit, you should write up a case study on the University of Birmingham, and how you have fostered that culture of compliance. I mean, I think that could be a neat contribution to the community. I think back of my time when I was overseeing the Central Registration Results Reporting program and there were these conversations about if we get fined for this study, that is out of compliance, even if its an NIH study, the grant recipient who is the institution is the person that is sanctioned. And the conversation would be okay. Well, the fine comes to the institution. The  institution is like, I'm not paying for it. Department can pay for it. Department sheep. I'm not paying for it. The investigator can pay for it, and so there's this kind of passing off, which goes completely against, Birgit, what you're saying, but I don't think that's uncommon, and I think when we talk about a culture of compliance Birgit to your point. Kind of really appreciating not only the positive results, but the negative results because we all learn from negative research results as well, and also drawing on a publication that Deborah Zarin, who was the director of clinical Trials.gov. When she left clinical trials.gov, she came to the MRCT Centre to pursue her research interests. She published a paper about noninformative trials, and thinking about how many trials do we have out there that are recruiting, enrolling patients that are actually not going to inform anything. And so if we think about clearing the clutter. Really having informative trials that can lead to some sort of change, hopefully, they get positive results. Maybe they don't, but that's all still important. That leads Birgit that culture of compliance that you're talking about. But I think you should write up a case study.

Birgit Whitman: I did a case study before for you once Till didn't I, about Bristol. So maybe I'll do another one. I actually share your point. So that's the other thing, I look after the PGR community here, so the postgraduate community and obviously you can have an Msc study and in theory if you apply WHO guidelines that should register but is it really the right thing that we register that and go through all of that and actually, it's an Msc study. It's a one year research project. It's highly unlikely to. It's really a learning curve for research rather than actually about the research results itself. So we need to be quite realistic of what we want out there to clutter everything up. It's already cluttered enough. You can't see the wood before the trees as some English people would say.

Gustav Nilsonne: There's one thing I wanted to say, while we still have a minute or 3. Clinical trials research is just the beginning. Other research, especially on humans, but also on animals and more broadly, has learned a lot from the registration mandates in clinical trials research. It took many years for the ICMJE recommendations to really come through the system and become fully accepted and enforced. That started somewhere around 20 years ago. From where I'm sitting it seems that pre-registration of other kinds of experimental research has really taken off in the last 5 years, approximately, and I think this debate about clinical trials is going to shape also how we work with other kinds of research designs. Maybe animal research as the first next step, but also other basic research experiments on humans and so on, which is exciting. Perhaps I'll take the opportunity also to say that I'm coming out of this debate slightly less convinced that sanctions are a good idea. I do believe they have a place, but I want to go back to Georg's horse riding allusion. Sometimes it's enough that you're holding a whip, and the horse knows it.

Georg Rüschemeyer: It's just the idea of the whip that we need, you don't actually want to use it.