• Eric Low
• Dr. Dan Ollendorf
• Dr. Wija Oortwijn

Guest Experts: Dr. Regina Skavron and Nicholas Hedberg

Leeza Osipenko: Introduction and questions to the panel 

If you look at any academic literature on HTA, the vast majority of publications focus exactly on processes and methods, and quite a few political issues are also geared towards dialogues about methods. Are we doing things right? This might sound like a boring part of the story, but this is a pivotal part because that's how we make decisions and create transparency and validity of those decisions in the end. We do have a lot of variation in processes and methods, no one doubts that, and there are reasons for it. I highlighted here five different reasons. Here are the key ones. The economic factor, for example there is a different ability to pay within the country, there are different levels of resources, there is different GDP per capita within the country, and the size of the country. Another factor is skills. This also connects to the size of the country. For example, you might have a wealthy country with very high GDP per capita, and you will be a small enough country to not necessitate full capacity for HTA assessment. Or you might have a very populous country which definitely would have benefited from HTA, but they don't have other resources to implement the system, and they don't have particular skills. So, for example, the UK historically developed schools where quite a lot of health economists have been trained in delivering this skill pool to the market. Other countries did not have that opportunity and did not develop enough staff to conduct this work. There are differences in the healthcare systems. Some systems are private, others have universal health coverage. There are many mixed systems, and the way the healthcare is organized and paid for regionally or nationally really varies across Europe and across the world. There are social factors like cultural factors, value judgements, how do we make decisions, what is of value, how much does a human life cost? Each society may have different perspectives on that and there are legal constraints. Systems with governments and constitutions that drive certain laws within the country, which enable or disable particular processes. If we look at methods, there's quite a lot of similarity across HTA agencies, especially when it comes to demonstration of clinical benefit. I think representatives from the industry would agree on that because any HTA agency you talk to want the same endpoints. They have a similar critique of clinical trial designs. They have similar questions on populations. Methods continue to be questioned and criticised. They are criticised in different ways. For example, since the moment of the appearance of quality it has been criticized, whether it's the right and fair measure. Articles keep coming up with regular frequency. The big question now for the new types of medicines, which are extremely expensive, such as gene therapies or personalised medicines which are made at the bedside. Should they be valued using the same methods? Answers keep coming up that maybe they shouldn't. When we have new methodologies trying to assess these therapies differently, how logical are these changes? Are they arbitrary? Are they legitimate? Are they grounded in theory or common sense? Or are they made up artificially? A lot of questions with methodology start coming up because of the changes of the landscape. In terms of processes, I think it would be fair to say that all HTA agencies improved over the last 20 years. I think we started making decisions faster. Faster to the time of marketing authorisation. There is some harmonisation, but the frustration is that each agency still has their own processes and there is no common template. It’s a lot of work, a lot of work for the applicants, a lot of work for the stakeholders, for the evaluators to go through many agencies or to go through many decision-making points. EUnetHTA was a great start to try to enable this harmonisation. The problem with EUnetHTA is that it was voluntary, and some successes were achieved. How sustainable they were was the question, and fortunately a very important step has taken place a few months ago, and we got a European Regulation on Health Technology Assessment, which will be implemented in 2025, but the work starts now. The whole idea of it is to harmonise processes, to make the workload less, to make decision making more uniform and more transparent, to avoid duplication of work, and to bring clarity to the procedures. It is a fantastic move and it'll take time. Only time can tell us what will happen. Let’s not forget the UK exited the European Union. It might not even look in this direction and continue with its own processes. There's a lot of HTA activity in in Canada and the United States. HTA activities are growing in low middle income countries. There is a lot of other HTA methods and processes that will not fall under this regulation, still making the world of HTA quite complicated. To kick off discussion. I'd like to ask the panel how much the difference in methodological approaches and processes contributes to a divergence in HTA recommendations across countries? Are we being inefficient, redundant, and bureaucratic by having different methodologies and processes in every country? Is the new regulation in Europe a solution? 

Nicholas Hedberg: From January 2025 we know that there will be common legislation in place. What you described as the voluntary work in EUnetHTA is now much less voluntary, and when you pointed out the legal factors, for example, all of us are aware that we can work together in EUnetHTA, but we must also respect our national legal boundaries. Now, the task from the commission, the council, and the parliament is to identify the challenges or the hurdles in your national legislation and use these three years to change them, when it comes to joint HTA within the scope that is then demanded by or set up a new regulation. I'm approaching your questions now slowly, but of course, we see that the economic part of HTA is very differently important. We will have to realise that the joint part is, for some of us, only the first half or even smaller than half, because that's before we start talking about costs and healthcare payment models and what we might know on the national level put in HTA, like we do in Sweden, for example. I would also like to add to your list of factors, I think expectations that we build up from the economic status we have in the country, the healthcare system we are used to, thee social factors we have with us from birth or being brought up will also give us different expectations for what the healthcare system is about. But once again to your questions. The difference in methods - we still see them. They come because we started on 28 different places or even more because in many member states this has been a regional business. Now we are merging from then 27 into 1, given the first half and given the relative effectiveness. So, we will probably still see some divergences in the methods of the health economic part. Of course, the devil will be in the detail in the first, the one where we are supposed to talk jointly and be aligned. There will be a number of issues that we will try to solve beforehand and that we might say, “ask insurance companies.” I don't hope that's going to be very common, but insurance companies sometimes say to you if you call them up and ask, “how much are you going to pay me if this happens?” Then the answer is, “we don't know, because that's never happened before.” So, some of the questions and issues on the joint work will have to be dealt with at the time when they arrive. Then, are we inefficient, redundant, and bureaucratic? No, we've never been more than we have needed, but I think we will all realise that a system must find a balance between democratic and bureaucratic. I've been working with this in Sweden for 20 years, so when we changed the legal system in 2002, I said to my colleagues, “I'm quite assured that we will gain in democracy, but we will lose in bureaucracy.” Which probably has played out right, but still on a decent level. So, a certain part of bureaucracy is needed, and redundancy I think is something we need to carry with us. Of course, a system with 27 actors has a capacity to be a bit more redundant than a single line system. That's going to be a challenge for the quality assurance of EUnetHTA, or the Joint HTA legislation. Is the regulation the solution? Not to all your need and questions, but to some of them. Especially if you're interested in HTA, to the early ones in each process. 

Regina Skavron: How do we deal with bureaucracy? I would like to point out the beauty of bureaucracy as well. We, in Germany, established a system 10 years ago where every product has to undergo the same procedure within six months. So, all the information is online. You can look it up, look at the decisions, and look at the hearings. It's all published and it’s very transparent. I have to say that transparency, secured by the bureaucracy, is something we owe the population. If you look at the expenditure on pharmaceuticals in Germany, we have 43 billion euros a year. That is about 20% of all health expenditure. So, 250 billion in Germany, which is almost 12% of the German GDP. That is a lot of money and it's public money. We have to be accountable for that money and how we spend it on products. So, the whole procedure is there to secure the same rules for all the products and also for the money we spend. I think you have to keep that in mind if you talk about HTA. It's not, “we don't work on procedures and methods just to talk about it.” It’s money and it’s health. Then again, I would like to go into detail on one part you mentioned and also you Nicholas - the economic evaluation, In Germany, for example, we do not do any kind of economic evaluation in HTA. All we look for are clinical benefits. If you look at the system, of course, we have to get a harmonisation within the European Union and we are we are going there. We are already on the way. We have to go. But then again, our legal framework is very different to most of the other countries and we are going to get closer. But I'm pretty sure we will not do any economic evaluation even in three years. So, the first part, the clinical evaluation will be streamlined. But then at the end, after the assessment, the appraisal will still be very national. Then again, the pharmaceutical companies, of course, they want the same procedures and the same templates. But if you ask, they probably don't want the same prices all over Europe. That, of course, we have to keep a balance. 

Dan Ollendorf: I've been thinking about this in terms of divergence of methods in particular. But it feels to me that if all of these diverse HTA bodies were looking at a reasonably robust set of evidence, two or three randomised clinical trials with consistent results and some pretty good documentation on potential harms, you might not see so much divergence in methods, but this really relates to some of the discussions we've had in our earlier sessions. You see divergence in methods, in particular in the deliberative activity, when the evidence is more thin, because it's probably human nature, more than anything else. There's a need to think about what you might be missing if what you're looking at is a single arm study of 10 patients. So, then you see these disparate approaches to thinking about aspects of treatment like novelty, for example. So, is this a new method of action or a new kind of delivery approach? In some jurisdictions that may be perceived as a benefit. In others, it may be perceived as a risk. In still others, it might be a mix of the two because there are a lot of unknowns. So, I think that divergence is perhaps reflective of the evidence that HTA bodies are dealing with. But one thing that I am I am hopeful about, with something like the EU regulation, is that there could be kind of a critical revisiting of when some of those special consideration should be put into place. So, for example, we already know that HTAs threshold for any special consideration around disease rarity is different than the threshold of the regulator, but each tier usually is concerned about rarity when there are issues like outcomes that are not well described, populations that are very small, and a limited set of clinical experts. Yet some of those methods approaches - some of those adjustments to methods are applied in situations where we do know the outcomes, we do have the experts, and trials can be at least of a reasonable size. So, maybe there could be this critical review of when do we really need to think in a special way about this and when can we use a traditional approach? 

Wija Oortwijn: I want to reflect on what Nicholas said about process or Democratic focus of countries. I think that it is key for what we are talking about because not all the countries that are into HTA are democratic. We also have to deal with systems that are less democratic and how can then they adhere to what we would say are standards of HTA methods or processes in order to work with us to be accountable to our societies. I think that's very challenging. So, when we look at the context, maybe the legal factor, how societies are structured. I think that's a very important point. First, look at who are the actors? What is the is the democratic focus of a country? Is it possible to implement and use the methods that we as a community think are standard? That is one reflection. The other reflection I had was what Regina said about transparency. I think transparency is key and we’ve known that for many years in the legislation it says inclusiveness and transparency. But what do we mean by that? In terms of transparency does it also mean that the information we put out to society, to patients, and to citizens - do they really understand what we mean? Do they really understand how the process works and how the methods were used? How you talk about clinical evidence, there are so many nuances in terms of uncertainty, how to interpret it all, and how to make that clear in order to be accountable to the truth. So, for me, that's why it's so intriguing and we might think that we use the same language and do understand it in the same way. I think there is a lot to be to be done, especially on methods and processes. Then what Dan said about the evidence that in some cases, clinical trials are used and then the evidence might be more secure or uncertain, and that that might be the highest level of evidence. But, I think that builds on the notion that we focus on the biomedical orientation of our health system. That can be challenged because why would randomised clinical trial evidence be more important than patient based evidence, for example? So, these are things that I think is also a paradigm shift that I think would be nice to discuss. Also, what it means in terms of endpoints, outcomes, and relevance of HTA for our society. This is key because we tend to forget and we focus so much on trying to get an accurate answer to a very small thing because we're focused mainly on the revelation. Well, there are so many other important aspects of HTA that are meaningful and important to take into consideration when we make a decision. 

Eric Low: I think that for me, the decoupling between the clinical assessment and the appraisal is important because I think that doing a clinical assessment at an EU level is one thing, but the appraisal by each individual countries is fraught with challenges. I think all EU governments are elected based on a democratic process against a manifesto, and it's up to them how they spend their tax revenues and so many different cultural, social, economic, political factors press down on what an individual country does with its tax revenue. I think even with a strong wind and move towards alignment, the decisions at a local level will be very different for a very long time. How that is harmonized is problematic. I think it's right to aim in that direction, but it will be difficult to get a certain level of harmonisation in how the appraisal is done at a local level. I think to Dan’s point particularly - this is an area I'm really interested in – is the HTA methodology comes under pressure because of inappropriate data and data that is just not fit for purpose. So, their methodology then has to deal with uncertainty and complexity, often unnecessarily so. I think for HTA assessment and health system appraisal to work in harmony, we require better data inputs. The better the inputs into the HTA system, the better downstream decision making and more consistent that decision making can be. Another point is to see this from the point of view of industry. In my experience, industry call for all kinds of things – from cancer drugs fund, innovative medicines funds, this modifier, that modifier - what decision makers and academics do is they take that on board and then they do, as Leeza alluded to, the kind of thinking around it and then they play it back and it's not what industry want. They then complain about it because it’s quite tight, it's academic, it's thoughtful, it's responsible, which it needs to be. When I think about the joint clinical assessments (and I've heard companies call for this for years) but I can't understand how a joint clinical assessment will always work in favour of a company, especially if that clinical assessment is not what people want to hear. I think it's problematic – the whole system will get caught up and slowed up with complaints and arguments from industry that that clinical assessment isn't as positive or not what we expected. In the more fragmented process that we have at the moment, there's quite a lot of strategy that goes into it, as Regina mentioned around, “we picked Germany first, maybe France next, the UK, the Netherlands, whatever.” They understand that they can game and play the system a little bit and they can use this price and that price. A joint clinical assessment sort of wipes that out, which I think is a good thing, it's an even playing field. But where industry don’t like the outcome of that, I think it would be problematic. 

Nicholas Hedberg: There are a number of things that are not clarified in the regulation text that cannot be dealt with on that high level and I know there are a number of terms and probably a number of different documents that I don't know by heart, but one kind of document is the delegated acts where the commission co-defines the details that shall be there under the broad regulation. So, I guess the commission has a lot of influence there and in this case, also, the coordination group that will be constituted by one voting representative from each member state will have a lot of impact. I think they can also decide on the standard operating procedures. So, a lot of the methodology will be not on the legal basis in EU, but on the delegated act level or in documents by the coordination group. So, it will be dealt with. I would say, the primary focus of the tender service contract Eunethta, where I am now the chair of the Consortium Executive Board, is to provide suggestions on the 19 activities to the commission for their judgement if they want to include that as a background to their delegated acts. I am aware I made a connection to the climate debate and realise that you can be either pragmatic or an idealist. Either you say, “we cannot know anything about this because we don't have the evidence we expect or that we are used to.” Or you can look at the same figures and say, “we must make sure that we can make use of this knowledge somehow, it is not what we wished for, but we must make use of it.” Without going too far, I think that analogy goes very well with the Green Party in Sweden, where we have one faction that says, “we must stand for the ideal ideological green values” and risk says, “no, we must be proper to be in the government, and that means being pragmatic.” In that way, you can judge differently based on the same kind of data and on the same kind of uncertainty because you have different strategies on how to come through. If you don't make a difference on your own desk, which quite a few people or only a few people do, then I think that can influence in this divergence. 

Dan Ollendorf: So, I was struck by Eric's comment around separating the clinical assessment from the appraisal process. It also got me thinking about the need to think past the appraisal itself. We talked about the evidence needs and challenges during the appraisal process, but there will always be uncertainty even when you have a robust data set and there may be - again based on the nature of the health system and the remit of the HTA body and its powers - what should be done next? So, there's a residual uncertainty, how can that be resolved in some way? So, you have, as Eric mentioned, the Innovative Medicines Fund and the Cancer Drugs Fund in England. Regina in Germany, there is the ability in certain circumstances to commission government funded research to try to resolve uncertainty post decision. In the U.S, there's no real vehicle for this other than the decisions that individual payers might make. But it seems as though again, I don't know if this will be part of the methods conversation as part of the EU regulation, but some sort of sense of how to address the value of information question. What additional information will help us resolve this uncertainty? Is it worth it to try to collect this additional information? And what sort of vehicle might we consider to provide access to the technology while those questions are being answered? 

Wija Oortwijn: This is an ongoing discussion that we now have within the working group between HTAI and the DIA on. Indeed, how do we conceptualise uncertainty throughout the lifecycle, but specifically in the HTA regulatory interface, but also for HTA decision-making. It’s not so easy. I'd like to reflect on what Nicholas said. You can be pragmatic, but if you have a different understanding of what we mean by the level of uncertainty where this takes place and how to make a typology of this, then you can also have a mismatch between the different stakeholders. So, I'm coming back to inclusiveness - if you do not have a common understanding of what kind of evidence or information we are looking for, what kind of additional evidence we are looking for to indeed resolve uncertainty, and even we do not understand what we mean by uncertainty, then I think that discussion is very confusing. For example, I would give you the example that we have in the working group between the regulators focusing on the clinical benefit assessment and HTA type people focusing on clinical or relative effectiveness or even effectiveness and appraisal. So, there are completely different standards in terms of what the level of evidence should be and how to resolve it later on in the life cycle. I'm not quite whether or not you can also address this gap between the regulatory interface and the HTA interface. If you do not have a common understanding from the beginning and do not include all the important aspects from the beginning, you might have problems later on when you are going to implement or use HTA for national decision-making. I'm not giving any answers, but I think methods and processes could help in this respect if we have more clarity on how to deal with it. 

Leeza Osipenko: We have very different affordability thresholds, very different level of resources, skills, and clinical practice. The question is – if we simplify the system and say everyone has absolutely identical processes and absolutely identical methods, we still allow a degree of different decision making because of affordability thresholds for example. Now, because we have quite a bit of divergence in processes and some divergence in methods, this perhaps contributes a lot more to diversity of decision making or maybe not. So, that's not a trial we can actually put forward. But when something starts it’s complex by design because we start in small groups. Each country started their own calendar, they started their own metric system, they started their own cultural preferences. Then globalisation comes and people start to think for me in order to talk to those people across the sea we need to find some common language, find some understanding and then step by step, we come to standardisation. We still have a strange system in the US for metrics and measurements, and then everyone sort of puts up with that. But the rest of the world uses metric system quite happily, and in science we came to that standardisation. That’s the starting point. Same thing with calendars. Every culture had their own calendar, and at some point they agreed on something. So, my question also is, are we at the beginning here where we have some hope that we will have some common metrics and common calendars, so to speak, for processes? Still, knowing that the actual decision making at the appraisal stage will differ for other factors. Maybe we shouldn’t have it but if we should have it, where are we in the timeline? Is this decades from now? Is this happening already? Who should be in charge? Because I think European regulation is a potential step forward for Europe. Should there be a global step, maybe there shouldn't be. But on the other hand, if you look at that, humans are pretty similar biologically. Are we over complicating things saying that we really must consider these reservations? 

Eric Low: The assessment and appraisal system, collectively, have got a very difficult balance to strike because on one hand, we want to pull through innovation, and we want to be pragmatic and flexible in how we do that in the right circumstance. But we also want to create a demand side and push back a little bit and say it's not the system's job to mark and reward unfairly and unnecessarily. And what concerns me a little bit, and I'm not sure that this is what Nicholas said, but what I've picked up a little bit was let's make something of it or let's put in an IMF for a CDF, which in a way you can look at the CDF and the IMF in two different ways. One, you could say this is the mediocre stuff, this is the stuff that's not that good. They didn’t do a good clinical development programme, they dropped the ball, but let's find a way to get it into the system because we're concerned about the political pressure of saying no to a potential innovation. You could argue the really good evidence - an appropriate level of uncertainty without an appropriate commercial deal gets through NICE. So, there's two ways to look at this CDF from the IMF, but collectively, what the UK does is it disincentivizes improvements in evidence development because there's always a slip road, there's always a way to get your drug to market - through this lever or that modifier. So what incentive is there for industry to bring better inputs to the system that are much more helpful to patients? What the system does is it manages financial risk; the uncertainties are still there a lot of the time and that transfers over to patients. I think there's a balance to be struck between this demand and supply conundrum, where there's an appropriate amount of flexibility and appropriate levels of pragmatism to pull through the stuff that deserves to come through. But we say to suppliers, “you need to change the way you think about your evidence development, it's no longer acceptable just to address regulatory questions for the FDA and expect health systems that have got different needs and different questions to do magic things with it.” We need to get better at striking that balance, and I think we're some way away from being able to do that. 

Regina Skavron: We are, on the European level, going to distinguish between assessment and appraisal, that's for sure. So, the assessment will be harmonised - the appraisal and the decision making will be still on a national level. If you ask, where are we in the timeline, I think we are on the way to harmonising the assessment and we are still far away from a harmonised appraisal and that is due to different systems. I have absolutely no idea if there will be changes on the appraisal level, but the appraisal and the reimbursement decisions on a national level are always linked to the healthcare system and to monitor with questions - how much is the system going to pay for it? As long as we have different systems, I doubt that the appraisal can be harmonised. But that's just my personal view. Ever since we started the system in Germany 10 years ago, we have we have this conversation, so companies come to Germany and say, “we have this beautiful study, placebo controlled and EMEA is fine with it, so please give us the reimbursement price we are asking for.” Then they probably won't get any additional benefit in the German system, which has a direct impact on the negotiation on the price. So, they will not get the price they are asking for. So, I think we are moving towards a different understanding that EMEA approval doesn't help you in Germany if you don't have a comparison that is appropriate in the German system. So, there is a monetary pressure to create better evidence, direct comparisons, longer studies. So, a 10 week, placebo controlled trial won't get you anywhere in Germany. You are still on the market but you don't get the price you are asking for. If you ask me, I think a financial incentive is probably the only thing that is going to work if you want to create better evidence. 

Nicholas Hedberg: I don't think that the discussion we have had so far makes a difference if you have the entry point of a patient or of a society. My take that you must realise there is a balance between bureaucracy and democracy - that's as valid for an individual patient as for a society. Then of course, HTA is also to protect the public. So, there might be there might be differences. I did use a political party as my example, but I do think it can be an unpolitical, distinguished situation if you have two therapies that are excluding each other, so you can take any of them to the patient, but you cannot give both. So, there's a one-shot treatment. Both have proven in the regulatory pathway that they are better than their own side effects. But there is no direct comparisons. Then I think from a purely scientific point of view for that individual patient, you can end up in a situation where one scientist says there is no data that can show us the way to the best agent of these two. So basically, we cannot say anything and what the clinician will do, then I don't know. Flip a coin, perhaps. There might be people that say from a purely scientific point of view we know it's not very good evidence, but we would favour one without taking any economic considerations. I think we must be open to either your idealistic or pragmatic also in your scientific approach. Then returning to your question, Regina, uprisal is the step after health economics, so that is far beyond the remits of the joint regulation. I don't think that the member states will ever give up the right to make their own decisions. Dan, I think you said something very important – value of information. I see that the research and work around value of information has a beautiful future. I would see the need of proper value information calculations for a number of reasons. Is it worth looking for or asking for better information on this drug? But also, when we see older substances agencies that got labels, for example, in the orphan drugs system, and the prescriber says it's so valuable to know that what we have used for a number of years is actually providing evidence in a clinical trial. I want to have a figure of that value of information because the price tag elevates dramatically.  

Dan Ollendorf: We've talked about this a number of times, and I know we brought this up before, but Nicholas, Wija, and I have been involved with the Global Policy Forum of HTAI for many years. There's always been a reluctance on the side of the HTA representatives to embrace the notion of standardisation. The recommendations that have come out of the forum historically have been a little more watered down because there's a sense that we shouldn't be too prescriptive in telling HTA bodies what to do. In recent years, we've pushed back on that a bit. So, we've talked about the idea that at least some principles, some general conceptual areas could result in some agreement. We had a topic a few years ago in the deliberative process. We created some principles that's then been expanded into joint work groups that Wija co-chairs for with a publication that will be coming out at some point, hopefully soon. I can envision the idea, and maybe this does come through something like the EU regulation or another collaborative body, I can envision the idea of not necessarily a reference case, but the notion of a set of guidelines and then within each of the categories in those guidelines, there can still be the potential for variation in any individual setting. Those of us that work with low- and middle-income countries that are trying to enshrine HTA or formalise it know that's often the starting point. What are the steps to take and make your own decisions, but here's some guidance, and here's how other bodies have done it? So perhaps that's a direction we could move in as a community. 

Wija Oortwijn: I also wanted to mention the work that we have been doing on the principals and also the work that some of the interest groups of HTAi have been working on - patient submission templates that patient groups can use when they submit their viewpoints or questions to appraisal committees, and they are used in practise. So, there is some common work on methods and processes that are used by some of the stakeholders. I think it's not undoable and like what Dan said, the work that you then have to do, it has to be on a level that can also be adaptable to the local context. That’s, I think, very important. One thing that I wanted to raise was - because I was inspired by Eric again – is about the wrong side, and still we are focusing a lot on the push side because a lot of the work that HTA agencies are doing is reviewing pharmacoeconomic dossiers and also works with products that are in the product pipeline. But there is very little attention for what are the common areas? For example, in Europe or in certain parts of Europe where we really want to focus on in terms of development of products or services or interventional technologies. I mean, why is there so much focus on the drug pipeline or the pipeline of new innovations, which is important for some of the countries, like the ones that I work and I know where Dan works in Central Eastern Europe or in Africa, they don't even have access to services. The demand and the needs are completely different in these areas. Also, I think we could focus, for example, on trying to have a standardized process for identifying where we feel there is a need for development instead of asking it to the industry. 

Nicholas Hedberg: The EU regulation has 2 equally large halves. One is for pharma, and one is for medical devices. I would guess it doesn't perhaps cover everything that's in this post and all the expectations here. But without doubt, the assessment of the new kind of technology will be important for us in the coming decade and we must go from the algebraic society to the algorithmic. I don't even know what that means, but we must stop judging from mean and mediums, and judging from individuals. That’s still something I don't understand the meaning of. But at least I understand those words. What does evidence mean then in that new world? So, let's do that together then. What I wanted to raise, because I did forget that in one of my last comments, we did talk about what would be in the regulation and someone touched upon the issue that probably follow up data will not be so pronounced in the regulation. Though, scientific advice or joint scientific consultation, as the term is now, especially parallel with EMEA, will be a major part of both the preparation. Regina and I will be situated to talk more about that from the rich GBA experience of HTA advice. But that is something that has been really pronounced in the future system, to focus on early interactions between providers and assessors to make sure that we come together and clarify the most important questions before we have set up the clinical studies and formulated our research questions. 

Eric Low: You've got the assessment appraisal, but then the clinical optimization bit that no one talks about it as well. Keep in mind that we're basing this assessment appraisal commercial decisions on a regulatory dataset designed to answer regulatory questions 5-10 years ago and thinking about the algorithm, maybe think about this - look in the context of the UK and NICE in the cancer. The clinical pathway looks like a donkey, it doesn’t look like a racehorse. It's what I would describe as a consequential pathway. It’s a pathway that has emerged by random bits of marketing authorisation coming to NICE and NICE doing their very best to try and assess it within the licensed indication and thinking about what are the downstream impacts on the pathway of doing this now and in the future. That’s a little crazy, in my opinion, and what we need is algorithms at the other side, to take the assessment and the appraisal and say right, how do we fit this and optimise it into logical, patient centric, clinical pathways that make sense? In doing that, what you're able to do is to identify gaps in that algorithm and even predict future gaps in that algorithm that then says that’s your demands side with any given cancer or rare disease by building these algorithms and looking at gaps. You can push that information back into the system and say, “where's the evidence in the pipeline?” You can then identify it, tag it and pull it through this system to plug that evidence gap. There's not enough thinking around what happens beyond the appraisal. How do we optimise these medicines to reflect real world populations that have been treated with different treatments than those in the clinical trial? How do we build these sorts of algorithms to predict and to highlight and to find what we need from supply in the future? So, there’s a whole other world beyond HTA that needs to be thought through as part of a systems approach or a lifecycle approach. I guess the other thing I meant to say earlier on is this need to move beyond the assessment of a technology, to the assessment of the technology and its clinical development programme, to help us understand the extent to which a particular technology qualifies for the pragmatism and flexibility, because if it's because the company did a very poor job developing their drug and forgot to do X, Y and Z, it's not the system's responsibility to fix that. So, we need to move beyond just the technology and assess the entire clinical development programme to see what could the company have done better or differently? It's not down to the taxpayer to pay the price for a poorly conducted clinical development programme that's produced uncertainty in the evidence. 

Leeza Osipenko: Two questions. One, building up exactly on what you have said. We deal a lot now with suboptimal evidence on early decisions, trying to make decisions faster, trying to get drugs to patients sooner, and having this massive gap in evidence after the drugs are in the market. The work is not harmonised across the bodies, but even within specific system enforcement of this data collection, enforcement of this knowledge of how the particular product will perform is pretty poor. We all know that it needs to be done, no one is in charge and this could be a very interesting ground for harmonisation and cross-country regulation, cross-country reporting as a part of post HTA which feeds into the original decision. So, maybe it's the way to introduce some joint work on methodology and processes - just the question of challenge to you. Another question is, for example, the point of this particular session is that we have a problem with methods and processes depending on the stakeholder. If you look from inside the agency, and if you ask NICE directly, I really doubt they will say they have a problem. They they're fine with their methods and they're fine with their processes. They do method improvement guides and as an agency, Regina will probably agree that Germany is doing just fine, so is Sweden. So, the agency itself has no problems, so who? Who is there that has a problem? Is it industry? I suspect they must have a massive headache going to across different countries. Patients, I doubt, because once again if it's a UK patient, they know how NICE works. Would patients in the UK worry that Sweden makes decisions differently? They might. Of course, if Sweden says yes to the drug while the UK says no, then the patient becomes extremely involved. So, my question to you, who is the key stakeholder? Or maybe this goes back to this dialogue of bureaucracy versus democracy because we are spending a lot of resources on this and maybe the stakeholders in developing countries who look and say, hold on, how much does the UK spend on this? How much does Germany spend all this? We can’t spend that. Where is the template? Is the shortcut? How can we do this? So, who needs this kind of harmonisation of methods and processes?  

Wija Oortwijn: I agree, but I think the problems that he highlighted might be because of the system, because we shape the system in a way that is fragmented, that everybody has their own kind of method. The regulators have their own methods, the standard of evidence that is needed is different. Of course, we have the HTA agencies. Now we think that we need to include stakeholders. I think we created something and that's facts, so we have to deal with how it is. I think that also reflects to what you were saying, Leeza - how can we maybe do it a bit better? I think that by looking more from a life cycle perspective and also from a societal perspective, we can maybe identify where the gaps or the hurdles are reflecting on who is in charge. Of course, that depends on who you ask. Like you said, the HTA agencies have their own remit, EMEA its own remit, the FDA has its own remit and national governments also have their own remit. They also have a budget to invest in certain disease areas. They also have their own innovation policies that they want to put forward. So, it depends, I think, and so the local context is key and I think we should be trying to see if we can identify processes that we can harmonise across the whole thing and who is in charge. I think that depends. It depends if you like the EU regulation on HTA. We have decided that this is something that we would like to harmonise. I'm not quite sure whether that will solve all the problems because it only focusses on a very tiny thing within the HTA process. So, it comes back to bureaucracy and democracy indeed. Ok, we might have some harmonisation on how we collect data on relative effect from a HTA perspective that will not necessarily solve the issue that we have with regulators. The gap in terms of the level of evidence and what they asked for in the push. We as a panel will not solve all the issues, but I think fundamentally we need to look more from the system’s perspective. could be the European Commission, but that will be very hard because it depends on the legal framework that we have in place and what we have agreed that the competence of the countries in Europe is still to organise their own health systems. So that will not change. I think we can only indeed maybe try to identify topics are part of the process that we try to harmonise, but I'm not quite sure if that would help us. I don't know, actually, because my few points still are to have it optimised like what Eric says, but then you need to have everybody who is part of the chain to have the same viewpoints, the same focus, and that will be very hard. For me, the most important part is why do we not start with what is here? The problem in terms of what is the relevant policy question that we want to address? Because we are investing lots of money in certain disease areas, which is fine, but we never we never discuss why. Why do we invest in a cancer drug fund? Why not in a diabetes fund? These things for me are very important and I'm very passionate about this, but I think this should be the focus of our society's perspective. Why do we invest in something and why not in something else? 

Dan Ollendorf: If there is greater consistency and appraisal and recommendation making, that's not necessarily something that industry would embrace especially if it's a universal no. Patients, on the other hand, might be more willing to live with that, and I think that they'll feel challenged with what is more likely to be the case, which is that some jurisdictions say yes, some say no, some say maybe and here's a special programme. So, if you are in the UK and you understand that NICE is the body that's making the decision, but the decision means you don't get access to something you feel that you desperately need, that's not going to necessarily sit well. I'm wondering, and again, I think we're talking well beyond what we think the remit of the EU regulation is, but it seems as though, in addition to taking systems approach that Wija is talking about, we need to take a globalised approach to understanding what the implications are for different patient populations. So, Eric, you've done so much work, so much important work in myeloma. This is a disease where if a patient is resistant to everything else that's currently there, they're out of options. If a new option comes in, that potentially could extend their life by another year or two, and it's not available because of a local decision, it's a huge challenge for the patient and if we're going to try to centralise some of this activity, can we centralise a discussion of the implications? That's the open question. 

Eric Low: When I try to simplify things as much as I possibly can across this sort of ecosystem and continuum. 2 things come to my mind. One is whoever makes the rules in which we all work, and it's normally government mostly responsible for the rules that we have to work in - a lot of this starts and ends with government and governments around the rules they make in which all the different players, actors, and stakeholders work with them. Most often we're all very good at working within the rules and remits that we're given. The other thing I've come back to, and I've mentioned this in previous discussions, is incentives and rewards. If you want to herd cats, just move the food. Until we get the right incentives, the right rewards, unified around a common goal that Wija mentioned, it's very difficult to work at a systems level. So, we need an appropriate set of rules and we need an appropriate set of reward incentives. Once they're in place, then we can start to think about making this seismic change that's needed to solve this. Otherwise, I think we're just going to tinker around the edges, and that may be enough. Marginal improvements in how we do this might be enough. But if we must do this at a systems level, we have to look at rules and we have to look at rewards. 

Leeza Osipenko: I think we do look at this with a varied degree of success at the national level. Perhaps the reason we have all this variation is because there is this lack of, not just the organisation, but the lack of interest they have, the lack of need, who would say, let me take it in my own hands? Of course, there's a lot of resistance because of national interest. So, perhaps a challenging question for Wija – is something like HTAi, can this be promoted and asked as a question? Or once again, just because it's it's a voluntary organisation, it has no legal power, there's no one to enact it because I think what we see in Europe, there's a very strong legislation based. We decided to create a European Union, we decided to create a European Parliament and that led European institutions, and this regulation on HTA is completely a result of that. That was a political step with a strong political backing. These things are not done voluntarily. We've seen Eunethta. Perhaps because there is no decision maker, proper incentives, or a strong enough need we have this divergence, which is simply inevitable. 

Wija Oortwijn: I think there are elements in our system that feeds how the system became. The focus on the hierarchy of the level of evidence, for example, all amongst us who studied medicine or pharmacy or biomedical sciences, we are taught that is the model to follow that we have to adhere to that level of hierarchy. So, if we are taught that a randomised clinical trial is the best evidence ever, it’s very hard to see that if there's uncertainty in the level of evidence that you might also consider other types of evidence. That is always seen as less or more subjective. I would contest that, I think now there is a paradigm shift coming because we are more inclusive and we work with patients and other stakeholders so we could challenge that. But that will take time. I know Nicholas knows also very well that even if you want to have a law in place, it will take many years. That's on a very high political level that has been established, which affects a lot of lobbying between all the member states. I'm not quite sure if we will ever obtain this, but at least it's one step forward, and I think I'm very happy to be part of Europe and to be part of the European Union and that we have this legislation in place because it's a very important step forward, at least in one aspect. Of course, we want to be ambitious, and we would like to see maybe an ideal world. But it will take a very long time.