• Eric Low
• Prof Carole Longson
• Dr. Wija Oortijn
• Dr. Dan Ollendorf

Guest Experts: Javier Guzman

Pilar Pinilla-Dominguez: Today, we'll be covering HTA in lower middle-income countries. I will just say that, as we know, many lower middle-income countries have historically relied on external support and funding to contribute to efforts to achieve universal health coverage. However, the current panorama challenges this framework with on one hand, increases in health care cost in almost everywhere. And then on the other, we have reductions in healthcare budgets and budgets have indeed been even more strange because of the current COVID-19 pandemic. That has also led to reduction in funding in many countries. Therefore, it is more crucial than ever, that countries and lower middle-income countries generate more healthcare on resources spent, and indeed contributing to efficiencies in healthcare spending, while still contributing to or having in mind the trade of equality and equity. Indeed, that can be achieved by applying evidence-based priority setting mechanisms and HTA as a tool to support fair and equitable allocation of scarce or limited healthcare resources is indeed a tool that can support those efforts. However, institutionalising and implementing HTA is very challenging, and particularly for low middle income countries can be even more challenging. First, it needs political will, and the commitment to support which can be sometimes unpopular but evidence-based decision. Also in low middle income countries, there is a lack in many countries in terms of capacity and capability at both human and monetary levels. There is a lack of local data that can be used to inform the analysis required for HTA. Sometimes there's a lack of interest from different stakeholders in transparency and accountability principles that are relevant for HTA. Sometimes we also have over reliance on HTA outputs of low income countries from high income countries that have very different health care systems, different epidemiological profiles and different indeed decision problems that are looked at in HTA. Finally, as HTA requires a more disciplinary approach and participation of different stakeholders, sometimes vested interests within the different stakeholders can hamper the successful implementation of HTA policies. Therefore, today, we would like to discuss why should lower middle income countries aim to institutionalise HTA? What are key problems and potential approaches that we can learn? Then we would like to move on to health systems strengthening. What are the key steps that low- and middle-income countries should consider when embarking HTA institutionalisation? What are key elements for success, including cost of entry, infrastructure, supply chain issues, that should be considered? Can we spot some good examples, and what can others learn from this? Then, what shall be in place in terms of political levers before implementing HTA in terms of embedding those HTA outputs into access and adoption policies? Finally, we would like to discuss how power dynamics have impacted health care policy agenda, in low income countries? How have these vested interests that we were discussing earlier, from the different stakeholders, handicap sometimes HTA efforts in lower middle-income countries? And how can these be mitigated? Finally, we would like to discuss how different are the challenges in low-, middle- and high-income countries? And whether they require different health systems strengthening approaches? What can be indeed adopted from other settings? And what can we all learn from each other?

Javier Guzman: I think it's a very timely discussion, it's a very interesting conversation, especially with the current financing climate the key fact is that money for health is not going to increase. So, health budgets will be stable at best and may be cut. So, it is very timely to talk about value for money. I guess your first question is a question that really resonated with me in terms of the value of institutionalising HTA for low- and middle-income countries. I think there are just different entry points, there are different problems, different countries want to solve when thinking or when using health technology assessment. Do you see for instance, the experience of Latin America, and there's a very big move towards the right to health, so right to health and how to deliver on it. Then that was basically the key element or factor in making decisions on what's in and what's out of this right to health package. But if you see the examples of sub-Saharan African countries, sometimes the question is, what should be included in the reimbursement list or the list that will be used by the central medical store? Or what should be included in the essential medicines list? In other examples for experiences like India, for instance, physicians have been interested in improving quality. So, for instance, the National cancer grid, a network of hospitals, decided to use health technology assessment to inform the standard treatment guidelines and improve clinical practice. So, the entry points are different, the problems are different, and I guess that is my first takeaway, which is, what is the window of opportunity? What is the angle of the entry point for this to be successful? Is it efficiency? Is it equity? Is it quality? Is it a combination of elements? Because my reflection is, this is as you said, a long-term plan, this is a process. Institutionalising explicit priority setting is not going to happen overnight. You mentioned the importance of political will. You mentioned the importance of having a good ecosystem that could be put in place to deliver on this aim. But unless there's a very good window of opportunity, and there is buy in from the political leadership, I think it's very difficult to make progress. The current climate is a climate that is fit for purpose because decision makers will have to make decisions under a very challenging environment because of the macroeconomic situation I mentioned before, post pandemic, with inflation and rising food prices and everything else.

Pilar Pinilla-Dominguez: I think that, apart from the point that you made in terms of quality, efficiency, equity, you also made the point of you introduce somehow within that window of opportunity concept the role of the different stakeholders, you mentioned, the politicians, you mentioned clinicians, you mentioned, patients, obviously you mentioned the HTA practitioners as well. So how can we all work with each other?  

Wija Oortijn: I think it starts with political will. Of course, with the problem that you would like to address and whether HTA is a potential impartial answer to that problem, because I think not all problems that you mentioned can be answered solely with HTA. I think the ecosystem is indeed very important so a sustainable health system will leverage I think, will help also institutionalise HTA in a certain country. I also fully agree with this long-term perspective that people might think that HTA is the golden bullet, but without having the foundation in a health system HTA will not fly, I would say, and we always tend to forget that countries like the UK and the Netherlands, where I live, are still developing their HTA system and optimising their HTA system. And of course, when we talk about lower middle-income countries where I also work myself, people always want to move very fast and that's not something that is going to happen. So, I think you should also meet expectations and be very clear about what HTA can deliver, and what it can’t deliver and what you need for that. What I also wanted to say as a first reflection is mainly about the system's perspective as such. Political wheel is one thing, that's very important, but it's also very important that the policymakers have a clear policy framework in place on which they can build a HTA process, one HTA system. For example, I'm now in Abu Dhabi, and I work with the Department of Health, and they don't have a very clear policy framework, in terms of what are the priorities, for example, and how that relates to the health needs of the population, even though it's very clear that efficiency here is the main problem. So, cost containment is the main problem, and they would like to use HTA for that. But still, it's very hard to understand how that would work if you don't have a relevant policy framework in place that clearly articulates the decision-making criteria, for example, in relation to the values that are underlying the system.

Dan Ollendorf: I think the window of opportunity that Javier mentions, really is an interesting way to think about it at this moment, because we are sort of facing a lot of financial challenges and austerity issues and health budgets in general. I think that may be an opportunity for us as a community to really bring this messaging around the benefits of institutionalising HTA home, in that, without it many investment decisions are made without a common frame of reference, without paying attention to the economic side of HTA and Javier as you've published yourself on what that often means is that health is left on the table, that there's health foregone. So, the way to try to take, what are going to be increasingly limited funds, and apply them in as efficient a way as possible, in part involves HTA. The other bit of messaging I might think about would be borrowed from high income settings when HTA evolved. In that, again, without a formal national commitment to doing HTA, decisions would be made in a much more decentralised fashion, certain parts of a given country might have access to a new and useful technology while others would not because of that decentralised decision making, and so the benefit of a centralised approach would be that there's at least the opportunity for a national conversation about the potential for new technology.

Carole Longson: The window of opportunity is a good framing for this discussion because you have to think through what that window of opportunity might be in those different settings that we might contemplate. I mean, if I'm thinking about it as a house, we've got lots of windows of opportunity. Dan picked up a couple and I'm going to talk specifically about the window of opportunity that created NICE, and it wasn't really about the money so much. It's quite interesting that the foundations of NICE, of course, NICE is well known for doing value for money assessments. But the political reason that NICE was established was the equity issue, that in the UK, there was not a system that allowed a person who's engaging with our national health service, to be to be able to be clear what they could access wherever they lived. That, in a sense, window of opportunity, that was back in 1999 in the UK, was an important political lever for thinking how we introduce a rational system that allows people to be able to access new technologies, starting with drugs, new technologies, no matter where they lived in a national system. So, if you're thinking then about transferring that concept across to different settings, you could start to develop, in a sense, the window of opportunity framework for why a country might be interested in establishing what is in essence, a set of tools or toolkits to enable them to think through how they might fairly, equitably, and economically distribute their finite healthcare resources. I think that perhaps talking about the implementation of health technology assessment in that type of context, takes us away from what tends to happen in these conversations, which is very quickly moving to the transferability of HTA processes from a high income setting to a low-income setting. That never works, the principles might work, but we can just discuss the principles without having to discuss the techniques. It would be quite useful, perhaps, if we could explore some of those principles under the window of opportunity and why different countries might wish to use this fantastic toolkit that we have in health technology assessment. It might be quite a different way, perhaps, of entering the conversation.

Eric Low: I think for me it's recognising that while there's commonality in issues and challenges that health systems have, there'll also be unique things within each country. So, for me, it's about first and foremost it’s about being very precise about what is the problem that needs a solution. It’s Then to figure out if HTA is the solution to that problem or is an alternative approach? Because I've often wondered in the UK context and an England context if NICE wasn't around, what would NHS England commercial team do? I think they'd get along just fine. Making different types of decisions around what's paid for, what's not paid for and how much we can pay for it. So, is there an alternative solution to HTA? Or is it some sort of hybrid type model, and then what type of HTA? And I just pick up on Carole's point about equality and equity, because for me they're two different things. So what equality is, is all patients are all members of the public get access to the same health care. Equity means that healthcare is customised and it's specific to your needs, whether they're social needs, health literacy needs, or disease needs. So that would say, if you have a particular type of aggressive cancer, or a rare disease, then you might need many more resources than another patient from that same cohort. That, for me, is where HTA falls down a little bit, it’s in the equity because it's not very good at customising and reflecting individual patients in the real world. It tends to be at least from a UK perspective, a one size fits all, irrespective of your risk factors and irrespective of your personal preferences. This is what's been shown to be cost effective. This is what you're going to get. Two other quick points is that low- and middle-income countries can learn what's been done well in other HTA, and what hasn't worked, and benefit from that. At the end of the day, my final point is HTA assessment and appraisal is dependent on inputs. If the inputs aren't appropriate for the decision context, it's very hard to get the right decision. So, we need to be clear, where does the evidence come from upon which low- and middle-income countries are making their decisions? Because if it's coming from a US based study for the FDA, it's going to be problematic, because it's already problematic for the UK and France and the Netherlands, never mind low- and middle-income countries.

Pilar Pinilla-Dominguez: I think that’s the other factor we're referring to is, what are the factors that you'll be considering of value? That's something that may well not be transferable across different jurisdiction, also making the point about HTA answering specific questions, but not all the questions, it's not going to solve that problem. It’s not going to solve health financing, per se. Then the very important issue about transferability of inputs.

AUDIENCE QUESTION: We need to learn; we need to be more effective if we are talking about effectiveness and making the value of HTA more visible. These are the days, and this is the time for us to make it not just visible, but the value and the best way for me in the context I'm working with not just with Latin America and the Caribbean, but also with Bangladesh and others. The best way in this context, critical context globally, but LMI sees mainly is to be able to demonstrate this HTA has a value in specific topics like health, equity, and nowadays, resources, infrastructure, we need to be able to transfer not just knowledge, but ways frameworks to the decision makers maybe in the government that this is knowledge and framework that is able to bring changes in this quality of services, accessibility to health organisations, and almost the economic sustainability of the Health Organization's and health research organisations as well in the government. Because there is the main area on which the stakeholders which are the decision makers are taking the decisions. The other aspect is I agree with a colleague when she said that it is time also to stop to think about the methodologies to implement HTA - yes, I know in the past and even now, we are focused on the discussion of which methodology, NICE methodology, a European one etc. But beyond that, the transfer of knowledge about HTA value and impact measurement, once we apply HTA is key to be transferred to the decision makers, and in that regard, multidisciplinary members in the government, on the side, not just the expert, but the expert which has been working in the side in different contexts. In this regard, we will be able to show HTA beyond a philosophic or scientific but so far from our reality discussion, etc.

Pilar Pinilla-Dominguez: I think that sometimes more established HTA agencies have failed somehow, to show the impact of HTA in a more systematic way. Just today, I was reading a paper on the value of HTA in Thailand, and an approach to quantify that in terms of health. So, it's so important, and it's such a powerful message to show the importance of HTA and the impact of it.

Carole Longson: One of the one of the things I want to pick up from the previous speaker is this concept of process or procedure, because I think it is one of the universal capabilities of health technology assessment, no matter how you do it, and how you apply it, what HTA allows is to have a procedural structure around making decisions. That is absolutely universally transferable, and it is one of the things that I found when I've gone around talking to different people from different countries - government officials or meetings. We get very quickly into the methods of HTA, should we use quality? Should we not use qualities? How does NICE cope with so much data, but we skim very quickly over what really matters, which is how to build a process, a procedural structure that allows the decision maker to feel comfortable with making quite difficult decisions about universal coverage, maybe, or whether or not a population is going to have access to an individual technology, that procedure, the way in which you do it, not just how you do it methodologically is so powerful. Of course, depending on how comfortable you are as either a decision maker or jurisdiction, about how far those ripples of procedure of procedure go. You can build layer upon layer of procedural inclusion, if I could put it that way, with your stakeholders, with different types of perspectives. It starts to then build up a very important conceptual framework that NICE has used right from the beginning, which is accountability for reasonableness. Which is one of the underpinning philosophical tenets of the way that NICE does things - is that if you can explain what you do, and why you do it's not about the decision, but how you go about undertaking decision making. If you can give people the chance to at least question the way you're doing that, what you have embedded is an important social grounding for health technology assessment itself. I’ve absolutely not talked about methodology, because we go too quickly into methodology. I'm talking very much about using HTA as a powerful tool to improve the process of decision making, and a powerful tool to improve the inclusivity of that decision making itself.

Javier Guzman: I wanted to comment on that point about process decision making. I very much agree with the potential benefits that the flip side of the conversation is how policymakers I know, are often not very keen to get into explicit the priority setting and how there is the need to have demand from society, sometimes in organised civil society, sometimes the citizens, sometimes just the system because often it's easier for policymakers to just say, and we've seen that everything is covered. Have this higher priority. And then you get into implicit rationing, and you get in to everything is covered in theory, but nothing is covered in practise. The big gap between the announcements and the reality. So, I do very much agree with the benefit. I just see that it sometimes is hard to get policymakers to be restrained, To some extent, or open to share some of the decision making with other stakeholders. Also, I think if we talk about a continuum in terms of how systems mature and our systems grow, participation is something that is not top of mind when you get into this priority setting HTA institutionalisation strategies. We do provide a lot of technical assistance and often I hear “give me the evidence to make decisions” and shouldn’t we invite or incorporate others to help appraise the evidence or help inform the decision? In a lot of cases, the decision makers are not interested. So, I see that as a key challenge and something that again is a long-term process. To some extent, we're talking about the rule of law here. We're talking about bigger issues of governance, bigger issues of democracy. 

Wija Oortijn: I just wanted to reflect on what Carol and Javier said in terms of process. What we see, and I was talking for my own profession in terms of trying to help countries institutionalise HTA and especially in low and middle income countries. But we do use a procedural framework for helping them in terms of thinking about how to use accountability for regionalism and setting up the whole HTA system. So, I also see HTA not only as doing the assessment, but the whole process of institutionalisation and governance and priority setting and identification of topics and monitoring and evaluation. And I think that is indeed what you also said. The monitoring and evaluation of how we implement HTA has not been the best. I mean, we have not been the best in in doing that. And in terms of the accountability for regionalism, yes that is a theoretical framework. And what we have tried to do is to translate that into a practical guidance or what we call evidence informed, deliberative processes. And we are implementing these now in different countries and we are very successful doing this in Pakistan and Iran. Now I'm in Abu Dhabi, but we also do it in Moldova and Kazakhstan, Ukraine. And we see that there is political will for obtaining at least some transparency in the process, but also in terms of involving different stakeholders. And I think what Javier was saying? Yes, in some countries it's very hard to involve stakeholders because they are not organised. For example, here also in Abu Dhabi, there is no patient organisation, for example. There's no umbrella organisation and patients are not yet organised themselves in order to give them the floor and how to involve them in all these processes and also how do you go about vested interests and interest of different stakeholders? I think that's very, very difficult. And I think that's also part of HTA. HTA is first about the process or the principles. And of course, the methods come with that. I mean, that's the technical expertise. And often countries lack technical expertise and how do you go about that? But without this foundation of indeed regional framework or a framework on how to institutionalise? I think, you can have methods, but then it doesn't make any sense. So, I think first you need to have a foundation on which you can build all the different layers. And that's very hard work for the countries themselves. Because look, what Javier was saying: some countries do not have a tradition of using evidence in policy making. So how can you then try to show that this is the way forward? And sometimes it's a very long journey back and forth, like in Kazakhstan, where I've been working since 2012. It's back and forth. It also depends on the government that is in the position to make these decisions. So sometimes you represent a more open government that is more open or prone to more deliberative decision making. And sometimes you have governments that are stricter in that. So, I think it all depends on the political cycle, and that makes it sometimes hard for us and especially for those in the country to do the work, of course. So, I think this political link to the political cycle of politicians, in some countries are very important.

Dan Ollendorf: One of my reactions could be when Javier and Wija talked about governments not listening to the evidence that's presented or everything's approved and nothing's funded, I could say, welcome to the U.S., where we share many of the challenges that we're talking about today. And in fact, maybe there are some lessons learnt for the U.S. as well as we talk through this. Talking about process, It's interesting, I'd have to say in my prior role as a chief scientist for an  HTA organisation, I probably spent about 35% of my time on science and 65% on process. The importance of process is not just that you are documenting exactly how you go about doing things so that it's predictable for stakeholders, but that you're identifying opportunities for stakeholders to engage directly in the process. Time and time again, stakeholders in particular patient groups tell me that the benefit is not knowing about the process and knowing when to engage. It's that somebody is speaking with them and listening to them. Because often at times they're feeling as though decisions and recommendations are made without their input. So, I know it's kind of skipping ahead to some of the keys to success, but one of the things I jotted down as I was reviewing the slide was formalisation of HTA really involves thinking about getting ahead of certain issues that are going to come up. And we've touched on some of them already. One is how to embed outreach to stakeholders and have them be as involved in your processes as they can possibly be, including, I might add, some of the deliberative aspects of HTA than Wija was speaking about. So there actually are some very good examples in middle income settings of interested stakeholder groups, whether it's patient groups or citizen groups being involved in deliberation to try to provide input on some of those equity questions that Eric brought up. Another and I think HTA in high income settings has struggled with this is getting ahead of what special cases might come up. What areas deserve special consideration so that if in a particular society there is a great degree of value placed on therapies for extremely rare disease or therapies for cancer, the issues that we've all dealt with in high income settings that those special considerations are documented and clear at the outset, rather than having to play some sort of catch up later. I think that Carol mentioned the accountability for reasonableness framework, touching on something that Javier here brought up if. In a particular setting where there are political challenges, perhaps extending that framework to some of the stakeholders who were then involved in turning a recommendation into a decision is something that's required. So, if we think about many middle-income settings, either have decentralised health authorities, multiple payers or both, then if a central HTA body is making a recommendation and one of those organisations decides not to adopt that recommendation, they should explain why. They should be very clear about why they're there not adopting the recommendation. And so that that's really an extension of this notion of accountability for reasonableness and explaining your rationale.

Eric Low: I guess in the absence of government setting health care priorities or in the absence of health care systems and the components of health system setting priorities, it kind of comes down to who shouts the loudest. It seems to get a disproportionate amount of the scarce resources. Dan mentioned cancer, so we know that the cancer lobby is massive, both on the public side, the patient advocacy side, and the industry side. But we know that many cancer drugs bring tiny benefits. We've discussed that in previous webinars. So, I guess what I'm saying is that there needs to be some degree of prioritisation, some framework in which to work. Otherwise, whoever shouts the loudest or has the biggest voice in low- and middle-income countries will walk in and win. The other kind of point is always to be mindful that each to is a point in a continuum from bench to bedside. It's not the final point. And for HTA guidance to be relevant in the real-world setting, to be adopted and implemented takes resources, it takes systems, it takes pharmacy, it takes data collection, it takes education, it takes prescribing, training, monitoring. There are all kinds of things that come after HTA that if they're not in place, you might as well not do the HTA at all because the health system will continue to bite costs and not outcomes. And then just to pick up on that point that I made also is where does the PPI come in? Because even in England, in Scotland, in Canada, it's still very difficult to do patient involvement in HTA, the opportunities are there. That often the sophistication of the advocacy it isn't what the HTA needs. If patient advocacy wants to impact on HTA decision making, it's much easier to influence the evidence development. It's not easy to influence evidence development, but that's where you want to have the biggest impact on HTAs, making sure that the inputs are going to be appraised and assessed, and are the best they can be because if they're not, patient input has marginal benefit. 

AUDIENCE QUESTION: I think first off, it's a bit of a challenging topic to discuss without having adequate representation from low-income countries and ministries here. Just to better understand some of those barriers. And those barriers are huge. I think two of the points I wanted to raise, how can we as a group help these countries? And one of them is data. And the second is access to training online. And when I say access, stuff that's freely available and that's something we should really do better at. Someone said points about NICE and it's not about money. I totally agree with that, because in Tanzania, it's not necessarily about money. It's just about the structures and the will of doing it. There are a lot of entry points to get in. Someone else to said something about decentralised system and that making it challenging. But for me that's been the window to say to districts in a region where I'm working with them. They know and they've asked for help to implement some sort of prioritised list. And in setting that list, it's almost impossible to access data. There’s a lot of data out there that's around complex combinations when it comes to diabetes and treatments. But they just want to know looking at metformin, where should that be on their list? What is the cost effectiveness? How do other countries have that and where is it prioritised? And there actually is nothing easily accessible out there. So, they're sitting there, they're wanting to prioritise and trying to look for this data and help them find that data. It's nearly impossible. I'm not so sure that these procedures are so transferable in two of the different countries I've been working on and developing their processes. We've changed them so many times. I think what is most important is and this is what I emphasised here in Tanzania, if you can substantiate your decisions and if you have had reasons for those decisions. Another point was on the resources where those are going to come from. I don't think there's an issue of money here. There is no money and that's agreed. It's around human resources. There just are not people. So, one of the people I'm working with is an anaesthesiologist here. They're 70 in the country and he's keen to work in HTA. But I know if he starts working in HTA, there’s going to be 69 anaesthesiologists in the country. So how do you kind of push that. Our budget constraints are $2 per capita in terms of medicine spending. That’s what we're trying to those are the realities. So, I know there's a push to establish and institutionalise HTA, but it's really thinking about what works for the country. What are those entry points? Can we start prioritising with the few key areas, and can we help the country or even the region save some money to get there? The goal should be getting more money for health and less about getting value for money for establishing HTA as a broader unit. So, I think we have a responsibility to see, can we make our data more accessible, and can we push our countries to freely providing that data? So, as well as saying, we're a developed country and we have a lot of resources behind us. Can we get our courses for free online for those that need us? 

Carole Longson: Gavin has given us the challenge of getting real and getting practical and not theoretical. And it is a really important sort of check in the conversations. And I was just thinking about that before Gavin put his points across. And I'm thinking through some of the some of the situations I've been in over the years where you've got you've got a politician in front of you from a from another country who's interested in talking about what NICE was up to and what it's processes is.  I'm just giving you a couple of insights into some of those conversations. When I started talking about process, their eyes completely clouded over, and they stopped listening basically because it sounded boring, and it sounded something that isn't going to solve their problems. Talking about HTA processes. But there were one or two little tricks I had up my sleeve that did tend to keep their eyes open. And one was, it's a process that takes away difficult decisions and gives you justification for the final decision you're going to need to need to make seem to seem to go down extremely well. The second point that tends to still keep that eye open was that having a structure around these decisions means that you probably can develop something that is really efficient and relatively low cost because you probably can borrow information from other places as long as you have a process that's tailored for your jurisdiction to then internalise that and make the decision. So, I've got them into talking about process really via talking about either making it easier for them personally in their in their political life or making it efficient for their department, because we can develop a very streamlined process that allows you to use all the information that's out there. I do accept the fact, you're right Gavin, it's sometimes difficult to source information from other jurisdictions in a way that's easily transferred into another place. But I'm just wondering if perhaps, again, some of the barriers to the institutionalisation of HTA is because of the way we talk about HTA itself. And I'm just wondering if there are different ways to be able to explain the usefulness of HTA that resonates much more in the jurisdiction that you're talking to rather than the language we tend to use, which is about “I've done it myself”, accountability for reasonableness, fairness. That doesn't tend to always go down very well 

Javier Guzman: Just wanted to comment on the point made by the comment made by Gavin on revenue generation versus allocated efficiency. And I think it's not an, either, or. I think there needs to be additional money for health, but there should be acute, explicit, systematic priority setting to improve the allocative efficiency. But I do take the point that I think it’s very hard, I think sometimes questionable, to talk about priority setting when you're talking about very necessary, cost effective, essential interventions. So, are we sacrificing elements that are essential for a health system? And that is clearly the case when you talk about very low income countries, and you talk about health expenditure per capita of less than $50 per person. There’s been some work by DCP in terms of assessing the cost of essential benefit package, the essential bare minimum and in $79 per capita per year. So, if you talk about health expenditure lower than that, you're talking about sacrificing things that should not be sacrificed. So that is a key element that should be in the conversation now. How do you work in under those difficult, constrained scenarios? Because the decisions would be made anyway. So, one could argue that priority setting is still relevant even in those settings where expenditure is low and difficult, decisions will be made. So, I understand the angle as well. But I just want to say the importance of yes, revenue generation and the alignment that should also happen with allocating efficiency.

Dan Ollendorf: So, at the risk of sounding too much like an advertisement, I think there are there are some things that could help, certainly in the kind of situation that Gavin was describing. There may be opportunities to identify courses or modules that could be freely available to individuals on the ground in each country. There are repositories of information on the cost effectiveness of interventions, and those are numerous. DCP3, W.H.O. Choice. My own centre at Tufts has recently revamped its cost effectiveness registry and is making the information available for free to LMIC researchers and policymakers as long as they fill out a researcher agreement. So, what's been evaluated in other settings and of course there are some caveats around that because cost effectiveness studies are funded by a variety of sources and it's often the funders agenda that they're addressing rather than pure priority setting needs in each country or region, but there are some resources available. In addition, while there are lots of challenges and concerns around the idea of adapting HTA elsewhere to a given setting, there has been some movement in development of a methodology to really think very concretely about what can and can't be adapted for similar decision problems in different settings. I have written previously about the idea of virtual HTA, which is really the sense that if a given set of policymakers or government officials has the sensibility to do HTA, but there just are not bodies on the ground to do it. There could be an extended team from elsewhere to help with that process. Could be a couple of situations that relate to proof of concept, or it could be a more involved process as well. So, I think there is the potential out there, certainly proceed with caution is probably the idea in all of these circumstances, but there are resources.

Wija Oortijn: I also wanted to say a few words about training and Dan also mentioned resources. But I think training is key and capacity building is key. And although there might be online trainings, I think of course it's also about using that expertise on the ground and build local capacity as much as possible. And for example, Kevin was referring to his experience in Tanzania, currently with some involved in HTA. But example now here in Abu Dhabi, I ask about the expertise in health economics, and there is only one health economics here in the country and they don't have the capacity. And I was also asking about training opportunities in terms of what kind of curriculum do they offer at the university? And there is of course training opportunities in clinical epidemiology, for example, and in public health, but not in health economics. And I think that's also something that we tend to forget is the link to students and to curriculums in the country itself. So, we tend to think in terms of HTA that there is no link with educational programmes and I think we could also try to change that model that we also need to invest in, in local capacity building in the countries themselves, linking HTA expertise maybe from other countries. So other countries providing master programmes like our staff and my university and that we could also work with other universities in other countries in trying to provide training online. And potentially that could be partially free of charge. We do train to trainers and for example, we are now going to do this with the HTA department in Ukraine. So, we will do a training to trainers. So, they will build the capacity through to train the trainers to teach their own people the HTA department. So, I think there are more and more options to share expertise and to learn from each other. And that brings me to another issue of training or at least sharing experience. I think also those, say policy makers or departments of health, who are interested in setting up HTA programmes or at least use HTA or how we would use it. I think that we should also, as a HTA community, have the responsibility to use our own networks and bring these people into context with others to learn from each other. Those, for example, in the region or in the same situation so that they can share expertise and use that learning into their own practise because it's not up to us to tell them what to do. But I think if you talk to a person in another HTA body or in a Department of Health with the same problem, it might also help you at least to understand what the challenges are. And they can also give maybe practical solutions. So, I think the sharing and the learning, and the capacity building can go hand in hand with the universities HTA community as well as with the policy makers. 

Pilar Pinilla-Dominguez: Sometimes we take for granted on these important links with academics and with academia. In that in some context, like in the UK, without that support, there will not be much that we will be able to achieve. When you don't have that body of academic networks already in place, it's very difficult to drop that the lead on capacity that is on the ground. And the other thing that I think, as you pointed out quite rightly, is the emergence of networks, of networks, of people having the same issues and the value of those connections in terms of advancing in HTA. And I think that now more than ever, we are seeing more of these networks happening and taking shape and hopefully bringing value to the conversation. I’ve just seen that comment on the chat as well about EUnetHTA and about the collaboration between high income countries and middle income countries, And I just wondered whether perhaps you Wija or anybody from the panel would like to comment on that. What are the lessons learnt from the start of EUnetHTA 2021 and the collaboration between high income countries and middle income countries and whether that will be documented? 

Carole Longson: I think it's too early to know. But in terms of inclusivity, of course, that has been the beauty of the EUnetHTA initiative all along, and all member states within the European Union were able to join in, irrespective of their own internal capability. And I think that that has been one of the most important features of EUnetHTA. And it's another thing that isn't really talked about. But you learn so much from understanding both the opportunities, but also the constraints that are that are in different places about how you might do things in your own jurisdiction irrespective of the resources that you have available. Because we always tend to have a mindset that is totally framed from the jurisdiction you're in, and it is only by talking to others about what they're doing, how they're doing it, why they're doing it that you can start to reframe your own way of doing things. And I can absolutely say that there have been many things that having a conversation with international colleagues and then taking that back into NICE over the years has been profoundly important. 

AUDIENCE QUESTION: How can we make sure that the process will be documented? Because you're fully right. It's just starting up and I've heard tenders out there. So maybe it's not now the right moment to assure that that the whole process will be documented because I'm not sure that will happen.

Wija Oortijn: Well, I'm not involved in EUnetHTA 2021, but we can talk to the person in charge, so that's not a problem. But I think it's a good point. And I think it would be great if you just send them a message and asking them about this. Because I think if we don't speak up and ask whether this will be documented, then they might not think about it. So, I think it's a good idea. I fully agree because like I said before, I think we tend to do things and we do not measure any impact. We do not think about the potential value of HTA or even in terms of what value collaboration means. And then we forget that and then it's over and we haven't set the baseline. And I think the baseline is very important. And, what we collaboratively  would like to achieve and monitor this along the path towards that goal and see whether the strategy needs to be adapted. So, I think that's what we would call an impact pathway. So, I think it's a fantastic comment that you made. And I will note this down and maybe we can talk to Nicolas. To see if he can take this on board. 

Javier Guzman: Yes. I wanted to talk a bit about networks and about the role of regional platforms. The first thing about networks is I think it's important to be clear about the goals of each of the networks. We've seen networks in, of course, in Europe but in low- and middle-income countries we’ve seen HTA Asia link, and I think that the lessons to learn are many in terms of what works, what doesn't and how you maintain momentum. At the moment we are in conversations with one of the IDSI partners. They're very interested in creating the HTA networks. So is that process that is happening and we are really interested in learning from all those lessons from the different networks. Now in terms of the conversation on regional approaches. There's always been this, I guess, challenge and a limitation that it is difficult to do assessments at the regional level. These decisions are made at the national level and therefore a capacity should be created at the national level. That makes sense. But at the same time, we cannot deny this reality that the regional platforms are getting a lot of traction and the decision making is taking place at the regional level. And regional approaches might be a way to leapfrog, move quickly to having some capacity you cannot create, and you don't want to create a NICE in every country of the world as much as you don't want to create know an FDA in every country in the world. So, what is the role of regional approaches? We, for instance, are supporting the African Union and the Africa CDC. They've created the health economy, the health economics programme. And the idea is to provide the continent with a platform that is supportive and can provide guidance. And guidance could be about methods, it could be about approaches, it could be about topics, but it could also be about pioneering countries to do assessments and informing each other about the best way to solve issues. So, I think regional approaches will be more relevant going forward. We've seen how the decision-making is regional to a large extent in Covid-19, for instance.  

Wija Oortijn: I like the point Javier raised in terms of regional networks and of course HTA Asia link is a voluntary network and they do not have any resources that support networks. I think that's one of the barriers for the HTA Asia link is that's what they have published about themselves and what they have told me. So, I think the purpose of networks is of course in HTA asia link is information sharing and organising conferences together to share experiences. And I think we could distinguish between information sharing or doing assessments together like the EUnetHTA are, or maybe doing other types of HTA activities like horizon scanning or using horizon scanning to inform and make reimbursement decisions, for example, that is undertaken in Europe by, for example, the Beneluxa initiative. And there are a few more. But even though there are some more networks or collaborations between countries, sometimes these are not very effective, or at least they have not done anything. So, it's sometimes also more for I don't know how to say this, but it looks more like a marketing tool or something. Okay. The intend to work together, but we should not forget that. Also, collaboration could be challenging. And EUnetHTA also show that it's not always easy to collaborate in terms of the national mandate that countries do have. So, I think we can make a distinction between information sharing, experience sharing and doing joint assessments or other activities in order to have more countervailing power, for example, in terms of doing negotiations with industry. And Carol, maybe you can also talk about and maybe Eric can talk about the UK in which different HTA bodies now work together with the regulatory authority as well as with industry to accelerate innovations. I think there are there are many ways of collaboration, and I think we could maybe identify all of this and what maybe also in terms of effectiveness, what is effective for what. I mean, collaboration for the sake of collaboration, I think that's not the way forward. I think we really should think about what is effective collaboration, how can we support that and what is each other's role in this? Because without resources for example, HTA Asia link, they really struggle with moving the network forward like EUnetHTA, which is unlike EUnetHTA, which is supported by PAHO, the secretariat is hosted by PAHO so they have more resources in terms of doing things. So, I think that we should not forget that resources also there can be a leverage for these networks. 

Pilar Pinilla-Dominguez: It's also the importance of those collaborations supported by the mentioned researchers, but also the human resources that needs to be committed to actually collaborate with each other. Sometimes that's not there, and we don't necessarily have the people who will then sit down, and who will do the actual collaboration. 

Eric Low: I think that there's lots of potential in networking and collaboration, but again, it needs to be clear about why and what's the objective, what's the problem it solves, and what type of collaboration is the information sharing, etc. And I think again, why I've kind of made the point is there's a difference between the appraisal of the evidence and the assessment of its value. The appraisal part is done at a local level. And there's all kinds of factors that will influence willingness to pay thresholds, for example, in different jurisdictions for different reasons. And these decisions are time sensitive. It depends on what else has happened in that country at that time, what else has happened in the economy legally, politically? Who is shouting the loudest? There's all kinds of things will impact on a country's willingness ultimately to pay for any new treatment that has been assessed through a HTA body. 

Dan Ollendorf: I think we do need to think again about the decision problem that we're trying to solve here. So, with regional networks it feels to me anyway that it'd be much more complex to think about regionally coordinated, essential benefit, package design or refinement. But when you're in the realm of single technology assessment, I think the benefit of informal networks is certainly already there. If there's something more formalised, that's really a statement to the fact that a sponsor or manufacturer is producing one package of evidence globally, just one. So, it's the same evidence, and it is something of a duplication of effort to have multiple syntheses of that same body of evidence. And if there is a way to think about what could be coordinated regionally, what should be left to the individual nations to do on their own, then that's potentially a way forward. The other linchpin, I think, for a successful regional network is tying it to pooled procurement. So, we have examples of this in high income settings for sure. But that, I think is an idea that if there are joint clinical assessments, maybe with individual economic evaluations, but that all rolls into a pooled procurement exercise. And I think the benefit of that wrapped all together to participating countries is self-evident.

Carole Longson: It's just a couple of additional points of collaboration because many of the things that I might have wanted to say would have been done very eloquently. And again, just touching on the experience, perhaps of NICE when it was in EUnetHTA and of NICE now as its collaborating post-Brexit with its nearer neighbours in Scotland, Wales, and Northern Ireland. And the first thing is absolutely deciding why you need to collaborate is, is key. For both of those examples, it was about avoiding duplication or is about avoiding duplication and a little bit about scarcity that there's a finite resource even in those countries that have very well developed and people may feel very well resourced in health technology assessment capability. That it's a limitless demand once you get going. So, both of those were around avoiding duplication and scarcity. But even so, when you've understood, the reason of collaborating is a very labour intensive trying to collaborate on meaningful activities. And so, the point about having to think about resourcing the collaboration, is essential because certainly when we started our interaction with EUnetHTA, we have no idea just how much how much effort we needed to put in to make the collaboration work well and to get what we needed ourselves as an individual agency back from that collaboration. So, a lot of thinking as well as a lot of a lot of physical effort. The same with the UK collaboration. NICE is sort of hosting the secretariat, which isn’t really a secretariat. It's a very small amount of a person's job now, but even so, really important. The final thing I would want to make. So firstly, you must know what you are collaborating about. Secondly, you must realise that to do it there is an overhead. And the third thing is, it's very difficult to collaborate on what you are doing as an agency as your real day job, which is the HTA itself. And you can see how long it's taken the journey for EUnetHTA to get to that point and even now, it's only a small component of a full HTA that has agreed to be produced collaboratively, but perhaps where the more meaningful collaboration can take place because people feel comfortable is around process development, is around sharing of processes, again to be able to both think about things on an individual country level, but maybe even to get very efficient across countries about the way that you do various processes of HTA. So that as, as Dan said, those people that are having to submit information into the process can be more effective. And the second thing is obviously on sort of methods or how you go about the technical aspects of HTA, those two things are very fruitful areas to start with, but you still must do the first two, which is know why you are collaborating and make sure you have the resource. Otherwise, the collaboration is very difficult to continue with. 

Pilar Pinilla-Dominguez: I think that you picture an ideal framework in which, if you have similar process in a structured way and then common methodology you create the window of opportunity, If I may use the term, for capacity being used from different areas and that exchangeability of resources that can come into support perhaps certain areas of the world that may not have that capacity at that time. 

Javier Guzman: I don't think we have lessons yet because this is a new initiative being set up and being led by African researchers and policymakers led by Cambridge. But there are basically looking at existing networks, trying to understand what the lessons are, what the success factors are. So, your point about funding for the HTA Asia link is something that they clearly took on board. The point of having clear terms of reference to decide what’s in and what’s out  in terms of what they would collaborate on, what they would not. It's also very much considering something of what Carol was saying in terms of the purpose of the funding also considered. So, it is a nascent initiative in a continent that is eager to learn, that is eager to understand what the value of HTA is. And a country where being able to conduct these this conducts these assessments and put in place their systems. So, I guess we'll have to wait a bit to see how they go, and that will clearly enlighten us in terms of their own experience. 

AUDIENCE QUESTION: From the experience of the W.H.O. Collaborating Centre for Health Technology Management in the last sixty years. We are more than ever sure that this is the time to build the capacity. This is a time to transfer knowledge not just through sophisticated models or frameworks, but with evidence collected from other countries and successful responses to needs. That's it. What in the collaborating regarding HTA, are the priorities of the country or the region? Let's talk about a country selected. Let's select three countries and then ask them the government, two or one priority and show them through  training with evidence, how much value in the impact measurements could bring in the application of HTA? And then this evidence are very clear drivers to make a change in decision making. 

Javier Guzman: Thank you for the opportunity to talk about the importance of systematic priorities, red tape, and deployment of HTA as a tool to solve some of the problems the low income countries are facing and the importance of learning from the successes and the mistakes from high income countries and vice versa. So great to feature a piece of the LMIC angle into the broader conversation. 

Dan Ollendorf: So, the value and impact of HTA is what we'll be discussing at next year's HTA Global Policy Forum. Because as we've all discussed, even in these established settings, we really struggle to try to document what the benefits have been. So, I think that, really above all else, we should be thinking about how the tools of the trade (This is reflecting on something that Carol said), but how the tools of the trade can be used to most benefit these settings where an evaluation of priorities and potentially some level of formalisation of an approach like this would make sense. But we must be very respectful of the context in which these processes are implemented to understand cultural differences and differences in expectation, differences in political systems, to make sure that if something really is not fit for purpose, we're not trying to shoehorn it in. 

Carole Longson: I have come out of this conversation with a very strong feeling that there isn't one cookbook. It’s not as straightforward as that. And maybe we have thought about it. We've got a set of tools and we just need to pick the right tools for an individual setting, understanding the concepts, understanding the context. But I think it's more complicated than that. I think you can't just have a set of tools and go into a country and think HTA tools and think that it's going to be relatively straightforward to apply them. We might again have to tear up the rule book and think again about how we are going to make best use of this wonderful asset we have, which is HTA to do good across the globe. 

Eric Low: I think for me, it's just about going slowly to go quickly and really being very thoughtful about the in-country environment, the decision context, the problem to solve. What might the right tools be to solve that specific problem that may or may not be HTA or it may be a recipe in the cookbook from Carol of what HTA is, but it will not be enough. There will need to be other sort of architecture around the case again that is specific to the country to make it work.

Wija Oortijn: I think the HTA operates within a system and the system perspective is key for countries. And I think indeed the question we should ask, I mean we as a community, what are we trying to solve? What is the problem here? And I think the problem the policy problem should be the leading question. And that could, of course, differs from country to country, even between countries and regions of countries. So, context, I think, is key. It's and always has been key. And I think we already know that HTA is not a blueprint and that we really must work with the countries to see what we can do. And to me, it's always a challenge when you go to a country to understand the context. And so, for me, working with local people is the most important lesson that I have learnt in my whole career. It's not for me to tell others what to do. It's just to ask questions. Questions that can be reflective for the decision makers and for other people doing the HTA on the ground. So, I think that's the learning and I think that works well. And I what I really would like to envision is also about sharing knowledge together and building a more resilient system in which HTA plays a key role, because I also believe in the approach. I would say HTA is part of the approach in strengthening a health system.