R for HTA 2024 Workshop - Robert Smith & Tom Ward - AssertHE

Summary

The speaker introduces the 'CT H' package developed for model review in health economics, emphasizing the shift from spreadsheets to script-based languages. The package includes checks for model validation and a visualization tool for understanding the model's function network. It also features a unique summary generation using large language models to explain individual functions. The presentation highlights the importance of communication, efficiency, and adaptability in model development, inviting contributions and feedback on the open-source tool.

Takeaways

• 🙌 The presentation introduces the CT H package developed for model review, emphasizing the transition from spreadsheet software to script-based programming languages like R.
• 📝 The speaker expresses gratitude to contributors who have helped in the development of the package, including those who have tested bugs and contributed to the manuscript.
• 🔄 The overarching aim is to shift the modeling pipeline to a more integrated system for statistical work, model development, visualization, reporting, and application creation.
• 📈 The importance of usability, communication, efficiency, adaptability, and future-proofing in model development is highlighted, with a focus on the need for intuitive interfaces and effective communication of complex clinical pathways.
• 🤖 The potential of large language models in health economic modeling is mentioned as a future direction, emphasizing the need for models to be adaptable and incorporate advanced technologies.
• 🔍 The assertHR package is introduced as a tool to help modelers review and build models, offering a series of checks and functionalities to ensure models are working as intended.
• 📊 The package includes functionality for visualizing the structure of a model through a function network plot, which helps in understanding how different functions within a model relate to each other.
• 🛠️ The visualization tool allows for the identification of functions, their documentation, corresponding tests, and code coverage, providing a quick overview of the model's structure and testing status.
• 🔗 The use of interactive HTML outputs for model visualization is discussed, enabling easy sharing and communication of the model structure with stakeholders.
• 🔑 The potential for generating summaries of model functions using large language models is explored, offering a way to understand individual components of a model in layman's terms.
• 🔍 The script concludes with a call for contributions and feedback on the CT H package, emphasizing the open-source nature of the project and the community's role in its development and improvement.

Q & A

What is the primary goal of the CT H package being discussed?

The primary goal of the CT H package is to shift the modeling pipeline from spreadsheet software like Excel to script-based programming languages, enabling a more integrated and efficient process for statistical work, model development, data visualization, reporting, and app creation.

Why is there a focus on moving away from Excel to script-based programming languages in model development?

The focus is on improving usability, communication, efficiency, adaptability, and future-proofing. Script-based languages allow for better code management, testing, and integration with other tools, which is essential for complex health economic models.

What does the speaker mean by 'shift the modeling pipeline'?

Shifting the modeling pipeline refers to transitioning from using multiple disparate tools like Excel, VBA, and PowerPoint to a unified system where all aspects of model development, analysis, and reporting are handled within script-based programming languages.

What is the significance of the assertHR package in the context of this presentation?

The assertHR package is significant as it provides a set of tools for modelers to review and build models more effectively. It includes functionality for conducting checks on model components and visualizing the structure of a model through function networks.

How does the assertHR package help in reviewing models?

The assertHR package helps in reviewing models by providing a series of checks that can be applied to model components, ensuring they meet certain criteria, and by visualizing the model's function network, which aids in understanding the relationships and flow between different parts of the model.

What is the role of the 'visualize project' function in the assertHR package?

The 'visualize project' function in the assertHR package allows users to plot the function network of a model, which helps in understanding how different functions within the model are interconnected and how they contribute to the overall model structure.

How can the assertHR package be used to improve communication about a model's development?

The assertHR package can be used to improve communication by generating summaries of model functions, identifying test coverage, and providing an interactive visualization of the model's structure. This makes it easier for stakeholders with varying levels of technical knowledge to understand the model.

What is the potential future direction for the assertHR package mentioned in the script?

The potential future direction includes expanding the testing functionality, integrating a chat-bot for interactive questions about model functions, fine-tuning a large language model for better function summaries, and possibly submitting the package for peer review and publication.

How does the speaker address concerns about the limitations of the assertHR package in verifying model accuracy?

The speaker acknowledges that the assertHR package is not a substitute for full verification checks and that it may not catch all errors, such as incorrect transition probabilities. However, it is intended to nudge modelers towards better testing practices and provide a starting point for identifying potential issues.

What is the current status of the assertHR package in terms of availability and contribution?

The assertHR package is currently available as an open-source development version on GitHub. The speaker encourages contributions, testing on various models, and feedback to improve the package further.

Outlines

00:00

📈 Shifting the Modeling Pipeline to Script-Based Languages

The speaker begins by expressing gratitude for the contributions to the development of the CT H package, which is aimed at enhancing model review. The overarching goal is to transition the modeling pipeline from spreadsheet software like Excel to script-based programming languages, with a focus on improving communication, efficiency, adaptability, and future-proofing. The speaker highlights the importance of usability and communication in health economic models and the need to cater to diverse audiences with varying levels of technical knowledge. The presentation also touches on the use of shiny apps and markdown files for automation and the structure of cost-effectiveness model packaging in R.

05:02

🔍 Introducing the assertHR Package for Model Review and Development

The speaker discusses the challenges of reviewing complex models with nested functions and numerous scripts, drawing a comparison with the clarity of Excel files. To address this, the assertHR package has been developed to assist modelers in reviewing and building models. The package offers a series of checks to ensure the model's correctness, as well as functionality to visualize the structure of a model. The speaker emphasizes the benefits of having a centralized package for common checks, rather than each modeler writing their own, and introduces the 'visualize.project' function for plotting the function network of a model.

10:02

🛠️ Visualizing and Summarizing Model Structure with assertHR

The speaker elaborates on the visualization feature of the assertHR package, which allows users to plot the network of functions within a model, helping to understand the relationships between functions. The package scans the project folder, identifies functions, documentation, and corresponding tests, and presents this information in an HTML output. This visualization aids in quickly identifying untested functions and understanding the model's structure. The speaker also introduces the use of a large language model to generate summaries of individual functions, providing a layperson's understanding of what each function does.

15:06

📚 Case Studies and Future Developments of the assertHR Package

The speaker shares examples of case studies where the assertHR package has been applied, including models from GSK and others, and acknowledges the contributions and feedback from the community. They outline future plans for the package, such as expanding testing functionality, integrating a chat-bot for interactive questioning about functions, and potentially fine-tuning a large language model for better responses. The speaker also invites contributions and feedback on the open-source software and mentions a linked open-access publication for further information.

20:06

🤔 Addressing Concerns and Future Considerations for assertHR

In the final paragraph, the speaker addresses a concern raised about the limitations of the package in providing a false sense of security, as it does not replace full verification checks. They acknowledge that while the package aids in visualization and some testing, it cannot catch all errors. The speaker also dismisses the idea of making the visualization an R Studio add-in, stating that there are no current plans for such integration.

Keywords

💡CT H package

The 'CT H package' refers to a specific software package being developed for model review in the context of health economics. It is designed to facilitate the transition from spreadsheet-based modeling to script-based programming languages. The package includes functionalities for testing and visualizing models, which is central to the video's theme of improving the modeling pipeline and ensuring quality control.

💡Modeling pipeline

The 'modeling pipeline' is the process of developing, reviewing, and implementing health economic models. The video emphasizes the goal of shifting this pipeline from spreadsheet software like Excel to script-based programming languages, which can enhance efficiency, adaptability, and communication of complex models.

💡Shiny apps

Shiny apps are interactive web applications built using the R programming language. In the context of the video, they represent a tool for creating intuitive interfaces that can communicate the modeling process to diverse audiences, from clinicians to patients.

💡Markdown

Markdown is a lightweight markup language used for formatting text. In the script, it is mentioned as a method for automating reporting in conjunction with Shiny apps, which helps in the documentation and communication of health economic models.

💡Health economic models

Health economic models are analytical tools used to evaluate the costs and health outcomes of different interventions or treatments. The video discusses the importance of these models in communicating complex clinical pathways and the impact of new interventions.

💡Usability

Usability in the context of the video refers to the ease of use and intuitiveness of the tools and interfaces developed for health economic modeling. It is one of the key considerations for the developers to ensure that the models can be understood and utilized by a wide range of audiences.

💡Efficiency

Efficiency in the script highlights the importance of improving computation time and streamlining the process of model development and review. It is tied to the goal of moving from multiple software applications to a unified package for health economic modeling.

💡Adaptability

Adaptability refers to the ability of the modeling tools to be flexible and easily modified to suit different needs or incorporate new features. The video discusses the importance of this concept in ensuring that models remain relevant and can incorporate future advancements, such as large language models.

💡Code coverage

Code coverage is a measure of how much of a codebase is tested by automated tests. In the video, it is used to assess the quality of the health economic models, indicating whether all parts of the model have been adequately tested and reviewed.

💡Large language models

Large language models are AI systems capable of generating human-like text based on input data. The video discusses the potential of integrating these models into health economic models for future-proofing and improving the summarization and understanding of complex functions within the models.

Highlights

Development of the CT H package for model review, aiming to shift from spreadsheet software to script-based programming languages.

Introduction of a package that allows for model quality checks and communication of the QC process.

The importance of model communication for various audiences, including modelers, clinicians, patients, and payers.

Key considerations for model development: usability, communication, efficiency, adaptability, and future-proofing.

The use of Shiny apps and markdown files for automating reporting in health economic models.

Overview of the assertHR package, designed to help modelers review and build models with a focus on quality checks.

The visualization of a model's function network to understand how functions fit together within the codebase.

The ability to generate a summary of any function using a large language model for better understanding.

Use of the 'visualize project' function to plot the function network and assess model structure.

The interactive nature of the HTML output, allowing users to explore function details and test locations.

Plans for future enhancements, including expanding testing functionality and integrating a chat-bot for function inquiries.

Open-source nature of the project, encouraging contributions and testing on various models.

Potential for fine-tuning a large language model with teaching material for improved function summaries.

The CT H package's role in nudging the industry towards better testing practices in health economic modeling.

The limitation that the package does not replace full verification checks and the need for clear communication of this.

No current plans to make the visualization an R Studio add-in, but openness to contributions for such development.

Transcripts

uh so everyone um thank you thank you

for coming um yeah I've been a bit busy

today so apologies if there's uh there's

a few uh last minute issues um okay so

first of all so we're going to talk

through the the CT H package uh which

we've been uh developing uh for our

model

review and wanted to say start by saying

uh thank you to to a lot of people who

contributed to this um either in kind of

contributing to the the manuscript which

is in progress and I'll share a a link

to the kind of draft on on the progress

uh in this presentation but also people

who who've directly um tested bugs and

things in the package and contributed to

the package

itself um so the overarching aim and I

guess the overarching aim here in

general and I I start a lot of

presentations with this slide is to

shift the the modeling pipeline uh from

spreadsheet software msxl to to script

based programming languages so from the

you know your survival analysis through

uh your say VBA code for a model

um to excel plots word outputs and

PowerPoint we want to shift that

pipeline that whole pipeline through to

a system where we are able to uh perform

our our stats work our model our Pops

and visuals our reports and our apps so

that's kind of like the big picture uh

kind of where can we get

to so uh through various Publications a

lot with uh with Paul and and well in

the past we've worked with how do you

build these shinies up shiny apps How do

automate some of this reporting in in

markdown files uh and then last year at

our HDA we presented on packaging of

cost effectance model so a structure by

which you can create a package for a

model uh in a set format and that kind

of um provides a structure and a

template for how to to encode uh models

in R um and that looks generally quite

similar to what most of the models that

I've see in in our and view INR uh end

up being like and so often we'll get

models where we have you know a big

folder with a load of functions a folder

with a load of tests Etc so we'll come

to that in a second so that's kind of

the the background of the the

introduction to where we're we're

starting

from yeah and um so before um Rob talks

about the the interesting stuff um I

wanted to give just a quick uh

perspective on why we're particularly

interested um in shifting towards R um

from a a farmer perspective so um for

any individual asset we work on we just

have a huge range of potential models uh

that might get built throughout the um

the assets development timeline um but

the thing I really want to focus on

there is not the breadth of the models

but the breadth of the audiences um that

we work with both during the development

phase and the output phase of the models

um you know we talk to modelers uh

clinicians patients payers uh people

with lots of technical knowledge people

with very little technical knowledge um

and one of the things we really value um

in those experiences is being able to

communicate the modeling process um so

you know when we build Health economic

models the whole idea behind it is that

we're building these models to uh

communicate a complicated clinical

pathway and um the impact and new

intervention might have on that pathway

um and so it's really important for us

um that we're able to communicate to the

right people at the right time um and

this just brings me on to guess what the

couple of the key considerations for us

so usability communication um just you

know the ability to develop intuitive

app style interfaces using things like

our shiny um and um you know ours focus

on data visual visualization sorry is um

is a huge bonus for us as we communicate

these ideas um the other two things

efficiency adaptability uh slightly more

more obvious obviously it's uh it's

great to have um an improvement in

computation time and you know as Rob

said pull everything under one package

rather than having to look at word excel

PowerPoint and kind of pull things in

and out of things um and then you know I

guess the last one touches on being

future proof um you know so we're

working um with a lot of large language

models these days and being able to

include those within Health economic

models is clearly going to be a future

direction for for modeling in general um

and so I'll hand back up I think he's

gonna uh talk through um uh this this

package of tools which hopefully will

help us not only QC models but also

communicate that um QC process which I

think is really

important thanks and thanks in

particular to to Tom and and YY who are

here who've been uh helping internally

to test some of what I'm going to talk

about on in some internal models so

we're testing it not only on some open

source models but some internal models

as well which has been really useful and

valuable so um you might imagine

somebody says can you review my model um

so it might be that you've got uh your

big our model and they go to an ERG or

uh it's a MTA and then you go to to

Industry and somebody needs to review

that that model and so those of you who

who have reviewed one of these quite

more complicated complex side of

modeling type models will know that you

end up with a whole ton of nested

functions it's very very hard you've got

a load of scripts and it's very very

hard to even start like where do you

start with this this process just

understanding where things fit together

once you get to the code you're kind of

you're right it's how do I even start to

approach this problem so quite often you

end up with you know tons of ask scripts

like what do I do with all this stuff

and this is probably one of well this is

one of the biggest criticisms of R

versus Excel because everybody opens an

Excel file and yes you've got a lot of

tabs but they're kind of Fairly clearly

labeled and you can kind of Click

through and see how things fit together

to some extent even if I think generally

it's it's

harder so often feel like this

so and then after panicking for a while

you kind of start moving through the

code trying to understand uh the code

base um but what would be really handy

to have is something that allows you to

help review that code and ensure it's

working as you you intend and as it as

it says in the

documentation so we built uh the assert

HR package and the idea of this is to

help modelers review uh and build

because obviously building a model you

should be kind of reviewing the model as

you're going along making sure it's it's

uh working as you intend uh you might

have internal reviewers in your your

organization and you might also want to

convey the model to other people and

explain it and so having tools that

allow you to do that are really useful

so there's two bits of functionality

that we uh that we really focus on so

first is a whole series of checks so

those of you that have built models in

our will know as you're writing your

code you'll often add in a line that's

just a check line after you built

something so you create a transition

probability Matrix and you then have a

line that says do all the rows of this

Su to one you create a trace do all the

rows some to one is this a numic is you

know is this the right type of object is

it the right dimensions all these little

tests that everybody writing

independently makes no sense whatsoever

what makes much much more sense is just

to have one package where we just have a

function which says check mark of Trace

check transition probability Matrix and

that's how this this um um kind of

project started was rather than

everybody writing their own checks let's

pull our resources we can all contribute

to this package which just then you just

install package check Trace check

transition proper to Matrix check that

run off PSAs all of that kind of thing

uh that everybody's kind of doing anyway

uh gets put on a package but then I got

really carried away and uh and then

number two started which is how to

summarize and visualize the structure of

a model so uh what this does is it

allows you to plot the function Network

um so any model in R if it's well

constructed should be a network of

functions so some functions are going to

call other functions um other functions

will be kind of Standalone um but often

you'll have nested functions where um

you have you know a whole chain of

functions so function one calls function

two which call function three which

calls function four and so understanding

how all those fit together uh is really

important so what we've done is built

this package which allows you to to pass

through your project folder for your

pack uh for your model and it will then

recognize it it will take a folder for

the functions and a folder for your

tests and it will will Loop through all

of the code and identify where the

functions are so exactly which line

they're specified on they're created um

and identify where the documentation for

that function is and identify if there's

a corresponding test in a test

folder and so as I said there's two

different ways to use this so at the

moment it's just on GitHub not Crown um

as a development version so you can just

do Dev tools install GitHub um and then

Library H and then somewhere in your

code so you've got all your model code

and then somewhere in your code you

might want to you know check that your

transition properties to array complies

to load rules and so if you look at the

that the documentation for that function

you'll see all of the the things that

we're checking there and so you can say

well I'm happy with this um this is what

you know lots of other people are using

so these these checks fine with me and

then it just Flags if there's any errors

so in this case uh there's an error with

this uh array and so it tells you the

transition probabilities are not valid

for these different Health States at

these different Cycles um so this has

been done before and it's just an

adaptation on on on all work um but

probably the more interesting bit is

what I'm going to talk about for most of

the rest of the session U which is to

visualize a network of functions and

there's a single function that you can

call here called visualize

project and what you have to give that

function is a project path so this is

the project directory top level where

are the functions the functions are in

here the r folder

where are the tests the tests are in the

test test that folder you could have

them in somewhere else you could have a

function folder called functions or a

test folder called tests whatever you

want to call it uh and then you just

need to uh say whether you want to run

coverage of of of functions to see what

the code coverage is like on those

functions so those are different kind of

arguments to the um the

function uh as I said there's a lot so

on on the first side there's there's a

lot of these different checks and

they're all kind of included in

documentation um yeah like they're

useful everyone's writing them don't

think that's super super interesting but

it's something that as a group I really

strongly believe that we should all be

pulling uh together so if you do want to

contribute a check that you're using

internally in your organization and

you're just you know you're using it

routinely uh then please uh send me a

pull request or send the code and we can

build it in and So eventually we'll have

a whole Lo of these checks for different

types of models So currently we just

have uh markup models that we could have

you know checks for um PSM for um b um

you know any type of model uh all in one

place so that everyone's kind of

validated these this bit of

code but probably more useful is is the

model review so the visualization so

this is what uh gets spit out in HTML

from the um from from the reviewer

function and so this is the Run six

model and the Run six model has five sub

functions so this the function in the

middle calls those five so the arrow

denotes that that function exists or is

called by the the function which the

arrow is pointing to and this is a very

simple model so these five functions are

called sequentially so there's a parent

and then five children and these two

functions here are not called by run six

model they used somewhere else in the

code base um but what we can see is that

that very quickly that those um six

functions in the middle all have test

associated with them so it's gone

through and it's recognized these are

our functions um these are the names and

these ones have tests they denoted in

green uh the one that's orange uh is

orange because has a test but its code

coverage is between 20 and 80% and the

one in red doesn't have a test so we can

very quickly see were our functions um

and do they have tests and how they how

do they fit together now obviously

that's a very simple model um but we can

click on any one of the nodes so any of

our functions and we see the function

name we see where the function is

located in our code base we see where

the test is located and we see the

coverage so that's kind of a like

summary of every constituent part of the

model within this HTML um output we can

create new functions so just for

discussion purposes so say we had a

meeting internally and we run this

visualizer we get this output for our

model and we say right we want to build

in this new part of the model or this

new algorithm into the model we can

construct our own nodes just in this

visualizer as a communication atol to

say you know now we're going to build

this new function that does this new

thing and that gets called by an

existing function and that that's our

kind of plan going forward so it's also

useful as a kind of planning tool for

your your modeling

work but what's um kind of really

interesting from a future proofing

perspective is we can generate a summary

of any of these these uh these functions

so the I symbol basically takes

you yes so the I symbol basically takes

you uh produces a popup which displays

the function code uh the uh this symbol

basically takes you to our studio to

that location the function as you expect

but this robot symbol basically uh it

generates a large language model summary

of the function so as um as we talked

about already large language models is

not very good if you provide thousands

of lens of code uh and tell it to

summarize it or tell it to talk about it

but if you provide it five lenses of

code suddenly it gets a lot better so if

you have a model that's built up of lots

of small algorithms and you send those

to be summarized suddenly uh you get a

much better uh output and result so

rather than getting you know sending

your entire model to be summarized

what's going on we just we want to

understand in Lay person summary what

does this one function do and so then it

it generates a a two paragraph summary

so just a short bit of text summarizing

what this function

does and the way this works is is that

when that button pressed the function

data is extracted so we take um the body

of the function the arguments and the RO

code store that all data uh create that

like pass that all together into a

prompt and then send that VI an API

request uh at the moment to open AI um

and receive a response reform format it

and then show it in the the

mod so it looks like this so from your

um clicking that button you then just

get this this short summary

um and so we've run this as a few case

studies um so we've done the nice RCC

model so thank you to to D and Darren

that we heard earlier uh first

presentation um we've done the six s

pack teaching model that we we published

on a couple of times um we've done the

cdx2 ca model Fernando published back in

2022 uh which is an our package uh and

and several others described there uh

and then um the GSK team have been doing

some internally and several of you have

contacted me and thank you very much for

implementing those internally and and

telling me about all the bugs in this

software uh it's been really handy and

for those of you have sent PO requests

um to to help improve it that's

fantastic um so the code for this is is

completely open source so here's an

example of one this is the the CDX to

biomarking biomarker testing um output

so you can see the the function network

is actually slightly bigger than that

and then a summary of one of the

functions generated this is the um uh

the the cheff micro simulation model um

that's built by D poad so this is kind

of what that looks like and then you can

see a summary here of where the function

is function location test location and

this is is is fully interactive um and

the same for the nice RCC Pathways pilot

so use coup more

minutes I don't know where I put it I'm

just trying to find

the okay that's

fine so yeah if you want um I can send

some some examples so render HTML output

um and so what that means is that you

can then share that visualization via

email to whoever you want to to share it

with um so in terms of next steps

um we welcome contributions testing the

software on your own models the software

is fully open source so you can just

download it from from GitHub and test it

on your models what we'd request is if

you do use it please let us know how it

went either say fine no problems or um

tell us what the the issues are so we

can help kind of improve this going

forward um we're going to expand some of

the the the testing functionality that

it does provide and also integrate so at

the moment uh it returns a two paragraph

summary of the of each function uh when

you click on there what we could do is

build in sort of a almost like a chat

bot so you can ask questions about the

function and and respond back and

forward which really cool uh and then

also we're looking at whether we can

fine-tune a l large language model

because we have a lot of teaching

material uh and so we can take that

teaching material and the code and then

fine-tune the large language model so

that it gives a better response than the

the model just developed by by open AI

um or you'd Hope anyway uh and then

finally uh submission at some

point so if you are interested in in

having a look and commenting or you do

have any comments I'm Consciousness late

and we want to get to the fob have a

look at the the Open Access publication

that's Linked In it's linked here and on

GitHub um and and just let us know feel

free to make comments on

there yeah

[Applause]

conscious that obviously everybody wants

to go to Pub so probably will limit this

to two questions so fingers

first have we got who wants to ask

one go ahead Rob so obviously love the

package the unit testing visualization

is great but my main concern would be

giving users a FSE sense of security

because this wouldn't pick this is not

equivalent to a full verification check

so it wouldn't pick up if the wrong

transition probabilities have been

pushed in the wrong place or the

treatment are being commuted

sorts of things so there could be

something massively wrong with the

model 100% pass so I think that caveat

needs to be much clear communting this

to absolutely yeah no so I mean if your

code is wrong it's still wrong right

like this is for visualizing the model

um it won't it won't necessarily Find

the Errors I guess what I would say is

as an industry we probably aren't very

good at testing our own code and like

people from software development would

probably come in and look at Health

econic models and be like like you guys

aren't doing a hell of a lot of testing

and so maybe this will nudge people

towards at least writing some tests even

if the tests don't pick up everything at

least there's some um Le a start so yeah

um do we have any questions on the

chat that one

yeah well that's brutal

from thanks John

okay so are are there any plans to make

your visualization in R Studio add in no

no I I don't think yeah to be rooted

honest

no I think that'd be great if I asked

you to call me up and want me to come

come come work on it then then great but

no not

currently yeah