R for HTA 2024 Workshop - Robert Smith & Tom Ward - AssertHE
Summary
TLDRThe speaker introduces the 'CT H' package developed for model review in health economics, emphasizing the shift from spreadsheets to script-based languages. The package includes checks for model validation and a visualization tool for understanding the model's function network. It also features a unique summary generation using large language models to explain individual functions. The presentation highlights the importance of communication, efficiency, and adaptability in model development, inviting contributions and feedback on the open-source tool.
Takeaways
- 🙌 The presentation introduces the CT H package developed for model review, emphasizing the transition from spreadsheet software to script-based programming languages like R.
- 📝 The speaker expresses gratitude to contributors who have helped in the development of the package, including those who have tested bugs and contributed to the manuscript.
- 🔄 The overarching aim is to shift the modeling pipeline to a more integrated system for statistical work, model development, visualization, reporting, and application creation.
- 📈 The importance of usability, communication, efficiency, adaptability, and future-proofing in model development is highlighted, with a focus on the need for intuitive interfaces and effective communication of complex clinical pathways.
- 🤖 The potential of large language models in health economic modeling is mentioned as a future direction, emphasizing the need for models to be adaptable and incorporate advanced technologies.
- 🔍 The assertHR package is introduced as a tool to help modelers review and build models, offering a series of checks and functionalities to ensure models are working as intended.
- 📊 The package includes functionality for visualizing the structure of a model through a function network plot, which helps in understanding how different functions within a model relate to each other.
- 🛠️ The visualization tool allows for the identification of functions, their documentation, corresponding tests, and code coverage, providing a quick overview of the model's structure and testing status.
- 🔗 The use of interactive HTML outputs for model visualization is discussed, enabling easy sharing and communication of the model structure with stakeholders.
- 🔑 The potential for generating summaries of model functions using large language models is explored, offering a way to understand individual components of a model in layman's terms.
- 🔍 The script concludes with a call for contributions and feedback on the CT H package, emphasizing the open-source nature of the project and the community's role in its development and improvement.
Q & A
What is the primary goal of the CT H package being discussed?
-The primary goal of the CT H package is to shift the modeling pipeline from spreadsheet software like Excel to script-based programming languages, enabling a more integrated and efficient process for statistical work, model development, data visualization, reporting, and app creation.
Why is there a focus on moving away from Excel to script-based programming languages in model development?
-The focus is on improving usability, communication, efficiency, adaptability, and future-proofing. Script-based languages allow for better code management, testing, and integration with other tools, which is essential for complex health economic models.
What does the speaker mean by 'shift the modeling pipeline'?
-Shifting the modeling pipeline refers to transitioning from using multiple disparate tools like Excel, VBA, and PowerPoint to a unified system where all aspects of model development, analysis, and reporting are handled within script-based programming languages.
What is the significance of the assertHR package in the context of this presentation?
-The assertHR package is significant as it provides a set of tools for modelers to review and build models more effectively. It includes functionality for conducting checks on model components and visualizing the structure of a model through function networks.
How does the assertHR package help in reviewing models?
-The assertHR package helps in reviewing models by providing a series of checks that can be applied to model components, ensuring they meet certain criteria, and by visualizing the model's function network, which aids in understanding the relationships and flow between different parts of the model.
What is the role of the 'visualize project' function in the assertHR package?
-The 'visualize project' function in the assertHR package allows users to plot the function network of a model, which helps in understanding how different functions within the model are interconnected and how they contribute to the overall model structure.
How can the assertHR package be used to improve communication about a model's development?
-The assertHR package can be used to improve communication by generating summaries of model functions, identifying test coverage, and providing an interactive visualization of the model's structure. This makes it easier for stakeholders with varying levels of technical knowledge to understand the model.
What is the potential future direction for the assertHR package mentioned in the script?
-The potential future direction includes expanding the testing functionality, integrating a chat-bot for interactive questions about model functions, fine-tuning a large language model for better function summaries, and possibly submitting the package for peer review and publication.
How does the speaker address concerns about the limitations of the assertHR package in verifying model accuracy?
-The speaker acknowledges that the assertHR package is not a substitute for full verification checks and that it may not catch all errors, such as incorrect transition probabilities. However, it is intended to nudge modelers towards better testing practices and provide a starting point for identifying potential issues.
What is the current status of the assertHR package in terms of availability and contribution?
-The assertHR package is currently available as an open-source development version on GitHub. The speaker encourages contributions, testing on various models, and feedback to improve the package further.
Outlines
📈 Shifting the Modeling Pipeline to Script-Based Languages
The speaker begins by expressing gratitude for the contributions to the development of the CT H package, which is aimed at enhancing model review. The overarching goal is to transition the modeling pipeline from spreadsheet software like Excel to script-based programming languages, with a focus on improving communication, efficiency, adaptability, and future-proofing. The speaker highlights the importance of usability and communication in health economic models and the need to cater to diverse audiences with varying levels of technical knowledge. The presentation also touches on the use of shiny apps and markdown files for automation and the structure of cost-effectiveness model packaging in R.
🔍 Introducing the assertHR Package for Model Review and Development
The speaker discusses the challenges of reviewing complex models with nested functions and numerous scripts, drawing a comparison with the clarity of Excel files. To address this, the assertHR package has been developed to assist modelers in reviewing and building models. The package offers a series of checks to ensure the model's correctness, as well as functionality to visualize the structure of a model. The speaker emphasizes the benefits of having a centralized package for common checks, rather than each modeler writing their own, and introduces the 'visualize.project' function for plotting the function network of a model.
🛠️ Visualizing and Summarizing Model Structure with assertHR
The speaker elaborates on the visualization feature of the assertHR package, which allows users to plot the network of functions within a model, helping to understand the relationships between functions. The package scans the project folder, identifies functions, documentation, and corresponding tests, and presents this information in an HTML output. This visualization aids in quickly identifying untested functions and understanding the model's structure. The speaker also introduces the use of a large language model to generate summaries of individual functions, providing a layperson's understanding of what each function does.
📚 Case Studies and Future Developments of the assertHR Package
The speaker shares examples of case studies where the assertHR package has been applied, including models from GSK and others, and acknowledges the contributions and feedback from the community. They outline future plans for the package, such as expanding testing functionality, integrating a chat-bot for interactive questioning about functions, and potentially fine-tuning a large language model for better responses. The speaker also invites contributions and feedback on the open-source software and mentions a linked open-access publication for further information.
🤔 Addressing Concerns and Future Considerations for assertHR
In the final paragraph, the speaker addresses a concern raised about the limitations of the package in providing a false sense of security, as it does not replace full verification checks. They acknowledge that while the package aids in visualization and some testing, it cannot catch all errors. The speaker also dismisses the idea of making the visualization an R Studio add-in, stating that there are no current plans for such integration.
Mindmap
Keywords
💡CT H package
💡Modeling pipeline
💡Shiny apps
💡Markdown
💡Health economic models
💡Usability
💡Efficiency
💡Adaptability
💡Code coverage
💡Large language models
Highlights
Development of the CT H package for model review, aiming to shift from spreadsheet software to script-based programming languages.
Introduction of a package that allows for model quality checks and communication of the QC process.
The importance of model communication for various audiences, including modelers, clinicians, patients, and payers.
Key considerations for model development: usability, communication, efficiency, adaptability, and future-proofing.
The use of Shiny apps and markdown files for automating reporting in health economic models.
Overview of the assertHR package, designed to help modelers review and build models with a focus on quality checks.
The visualization of a model's function network to understand how functions fit together within the codebase.
The ability to generate a summary of any function using a large language model for better understanding.
Use of the 'visualize project' function to plot the function network and assess model structure.
The interactive nature of the HTML output, allowing users to explore function details and test locations.
Plans for future enhancements, including expanding testing functionality and integrating a chat-bot for function inquiries.
Open-source nature of the project, encouraging contributions and testing on various models.
Potential for fine-tuning a large language model with teaching material for improved function summaries.
The CT H package's role in nudging the industry towards better testing practices in health economic modeling.
The limitation that the package does not replace full verification checks and the need for clear communication of this.
No current plans to make the visualization an R Studio add-in, but openness to contributions for such development.
Transcripts
uh so everyone um thank you thank you
for coming um yeah I've been a bit busy
today so apologies if there's uh there's
a few uh last minute issues um okay so
first of all so we're going to talk
through the the CT H package uh which
we've been uh developing uh for our
model
review and wanted to say start by saying
uh thank you to to a lot of people who
contributed to this um either in kind of
contributing to the the manuscript which
is in progress and I'll share a a link
to the kind of draft on on the progress
uh in this presentation but also people
who who've directly um tested bugs and
things in the package and contributed to
the package
itself um so the overarching aim and I
guess the overarching aim here in
general and I I start a lot of
presentations with this slide is to
shift the the modeling pipeline uh from
spreadsheet software msxl to to script
based programming languages so from the
you know your survival analysis through
uh your say VBA code for a model
um to excel plots word outputs and
PowerPoint we want to shift that
pipeline that whole pipeline through to
a system where we are able to uh perform
our our stats work our model our Pops
and visuals our reports and our apps so
that's kind of like the big picture uh
kind of where can we get
to so uh through various Publications a
lot with uh with Paul and and well in
the past we've worked with how do you
build these shinies up shiny apps How do
automate some of this reporting in in
markdown files uh and then last year at
our HDA we presented on packaging of
cost effectance model so a structure by
which you can create a package for a
model uh in a set format and that kind
of um provides a structure and a
template for how to to encode uh models
in R um and that looks generally quite
similar to what most of the models that
I've see in in our and view INR uh end
up being like and so often we'll get
models where we have you know a big
folder with a load of functions a folder
with a load of tests Etc so we'll come
to that in a second so that's kind of
the the background of the the
introduction to where we're we're
starting
from yeah and um so before um Rob talks
about the the interesting stuff um I
wanted to give just a quick uh
perspective on why we're particularly
interested um in shifting towards R um
from a a farmer perspective so um for
any individual asset we work on we just
have a huge range of potential models uh
that might get built throughout the um
the assets development timeline um but
the thing I really want to focus on
there is not the breadth of the models
but the breadth of the audiences um that
we work with both during the development
phase and the output phase of the models
um you know we talk to modelers uh
clinicians patients payers uh people
with lots of technical knowledge people
with very little technical knowledge um
and one of the things we really value um
in those experiences is being able to
communicate the modeling process um so
you know when we build Health economic
models the whole idea behind it is that
we're building these models to uh
communicate a complicated clinical
pathway and um the impact and new
intervention might have on that pathway
um and so it's really important for us
um that we're able to communicate to the
right people at the right time um and
this just brings me on to guess what the
couple of the key considerations for us
so usability communication um just you
know the ability to develop intuitive
app style interfaces using things like
our shiny um and um you know ours focus
on data visual visualization sorry is um
is a huge bonus for us as we communicate
these ideas um the other two things
efficiency adaptability uh slightly more
more obvious obviously it's uh it's
great to have um an improvement in
computation time and you know as Rob
said pull everything under one package
rather than having to look at word excel
PowerPoint and kind of pull things in
and out of things um and then you know I
guess the last one touches on being
future proof um you know so we're
working um with a lot of large language
models these days and being able to
include those within Health economic
models is clearly going to be a future
direction for for modeling in general um
and so I'll hand back up I think he's
gonna uh talk through um uh this this
package of tools which hopefully will
help us not only QC models but also
communicate that um QC process which I
think is really
important thanks and thanks in
particular to to Tom and and YY who are
here who've been uh helping internally
to test some of what I'm going to talk
about on in some internal models so
we're testing it not only on some open
source models but some internal models
as well which has been really useful and
valuable so um you might imagine
somebody says can you review my model um
so it might be that you've got uh your
big our model and they go to an ERG or
uh it's a MTA and then you go to to
Industry and somebody needs to review
that that model and so those of you who
who have reviewed one of these quite
more complicated complex side of
modeling type models will know that you
end up with a whole ton of nested
functions it's very very hard you've got
a load of scripts and it's very very
hard to even start like where do you
start with this this process just
understanding where things fit together
once you get to the code you're kind of
you're right it's how do I even start to
approach this problem so quite often you
end up with you know tons of ask scripts
like what do I do with all this stuff
and this is probably one of well this is
one of the biggest criticisms of R
versus Excel because everybody opens an
Excel file and yes you've got a lot of
tabs but they're kind of Fairly clearly
labeled and you can kind of Click
through and see how things fit together
to some extent even if I think generally
it's it's
harder so often feel like this
so and then after panicking for a while
you kind of start moving through the
code trying to understand uh the code
base um but what would be really handy
to have is something that allows you to
help review that code and ensure it's
working as you you intend and as it as
it says in the
documentation so we built uh the assert
HR package and the idea of this is to
help modelers review uh and build
because obviously building a model you
should be kind of reviewing the model as
you're going along making sure it's it's
uh working as you intend uh you might
have internal reviewers in your your
organization and you might also want to
convey the model to other people and
explain it and so having tools that
allow you to do that are really useful
so there's two bits of functionality
that we uh that we really focus on so
first is a whole series of checks so
those of you that have built models in
our will know as you're writing your
code you'll often add in a line that's
just a check line after you built
something so you create a transition
probability Matrix and you then have a
line that says do all the rows of this
Su to one you create a trace do all the
rows some to one is this a numic is you
know is this the right type of object is
it the right dimensions all these little
tests that everybody writing
independently makes no sense whatsoever
what makes much much more sense is just
to have one package where we just have a
function which says check mark of Trace
check transition probability Matrix and
that's how this this um um kind of
project started was rather than
everybody writing their own checks let's
pull our resources we can all contribute
to this package which just then you just
install package check Trace check
transition proper to Matrix check that
run off PSAs all of that kind of thing
uh that everybody's kind of doing anyway
uh gets put on a package but then I got
really carried away and uh and then
number two started which is how to
summarize and visualize the structure of
a model so uh what this does is it
allows you to plot the function Network
um so any model in R if it's well
constructed should be a network of
functions so some functions are going to
call other functions um other functions
will be kind of Standalone um but often
you'll have nested functions where um
you have you know a whole chain of
functions so function one calls function
two which call function three which
calls function four and so understanding
how all those fit together uh is really
important so what we've done is built
this package which allows you to to pass
through your project folder for your
pack uh for your model and it will then
recognize it it will take a folder for
the functions and a folder for your
tests and it will will Loop through all
of the code and identify where the
functions are so exactly which line
they're specified on they're created um
and identify where the documentation for
that function is and identify if there's
a corresponding test in a test
folder and so as I said there's two
different ways to use this so at the
moment it's just on GitHub not Crown um
as a development version so you can just
do Dev tools install GitHub um and then
Library H and then somewhere in your
code so you've got all your model code
and then somewhere in your code you
might want to you know check that your
transition properties to array complies
to load rules and so if you look at the
that the documentation for that function
you'll see all of the the things that
we're checking there and so you can say
well I'm happy with this um this is what
you know lots of other people are using
so these these checks fine with me and
then it just Flags if there's any errors
so in this case uh there's an error with
this uh array and so it tells you the
transition probabilities are not valid
for these different Health States at
these different Cycles um so this has
been done before and it's just an
adaptation on on on all work um but
probably the more interesting bit is
what I'm going to talk about for most of
the rest of the session U which is to
visualize a network of functions and
there's a single function that you can
call here called visualize
project and what you have to give that
function is a project path so this is
the project directory top level where
are the functions the functions are in
here the r folder
where are the tests the tests are in the
test test that folder you could have
them in somewhere else you could have a
function folder called functions or a
test folder called tests whatever you
want to call it uh and then you just
need to uh say whether you want to run
coverage of of of functions to see what
the code coverage is like on those
functions so those are different kind of
arguments to the um the
function uh as I said there's a lot so
on on the first side there's there's a
lot of these different checks and
they're all kind of included in
documentation um yeah like they're
useful everyone's writing them don't
think that's super super interesting but
it's something that as a group I really
strongly believe that we should all be
pulling uh together so if you do want to
contribute a check that you're using
internally in your organization and
you're just you know you're using it
routinely uh then please uh send me a
pull request or send the code and we can
build it in and So eventually we'll have
a whole Lo of these checks for different
types of models So currently we just
have uh markup models that we could have
you know checks for um PSM for um b um
you know any type of model uh all in one
place so that everyone's kind of
validated these this bit of
code but probably more useful is is the
model review so the visualization so
this is what uh gets spit out in HTML
from the um from from the reviewer
function and so this is the Run six
model and the Run six model has five sub
functions so this the function in the
middle calls those five so the arrow
denotes that that function exists or is
called by the the function which the
arrow is pointing to and this is a very
simple model so these five functions are
called sequentially so there's a parent
and then five children and these two
functions here are not called by run six
model they used somewhere else in the
code base um but what we can see is that
that very quickly that those um six
functions in the middle all have test
associated with them so it's gone
through and it's recognized these are
our functions um these are the names and
these ones have tests they denoted in
green uh the one that's orange uh is
orange because has a test but its code
coverage is between 20 and 80% and the
one in red doesn't have a test so we can
very quickly see were our functions um
and do they have tests and how they how
do they fit together now obviously
that's a very simple model um but we can
click on any one of the nodes so any of
our functions and we see the function
name we see where the function is
located in our code base we see where
the test is located and we see the
coverage so that's kind of a like
summary of every constituent part of the
model within this HTML um output we can
create new functions so just for
discussion purposes so say we had a
meeting internally and we run this
visualizer we get this output for our
model and we say right we want to build
in this new part of the model or this
new algorithm into the model we can
construct our own nodes just in this
visualizer as a communication atol to
say you know now we're going to build
this new function that does this new
thing and that gets called by an
existing function and that that's our
kind of plan going forward so it's also
useful as a kind of planning tool for
your your modeling
work but what's um kind of really
interesting from a future proofing
perspective is we can generate a summary
of any of these these uh these functions
so the I symbol basically takes
you yes so the I symbol basically takes
you uh produces a popup which displays
the function code uh the uh this symbol
basically takes you to our studio to
that location the function as you expect
but this robot symbol basically uh it
generates a large language model summary
of the function so as um as we talked
about already large language models is
not very good if you provide thousands
of lens of code uh and tell it to
summarize it or tell it to talk about it
but if you provide it five lenses of
code suddenly it gets a lot better so if
you have a model that's built up of lots
of small algorithms and you send those
to be summarized suddenly uh you get a
much better uh output and result so
rather than getting you know sending
your entire model to be summarized
what's going on we just we want to
understand in Lay person summary what
does this one function do and so then it
it generates a a two paragraph summary
so just a short bit of text summarizing
what this function
does and the way this works is is that
when that button pressed the function
data is extracted so we take um the body
of the function the arguments and the RO
code store that all data uh create that
like pass that all together into a
prompt and then send that VI an API
request uh at the moment to open AI um
and receive a response reform format it
and then show it in the the
mod so it looks like this so from your
um clicking that button you then just
get this this short summary
um and so we've run this as a few case
studies um so we've done the nice RCC
model so thank you to to D and Darren
that we heard earlier uh first
presentation um we've done the six s
pack teaching model that we we published
on a couple of times um we've done the
cdx2 ca model Fernando published back in
2022 uh which is an our package uh and
and several others described there uh
and then um the GSK team have been doing
some internally and several of you have
contacted me and thank you very much for
implementing those internally and and
telling me about all the bugs in this
software uh it's been really handy and
for those of you have sent PO requests
um to to help improve it that's
fantastic um so the code for this is is
completely open source so here's an
example of one this is the the CDX to
biomarking biomarker testing um output
so you can see the the function network
is actually slightly bigger than that
and then a summary of one of the
functions generated this is the um uh
the the cheff micro simulation model um
that's built by D poad so this is kind
of what that looks like and then you can
see a summary here of where the function
is function location test location and
this is is is fully interactive um and
the same for the nice RCC Pathways pilot
so use coup more
minutes I don't know where I put it I'm
just trying to find
the okay that's
fine so yeah if you want um I can send
some some examples so render HTML output
um and so what that means is that you
can then share that visualization via
email to whoever you want to to share it
with um so in terms of next steps
um we welcome contributions testing the
software on your own models the software
is fully open source so you can just
download it from from GitHub and test it
on your models what we'd request is if
you do use it please let us know how it
went either say fine no problems or um
tell us what the the issues are so we
can help kind of improve this going
forward um we're going to expand some of
the the the testing functionality that
it does provide and also integrate so at
the moment uh it returns a two paragraph
summary of the of each function uh when
you click on there what we could do is
build in sort of a almost like a chat
bot so you can ask questions about the
function and and respond back and
forward which really cool uh and then
also we're looking at whether we can
fine-tune a l large language model
because we have a lot of teaching
material uh and so we can take that
teaching material and the code and then
fine-tune the large language model so
that it gives a better response than the
the model just developed by by open AI
um or you'd Hope anyway uh and then
finally uh submission at some
point so if you are interested in in
having a look and commenting or you do
have any comments I'm Consciousness late
and we want to get to the fob have a
look at the the Open Access publication
that's Linked In it's linked here and on
GitHub um and and just let us know feel
free to make comments on
there yeah
[Applause]
conscious that obviously everybody wants
to go to Pub so probably will limit this
to two questions so fingers
first have we got who wants to ask
one go ahead Rob so obviously love the
package the unit testing visualization
is great but my main concern would be
giving users a FSE sense of security
because this wouldn't pick this is not
equivalent to a full verification check
so it wouldn't pick up if the wrong
transition probabilities have been
pushed in the wrong place or the
treatment are being commuted
sorts of things so there could be
something massively wrong with the
model 100% pass so I think that caveat
needs to be much clear communting this
to absolutely yeah no so I mean if your
code is wrong it's still wrong right
like this is for visualizing the model
um it won't it won't necessarily Find
the Errors I guess what I would say is
as an industry we probably aren't very
good at testing our own code and like
people from software development would
probably come in and look at Health
econic models and be like like you guys
aren't doing a hell of a lot of testing
and so maybe this will nudge people
towards at least writing some tests even
if the tests don't pick up everything at
least there's some um Le a start so yeah
um do we have any questions on the
chat that one
yeah well that's brutal
from thanks John
okay so are are there any plans to make
your visualization in R Studio add in no
no I I don't think yeah to be rooted
honest
no I think that'd be great if I asked
you to call me up and want me to come
come come work on it then then great but
no not
currently yeah
Weitere ähnliche Videos ansehen
10-Minute Tutorial: Patient-Level Prediction or "PLP" (Jenna Reps)
Google NotebookLM in italiano! Guida completa alla super AI
Shannon Weaver Model of Communication
Create Anything with LLAMA 3.1 Agents - Powered by Groq API
Lesson 2: Models of Communication | Oral Communication in Context
Fine-Tune Your Own Tiny-Llama on Custom Dataset
5.0 / 5 (0 votes)