Systematic reviews in Elicit | Screening & extraction
Summary
TLDRThe video script presents a comprehensive guide on utilizing the Elicit AI tool for streamlining the screening and data extraction phases of systematic reviews, rapid reviews, scoping reviews, and meta-analyses. It showcases Elicit's capabilities, such as AI-powered data extraction from PDFs and tables, custom column creation, filtering, and high-accuracy modes. The script highlights Elicit's potential to save time, increase accuracy, and facilitate a more systematic approach to literature reviews. It also emphasizes the importance of reviewing Elicit's work and offers insights from internal testing, demonstrating improved accuracy compared to manual extraction by trained research staff.
Takeaways
- ๐ Elicit is a tool that uses AI and language models to assist with data extraction and screening for systematic reviews, rapid reviews, scoping reviews, meta-analyses, and literature reviews.
- ๐ Elicit can extract data from PDFs and tables, a unique feature compared to other AI tools.
- ๐ผ While Elicit automates some tasks, users should still carefully review its work and integrate it thoughtfully into their workflows.
- ๐ญ Elicit has been shown to achieve higher accuracy than trained research assistants in identifying relevant papers and extracting data.
- ๐ The tool allows users to filter and sort extracted data based on custom criteria and formatting.
- ๐พ Users can download extracted data as a CSV file for further review and annotation.
- ๐ค Elicit offers a high accuracy mode for improved precision in data extraction, at a higher computational cost.
- ๐ต The tool keeps track of the user's work and progress, allowing them to pick up from where they left off.
- ๐ซ Elicit recommends uploading fewer than 100 papers at a time to avoid performance issues.
- ๐ฌ The team behind Elicit offers best practices, tips, and unreleased features for systematic review projects.
Q & A
What is Elicit, and who co-founded it?
-Elicit is a tool that uses AI, specifically generative AI and language models, for screening and data extraction in systematic reviews and similar projects. It was co-founded by Jung Juan.
How does Elicit aim to assist researchers?
-Elicit aims to save researchers time by automating the data extraction process from PDFs, freeing them to focus on synthesizing information and critical thinking, rather than manual copying and pasting.
What types of reviews and analyses can Elicit be used for?
-Elicit can be used for systematic reviews, rapid reviews, scoping reviews, meta-analyses, or any project requiring a systematic approach to literature review.
Can Elicit handle the extraction of data from tables in PDFs?
-Yes, Elicit has a unique feature that allows for the extraction of data from tables in PDFs, which is important for research.
What is the recommended limit for the number of papers to process at once in Elicit?
-It is recommended to stay under about 100 papers at a time for data extraction to avoid slowing down the app.
How can users upload papers to Elicit?
-Users can upload papers by dragging and dropping PDFs, selecting multiple PDFs from their file picker, or uploading papers from Zotero.
What does Elicit do when it's not confident in its data extraction accuracy?
-When Elicit is not confident about its data extraction accuracy, it flags the data so users can double-check its work.
How does Elicit ensure the privacy of the papers uploaded into its system?
-Papers uploaded into Elicit remain entirely private to the user; they are not shared with anyone else or uploaded for public access.
What advantage does Elicit claim over manual data extraction by trained research staff?
-Elicit claims to have better accuracy in identifying relevant papers and extracting data compared to manual extraction by trained research staff, with internal testing showing higher retrieval rates and accuracy.
What should users do if they are about to embark on a serious review project using Elicit?
-Users planning to undertake a serious review project using Elicit are encouraged to contact the Elicit team for best practices, tips, and information on batch jobs outside of the app for time-saving.
Outlines
📹 Introduction to Elicit for Systematic Reviews
This paragraph introduces Elicit, a tool that uses AI and language models to assist with the screening and data extraction steps of systematic reviews, rapid reviews, scoping reviews, meta-analyses, and literature reviews. The speaker explains that while Elicit can save a significant amount of time, its work should be carefully reviewed as the technology is still in its early stages. They claim that Elicit's accuracy often surpasses that of trained research assistants in internal testing.
🔍 Screening and Filtering Papers in Elicit
This paragraph demonstrates how to screen and filter papers in Elicit based on specific criteria. The speaker shows how to extract data related to population characteristics, age, region, and create custom columns for specific formatting needs. They explain how to filter the results based on these extracted data points, download the data as a CSV file for further review, and make notes or corrections as needed. The speaker also discusses the benefits of using high accuracy mode, albeit at a higher cost, for improved extraction accuracy.
📊 Accuracy, Privacy, and Best Practices in Elicit
In this paragraph, the speaker shares some test results comparing Elicit's accuracy to that of trained research assistants in identifying relevant papers and extracting data. They claim that Elicit outperformed human researchers in both scenarios, often being 13-26% more accurate. The speaker also mentions that Elicit saves user work and provides privacy for uploaded papers. They encourage users to reach out for best practices, new features, and evaluation assistance for systematic reviews.
Mindmap
Keywords
💡Systematic review
💡Data extraction
💡Screening
💡AI-assisted review
💡High accuracy mode
💡Custom columns
💡Filtering
💡CSV export
💡Accuracy testing
💡Confidence flagging
Highlights
Elicit uses AI and language models to automate data extraction from PDFs for systematic reviews, rapid reviews, scoping reviews, meta-analyses, and literature reviews.
Elicit can extract data from tables in PDFs, a unique feature compared to other AI tools.
Screening workflow: Use predefined or custom columns to extract relevant information (e.g., population characteristics, age, region, continent) from papers and filter based on inclusion criteria.
Download extracted data as a CSV file for further review, editing, and notes.
For screening a large number of papers, adding columns is the most cost-effective approach. For data extraction, enable high accuracy mode for better results, especially when extracting from tables.
Papers uploaded to Elicit are private and not shared with other users.
Elicit retrieved 96% of relevant papers compared to 92% by trained research assistants in a 5,000-paper screening test.
In a data extraction test, Elicit achieved 98% accuracy compared to 72% by trained team members.
Elicit was 13-26% more accurate than manual approaches across various data fields.
Work is saved in the sidebar and can be accessed later without additional cost.
For systematic review projects, users are encouraged to reach out to the Elicit team for best practices, tips, and unreleased features.
Elicit team also works with teams to evaluate Elicit for systematic reviews.
High accuracy mode makes about half the error compared to regular mode but is more expensive.
Papers can be uploaded via PDF or by connecting to a Zotero integration.
When Elicit is not confident about its answer, it flags the result for the user to double-check.
Transcripts
hi I'm Jung Juan one of the co-founders
of elicit and today I want to show you
how you can use elicit for the screening
and data extraction steps of projects
like systematic reviews uh as well as
any of the kind of similar versions of
systematic reviews like rapid reviews
scoping reviews meta analyses um and
even if you just want to take a more
systematic approach to your literature
review I'm hoping that some of the
features and workflows I show you today
can really help you save a ton of time
and free you up to do uh to spend more
of your research hours on synthesizing
the information or or thinking more
critically about all of it instead of
copying and pasting data from PDFs as
you may know elicit uses um AI uh
generative Ai and language models to do
a lot of this data extraction work these
are still pretty early Technologies so
you should expect to spend quite a bit
of time reviewing all of it List's work
by no means is this like an automate and
forget it type of um uh experience like
you should you should be pretty
thoughtful about how you're integrating
these into your into your workflows and
um be careful to check elicits work that
being said we have done a decent amount
of internal testing comparing elissa's
accuracy to the accuracy of trained
research staff research assistants
manual data extraction and in a lot of
cases we really are beating a lot of uh
human accuracy um so I think it's really
promising and um you know all of this
extraction work takes a lot of time so
I'm hoping we can find good ways for
Alysa to augment you and and kind of
Accel all the work that you're
doing so the main workflow I'll be
focused on today is this extract data
from PDF's workflow um I have a bunch of
papers uploaded into my library already
if you click upload papers there's a way
to drag and drop PDFs you can drop a
bunch of them at once you don't have to
drop uh add one PDF at a time you can
select a bunch from your uh file picker
and upload many at once um you can also
go directly to your
library which you can find in your
sidebar and upload papers here or upload
papers from zoto we have another uh
video that will show you how you can do
that um so here I have about 39 papers
they're they're not in the same domain
so it's pretty unlikely that you would
ever do a review of a a group of papers
as diverse as this but these are the
papers that I have so I'll just use them
to showcase the features I'm going to
select all of them there's 40 there's
about
39 we typically recommend St under about
a 100 at a time if you are extracting
data from about 100 papers and
extracting lots of data it can
definitely the app starts to get a
little bit slow so starting with with
smaller numbers if you can um is a is is
a good best practice if you are about to
embark on a pretty serious review
project reach out to our team you can
email us at info@ elicit
docomo have ways of um running kind of
batch jobs outside of the app that can
be it can save you a lot of time be a
lot easier
so here you can see all the papers have
loaded in and now if you click into a
paper um you can see all the text here
and then you can also see that we
extract tables this is a feature that's
very unique to elicit really important
for research obviously but no other AI
tool has this um so you can use data
from tables as well and extract the data
in the contents of
tables um so I'll just go through an
example of how you might screen down all
of these uh papers so there's a lot here
you don't know exactly how many are
relevant and presumably you have some
criteria by which you're determining
whether a paper is relevant to your
review or not um so let's say for
example that criteria is population
based uh you have a bunch of columns
here that you can you can um use to
understand uh extract data from the
papers and understand more about what
the papers did um so if your if your
inclusion criteria are kind of
population Focus you can start with a
column like population characteristics
this is a pretty open-ended column it'll
just give you information about all the
different populations uh discussed in
the papers you can see there's a lot of
content in
here and as with in elicit if you click
on an answer you can always see uh the
sources and where the information came
from so this is a really great way to
check elicits work um and you can see
the most relevant quotes here and you
can tap through a bunch of them and then
you can also open the paper and see the
information in
context so that's say um I first start
I'm not exactly sure what I'm looking
for so I'm going to start with a kind of
open-ended column population
characteristics more generally um and
then I realize okay you know I should uh
if you have specific inclusion criteria
you might be able to skip some of these
steps but I'll just kind of go step by
step um just to make the point so now um
I think I'm noticing that the you know
populations differ along many dimensions
right there's age mentioned here gender
um region and certainly like lots of
other details uh so I might want to
drill down a little bit deeper and maybe
I'll ask specifically about participant
age that's another column that I can add
again all of those columns um are added
here when um elicit is not confident
about its answer it'll throw this flag
so you can double check so it's possible
it is mentioned in this paper um and
this uh this column just didn't pick up
on that so you might want to come back
and review that more carefully later and
again you can check click on the to um
see to double check list it's work so I
have maybe you know specifically a
specific field for age um uh I might
also want to do a field for uh region
maybe and then I'm getting a bunch of
different regions um and I'm noticing
that the regions are kind of on
different levels of granularity so maybe
I want to you know ultimately I kind of
want to you know include some papers and
exclude some paper so I want the
formatting to be a little bit consistent
um so I can ask my a custom uh question
here and and kind of extract fields in a
very custom way um so I can ask
something like what was the continent
where the study took
place um and I'll give instructions to
specifically follow to follow a specific
format so answer as one of uh
America
America go through the continents
here and I don't know if we'll have
Antarctica in any of these papers but
I'll do that in the interest of
completeness so now continent for
example was not uh one of our predefined
columns um but uh I was able to create a
custom column for my specific use case
give it formatting Direction so that I
could get a specific type of answer and
now from here I can filter by the
results of this column so I can just
filter for paper
that took place in
Africa um I can filter for and and maybe
Asia so I can include both Africa and
Asia and then if I delete these I will
you know see the full results
again um this column filter is a keyword
match so you do need to make sure that
the contents of the cell have the
keyword that you're filtering
by um let's see so that's a great way so
let's say if you had a region criteria
that you were screening by you can
extract the data format it and filter by
that
data um next in the screening process um
you might want to download this as a CSV
so that you can kind of indicate which
papers you've already reviewed or maybe
as you're going through the kind of
reviewing the quotes and reviewing the
listed work there's additional context
you're picking up on that you might want
to note in your um in your review
process so then you can download CSV
that's pretty pretty straightforward um
open it up in you know spreadsheets
Excel whatever is easiest for
you uh and then you'll get a spreadsheet
like this with the title the authors a
bunch of metadata uh each
column uh that you extracted here
population characteristics age region
continent where the study took place all
of the supporting quotes that we found
in the paper um as well as some
reasoning if that's if that ends up
being helpful as well as reasoning
basically so in cases where um unist it
might say not applicable or not
mentioned um will'll also kind of
explain the reasoning why that might be
the case we can you know share related
quotes um even if they don't directly
answer your question so from here you
might want to let's say um add a column
like reviewed or something and then say
reviewed or in progress um you can as
you go through you might want to check
elicit answers maybe you know if you
find that it's not 19 individuals I mean
or maybe I don't know if any of these
end up being you you want to add more
context you can add more context or make
Corrections um and basically
spreadsheets are going to for now
spreadsheets are are you know probably
going to be where you want to make more
edits um or directly make your notes
over time we definitely want to make
that more native to ELA but that's quite
complicated so right now spreadsheets
are probably better for you obviously
you can do some filtering and sorting
and spreadsheets as
well um a couple notes uh about how this
works
so um I think for screening if you're
going to do a lot of papers it's best to
uh this just adding columns is the
cheapest way um you'll definitely get
higher accuracy if you turn on high
accuracy mode you can do that by
clicking on any of these kind of
Bullseye buttons here um or over here or
by toggling this high accuracy mode here
um high accuracy mode um is uh is like
makes about half the error as kind of
regular mode but is also quite a bit
more expensive um so this probably makes
more sense for something like data
extraction or when you get into the
later uh stages of the process um for
screening if you're doing it for a large
number of papers you might want to do
like a rougher first
pass so extraction would work pretty
similarly um again I think the only
difference would be you you are likely
going to want to run high in high
accuracy mode when you get to the
extraction step so you can just turn
that on um once you do that you'll start
to be able to use information from the
tables as well um so for example when
you click on the source quotes you'll
start to see that it might be extracting
data from tables so if you need detailed
effect sizes um or other dimensions that
are mentioned mostly in the tables
you'll you'll want to run things in high
accuracy
mode so that's kind of the workflow for
screening and extraction um so again the
kind of assumption here is that you have
found your papers through whatever
search methodology that you've set out
it can you could have found your papers
in the lit or you might have found them
from other sources you can upload them
into elicit when you do that they're not
going to be shared with anyone else so
your papers will be entirely private to
you um and you know they're not going to
get uploaded no other users will see
them it's not a means of publishing
papers um it's just a way for you to
speed up your data analysis um you can
upload by PDF or by connecting users
hero integration Again by going back to
your library
and then you can select the papers that
you are most interested in um or select
the papers that are relevant and uh kind
of extract data to screen or uh or you
know PR prep for a meta analysis or some
other kind of data analysis I can show
you just really quickly some of the
testing that we've done with different
teams working on systematic reviews uh
there was one team that was trying to
screen about 5,000 papers we compared
our approach to some work that they had
done manually and we were actually able
to retrieve over 96% of all of the
papers that they consider to be relevant
um the kind of human research assistance
that they had trained only achieved
about 92% so elicit ability to identify
Rel papers was higher than the um than
delegating it to a bunch of um trained
research assistants and it was obviously
significantly cheaper significantly
faster much more Dynamic uh and the same
thing with data extraction again working
with a team that um was doing a lot of
extraction man manually um and we found
that elicit was about had about 98%
accuracy whereas a lot of the trained
members um or only 72% accurate um this
was especially true when elicit was
pretty confident so and when it wasn't
confident elicit would throw the flag so
that the team knew how to double check
its work so in a lot of cases there was
some disagreement between elicit and um
the manual approach um and when when the
teams kind of took a second look at
those answers it turned out Alyssa was
more accurate um so
uh yeah and I think just kind of um that
was yes that's kind of the overall
accuracy and then in general When
comparing to a lot of the um manual
approaches Alissa was often 13 to 26%
more accurate um for like an array of
different fields and data fields and um
and different
columns um another benefit is that you
know we're going to save all of your
work here in the sidebar so you can
always go back and pick up from where
you left off um it doesn't cost any
credits to reopen this View and then if
you continue and add more columns that
will cost credits if you're interested
in doing a systematic review with elicit
please reach out to us we would love to
give you best practices and tips uh as
well as a bunch of new features that we
might not not have released publicly yet
um and if you're generally interested in
evaluating elicit a systematically so
that it can be used for more systematic
reviews let us know we also work with
teams to do that type of work as well
thank
you
Ver Más Videos Relacionados
Summary Measures Used in Systematic Reviews
Understanding 'Levels of Evidence' - What are Levels of Evidence?
[Pengantar] Systematic Review dengan PRISMA Protocol
Solve Complex Fuzzy Name Variations with WinPure AI Data Match | 05 Minute Guide
Best Way to Make Business | Market Research | Marketing 2.0
Learn 80% of Perplexity in under 10 minutes!
5.0 / 5 (0 votes)