Systematic reviews in Elicit | Screening & extraction

Summary

The video script presents a comprehensive guide on utilizing the Elicit AI tool for streamlining the screening and data extraction phases of systematic reviews, rapid reviews, scoping reviews, and meta-analyses. It showcases Elicit's capabilities, such as AI-powered data extraction from PDFs and tables, custom column creation, filtering, and high-accuracy modes. The script highlights Elicit's potential to save time, increase accuracy, and facilitate a more systematic approach to literature reviews. It also emphasizes the importance of reviewing Elicit's work and offers insights from internal testing, demonstrating improved accuracy compared to manual extraction by trained research staff.

Takeaways

• ๐ Elicit is a tool that uses AI and language models to assist with data extraction and screening for systematic reviews, rapid reviews, scoping reviews, meta-analyses, and literature reviews.
• ๐ Elicit can extract data from PDFs and tables, a unique feature compared to other AI tools.
• ๐ผ While Elicit automates some tasks, users should still carefully review its work and integrate it thoughtfully into their workflows.
• ๐ญ Elicit has been shown to achieve higher accuracy than trained research assistants in identifying relevant papers and extracting data.
• ๐ The tool allows users to filter and sort extracted data based on custom criteria and formatting.
• ๐พ Users can download extracted data as a CSV file for further review and annotation.
• ๐ค Elicit offers a high accuracy mode for improved precision in data extraction, at a higher computational cost.
• ๐ต The tool keeps track of the user's work and progress, allowing them to pick up from where they left off.
• ๐ซ Elicit recommends uploading fewer than 100 papers at a time to avoid performance issues.
• ๐ฌ The team behind Elicit offers best practices, tips, and unreleased features for systematic review projects.

Q & A

What is Elicit, and who co-founded it?

Elicit is a tool that uses AI, specifically generative AI and language models, for screening and data extraction in systematic reviews and similar projects. It was co-founded by Jung Juan.

How does Elicit aim to assist researchers?

Elicit aims to save researchers time by automating the data extraction process from PDFs, freeing them to focus on synthesizing information and critical thinking, rather than manual copying and pasting.

What types of reviews and analyses can Elicit be used for?

Elicit can be used for systematic reviews, rapid reviews, scoping reviews, meta-analyses, or any project requiring a systematic approach to literature review.

Can Elicit handle the extraction of data from tables in PDFs?

Yes, Elicit has a unique feature that allows for the extraction of data from tables in PDFs, which is important for research.

What is the recommended limit for the number of papers to process at once in Elicit?

It is recommended to stay under about 100 papers at a time for data extraction to avoid slowing down the app.

How can users upload papers to Elicit?

Users can upload papers by dragging and dropping PDFs, selecting multiple PDFs from their file picker, or uploading papers from Zotero.

What does Elicit do when it's not confident in its data extraction accuracy?

When Elicit is not confident about its data extraction accuracy, it flags the data so users can double-check its work.

How does Elicit ensure the privacy of the papers uploaded into its system?

Papers uploaded into Elicit remain entirely private to the user; they are not shared with anyone else or uploaded for public access.

What advantage does Elicit claim over manual data extraction by trained research staff?

Elicit claims to have better accuracy in identifying relevant papers and extracting data compared to manual extraction by trained research staff, with internal testing showing higher retrieval rates and accuracy.

What should users do if they are about to embark on a serious review project using Elicit?

Users planning to undertake a serious review project using Elicit are encouraged to contact the Elicit team for best practices, tips, and information on batch jobs outside of the app for time-saving.

Outlines

00:00

📹 Introduction to Elicit for Systematic Reviews

This paragraph introduces Elicit, a tool that uses AI and language models to assist with the screening and data extraction steps of systematic reviews, rapid reviews, scoping reviews, meta-analyses, and literature reviews. The speaker explains that while Elicit can save a significant amount of time, its work should be carefully reviewed as the technology is still in its early stages. They claim that Elicit's accuracy often surpasses that of trained research assistants in internal testing.

05:01

🔍 Screening and Filtering Papers in Elicit

This paragraph demonstrates how to screen and filter papers in Elicit based on specific criteria. The speaker shows how to extract data related to population characteristics, age, region, and create custom columns for specific formatting needs. They explain how to filter the results based on these extracted data points, download the data as a CSV file for further review, and make notes or corrections as needed. The speaker also discusses the benefits of using high accuracy mode, albeit at a higher cost, for improved extraction accuracy.

10:02

📊 Accuracy, Privacy, and Best Practices in Elicit

In this paragraph, the speaker shares some test results comparing Elicit's accuracy to that of trained research assistants in identifying relevant papers and extracting data. They claim that Elicit outperformed human researchers in both scenarios, often being 13-26% more accurate. The speaker also mentions that Elicit saves user work and provides privacy for uploaded papers. They encourage users to reach out for best practices, new features, and evaluation assistance for systematic reviews.

Keywords

💡Systematic review

A systematic review is a type of literature review that aims to identify, evaluate, and synthesize all available research relevant to a particular research question or topic. The transcript emphasizes how Elicit can be used for screening and data extraction steps involved in systematic reviews, as well as similar types of reviews like rapid reviews, scoping reviews, and meta-analyses. For example, "today I want to show you, how you can use elicit for the screening, and data extraction steps of projects, like systematic reviews uh as well as, any of the kind of similar versions of, systematic reviews like rapid reviews, scoping reviews meta analyses."

💡Data extraction

Data extraction refers to the process of extracting relevant data or information from research papers or studies to be used in a systematic review or meta-analysis. The transcript highlights Elicit's capability to extract data from PDFs and tables using AI and language models. For instance, "elicit uses um AI uh, generative Ai and language models to do, a lot of this data extraction work" and "you can use data, from tables as well and extract the data, in the contents of, tables."

💡Screening

Screening is the process of evaluating research papers or studies to determine their relevance and inclusion in a systematic review or meta-analysis based on predefined criteria. The transcript demonstrates how Elicit can be used to screen papers by extracting data on specific criteria such as population characteristics, age, and region, and then filtering the papers based on those criteria. For example, "so here you can see all the papers have, loaded in and now if you click into a, paper um you can see all the text here, and then you can also see that we, extract tables this is a feature that's, very unique to elicit really important, for research obviously but no other AI, tool has this um so you can use data, from tables as well and extract the data, in the contents of, tables."

💡AI-assisted review

AI-assisted review refers to the use of artificial intelligence (AI) and machine learning technologies to assist in the process of conducting systematic reviews or literature reviews. The transcript emphasizes how Elicit utilizes generative AI and language models to automate and accelerate tasks such as data extraction and screening, while still requiring human oversight and validation. For example, "these, are still pretty early Technologies so, you should expect to spend quite a bit, of time reviewing all of it List's work, by no means is this like an automate and, forget it type of um uh experience like, you should you should be pretty, thoughtful about how you're integrating, these into your into your workflows and, um be careful to check elicits work."

💡High accuracy mode

High accuracy mode is a setting or feature in Elicit that aims to increase the accuracy of the data extraction process by employing more computationally intensive algorithms or models. The transcript suggests that high accuracy mode should be used for data extraction or later stages of the review process when higher precision is required, but it may be more resource-intensive. For example, "high accuracy mode um is uh is like, makes about half the error as kind of, regular mode but is also quite a bit, more expensive um so this probably makes, more sense for something like data, extraction or when you get into the, later uh stages of the process."

💡Custom columns

Custom columns refer to the ability to create and extract specific data fields or variables that are not part of Elicit's predefined set of columns. The transcript demonstrates how users can create custom columns by asking specific questions and providing formatting instructions to extract data in a desired format, such as extracting the continent where a study took place. This allows users to tailor the data extraction process to their specific research needs. For example, "so I can ask my a custom uh question, here and and kind of extract fields in a, very custom way um so I can ask, something like what was the continent, where the study took, place um and I'll give instructions to, specifically follow to follow a specific, format."

💡Filtering

Filtering refers to the process of selecting or excluding papers or studies based on certain criteria or extracted data fields. The transcript demonstrates how Elicit allows users to filter the list of papers based on the values in extracted columns, such as filtering for papers that took place in specific continents or regions. This is an essential step in the screening process of a systematic review. For example, "so now continent for, example was not uh one of our predefined, columns um but uh I was able to create a, custom column for my specific use case, give it formatting Direction so that I, could get a specific type of answer and, now from here I can filter by the, results of this column so I can just, filter for paper, that took place in, Africa um I can filter for and and maybe, Asia so I can include both Africa and, Asia."

💡CSV export

CSV (Comma-Separated Values) export refers to the ability to download the extracted data and metadata from Elicit into a CSV file format, which can be opened and edited in spreadsheet software like Excel or Google Sheets. The transcript highlights this feature as a way for users to review and make additional notes or corrections on the extracted data during the review process. For example, "next in the screening process um, you might want to download this as a CSV, so that you can kind of indicate which, papers you've already reviewed or maybe, as you're going through the kind of, reviewing the quotes and reviewing the, listed work there's additional context, you're picking up on that you might want, to note in your um in your review, process."

💡Accuracy testing

Accuracy testing refers to the process of evaluating and comparing the accuracy of Elicit's data extraction and screening capabilities against manual approaches or human research assistants. The transcript provides examples of internal testing conducted by Elicit, which showed that Elicit achieved higher accuracy rates (96% for screening and 98% for data extraction) compared to trained research assistants (92% and 72%, respectively). This highlights the potential benefits of using AI-assisted tools like Elicit in systematic review processes. For example, "there was one team that was trying to, screen about 5,000 papers we compared, our approach to some work that they had, done manually and we were actually able, to retrieve over 96% of all of the, papers that they consider to be relevant, um the kind of human research assistance, that they had trained only achieved, about 92% so elicit ability to identify, Rel papers was higher than the um than, delegating it to a bunch of um trained, research assistants."

💡Confidence flagging

Confidence flagging refers to Elicit's ability to indicate when it is not confident about its answer or data extraction, allowing users to double-check and validate those instances. The transcript mentions that when Elicit is not confident, it will throw a flag, prompting users to review that particular answer or data point more carefully. This feature helps ensure that users are aware of potential inaccuracies and can take appropriate action. For example, "again all of those columns um are added, here when um elicit is not confident, about its answer it'll throw this flag, so you can double check so it's possible, it is mentioned in this paper um and, this uh this column just didn't pick up, on that so you might want to come back, and review that more carefully later."

Highlights

Elicit uses AI and language models to automate data extraction from PDFs for systematic reviews, rapid reviews, scoping reviews, meta-analyses, and literature reviews.

Elicit can extract data from tables in PDFs, a unique feature compared to other AI tools.

Screening workflow: Use predefined or custom columns to extract relevant information (e.g., population characteristics, age, region, continent) from papers and filter based on inclusion criteria.

Download extracted data as a CSV file for further review, editing, and notes.

For screening a large number of papers, adding columns is the most cost-effective approach. For data extraction, enable high accuracy mode for better results, especially when extracting from tables.

Papers uploaded to Elicit are private and not shared with other users.

Elicit retrieved 96% of relevant papers compared to 92% by trained research assistants in a 5,000-paper screening test.

In a data extraction test, Elicit achieved 98% accuracy compared to 72% by trained team members.

Elicit was 13-26% more accurate than manual approaches across various data fields.

Work is saved in the sidebar and can be accessed later without additional cost.

For systematic review projects, users are encouraged to reach out to the Elicit team for best practices, tips, and unreleased features.

Elicit team also works with teams to evaluate Elicit for systematic reviews.

High accuracy mode makes about half the error compared to regular mode but is more expensive.

Papers can be uploaded via PDF or by connecting to a Zotero integration.

When Elicit is not confident about its answer, it flags the result for the user to double-check.

Transcripts

hi I'm Jung Juan one of the co-founders

of elicit and today I want to show you

how you can use elicit for the screening

and data extraction steps of projects

like systematic reviews uh as well as

any of the kind of similar versions of

systematic reviews like rapid reviews

scoping reviews meta analyses um and

even if you just want to take a more

systematic approach to your literature

review I'm hoping that some of the

features and workflows I show you today

can really help you save a ton of time

and free you up to do uh to spend more

of your research hours on synthesizing

the information or or thinking more

critically about all of it instead of

copying and pasting data from PDFs as

you may know elicit uses um AI uh

generative Ai and language models to do

a lot of this data extraction work these

are still pretty early Technologies so

you should expect to spend quite a bit

of time reviewing all of it List's work

by no means is this like an automate and

forget it type of um uh experience like

you should you should be pretty

thoughtful about how you're integrating

these into your into your workflows and

um be careful to check elicits work that

being said we have done a decent amount

of internal testing comparing elissa's

accuracy to the accuracy of trained

research staff research assistants

manual data extraction and in a lot of

cases we really are beating a lot of uh

human accuracy um so I think it's really

promising and um you know all of this

extraction work takes a lot of time so

I'm hoping we can find good ways for

Alysa to augment you and and kind of

Accel all the work that you're

doing so the main workflow I'll be

focused on today is this extract data

from PDF's workflow um I have a bunch of

papers uploaded into my library already

if you click upload papers there's a way

to drag and drop PDFs you can drop a

bunch of them at once you don't have to

drop uh add one PDF at a time you can

select a bunch from your uh file picker

and upload many at once um you can also

go directly to your

library which you can find in your

sidebar and upload papers here or upload

papers from zoto we have another uh

video that will show you how you can do

that um so here I have about 39 papers

they're they're not in the same domain

so it's pretty unlikely that you would

ever do a review of a a group of papers

as diverse as this but these are the

papers that I have so I'll just use them

to showcase the features I'm going to

select all of them there's 40 there's

about

39 we typically recommend St under about

a 100 at a time if you are extracting

data from about 100 papers and

extracting lots of data it can

definitely the app starts to get a

little bit slow so starting with with

smaller numbers if you can um is a is is

a good best practice if you are about to

embark on a pretty serious review

project reach out to our team you can

email us at info@ elicit

docomo have ways of um running kind of

batch jobs outside of the app that can

be it can save you a lot of time be a

lot easier

so here you can see all the papers have

loaded in and now if you click into a

paper um you can see all the text here

and then you can also see that we

extract tables this is a feature that's

very unique to elicit really important

for research obviously but no other AI

tool has this um so you can use data

from tables as well and extract the data

in the contents of

tables um so I'll just go through an

example of how you might screen down all

of these uh papers so there's a lot here

you don't know exactly how many are

relevant and presumably you have some

criteria by which you're determining

whether a paper is relevant to your

review or not um so let's say for

example that criteria is population

based uh you have a bunch of columns

here that you can you can um use to

understand uh extract data from the

papers and understand more about what

the papers did um so if your if your

inclusion criteria are kind of

population Focus you can start with a

column like population characteristics

this is a pretty open-ended column it'll

just give you information about all the

different populations uh discussed in

the papers you can see there's a lot of

content in

here and as with in elicit if you click

on an answer you can always see uh the

sources and where the information came

from so this is a really great way to

check elicits work um and you can see

the most relevant quotes here and you

can tap through a bunch of them and then

you can also open the paper and see the

information in

context so that's say um I first start

I'm not exactly sure what I'm looking

for so I'm going to start with a kind of

open-ended column population

characteristics more generally um and

then I realize okay you know I should uh

if you have specific inclusion criteria

you might be able to skip some of these

steps but I'll just kind of go step by

step um just to make the point so now um

I think I'm noticing that the you know

populations differ along many dimensions

right there's age mentioned here gender

um region and certainly like lots of

other details uh so I might want to

drill down a little bit deeper and maybe

I'll ask specifically about participant

age that's another column that I can add

again all of those columns um are added

here when um elicit is not confident

about its answer it'll throw this flag

so you can double check so it's possible

it is mentioned in this paper um and

this uh this column just didn't pick up

on that so you might want to come back

and review that more carefully later and

again you can check click on the to um

see to double check list it's work so I

have maybe you know specifically a

specific field for age um uh I might

also want to do a field for uh region

maybe and then I'm getting a bunch of

different regions um and I'm noticing

that the regions are kind of on

different levels of granularity so maybe

I want to you know ultimately I kind of

want to you know include some papers and

exclude some paper so I want the

formatting to be a little bit consistent

um so I can ask my a custom uh question

here and and kind of extract fields in a

very custom way um so I can ask

something like what was the continent

where the study took

place um and I'll give instructions to

specifically follow to follow a specific

format so answer as one of uh

America

America go through the continents

here and I don't know if we'll have

Antarctica in any of these papers but

I'll do that in the interest of

completeness so now continent for

example was not uh one of our predefined

columns um but uh I was able to create a

custom column for my specific use case

give it formatting Direction so that I

could get a specific type of answer and

now from here I can filter by the

results of this column so I can just

filter for paper

that took place in

Africa um I can filter for and and maybe

Asia so I can include both Africa and

Asia and then if I delete these I will

you know see the full results

again um this column filter is a keyword

match so you do need to make sure that

the contents of the cell have the

keyword that you're filtering

by um let's see so that's a great way so

let's say if you had a region criteria

that you were screening by you can

extract the data format it and filter by

that

data um next in the screening process um

you might want to download this as a CSV

so that you can kind of indicate which

papers you've already reviewed or maybe

as you're going through the kind of

reviewing the quotes and reviewing the

listed work there's additional context

you're picking up on that you might want

to note in your um in your review

process so then you can download CSV

that's pretty pretty straightforward um

open it up in you know spreadsheets

Excel whatever is easiest for

you uh and then you'll get a spreadsheet

like this with the title the authors a

bunch of metadata uh each

column uh that you extracted here

population characteristics age region

continent where the study took place all

of the supporting quotes that we found

in the paper um as well as some

reasoning if that's if that ends up

being helpful as well as reasoning

basically so in cases where um unist it

might say not applicable or not

mentioned um will'll also kind of

explain the reasoning why that might be

the case we can you know share related

quotes um even if they don't directly

answer your question so from here you

might want to let's say um add a column

like reviewed or something and then say

reviewed or in progress um you can as

you go through you might want to check

elicit answers maybe you know if you

find that it's not 19 individuals I mean

or maybe I don't know if any of these

end up being you you want to add more

context you can add more context or make

Corrections um and basically

spreadsheets are going to for now

spreadsheets are are you know probably

going to be where you want to make more

edits um or directly make your notes

over time we definitely want to make

that more native to ELA but that's quite

complicated so right now spreadsheets

are probably better for you obviously

you can do some filtering and sorting

and spreadsheets as

well um a couple notes uh about how this

works

so um I think for screening if you're

going to do a lot of papers it's best to

uh this just adding columns is the

cheapest way um you'll definitely get

higher accuracy if you turn on high

accuracy mode you can do that by

clicking on any of these kind of

Bullseye buttons here um or over here or

by toggling this high accuracy mode here

um high accuracy mode um is uh is like

makes about half the error as kind of

regular mode but is also quite a bit

more expensive um so this probably makes

more sense for something like data

extraction or when you get into the

later uh stages of the process um for

screening if you're doing it for a large

number of papers you might want to do

like a rougher first

pass so extraction would work pretty

similarly um again I think the only

difference would be you you are likely

going to want to run high in high

accuracy mode when you get to the

extraction step so you can just turn

that on um once you do that you'll start

to be able to use information from the

tables as well um so for example when

you click on the source quotes you'll

start to see that it might be extracting

data from tables so if you need detailed

effect sizes um or other dimensions that

are mentioned mostly in the tables

you'll you'll want to run things in high

accuracy

mode so that's kind of the workflow for

screening and extraction um so again the

kind of assumption here is that you have

found your papers through whatever

search methodology that you've set out

it can you could have found your papers

in the lit or you might have found them

from other sources you can upload them

into elicit when you do that they're not

going to be shared with anyone else so

your papers will be entirely private to

you um and you know they're not going to

get uploaded no other users will see

them it's not a means of publishing

papers um it's just a way for you to

speed up your data analysis um you can

upload by PDF or by connecting users

hero integration Again by going back to

your library

and then you can select the papers that

you are most interested in um or select

the papers that are relevant and uh kind

of extract data to screen or uh or you

know PR prep for a meta analysis or some

other kind of data analysis I can show

you just really quickly some of the

testing that we've done with different

teams working on systematic reviews uh

there was one team that was trying to

screen about 5,000 papers we compared

our approach to some work that they had

done manually and we were actually able

to retrieve over 96% of all of the

papers that they consider to be relevant

um the kind of human research assistance

that they had trained only achieved

about 92% so elicit ability to identify

Rel papers was higher than the um than

delegating it to a bunch of um trained

research assistants and it was obviously

significantly cheaper significantly

faster much more Dynamic uh and the same

thing with data extraction again working

with a team that um was doing a lot of

extraction man manually um and we found

that elicit was about had about 98%

accuracy whereas a lot of the trained

members um or only 72% accurate um this

was especially true when elicit was

pretty confident so and when it wasn't

confident elicit would throw the flag so

that the team knew how to double check

its work so in a lot of cases there was

some disagreement between elicit and um

the manual approach um and when when the

teams kind of took a second look at

those answers it turned out Alyssa was

more accurate um so

uh yeah and I think just kind of um that

was yes that's kind of the overall

accuracy and then in general When

comparing to a lot of the um manual

approaches Alissa was often 13 to 26%

more accurate um for like an array of

different fields and data fields and um

and different

columns um another benefit is that you

know we're going to save all of your

work here in the sidebar so you can

always go back and pick up from where

you left off um it doesn't cost any

credits to reopen this View and then if

you continue and add more columns that

will cost credits if you're interested

in doing a systematic review with elicit

please reach out to us we would love to

give you best practices and tips uh as

well as a bunch of new features that we

might not not have released publicly yet

um and if you're generally interested in

evaluating elicit a systematically so

that it can be used for more systematic

reviews let us know we also work with

teams to do that type of work as well

thank

you