How to preregister a study on as predicted
Summary
TLDREmily demonstrates the process of pre-registering a study on askpredicted.org, a platform promoting open science and reproducible research. She guides viewers through entering author details, hypotheses, data collection plans, and analysis methods. Emily emphasizes the importance of specificity to prevent data manipulation and suggests documenting secondary predictions and demographics. She also shares her experience with platform sampling inaccuracies and how to anonymize pre-registrations for peer review.
Takeaways
- 📝 Pre-registration is a part of the open science movement, promoting transparency and reproducibility in research.
- 🔍 To pre-register on askpredicted.org, researchers must answer a set of questions about their study before collecting data.
- 👥 Researchers need to provide names, email addresses, and institutions of all authors involved in the study.
- 🚫 Ideally, no data should have been collected before pre-registration to ensure the integrity of the research process.
- 📜 Hypotheses should be explicitly stated, aligning with the planned tests to prevent data manipulation.
- 📊 Researchers should pre-register the dependent variable (DV), number of conditions, and specific analyses such as t-tests, ANOVA, or regression.
- 🔍 Outliers and exclusion criteria, like attention checks, must be specified in the pre-registration to avoid bias.
- 🔢 Sample size and the method of determining it, such as the number of participants recruited per condition, should be clearly stated.
- 📈 Pre-registration helps in specifying secondary predictions and exploratory questions, which are not essential to the main hypothesis.
- 📑 After completing the pre-registration, researchers can preview, edit, and accept it before it's sent to all authors for final acceptance.
- 🔒 Once all authors accept the pre-registration, it becomes part of the public record, enhancing the credibility of the study.
Q & A
What is the purpose of pre-registration in research?
-Pre-registration is part of a movement to promote open science, open data, and reproducible research. It helps ensure transparency and prevent data manipulation by specifying the research plan before data collection begins.
What is 'askpredicted.org' and how does it relate to pre-registration?
-'askpredicted.org' is a platform created by researchers at Wharton where researchers can create new pre-registrations and answer a set of questions to formalize their research plan and hypotheses.
Why should data collection be avoided before pre-registering a study?
-Ideally, data collection should not occur before pre-registration because the process assumes it is the first time data is being collected, ensuring the integrity and originality of the research findings.
What information is required when pre-registering a study on askpredicted.org?
-Researchers need to provide the names, email addresses, and institutions of the authors involved in the study, specify whether data has been collected, and detail the hypotheses, conditions, analyses, and any exclusions or outliers.
Why is it important to be explicit about the analyses and covariates in a pre-registration?
-Being explicit about the analyses and covariates helps reduce the chance of p-hacking, which is manipulating data to achieve statistically significant results. It also ensures that the research is reproducible and transparent.
What is the significance of specifying outliers or exclusions in a pre-registration?
-Specifying outliers or exclusions is important because it sets the criteria for participant inclusion or exclusion in the study. This transparency helps others understand the sample selection process and prevents bias.
How should the sample size be determined and specified in a pre-registration?
-The sample size should be determined based on the research design and specified in the pre-registration, including how participants will be recruited and any potential oversampling to account for platform inaccuracies.
What is the process of making a pre-registration anonymous for peer review?
-To make a pre-registration anonymous for peer review, researchers can use the 'make anonymous pdf' feature on askpredicted.org, which removes identifying information, making it suitable for blind review.
How can researchers recycle the text from a pre-registration for their final paper?
-Researchers can copy and paste the text from their pre-registration into a blank word document, where they can further develop their answers to the pre-registration questions. This text can later be used in the write-up of the study and the paper.
What is the role of attention checks in a pre-registered study?
-Attention checks are used to ensure that participants are engaging with the study material attentively. Researchers should specify in the pre-registration if they plan to exclude participants based on attention check results.
How does the process of accepting a pre-registration work among the authors?
-Once a pre-registration is completed and previewed by the lead researcher, it must be accepted by all authors involved. This acceptance is typically done through a personal or institutional email, and once all authors have accepted, the pre-registration is finalized and can be made public.
Outlines
📝 Pre-Registration Process on askPredicted.org
Emily provides a step-by-step guide on how to pre-register a study on askPredicted.org, a platform developed by researchers at Wharton. The pre-registration process is part of the open science movement, promoting transparency and reproducibility. It involves entering author details, confirming data collection status, explicitly pre-registering hypotheses, dependent variables, conditions, and analyses. Emily emphasizes the importance of being detailed to avoid p-hacking and suggests pre-registering outliers, exclusions, and sample size determination. She also mentions the need to account for platform-specific over or under recruitment and to specify any additional measures or exploratory questions. The process concludes with a preview, acceptance by all authors, and the listing of the pre-registration.
🔍 Reviewing and Anonymizing Pre-Registrations
Emily demonstrates how to review and make pre-registrations public on her main website. She explains the process of sending herself an email to access her pre-registrations and shows how to make them anonymous for peer review. The anonymization is crucial for blind review to ensure the study's integrity. Emily also shares her method of working with pre-registrations, starting with a blank document where she copies and pastes questions from the askPredicted.org website. She then writes detailed answers, including hypotheses, study setup, main predictions, and specific tests to be conducted. She provides an example of a study on admissions, detailing the prediction, conditions, and tests, and mentions the inclusion of additional measures and demographics. Emily concludes by explaining how she recycles her pre-registration text for the write-up of her paper.
Mindmap
Keywords
💡Pre-registration
💡Open Science
💡Reproducibility
💡Hypotheses
💡Data Collection
💡DV (Dependent Variable)
💡Analyses
💡Covariates
💡Sample Size
💡Anonymization
💡Attention Checks
💡Secondary Predictions
💡Demographics
Highlights
Introduction to pre-registration on askpredicted.org by Emily.
Pre-registration is part of the open science movement for reproducible research.
Creating a new pre-registration involves answering a set of structured questions.
Authors must provide their names, email addresses, and institutions for the pre-registration.
Data collection status should be reported, ideally with no data collected prior to pre-registration.
Hypotheses should be pre-registered explicitly to align with later tests.
Details on the dependent variable (DV), conditions, and analyses must be specified.
Pre-registration reduces the potential for p-hacking and encourages transparent research practices.
Outliers and exclusion criteria must be pre-registered, including attention checks.
Sample size and determination method should be clearly stated in the pre-registration.
Platform-specific deviations in participant numbers are acknowledged.
Any secondary predictions or exploratory questions should also be pre-registered.
The process includes naming the pre-registration and specifying the study type.
Pre-registration can be previewed, edited, and accepted by all authors involved.
Once accepted, pre-registrations are listed and can be made public for review.
Demonstration of making a pre-registration anonymous for peer review purposes.
Example of working with pre-registrations using a blank word document for drafting.
Details on how to recycle pre-registration text for the write-up of the paper.
Specificity in pre-registration is emphasized, including exact wording and conditions.
Instructions on how to handle deviations due to platform over or under sampling.
Inclusion of additional measures and demographics in the pre-registration process.
Final steps in making the pre-registration public on askpredicted.org.
Transcripts
hi guys this is emily um today i want to
show to you how to do a pre-registration
as predicted
as predicted was created by a set of
researchers
researchers at wharton and this
pre-registration is kind of part of a
movement
to have open science open data and also
to be
able to do reproducible research so
on askpredicted.org you'll want to
create a new pre-registration and then
answer a set of questions
you can also just try things out if you
just want to test something
um first you'll have to put in the names
and email addresses and institutions of
the authors who are part of the project
or
the study that you're pre-registering
then you uh will have to say
whether you've already collected data um
ideally
you obviously should have not collected
any data because the pre-registration
um assumes that this is the first time
that you're collecting your data so you
always want to click no
then you want to pre-register your
hypotheses you should be
with this as explicit as possible um
pre-registering kind of in line with
what your later um tests will be as well
you want to pre-register the dv the
number of conditions on your
study the exact analyses you want to run
so that could be that you will run a
t-test or that you will run an anova
or regression you want to include all of
the different covariates you might want
to use in a regression
and just be very explicit about the
exact analysis you want to run because
this obviously then reduces the
um ability to p hack your data and it's
a it's a good tool to just make yourself
or tie your own hands
um then um
you also want to pre-register any
outliers or exclusions so
assuming that you do some sort of an
attention check
and you plan on excluding people you
also have to
specify that here
if you exclude people who for example
didn't
provide like good answers to like an
open response question you want to put
that down here as well
next you want to specify the sample size
and how you determine that so
for example if you collect 200
participants on mturk or on prolific
academic
you'll want to write that down i usually
say how many participants i plan on
recruiting per condition here
yeah one thing i've actually recently
noticed which i i would recommend that
you do too
is that a lot of these platforms
actually don't um sample exactly the
number that you plan
so um for example uh on mturk i've
noticed that they always sample like
five people extra or so and especially
especially on like these um panels like
um mtrc panels and quilt tricks panels
uh they really over recruit people and
i actually like oversampled by over 200
people
in one study and so i would say that you
say you specify
the the amount of people in your
platform but then obviously any over
under recruiting of that platform
is due to the platform and not due to
your own choice
um and then finally you also want to
pre-register anything else that you
might be doing so
up here you only specify the like the
main analyses that um
that answer the main hypotheses or um
but not any other questions that you
might be doing so
if you have like a secondary prediction
that is not essential to your
um to your main hypothesis it might even
be exploratory
you want to put that down here as well
um i also always write down that i'm
collecting like my demographics
here or often like i also specify my
attention checks down here
because i don't plan on actually
excluding any participants but i usually
still collect attention checks just to
see how good my data is
and so then i'll say that i'll analyze
my
my data with with the attention check
and without the attention check in the
in a sense
finally um you wanna just give your
your um preregistration a name and then
specify
what the what type of um study it is
um and then okay i can't go to the
preview but
once um i've done this i can actually
preview my
pre-registration and then i will be able
to
like re-read my pre-registration and
make any edits
and then i will accept my
pre-registration and the
pre-registration will then be sent to
all of the authors that i mentioned up
here
and then um all of the answers uh all of
the authors have to accept this
pre-registration
and once we've all hit accept in our
personal email or in our institution
then the pre-registration will be in the
list of pre-registrations
and so for me
they would end up on like my main
website and i could then review them and
make them public i don't know if i can
show you now i think i first have to go
back into my email
let's see
so in order to review my studies i would
have to
send myself an email so i just did that
and then go to my pre-registrations
i click on them here and then
i get my list of pre-registrations and
depending on
the stage of the pre-registration it'll
either be
anonymized already or not so if i
haven't made it anonymous this is just
something that i can see myself
so for example if i had just submitted
something i could see my free
registration
this is a pre-registration that i made a
few days ago
and if i then want to make it public for
example for
review for blind review i i hit
make anonymous pdf and then
if people insert this link they actually
end up seeing my pre-registration
um and that's what i will actually then
end up putting into my paper
and this would be accessible and this is
anonymized which means that it's
okay for peer review because um
it doesn't say who was creating this
project or the study
so i wanted to show you an example of
how i work on pre-registrations
um before i actually put in the
information and i usually start with a
blank
word document like this one where i just
copy paste
my questions from the pre-registration
from the as predicted website
and then i write my text to answer the
questions
so for this project um it's a project on
admissions and
um i i
noted down here the the hypotheses
um i usually still kind of say what the
main
setup is of the study i do that and
later i can use this text also for my
write-up of the paper
in the study of the study and the paper
so and even though it takes a bit of
time
i can recycle it and then this is the
main prediction so here i predict that
participants will be more likely to
infer that their friend dislikes i
deliver degrees
when the friend visited two as opposed
to 10 countries and that
i then say is going to be tested through
a t-test
um and i also conduct like a difference
measure
um between two two measures here so
as you can see i'm very specific with my
with my pre-registration i say exactly
what will be measured i usually
explicitly actually write down the exact
words
um and then from what to what i'm
measuring it
i i specify the conditions i'm assigned
to what tests i'm doing
um and then here i say okay i want to
collect 300 participants from
from mturk um and then this is what i've
recently started to add it's um
deviations from the skull will be due to
over and under
sampling from the platform and then this
is my anything else so
um a couple of other measures that are
not that not that relevant to the main
question being asked
um i also say that i'm going to measure
whether people have ever been to greece
in italy and
i collect demographics this is kind of
my example
and then i would just copy paste that
into the into the questionnaire on as
predictive making it public later
so yeah that was it um how to
pre-register something on
askpredicted.org
Voir Plus de Vidéos Connexes
Registration & Preregistration
Data Collection Stratergy For Machine Learning Projects With API's- RapidAPI
Cách QUẢN LÝ THỜI GIAN HỌC TẬP cho HS LƯỜI nhưng BẬN RỘN | Học tới đâu giỏi tới đó
Truth in Data Science | Jaya Tripathi | TEDxYouth@BHS
Graphing Guide for chemistry
How to Write a Research Methodology in 4 Steps | Scribbr 🎓
5.0 / 5 (0 votes)