How to preregister a study on as predicted

Summary

Emily demonstrates the process of pre-registering a study on askpredicted.org, a platform promoting open science and reproducible research. She guides viewers through entering author details, hypotheses, data collection plans, and analysis methods. Emily emphasizes the importance of specificity to prevent data manipulation and suggests documenting secondary predictions and demographics. She also shares her experience with platform sampling inaccuracies and how to anonymize pre-registrations for peer review.

Takeaways

• 📝 Pre-registration is a part of the open science movement, promoting transparency and reproducibility in research.
• 🔍 To pre-register on askpredicted.org, researchers must answer a set of questions about their study before collecting data.
• 👥 Researchers need to provide names, email addresses, and institutions of all authors involved in the study.
• 🚫 Ideally, no data should have been collected before pre-registration to ensure the integrity of the research process.
• 📜 Hypotheses should be explicitly stated, aligning with the planned tests to prevent data manipulation.
• 📊 Researchers should pre-register the dependent variable (DV), number of conditions, and specific analyses such as t-tests, ANOVA, or regression.
• 🔍 Outliers and exclusion criteria, like attention checks, must be specified in the pre-registration to avoid bias.
• 🔢 Sample size and the method of determining it, such as the number of participants recruited per condition, should be clearly stated.
• 📈 Pre-registration helps in specifying secondary predictions and exploratory questions, which are not essential to the main hypothesis.
• 📑 After completing the pre-registration, researchers can preview, edit, and accept it before it's sent to all authors for final acceptance.
• 🔒 Once all authors accept the pre-registration, it becomes part of the public record, enhancing the credibility of the study.

Q & A

What is the purpose of pre-registration in research?

Pre-registration is part of a movement to promote open science, open data, and reproducible research. It helps ensure transparency and prevent data manipulation by specifying the research plan before data collection begins.

What is 'askpredicted.org' and how does it relate to pre-registration?

'askpredicted.org' is a platform created by researchers at Wharton where researchers can create new pre-registrations and answer a set of questions to formalize their research plan and hypotheses.

Why should data collection be avoided before pre-registering a study?

Ideally, data collection should not occur before pre-registration because the process assumes it is the first time data is being collected, ensuring the integrity and originality of the research findings.

What information is required when pre-registering a study on askpredicted.org?

Researchers need to provide the names, email addresses, and institutions of the authors involved in the study, specify whether data has been collected, and detail the hypotheses, conditions, analyses, and any exclusions or outliers.

Why is it important to be explicit about the analyses and covariates in a pre-registration?

Being explicit about the analyses and covariates helps reduce the chance of p-hacking, which is manipulating data to achieve statistically significant results. It also ensures that the research is reproducible and transparent.

What is the significance of specifying outliers or exclusions in a pre-registration?

Specifying outliers or exclusions is important because it sets the criteria for participant inclusion or exclusion in the study. This transparency helps others understand the sample selection process and prevents bias.

How should the sample size be determined and specified in a pre-registration?

The sample size should be determined based on the research design and specified in the pre-registration, including how participants will be recruited and any potential oversampling to account for platform inaccuracies.

What is the process of making a pre-registration anonymous for peer review?

To make a pre-registration anonymous for peer review, researchers can use the 'make anonymous pdf' feature on askpredicted.org, which removes identifying information, making it suitable for blind review.

How can researchers recycle the text from a pre-registration for their final paper?

Researchers can copy and paste the text from their pre-registration into a blank word document, where they can further develop their answers to the pre-registration questions. This text can later be used in the write-up of the study and the paper.

What is the role of attention checks in a pre-registered study?

Attention checks are used to ensure that participants are engaging with the study material attentively. Researchers should specify in the pre-registration if they plan to exclude participants based on attention check results.

How does the process of accepting a pre-registration work among the authors?

Once a pre-registration is completed and previewed by the lead researcher, it must be accepted by all authors involved. This acceptance is typically done through a personal or institutional email, and once all authors have accepted, the pre-registration is finalized and can be made public.

Outlines

00:00

📝 Pre-Registration Process on askPredicted.org

Emily provides a step-by-step guide on how to pre-register a study on askPredicted.org, a platform developed by researchers at Wharton. The pre-registration process is part of the open science movement, promoting transparency and reproducibility. It involves entering author details, confirming data collection status, explicitly pre-registering hypotheses, dependent variables, conditions, and analyses. Emily emphasizes the importance of being detailed to avoid p-hacking and suggests pre-registering outliers, exclusions, and sample size determination. She also mentions the need to account for platform-specific over or under recruitment and to specify any additional measures or exploratory questions. The process concludes with a preview, acceptance by all authors, and the listing of the pre-registration.

05:02

🔍 Reviewing and Anonymizing Pre-Registrations

Emily demonstrates how to review and make pre-registrations public on her main website. She explains the process of sending herself an email to access her pre-registrations and shows how to make them anonymous for peer review. The anonymization is crucial for blind review to ensure the study's integrity. Emily also shares her method of working with pre-registrations, starting with a blank document where she copies and pastes questions from the askPredicted.org website. She then writes detailed answers, including hypotheses, study setup, main predictions, and specific tests to be conducted. She provides an example of a study on admissions, detailing the prediction, conditions, and tests, and mentions the inclusion of additional measures and demographics. Emily concludes by explaining how she recycles her pre-registration text for the write-up of her paper.

Keywords

💡Pre-registration

Pre-registration is a process in scientific research where the design of a study is documented and made public before the data is collected. This helps to ensure transparency, prevent data manipulation, and promote reproducibility in research. In the video, Emily explains the steps to create a pre-registration on askpredicted.org, emphasizing its importance in the open science movement.

💡Open Science

Open science refers to the practice of making scientific research, data, and methodology publicly accessible to encourage collaboration, transparency, and reproducibility. The video script discusses pre-registration as a part of this movement, aiming to standardize research practices and make them more reliable.

💡Reproducibility

Reproducibility in research means that other scientists should be able to obtain the same results by repeating the same study using the same methods. The script highlights the role of pre-registration in enhancing reproducibility by reducing the chances of 'p-hacking' or selective reporting of results.

💡Hypotheses

Hypotheses are proposed explanations or predictions made in a scientific study that can be tested through experiments or observations. In the context of the video, Emily instructs viewers to pre-register their hypotheses explicitly to align with later tests and analyses, ensuring a clear and unbiased research direction.

💡Data Collection

Data collection is the process of gathering information and measurements required for analysis in a research study. The script mentions that ideally, data should not have been collected before pre-registration to maintain the integrity and originality of the research findings.

💡DV (Dependent Variable)

The dependent variable is the variable in an experiment that is expected to change as a result of the independent variable. In the script, Emily advises pre-registering the DV and the conditions of the study to clarify what outcomes are being measured and how they will be analyzed.

💡Analyses

Analyses refer to the methods used to interpret and draw conclusions from collected data. The video script details the need to pre-register the exact analyses, such as t-tests, ANOVA, or regression, to ensure transparency and prevent data manipulation.

💡Covariates

Covariates are variables in a statistical model that may affect the relationship between the independent and dependent variables. Emily mentions pre-registering any covariates that might be used in a regression analysis to control for potential confounding factors.

💡Sample Size

Sample size is the number of participants or observations included in a study. The script explains the importance of specifying the sample size in a pre-registration, such as recruiting a certain number of participants from platforms like MTurk or Prolific, to ensure the study has adequate statistical power.

💡Anonymization

Anonymization is the process of removing personally identifiable information from data to protect privacy and ensure unbiased review. In the video, Emily describes making the pre-registration anonymous for peer review by creating an anonymized PDF that does not reveal the creators of the study.

💡Attention Checks

Attention checks are measures used in research to ensure that participants are engaging with the study material attentively. The script mentions pre-registering any attention checks and exclusion criteria, such as excluding participants who did not provide good answers to open response questions.

💡Secondary Predictions

Secondary predictions are additional hypotheses that are not the main focus of a study but are still explored for potential insights. Emily suggests pre-registering these secondary predictions to maintain transparency and allow for exploratory analysis.

💡Demographics

Demographics are statistical data about a population's characteristics, such as age, gender, or income. In the script, Emily mentions collecting demographics as part of her study and pre-registering this information to provide a comprehensive understanding of the study's sample.

Highlights

Introduction to pre-registration on askpredicted.org by Emily.

Pre-registration is part of the open science movement for reproducible research.

Creating a new pre-registration involves answering a set of structured questions.

Authors must provide their names, email addresses, and institutions for the pre-registration.

Data collection status should be reported, ideally with no data collected prior to pre-registration.

Hypotheses should be pre-registered explicitly to align with later tests.

Details on the dependent variable (DV), conditions, and analyses must be specified.

Pre-registration reduces the potential for p-hacking and encourages transparent research practices.

Outliers and exclusion criteria must be pre-registered, including attention checks.

Sample size and determination method should be clearly stated in the pre-registration.

Platform-specific deviations in participant numbers are acknowledged.

Any secondary predictions or exploratory questions should also be pre-registered.

The process includes naming the pre-registration and specifying the study type.

Pre-registration can be previewed, edited, and accepted by all authors involved.

Once accepted, pre-registrations are listed and can be made public for review.

Demonstration of making a pre-registration anonymous for peer review purposes.

Example of working with pre-registrations using a blank word document for drafting.

Details on how to recycle pre-registration text for the write-up of the paper.

Specificity in pre-registration is emphasized, including exact wording and conditions.

Instructions on how to handle deviations due to platform over or under sampling.

Inclusion of additional measures and demographics in the pre-registration process.

Final steps in making the pre-registration public on askpredicted.org.

Transcripts

hi guys this is emily um today i want to

show to you how to do a pre-registration

as predicted

as predicted was created by a set of

researchers

researchers at wharton and this

pre-registration is kind of part of a

movement

to have open science open data and also

to be

able to do reproducible research so

on askpredicted.org you'll want to

create a new pre-registration and then

answer a set of questions

you can also just try things out if you

just want to test something

um first you'll have to put in the names

and email addresses and institutions of

the authors who are part of the project

or

the study that you're pre-registering

then you uh will have to say

whether you've already collected data um

ideally

you obviously should have not collected

any data because the pre-registration

um assumes that this is the first time

that you're collecting your data so you

always want to click no

then you want to pre-register your

hypotheses you should be

with this as explicit as possible um

pre-registering kind of in line with

what your later um tests will be as well

you want to pre-register the dv the

number of conditions on your

study the exact analyses you want to run

so that could be that you will run a

t-test or that you will run an anova

or regression you want to include all of

the different covariates you might want

to use in a regression

and just be very explicit about the

exact analysis you want to run because

this obviously then reduces the

um ability to p hack your data and it's

a it's a good tool to just make yourself

or tie your own hands

um then um

you also want to pre-register any

outliers or exclusions so

assuming that you do some sort of an

attention check

and you plan on excluding people you

also have to

specify that here

if you exclude people who for example

didn't

provide like good answers to like an

open response question you want to put

that down here as well

next you want to specify the sample size

and how you determine that so

for example if you collect 200

participants on mturk or on prolific

academic

you'll want to write that down i usually

say how many participants i plan on

recruiting per condition here

yeah one thing i've actually recently

noticed which i i would recommend that

you do too

is that a lot of these platforms

actually don't um sample exactly the

number that you plan

so um for example uh on mturk i've

noticed that they always sample like

five people extra or so and especially

especially on like these um panels like

um mtrc panels and quilt tricks panels

uh they really over recruit people and

i actually like oversampled by over 200

people

in one study and so i would say that you

say you specify

the the amount of people in your

platform but then obviously any over

under recruiting of that platform

is due to the platform and not due to

your own choice

um and then finally you also want to

pre-register anything else that you

might be doing so

up here you only specify the like the

main analyses that um

that answer the main hypotheses or um

but not any other questions that you

might be doing so

if you have like a secondary prediction

that is not essential to your

um to your main hypothesis it might even

be exploratory

you want to put that down here as well

um i also always write down that i'm

collecting like my demographics

here or often like i also specify my

attention checks down here

because i don't plan on actually

excluding any participants but i usually

still collect attention checks just to

see how good my data is

and so then i'll say that i'll analyze

my

my data with with the attention check

and without the attention check in the

in a sense

finally um you wanna just give your

your um preregistration a name and then

specify

what the what type of um study it is

um and then okay i can't go to the

preview but

once um i've done this i can actually

preview my

pre-registration and then i will be able

to

like re-read my pre-registration and

make any edits

and then i will accept my

pre-registration and the

pre-registration will then be sent to

all of the authors that i mentioned up

here

and then um all of the answers uh all of

the authors have to accept this

pre-registration

and once we've all hit accept in our

personal email or in our institution

email

then the pre-registration will be in the

list of pre-registrations

and so for me

they would end up on like my main

website and i could then review them and

make them public i don't know if i can

show you now i think i first have to go

back into my email

let's see

so in order to review my studies i would

have to

send myself an email so i just did that

and then go to my pre-registrations

i click on them here and then

i get my list of pre-registrations and

depending on

the stage of the pre-registration it'll

either be

anonymized already or not so if i

haven't made it anonymous this is just

something that i can see myself

so for example if i had just submitted

something i could see my free

registration

this is a pre-registration that i made a

few days ago

and if i then want to make it public for

example for

review for blind review i i hit

make anonymous pdf and then

if people insert this link they actually

end up seeing my pre-registration

um and that's what i will actually then

end up putting into my paper

and this would be accessible and this is

anonymized which means that it's

okay for peer review because um

it doesn't say who was creating this

project or the study

so i wanted to show you an example of

how i work on pre-registrations

um before i actually put in the

information and i usually start with a

blank

word document like this one where i just

copy paste

my questions from the pre-registration

from the as predicted website

and then i write my text to answer the

questions

so for this project um it's a project on

admissions and

um i i

noted down here the the hypotheses

um i usually still kind of say what the

main

setup is of the study i do that and

later i can use this text also for my

write-up of the paper

in the study of the study and the paper

so and even though it takes a bit of

time

i can recycle it and then this is the

main prediction so here i predict that

participants will be more likely to

infer that their friend dislikes i

deliver degrees

when the friend visited two as opposed

to 10 countries and that

i then say is going to be tested through

a t-test

um and i also conduct like a difference

measure

um between two two measures here so

as you can see i'm very specific with my

with my pre-registration i say exactly

what will be measured i usually

explicitly actually write down the exact

words

um and then from what to what i'm

measuring it

i i specify the conditions i'm assigned

to what tests i'm doing

um and then here i say okay i want to

collect 300 participants from

from mturk um and then this is what i've

recently started to add it's um

deviations from the skull will be due to

over and under

sampling from the platform and then this

is my anything else so

um a couple of other measures that are

not that not that relevant to the main

question being asked

um i also say that i'm going to measure

whether people have ever been to greece

in italy and

i collect demographics this is kind of

my example

and then i would just copy paste that

into the into the questionnaire on as

predictive making it public later

so yeah that was it um how to

pre-register something on

askpredicted.org