SwaGrader -- evaluation of the students, by the students, for the MOOCs

Summary

This presentation introduces Sol Cratera, a peer grading tool designed to address common issues in large class evaluations. It utilizes the Tropican mechanism to discourage strategic behaviors like grade inflation or reduction, ensuring fair assessments. Developed by researchers from IIT Kanpur and UC Davis, the tool allows for class creation, student enrollment, and assignment management. It features a grading method where students can be both graders and gradees, with the ability to upload and split assignments. Tropican uses 'probes' for quality control, and the tool provides progress tracking and regrading requests. The presentation concludes with a demonstration of the final grade distribution.

Takeaways

• 📚 Peer grading is a method where students evaluate each other's work.
• 🔍 Traditional peer grading tools may overlook potential issues like strategic behavior by graders.
• 🌟 The tool 'Sol Cratera' introduces a mechanism called 'Tropicana' to discourage strategic behaviors in peer grading.
• 📈 Tropicana ensures that it's in the best interest of peer graders to grade honestly.
• 👥 Developed by researchers from IIT Kanpur and UC Davis, the tool aims to improve the peer grading process.
• 🚀 The tool allows for user roles such as students, teaching assistants, and instructors, each with different permissions.
• 📝 Assignments can be created and uploaded with answer keys for graders to reference.
• 📑 The tool automatically splits bulk assignments into individual submissions for grading.
• 🔍 Tropicana uses 'probes' to monitor the quality of peer grading, with instructor-graded assignments as the gold standard.
• 📊 Instructors can monitor the progress of the peer grading process and view individual grader performance.
• 🔒 Once completed, grades are frozen, and a final Tropicana score is provided for the peer grading exercise.

Q & A

What is peer grading?

Peer grading is the evaluation of students' work by their peers, which is an efficient and scalable method for assessing large classes, especially in online education and massive open online courses.

What are the potential issues with traditional peer grading tools?

Traditional peer grading tools may ignore important behavioral aspects such as the possibility of graders competing or cooperating, which could lead to inflated or reduced scores, or graders not putting in meaningful effort, resulting in unrepresentative grading.

What is the Tropican mechanism?

Tropican is a mechanism introduced to ensure that it is in the best interest of the peer grader not to engage in strategic behaviors that could compromise the integrity of the grading process.

Who developed the Tropican mechanism and the Sol Cratera tool?

The Tropican mechanism and the Sol Cratera tool were developed by a group of researchers from IIT Kanpur and UC Davis.

What does Sol Cratera mean and how does it relate to peer grading?

Sol Cratera, which means 'self grader', is a tool that uses the Tropican mechanism for peer grading, aiming to ensure fair and representative grading by students.

How does a user create a class and enroll students in Sol Cratera?

Users can create a class and enroll students by uploading a roster, which sends email confirmations to all participants.

What role does the instructor play in the peer grading process in Sol Cratera?

The instructor can choose the grading method, upload assignments and answer keys, set maximum marks for questions, upload solutions, and monitor the progress of the peer grading process.

What are 'probes' in the context of the Tropican mechanism?

Probes are a special set of papers used by the Tropican mechanism to probe the graders. They are selected from submitted assignments, graded by the instructor, and serve as the gold standard for grading.

How does the instructor monitor the progress of peer grading in Sol Cratera?

The instructor can monitor the progress of peer grading by viewing the grades awarded to papers assigned to graders and tracking the overall progress of each peer grader.

What is the purpose of the regrading request feature in Sol Cratera?

The regrading request feature allows students to request a reevaluation of their grades, which is a standard feature in many grading systems to ensure fairness and address potential grading errors.

What is the significance of the Tropican score given at the end of the peer grading exercise?

The Tropican score is significant as it represents the mechanism's evaluation of the entire peer grading exercise, ensuring the integrity and quality of the grading process.

Outlines

00:00

📚 Introduction to Peer Grading and Sol Cratera

This paragraph introduces the concept of peer grading, which is the evaluation of students by their peers, and highlights its efficiency in large class settings. It discusses the shortcomings of traditional peer grading tools, such as the potential for graders to engage in strategic behaviors that can skew grading results. To address these issues, the presenter introduces 'Tropicana', a mechanism designed to discourage such behaviors. The tool for peer grading, named 'Sol Cratera', is explained as a self-grader that utilizes the Tropicana mechanism. Developed by a collaborative team from IIT Kanpur and UC Davis, the tool is showcased with a demo that includes user roles, class creation, student enrollment, assignment management, and the grading process. The paragraph concludes with a mention of the tool's capability to handle assignments in bulk and the integration of the Tropicana mechanism for peer grading.

05:04

🔍 Tropicana Mechanism and Grading Process

The second paragraph delves into the specifics of the Tropicana mechanism, which is central to the peer grading tool. It explains the use of 'probes', a set of papers that serve as a benchmark for grading. These probes are graded by instructors to establish a gold standard. The paragraph also covers the instructor's role in monitoring the peer grading process, the ability for graders to assign marks to papers, and the visibility of each grader's progress. The presentation concludes with a mention of the regrading request feature, which is a standard aspect of the grading process, and the finalization of grades for all students involved in peer grading. The Tropicana mechanism provides an additional layer of evaluation, known as 'Tropic Epona', which assesses the overall peer grading exercise.

Keywords

💡Peer Grading

Peer grading is a method of evaluation where students assess the work of their peers. It is highlighted in the script as an efficient and scalable way to evaluate large classes, especially in the context of online education. The script discusses how traditional peer grading tools may overlook certain behavioral aspects, which the tool 'Sol Cratera' aims to address.

💡Strategic Behaviors

Strategic behaviors refer to the actions taken by peer graders that may not align with the intended purpose of fair and accurate evaluation. The script mentions how graders might collaborate to inflate or reduce scores or put in minimal effort, leading to grading that does not reflect actual performance. The tool 'Sol Cratera' is designed to discourage such behaviors.

💡Tropicana Mechanism

The Tropicana mechanism is introduced as a solution to ensure that peer graders act in their best interest by not engaging in strategic behaviors. It uses a set of special papers called 'probes' to monitor the grading process and maintain integrity. This mechanism is central to the theme of the video, as it underpins the tool's effectiveness in peer grading.

💡Sol Cratera

Sol Cratera is the name of the peer grading tool being demonstrated. It is described as a self-grader that utilizes the Tropicana mechanism to ensure fair and accurate peer evaluations. The tool is designed to handle various aspects of the grading process, from enrollment to final grading, and is a key component of the video's message.

💡Probes

In the context of the Tropicana mechanism, probes are a special set of papers that are used to evaluate the integrity of the peer grading process. They serve as a 'gold standard' against which the accuracy of peer graders can be measured. The script mentions that these probes are graded by the instructor and are a critical part of ensuring the quality of peer grading.

💡Roles

Roles in the script refer to the different帽子 that users can have within the Sol Cratera tool, such as student, teaching assistant, or instructor. These roles determine the functions and permissions a user has within the tool, which is essential for understanding how the tool manages and organizes the grading process across various participants.

💡Bulk Assignment Upload

Bulk assignment upload is a feature of Sol Cratera that allows instructors to upload multiple assignments at once, which can then be split into individual submissions. This feature is mentioned as a time-saving and efficient way to handle large volumes of assignments, particularly when they are scanned together.

💡Grading Method

The grading method in the script refers to the options available to instructors for evaluating student work, such as traditional grading by teaching assistants or instructors, or peer grading facilitated by the tool. The choice of grading method is crucial as it determines the approach to student evaluation within a course.

💡Answer Key

The answer key is mentioned in the context of creating assignments within Sol Cratera. It is a document that provides the correct solutions to the problems in an assignment, which is important for ensuring consistency and accuracy during the grading process, especially when peer graders are involved.

💡Regrading Request

A regrading request is a standard feature in many grading systems, including Sol Cratera, that allows students to request a review of their graded work if they believe there was an error in the initial evaluation. This concept is briefly mentioned in the script as part of the overall grading process.

💡Tropicana Score

The Tropicana score is a metric given by the Tropicana mechanism at the end of the peer grading exercise. It is not explicitly defined in the script, but it can be inferred as a measure of the quality or effectiveness of the peer grading process, reflecting the tool's ability to maintain the integrity of peer evaluations.

Highlights

Introduction to peer grading as an efficient and scalable evaluation method for large classes.

Discussion on the limitations of traditional peer grading tools, including potential for collusion or lack of effort.

Introduction of the 'Tropicana' mechanism designed to deter strategic behaviors in peer grading.

Availability of details on the 'Tropicana' mechanism in a research paper for interested viewers.

Presentation of 'Sol Cratera', a peer grading tool utilizing the 'Tropicana' mechanism.

Collaboration between researchers from IIT Kanpur and UC Davis in developing the 'Tropicana' mechanism and 'Sol Cratera'.

Demonstration of user roles and the ability to switch between them within the tool.

Feature to create classes and enroll students by uploading a roster with email confirmations.

Instructor's ability to choose between traditional grading and peer grading methods.

Simplified assignment creation process with the option to upload assignments and answer keys.

Bulk upload and automatic splitting of assignments into individual submissions.

Explanation of the peer grading mechanism 'Tropicana' and its parameters set by the instructor.

Use of 'probes' in the 'Tropicana' mechanism to ensure accurate grading standards.

Instructor's capability to monitor the progress of the peer grading process.

Graders' interface for awarding grades and tracking individual progress.

Inclusion of a regrading request feature as part of the peer grading process.

Finalization of grades and the provision of a 'Tropicana' score for the peer grading exercise.

Conclusion of the presentation with a thank you note.

Transcripts

hello viewers in this short presentation

I am going to give a demo of our peer

grading tool first let us understand

what peer grading is it is evaluation of

the students by the students and this is

one of the most efficient and scalable

ways of evaluating large classes even

though this innovative evaluation method

is becoming rapidly popular in the age

of online education and massive open

online courses often a few important

behavioral aspects are ignored in the

traditional peer grading tools for

instance peer leaders are students in a

course and they can compete or cooperate

with each other which means that graders

can collaboratively create inflate or

deliberately reduce scores of others or

they can simply put no effort to create

meaningfully which may lead to a grading

which is not at all representative of

their performances so to deal with all

these problems we introduce a mechanism

called tropica that ensures that it's

the best response the best interest of

the peer grader not to engage in any of

such strategic behaviors that we

discussed in the beginning of this slide

so the details of this mechanism and its

properties are available in this paper

if you are interested please take a look

in this look into this paper so our tool

for peer grading is called Sol crater

which means self grader that uses

Tripucka mechanism as it's hard a group

of researchers from iit kanpur and UC

davis is involved in developing the

mechanism and the tool let me walk you

through this tool the landing page gives

an user to log in or sign up the user

may have one or more roles for different

courses in this demo we show all the

roles true though in practice there will

be only a selected subset of rows

for one individual for instance the

student a particular student may may be

student in certain courses and teaching

assistant for certain other courses

similarly an instructor may not be a

student or a teaching assistant in any

other courses but this is just to show

you the the exhaustive picture so the

users can create a class enroll students

by sending by uploading the roster which

will send email confirmation to all the

participants so this page shows

different assignments of the course in

their and their statuses the the

important part is the grading method

this tool allows the instructor to

choose the usual grading method where

the TAS or the instructor rates

themselves or the peer grading which

happens by at Rebecca so creation of

assignments is also easy the instructor

can upload the assignment with the

answer key and this is shown to all

graders during grading this page shows

how the maximum marks are assigned to

the questions once this is set the

entire solution PDF can be can be

uploaded in bulk which can later be

split into individual submissions the

tool does it for you so for instance if

there are handwritten assignments they

can they can be scanned together and

upload it on this tool so this page

which is currently a work in progress

shows that this the that bulk assignment

is splitted into smaller assignments the

crucial component of this tool is the

peer grading mechanism Tropican which

needs certain parameters to be entered

and this is entered by the instructor on

this page

Tripucka uses a special set of papers

called probes which is used to probe DPR

graders in this page those papers are

uploaded the probes can be selected from

the submitted assignments are graded by

the instructor and serves as the gold

standard of grading and this is one of

the important aspects of tropica

mechanism now the instructor can also

monitor the progress of the peer grading

process and graders can our award grades

to the papers assigned to them by

entering the marks in the appropriate

boxes the progress of each peer grader

can also be viewed so the final part of

this is is the regrading request which

is a standard thing and this is the

students view so finally all the grades

are frozen for all the students who are

also peer graders the important part is

that there is a tropic Epona's which is

also which is given by this mechanism

for this whole peer grading exercise so

and that brings us to the end of the

presentation thank you very much