SwaGrader -- evaluation of the students, by the students, for the MOOCs
Summary
TLDRThis presentation introduces Sol Cratera, a peer grading tool designed to address common issues in large class evaluations. It utilizes the Tropican mechanism to discourage strategic behaviors like grade inflation or reduction, ensuring fair assessments. Developed by researchers from IIT Kanpur and UC Davis, the tool allows for class creation, student enrollment, and assignment management. It features a grading method where students can be both graders and gradees, with the ability to upload and split assignments. Tropican uses 'probes' for quality control, and the tool provides progress tracking and regrading requests. The presentation concludes with a demonstration of the final grade distribution.
Takeaways
- 📚 Peer grading is a method where students evaluate each other's work.
- 🔍 Traditional peer grading tools may overlook potential issues like strategic behavior by graders.
- 🌟 The tool 'Sol Cratera' introduces a mechanism called 'Tropicana' to discourage strategic behaviors in peer grading.
- 📈 Tropicana ensures that it's in the best interest of peer graders to grade honestly.
- 👥 Developed by researchers from IIT Kanpur and UC Davis, the tool aims to improve the peer grading process.
- 🚀 The tool allows for user roles such as students, teaching assistants, and instructors, each with different permissions.
- 📝 Assignments can be created and uploaded with answer keys for graders to reference.
- 📑 The tool automatically splits bulk assignments into individual submissions for grading.
- 🔍 Tropicana uses 'probes' to monitor the quality of peer grading, with instructor-graded assignments as the gold standard.
- 📊 Instructors can monitor the progress of the peer grading process and view individual grader performance.
- 🔒 Once completed, grades are frozen, and a final Tropicana score is provided for the peer grading exercise.
Q & A
What is peer grading?
-Peer grading is the evaluation of students' work by their peers, which is an efficient and scalable method for assessing large classes, especially in online education and massive open online courses.
What are the potential issues with traditional peer grading tools?
-Traditional peer grading tools may ignore important behavioral aspects such as the possibility of graders competing or cooperating, which could lead to inflated or reduced scores, or graders not putting in meaningful effort, resulting in unrepresentative grading.
What is the Tropican mechanism?
-Tropican is a mechanism introduced to ensure that it is in the best interest of the peer grader not to engage in strategic behaviors that could compromise the integrity of the grading process.
Who developed the Tropican mechanism and the Sol Cratera tool?
-The Tropican mechanism and the Sol Cratera tool were developed by a group of researchers from IIT Kanpur and UC Davis.
What does Sol Cratera mean and how does it relate to peer grading?
-Sol Cratera, which means 'self grader', is a tool that uses the Tropican mechanism for peer grading, aiming to ensure fair and representative grading by students.
How does a user create a class and enroll students in Sol Cratera?
-Users can create a class and enroll students by uploading a roster, which sends email confirmations to all participants.
What role does the instructor play in the peer grading process in Sol Cratera?
-The instructor can choose the grading method, upload assignments and answer keys, set maximum marks for questions, upload solutions, and monitor the progress of the peer grading process.
What are 'probes' in the context of the Tropican mechanism?
-Probes are a special set of papers used by the Tropican mechanism to probe the graders. They are selected from submitted assignments, graded by the instructor, and serve as the gold standard for grading.
How does the instructor monitor the progress of peer grading in Sol Cratera?
-The instructor can monitor the progress of peer grading by viewing the grades awarded to papers assigned to graders and tracking the overall progress of each peer grader.
What is the purpose of the regrading request feature in Sol Cratera?
-The regrading request feature allows students to request a reevaluation of their grades, which is a standard feature in many grading systems to ensure fairness and address potential grading errors.
What is the significance of the Tropican score given at the end of the peer grading exercise?
-The Tropican score is significant as it represents the mechanism's evaluation of the entire peer grading exercise, ensuring the integrity and quality of the grading process.
Outlines
📚 Introduction to Peer Grading and Sol Cratera
This paragraph introduces the concept of peer grading, which is the evaluation of students by their peers, and highlights its efficiency in large class settings. It discusses the shortcomings of traditional peer grading tools, such as the potential for graders to engage in strategic behaviors that can skew grading results. To address these issues, the presenter introduces 'Tropicana', a mechanism designed to discourage such behaviors. The tool for peer grading, named 'Sol Cratera', is explained as a self-grader that utilizes the Tropicana mechanism. Developed by a collaborative team from IIT Kanpur and UC Davis, the tool is showcased with a demo that includes user roles, class creation, student enrollment, assignment management, and the grading process. The paragraph concludes with a mention of the tool's capability to handle assignments in bulk and the integration of the Tropicana mechanism for peer grading.
🔍 Tropicana Mechanism and Grading Process
The second paragraph delves into the specifics of the Tropicana mechanism, which is central to the peer grading tool. It explains the use of 'probes', a set of papers that serve as a benchmark for grading. These probes are graded by instructors to establish a gold standard. The paragraph also covers the instructor's role in monitoring the peer grading process, the ability for graders to assign marks to papers, and the visibility of each grader's progress. The presentation concludes with a mention of the regrading request feature, which is a standard aspect of the grading process, and the finalization of grades for all students involved in peer grading. The Tropicana mechanism provides an additional layer of evaluation, known as 'Tropic Epona', which assesses the overall peer grading exercise.
Mindmap
Keywords
💡Peer Grading
💡Strategic Behaviors
💡Tropicana Mechanism
💡Sol Cratera
💡Probes
💡Roles
💡Bulk Assignment Upload
💡Grading Method
💡Answer Key
💡Regrading Request
💡Tropicana Score
Highlights
Introduction to peer grading as an efficient and scalable evaluation method for large classes.
Discussion on the limitations of traditional peer grading tools, including potential for collusion or lack of effort.
Introduction of the 'Tropicana' mechanism designed to deter strategic behaviors in peer grading.
Availability of details on the 'Tropicana' mechanism in a research paper for interested viewers.
Presentation of 'Sol Cratera', a peer grading tool utilizing the 'Tropicana' mechanism.
Collaboration between researchers from IIT Kanpur and UC Davis in developing the 'Tropicana' mechanism and 'Sol Cratera'.
Demonstration of user roles and the ability to switch between them within the tool.
Feature to create classes and enroll students by uploading a roster with email confirmations.
Instructor's ability to choose between traditional grading and peer grading methods.
Simplified assignment creation process with the option to upload assignments and answer keys.
Bulk upload and automatic splitting of assignments into individual submissions.
Explanation of the peer grading mechanism 'Tropicana' and its parameters set by the instructor.
Use of 'probes' in the 'Tropicana' mechanism to ensure accurate grading standards.
Instructor's capability to monitor the progress of the peer grading process.
Graders' interface for awarding grades and tracking individual progress.
Inclusion of a regrading request feature as part of the peer grading process.
Finalization of grades and the provision of a 'Tropicana' score for the peer grading exercise.
Conclusion of the presentation with a thank you note.
Transcripts
hello viewers in this short presentation
I am going to give a demo of our peer
grading tool first let us understand
what peer grading is it is evaluation of
the students by the students and this is
one of the most efficient and scalable
ways of evaluating large classes even
though this innovative evaluation method
is becoming rapidly popular in the age
of online education and massive open
online courses often a few important
behavioral aspects are ignored in the
traditional peer grading tools for
instance peer leaders are students in a
course and they can compete or cooperate
with each other which means that graders
can collaboratively create inflate or
deliberately reduce scores of others or
they can simply put no effort to create
meaningfully which may lead to a grading
which is not at all representative of
their performances so to deal with all
these problems we introduce a mechanism
called tropica that ensures that it's
the best response the best interest of
the peer grader not to engage in any of
such strategic behaviors that we
discussed in the beginning of this slide
so the details of this mechanism and its
properties are available in this paper
if you are interested please take a look
in this look into this paper so our tool
for peer grading is called Sol crater
which means self grader that uses
Tripucka mechanism as it's hard a group
of researchers from iit kanpur and UC
davis is involved in developing the
mechanism and the tool let me walk you
through this tool the landing page gives
an user to log in or sign up the user
may have one or more roles for different
courses in this demo we show all the
roles true though in practice there will
be only a selected subset of rows
for one individual for instance the
student a particular student may may be
student in certain courses and teaching
assistant for certain other courses
similarly an instructor may not be a
student or a teaching assistant in any
other courses but this is just to show
you the the exhaustive picture so the
users can create a class enroll students
by sending by uploading the roster which
will send email confirmation to all the
participants so this page shows
different assignments of the course in
their and their statuses the the
important part is the grading method
this tool allows the instructor to
choose the usual grading method where
the TAS or the instructor rates
themselves or the peer grading which
happens by at Rebecca so creation of
assignments is also easy the instructor
can upload the assignment with the
answer key and this is shown to all
graders during grading this page shows
how the maximum marks are assigned to
the questions once this is set the
entire solution PDF can be can be
uploaded in bulk which can later be
split into individual submissions the
tool does it for you so for instance if
there are handwritten assignments they
can they can be scanned together and
upload it on this tool so this page
which is currently a work in progress
shows that this the that bulk assignment
is splitted into smaller assignments the
crucial component of this tool is the
peer grading mechanism Tropican which
needs certain parameters to be entered
and this is entered by the instructor on
this page
Tripucka uses a special set of papers
called probes which is used to probe DPR
graders in this page those papers are
uploaded the probes can be selected from
the submitted assignments are graded by
the instructor and serves as the gold
standard of grading and this is one of
the important aspects of tropica
mechanism now the instructor can also
monitor the progress of the peer grading
process and graders can our award grades
to the papers assigned to them by
entering the marks in the appropriate
boxes the progress of each peer grader
can also be viewed so the final part of
this is is the regrading request which
is a standard thing and this is the
students view so finally all the grades
are frozen for all the students who are
also peer graders the important part is
that there is a tropic Epona's which is
also which is given by this mechanism
for this whole peer grading exercise so
and that brings us to the end of the
presentation thank you very much
Посмотреть больше похожих видео
Turnitin Class id and Enrolment key 2024 |Full Activate| Working Free Account
LearnMate - Grader Report (LM 7.3.2.7)
Assessment in Learning Helen Bihag Boholano TeacherHelen TV
LU 1 Memahami Asesmen - Metode Asesmen
How can I manage my SSL certificates!? Look no further!
Usare l’AI per prendere appunti da qualsiasi video (TUTORIAL)
5.0 / 5 (0 votes)