Literably's Scoring/Grading Methodology (ORF assessment) Follow
Overview: At Literably, we are committed to providing teachers, schools, and districts with accurate assessment results within 24 hours - in order to help inform instruction and drive student achievement.
Below is information about how we score oral reading fluency (ORF) assessments on Literably. If you have any questions about this methodology or how a specific assessment was scored, please reach out to your Customer Success Manager or support@literably.com.
Scoring/Grading Methodology:
Each assessment on Literably is broken down into 3 main parts:
The reading transcript and multiple-choice questions are scored by Literably in order to produce a "graded assessment." If students complete the retell (which is optional), teachers can see that at the bottom of each graded assessment. Teachers also have the option to score the retell themselves, which will then also be included within the graded assessment.
1. The reading transcript:
When students take a Literably assessment, they record themselves reading the assessment text. Generally, Literably will score the first 2 minutes of this recording for texts up to the middle of 1st grade level, and the first 1 minute for higher levels. Sometimes, districts/schools select different subscription plans that score 2 minutes for all levels (more details here).
Teachers can choose to have students' recordings cut off after 1 or 2 minutes, or they can use Literably's default setting - which allow students to continue reading out loud until they finish the reading passage. To adjust this setting, click on the red "Class Settings" gear icon ( ) and check/uncheck the option in the pop-up window.
To produce the reading transcript, each student recording is listened to and scored by a Literably-certified human grader (details about our scoring conventions here). Literably only selects graders who demonstrate exceptional accuracy in transcribing oral recordings. Even after accepting a grader, Literably conducts random checks to ensure all graders are maintaining exceptional accuracy.
In addition, Literably recently started using speech recognition to help score small portions (<10%) of some fluency recordings. This new technology is being deployed primarily on the easiest-to-grade parts of some student recordings.
Our goal is to combine the expertise and human touch of our Literably-certified graders with the efficiency of speech recognition, in order to produce extremely accurate reading transcripts that are delivered to teachers as quickly as possible.
2. Five multiple-choice comprehension questions
Because there is only one right answer to each multiple-choice question, Literably's technology is able to quickly and automatically grade these questions. Literably's certified graders do not score the multiple-choice question portion of each assessment.
3. The Retell
The retell is optional for students to complete and:
- Does not play a factor in how Literably automatically levels and re-levels students
- Is not scored by Literably
Literably does not score the retell, because we have not found an objective and satisfactory way to do so given the variety of students, texts, and partners that we work with. We do allow teachers to score the retell themselves and provide a retell rubric on each assessment to help teachers who choose to do so.