Designing Rubrics That Work

by Dr. Suzanne L. Pieper, Coordinator of Assessment

An essay by Vincent Tinto in Inside Higher Ed reminds us that college success is built “one class and one course at a time.” Tinto further points out that in successful college classrooms, students get frequent, high-quality feedback on their products and performances. One way instructors can provide that feedback is by using rubrics to assess student work. Rubrics are efficient for making instructors’ expectations explicit and promoting fairness and consistency.

Why don’t more instructors use rubrics? A major obstacle is the amount of time it takes to construct a good one. A tempting shortcut is to choose a rubric from the wide variety of those online and in print. But how do you know if the rubric you choose is a good one? And how do you know if it will work for you and your students? Three questions will help you to choose and use the best rubric for your class.

What am I looking for?

Think about three to five criteria that you could use to assess student responses to a performance task. In What’s Wrong—and What’s Right—with Rubrics, W. James Popham says it’s tempting to describe all possible assessment criteria, but it’s best to keep your rubric brief. When designing or selecting a rubric, ask yourself, “What are the most important elements of this assignment that demonstrate student learning?”

Let’s imagine that you want students to be able to write a research paper. You find an online rubric that lists the following criteria for assessing a research paper:

• Purpose
• Content
• Organization
• Feel
• Tone
• Sentence structure
• Word choice
• Mechanics
• Use of references
• Quality of references
• Use of most recent edition of MLA Handbook for Writers of Research Papers

You wonder if you want to assess twelve criteria. You also notice that some of the criteria aren’t distinct. What is “feel,” and how does it differ from “tone”? When you look at the intended learning outcomes for your course, and you scrutinize the research paper assignment you’ve given to your students, you discover that five criteria are most important:

• Purpose
• Organization
• Content
• Mechanics
• Use of references

Now you have a manageable number of rubric criteria that will guide your students to improve their skills in writing research papers. An added bonus: you can use these same criteria for a variety of writing assignments in your course.

What is the possible range of student products/performances?

Rubrics need to accommodate the entire range of possible student responses, but how do you decide how many separate levels of performance you want to recognize in your rubric? The best way is to review actual student work. Start by sorting the work into upper range and lower range responses, and then further sort the work as needed. How many “piles” do you have? The number of piles should give you an idea of how many performance levels you will need in your rubric. Each performance level needs to be clearly distinct from the next so that there is no question about which level a particular piece of student work meets. Most rubrics allow three to five levels; if you include more levels, you might find it difficult to clearly distinguish the levels. You also want to think about how to label the performance levels. The labels should make clear the distinctions among levels but not discourage students. Here are some examples.

Let’s say you want to assess discussion posts in your online course. You know what you’re looking for, and you’ve identified five levels of student performances from your review of prior student discussions. However, you aren’t sure how to describe the levels of student performance. Say you find a collection of performance level descriptions online, and you think one set of descriptions shows promise: accomplished, advancing, developing, beginning, no concept.

Ask yourself how you would distinguish between “beginning” and “developing.” Also think about how your students might react to being described as having “no concept.” You realize that the set of descriptions seems to apply to student development rather than to the students’ work, so you instead might settle on five different performance levels for online discussion: excellent, good, average, fair, and poor. These levels reflect the range of your students’ performances, make it possible to distinguish between performance levels, and are not discouraging to students.

How do I describe what I am looking for at every point in the range of student products/performances?

Now that you have identified what you are looking for and the possible range of student performances, you are ready for the final step in adopting or adapting your rubric: the descriptions. The descriptions are the “meat” of the rubric because they explicitly detail what a student needs to do to get a score at each scale point. They also provide instructors with clear guidelines for improving student learning. Descriptions need to be consistent, distinct, and written “in plain English” so students can understand them.

Generic example of clearly distinct levels
of performance

Top Level	Middle Level	Low Level
Do X, Y, and Z	Do X and Y	Do X

Let’s say you want to assess oral presentations in your course. You know what you’re looking for, you’ve identified and labeled three levels of student performance, but you’re not sure how to describe exactly what a student needs to do to get a score at each point in the scale. You find an oral communication rubric in a resource book that at first glance looks perfect! When you inspect the rubric more carefully, however, you notice that the descriptions don’t always focus on the same characteristics across performance levels. For example, you want to assess pacing, yet pacing is described as “paced for audience understanding” in the high performance level, “sometimes too fast or too slow” in the middle performance level, and not mentioned at all in the low performance level.

You revise the descriptions of pacing so that the characteristic is addressed at all performance levels, and that the value of the characteristic changes in a measurable way between adjacent levels. Drawing from an example given by Robin Tierney and Marielle Simon in What’s still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels, you modify the rubric so that at the high performance level, the student “always paces for audience understanding.” At the middle performance level, the student “sometimes paces for audience understanding.” At the low performance level, the student “doesn’t pace for audience understanding.” With a few more revisions, you can use your rubric to effectively assess your students’ oral presentations. Additionally, when you share the rubric with your students, they can use it to better understand your expectations and deliver higher quality presentations.

Example of clearly distinct levels
of performance

Excellent	Acceptable	Needs Improvement
Always paces for audience understanding	Sometimes paces for audience understanding	Doesn’t pace for audience understanding

What successes have you had with designing rubrics for your courses? What challenges have you encountered? Please share your comments.

References

Popham, W. J. (1997). What’s wrong—and what’s right—with rubrics. Educational Leadership, 55, 72-75.

Tierney, Robin & Marielle Simon (2004). What’s still wrong with rubrics: focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2).

Using Assessment to Engage Your Students

by Sue Pieper
Coordinator for Assessment, e-Learning Center at Northern Arizona University

How can we get students to attend our classes?

How can we get students to participate in class?

How can we prevent students from cheating?

These questions dominated the conversations at a recent Course Redesign (CR) Conference, and faculty in the small group sessions on assessment were adamant about getting some answers. While these were very real concerns for course instructors, now, a few weeks after the conference, I continue to wonder whether their questions were getting at the core of the issue. I wonder if these concerns might, in fact, be symptoms of an underlying “disease” infecting our students—particularly students in large classes. Could our students be suffering from a lack of engagement?

What is student engagement, really? Researchers and educators have tossed around the term for years. Definitions of student engagement have varied greatly, from students’ time-on-task behaviors (Brophy, 1983) to their use of metacognition and self-regulation (Pintrich and De Groot, 1990). Most definitions have included some element of students connecting with and thinking deeply about the course content. These are desirable outcomes for any classroom instructor, but they are not always easy to achieve, especially if you are teaching a class of 100+ students.

What’s an Instructor to Do?

There are plenty of ideas out there in the literature. I did a quick search of an education database for “student engagement” and found over 5,000 entries! The problem is in knowing where to begin. So here’s my attempt to provide assistance: three steps for using assessment to create a more engaging course. These steps are based on the CR Conference discussions as well as my work consulting with faculty who are designing assessments for their online, hybrid, and web-enhanced courses.

Take a look at what you’re already doing.

You can incorporate questions about assessment in your end-of-the-course evaluations. Better yet, throughout the semester you can ask the students about how the course assessments are going. Psychology professor Melissa Birkett recently developed a survey that she’ll administer at mid-semester in her PSY 255 course, Introduction to Cognitive and Behavioral Neuroscience. The survey asks students about a new portfolio project she’s incorporated into the course. In addition to some scaled items, Melissa asks students to write short answers to items such as “One aspect of this project I enjoyed was…” “One aspect of this project I found challenging was…” and “I learned the most from this project by…” By administering the survey mid-semester, Melissa will be able to make small modifications before the course ends, if necessary, as well as plan for next semester’s portfolio project. For other methods of gathering students’ reactions to your course assessments, check out the chapter on Techniques for Assessing Learner Reactions to Instruction in Thomas A. Angelo and K. Patricia Cross’s Classroom Assessment Techniques: A Handbook for College Teachers.

Think about what you might do differently.

Which course assessment strategies are working for you, and which need some work? CR Conference participants provided lots of ideas for rethinking how you use assessment in your course. Nora Dunbar, Psychology lecturer and CR Conference participant, suggested thinking about incorporating collaborative work. If you’ve historically used individual assignments and assessments, consider including group/pair work. Students learn from sharing ideas. You, the instructor, will benefit, too. If the group, rather than each individual student, is responsible for a discussion post or final project, then you will have fewer final products to assess.Paula Garcia McAllister, Director of Northern Arizona University’s Institutional Review Board, suggested using short writing assignments in courses. If you’ve routinely used only mid-term and final multiple-choice exams to assess students, think about integrating several brief papers throughout the semester and using a rubric. Think about your criteria for assessing students, and then in the rubric, for each criterion describe the performance levels that correspond to the grade or points you will award. Up-front work in constructing a solid rubric leads to huge payoffs for you and your students. Your students will know exactly what kind of performance you expect from them, and you will spend less time assessing. The quality of students’ work is also likely to improve as the semester progresses. For a quick overview of how to make and use a rubric, take a look at Peter Gow’s slide show.

Some of these ideas are discussed in more depth in the Tuesday Tips on Teaching with Technology podcasts Assessment and Large Course Redesign with Wally Nolan and Sue Pieper and Pedagogy and Large Course Redesign with John Doherty and Wally Nolan. These podcasts are available as part of a podcast series (iTunes U link) on using technology to improve teaching and learning. You might also find two books from the Jossey-Bass Guides to Online Teaching and Learning series helpful: Rita-Marie Conrad and J. Ana Donaldson’s Engaging the Online Learner and Rena M. Palloff and Keith Pratt’s Assessing the Online Learner.

Consider whether technology could help.

Once you’ve decided what you’d like to change about the assessments in your course, think about how technology might assist you. Cathryn Ellis, Theatre professor and CR Conference participant, added Blackboard Vista quizzes before the start of her class and found that drop/fail/withdraw grades decreased and student understanding of the course material increased. Short quizzes administered online throughout the semester provide immediate and ongoing feedback and help students to learn. They also help you know how and where to focus instruction. Other instructors are experimenting with “clickers,” automatic response systems that can be used to promote collaboration and critical thinking among students. For more about clickers, see John Doherty and Wally Nolan’s blog post, Pedagogy and Large Course Redesign. You might also want to check out a couple of resource sites that provide information about how to implement clickers and the impact of clickers on student learning. For more about web tools that could be used for assessment, take a look at Educause Quarterly Magazine, Volume 2, Number 4, 2009. The entire issue is devoted to technology tools and student engagement.

After you’ve implemented a new assessment technique, you’ll need to ask students again how your course assessments are working for them. If you continue to follow the steps listed above, you will be able to improve your course over time. However, these ideas are only the beginning of a treatment for lack of student engagement in our courses. I’d like to hear about what you’re doing to promote student engagement in your classes. Have you tried any of these strategies or techniques in your classes? How have you engaged students through assessment? Feel free to share your ideas in the comments.

Resources

Brophy, J. (1983). Conceptualizing student motivation. Educational Psychologist, 18 (3), 200-215.

Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82 (1): 33-40.