Designing Rubrics That Work

by Dr. Suzanne L. Pieper, Coordinator of Assessment

An essay by Vincent Tinto in Inside Higher Ed reminds us that college success is built “one class and one course at a time.” Tinto further points out that in successful college classrooms, students get frequent, high-quality feedback on their products and performances. One way instructors can provide that feedback is by using rubrics to assess student work. Rubrics are efficient for making instructors’ expectations explicit and promoting fairness and consistency.

Why don’t more instructors use rubrics? A major obstacle is the amount of time it takes to construct a good one. A tempting shortcut is to choose a rubric from the wide variety of those online and in print. But how do you know if the rubric you choose is a good one? And how do you know if it will work for you and your students? Three questions will help you to choose and use the best rubric for your class.

What am I looking for?

Think about three to five criteria that you could use to assess student responses to a performance task. In What’s Wrong—and What’s Right—with Rubrics, W. James Popham says it’s tempting to describe all possible assessment criteria, but it’s best to keep your rubric brief. When designing or selecting a rubric, ask yourself, “What are the most important elements of this assignment that demonstrate student learning?”

Let’s imagine that you want students to be able to write a research paper. You find an online rubric that lists the following criteria for assessing a research paper:

• Purpose
• Content
• Organization
• Feel
• Tone
• Sentence structure
• Word choice
• Mechanics
• Use of references
• Quality of references
• Use of most recent edition of MLA Handbook for Writers of Research Papers

You wonder if you want to assess twelve criteria. You also notice that some of the criteria aren’t distinct. What is “feel,” and how does it differ from “tone”? When you look at the intended learning outcomes for your course, and you scrutinize the research paper assignment you’ve given to your students, you discover that five criteria are most important:

• Purpose
• Organization
• Content
• Mechanics
• Use of references

Now you have a manageable number of rubric criteria that will guide your students to improve their skills in writing research papers. An added bonus: you can use these same criteria for a variety of writing assignments in your course.

What is the possible range of student products/performances?

Rubrics need to accommodate the entire range of possible student responses, but how do you decide how many separate levels of performance you want to recognize in your rubric? The best way is to review actual student work. Start by sorting the work into upper range and lower range responses, and then further sort the work as needed. How many “piles” do you have? The number of piles should give you an idea of how many performance levels you will need in your rubric. Each performance level needs to be clearly distinct from the next so that there is no question about which level a particular piece of student work meets. Most rubrics allow three to five levels; if you include more levels, you might find it difficult to clearly distinguish the levels. You also want to think about how to label the performance levels. The labels should make clear the distinctions among levels but not discourage students. Here are some examples.

Let’s say you want to assess discussion posts in your online course. You know what you’re looking for, and you’ve identified five levels of student performances from your review of prior student discussions. However, you aren’t sure how to describe the levels of student performance. Say you find a collection of performance level descriptions online, and you think one set of descriptions shows promise: accomplished, advancing, developing, beginning, no concept.

Ask yourself how you would distinguish between “beginning” and “developing.” Also think about how your students might react to being described as having “no concept.” You realize that the set of descriptions seems to apply to student development rather than to the students’ work, so you instead might settle on five different performance levels for online discussion: excellent, good, average, fair, and poor. These levels reflect the range of your students’ performances, make it possible to distinguish between performance levels, and are not discouraging to students.

How do I describe what I am looking for at every point in the range of student products/performances?

Now that you have identified what you are looking for and the possible range of student performances, you are ready for the final step in adopting or adapting your rubric: the descriptions. The descriptions are the “meat” of the rubric because they explicitly detail what a student needs to do to get a score at each scale point. They also provide instructors with clear guidelines for improving student learning. Descriptions need to be consistent, distinct, and written “in plain English” so students can understand them.

Generic example of clearly distinct levels
of performance

Top Level	Middle Level	Low Level
Do X, Y, and Z	Do X and Y	Do X

Let’s say you want to assess oral presentations in your course. You know what you’re looking for, you’ve identified and labeled three levels of student performance, but you’re not sure how to describe exactly what a student needs to do to get a score at each point in the scale. You find an oral communication rubric in a resource book that at first glance looks perfect! When you inspect the rubric more carefully, however, you notice that the descriptions don’t always focus on the same characteristics across performance levels. For example, you want to assess pacing, yet pacing is described as “paced for audience understanding” in the high performance level, “sometimes too fast or too slow” in the middle performance level, and not mentioned at all in the low performance level.

You revise the descriptions of pacing so that the characteristic is addressed at all performance levels, and that the value of the characteristic changes in a measurable way between adjacent levels. Drawing from an example given by Robin Tierney and Marielle Simon in What’s still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels, you modify the rubric so that at the high performance level, the student “always paces for audience understanding.” At the middle performance level, the student “sometimes paces for audience understanding.” At the low performance level, the student “doesn’t pace for audience understanding.” With a few more revisions, you can use your rubric to effectively assess your students’ oral presentations. Additionally, when you share the rubric with your students, they can use it to better understand your expectations and deliver higher quality presentations.

Example of clearly distinct levels
of performance

Excellent	Acceptable	Needs Improvement
Always paces for audience understanding	Sometimes paces for audience understanding	Doesn’t pace for audience understanding

What successes have you had with designing rubrics for your courses? What challenges have you encountered? Please share your comments.

References

Popham, W. J. (1997). What’s wrong—and what’s right—with rubrics. Educational Leadership, 55, 72-75.

Tierney, Robin & Marielle Simon (2004). What’s still wrong with rubrics: focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2).

Helping Students to Succeed

by Dr. Sue Pieper, Coordinator of Assessment

With more students struggling academically, particularly in the first year, and fewer students persisting to graduation, many of us in higher education are asking how we can help our students to succeed. One way is to teach students to “think about their thinking.” In other words, we can teach them to develop metacognitive skills that will help them to become aware of their own thinking processes and use that self-awareness to regulate those processes. Researchers and practitioners agree that metacognition is critical to academic success.

In a previous blog post, I promised to provide some easy-to-implement strategies for teaching metacognition in face-to-face and online courses. When I reviewed the research and talked to colleagues about what they were doing to promote metacognition in their classes, a theme emerged: The best way to teach metacognition is to do it in conjunction with activities and assignments that are already a part of your class. Here are some strategies for teaching your students to think about their thinking when they take exams, listen to lectures, or work on writing assignments in your class.

Taking Exams

College students are often unaware of what they know and don’t know (Zabrucky and Bays, 2011). When taking exams, students frequently overestimate their level of understanding and readiness to take a test. First-year students in particular report that “looking over their notes” before an exam has worked well for them in the past (Ruban and Reis, 2006), and they are shocked when they receive exam scores that are lower than expected.

Karen Zabrucky and Rebecca Bays suggest that we can help students better understand what they know and don’t know by asking them to predict their exam scores right before they take an exam and then also estimate their scores right after taking an exam but before receiving their grades. Students can then compare their predictions and estimates with their actual exam scores. The authors also suggest that instructors ask students questions about how they studied for an exam and whether they felt they were adequately prepared to take the exam. These questions prompt students to reflect on both their level of preparation for an exam and the consequences of their level of preparation.

Listening to Lectures

Shawn Nordell (2009) conducted research with students in a large introductory biology course and found that most students had difficulty recalling course knowledge. When students were asked to write down two or three of the main points discussed in a lecture and readings, most students had no response at all or could remember only a key word or phrase.

Marsha Lovett (2008) described a technique called “wrappers,” activities that wrap around a learning activity or assignment and can be used to foster students’ metacognitive skills, including the recall of course knowledge. Instructors can “wrap” a lecture by presenting tips on active listening before the lecture, having students write down the three key ideas from the lecture immediately after the lecture, and then giving students a list of the three key ideas from the lecture for students to self-check. Lovett found that over time the students’ three key ideas increasingly matched those of the instructor.

Working on Writing Assignments

One way to prompt students to reflect on their writing is to provide them with questions for self-assessment. Before students begin to write, ask them to answer questions such as “What are my goals for this writing assignment?” or “What do I need to do to prepare to write?” You can also ask students to answer questions right after they write. Here are some suggested questions, adapted from a questionnaire used in a large-scale university writing assessment at Truman State University:

• How do you feel about your finished writing sample?
• How representative is this sample of your writing?
• Describe your writing process.
• What do you feel is especially strong about your writing sample?
• What do you feel could be improved in your writing sample?

The Writing Place at Northwestern University offers good examples of self-assessment questions and a worksheet for students to use when evaluating their own writing.

All of these strategies help students to practice their metacognitive skills and grow as learners. Have you tried any of these strategies in your online or face-to-face class? What worked, and what didn’t work? What other methods have you used to encourage students to think about their thinking? Please comment on your experiences with teaching your students how to learn.

References

Lovett, M. C. (2008, May 5). EDUCAUSE Learning Initiative Events. Retrieved May 18, 2008, from Metacognition and Monitoring: Understanding and Improving Students’ Skills for Learning.

Nordell, S. E. (2009). Learning how to learn: A model for teaching students learning strategies. Bioscene: Journal of College Biology Teaching, 35 – 42.

Ruban, L., & Reis, S. M. (2006). “Patterns of Self-Regulation: Patterns of Self-Regulatory Strategy Use among Low-Achieving and High-Achieving University Students. Roeper Review, 148 – 156.

Zabrucky, K. M., & Bays, R. (2011). Helping students know what they know and do not know. College Teaching, 123.

Rethinking Your Course with Student Learning in Mind

by Dr. Sue Pieper, Coordinator of Assessment

It’s official! During this year, Northern Arizona University will be moving to a new learning management system, Blackboard Learn. As you move your courses to the new system or use the new system for the first time, take advantage of the opportunity to rethink your courses, adjusting them as needed to incorporate a learner-centered approach in line with the university’s values.

Effective learner-centered courses are distinguished by five characteristics outlined by Maryellen Weimer in Learner-Centered Teaching: Five Key Changes to Practice (2002). According to Weimer, who also edits The Teaching Professor,  the following course elements change when teaching becomes learner-centered:

• The balance of power
• The function of content
• The role of the teacher
• The responsibility for learning
• The purposes and processes of evaluation

In a learner-centered course power is shared among teachers and students, resulting in a more engaging and motivating course environment. Content in a learner-centered course is used to promote students’ knowledge as well as their learning how to learn. The role of the teacher is to guide and facilitate student learning, with the students assuming the primary responsibility for learning. The purpose of evaluation includes not only grading, but also fostering improved student learning (Weimer, 2002).

Let’s consider some ways you can incorporate learner-centered teaching in your
own courses.

The Syllabus Quiz

The syllabus is an essential part of every course, providing information about learning outcomes, required textbooks and materials, a course calendar, and student and instructor responsibilities. We ask students to read the syllabus at the beginning of a course. Then we typically spend class time reviewing the syllabus, or we spend time answering multiple questions because students haven’t read the syllabus.

An alternative approach is to require students to pass a syllabus quiz, which can be set up as a brief true/false or multiple-choice online quiz that can be graded automatically by the learning management system. Many instructors have found that using a syllabus quiz as a “gateway” is effective. Students must pass the quiz with an acceptable score in order to gain access to the rest of the course. As a result, responsibility for reading and understanding the syllabus is placed on the students, not the teacher.

Self- and Peer-Evaluation of Group Work

Group work can be challenging in a face-to-face course, and effective group work is even more challenging in an online environment when instructors and students are at a distance.

If you want to improve group work in your course, consider involving students in not only completing the group assignments, but also in evaluating how they function as a group. First, teach them about the dynamics of group work and the various roles in a group. Then teach them how to assess their participation in their groups.

Rubrics — documents that describe an instructor’s grading criteria and levels of performance expected from students for each grade possible in an assignment — are very helpful for both instructors and students in assessing group work. An example of a team and leadership skills rubric, which assesses criteria such as group organization and coordination, is available from the e-Learning Center’s website.

When you create a rubric and share it with your students, you enable them to take responsibility for evaluating their own and their group members’ contributions to the group. Consequently, students learn content while also developing an awareness of how they work with others.

Student-Designed Exam Questions

Designing exams has traditionally been the purview of instructors, but what if we ask students to design an exam? Weimer offers examples, including a math instructor’s experience with giving students an end-of-the-course option to develop a final exam. She evaluated the assignment on criteria such as how well the exam questions corresponded with the intended student learning outcomes for the course, the solutions for the problems, and the point values assigned to the problems, taking into account their relative importance in the course content. The students told the instructor that they spent more time designing the exam than they would have spent studying for it.

Involving students in constructing exams and other course evaluations transforms the goal of assessment from just obtaining a grade to promoting student learning and development. Especially in an online environment, where students can sometimes feel isolated, designing exams, particularly as a group, can be an engaging and empowering course assignment.

Assess Your Courses

Are you incorporating learner-centered outcomes, assignments, and assessments? Are students assuming the primary responsibility for learning? Are you taking the role of guide and facilitator? If you think you could make some course improvements in these areas, try one of the ideas described here. For assistance in changing to a learner-centered approach or choosing the best learning management tools to accomplish your goals, contact us at the e-Learning Center. We’re here to help!

Using Assessment to Engage Your Students

by Sue Pieper
Coordinator for Assessment, e-Learning Center at Northern Arizona University

How can we get students to attend our classes?

How can we get students to participate in class?

How can we prevent students from cheating?

These questions dominated the conversations at a recent Course Redesign (CR) Conference, and faculty in the small group sessions on assessment were adamant about getting some answers. While these were very real concerns for course instructors, now, a few weeks after the conference, I continue to wonder whether their questions were getting at the core of the issue. I wonder if these concerns might, in fact, be symptoms of an underlying “disease” infecting our students—particularly students in large classes. Could our students be suffering from a lack of engagement?

What is student engagement, really? Researchers and educators have tossed around the term for years. Definitions of student engagement have varied greatly, from students’ time-on-task behaviors (Brophy, 1983) to their use of metacognition and self-regulation (Pintrich and De Groot, 1990). Most definitions have included some element of students connecting with and thinking deeply about the course content. These are desirable outcomes for any classroom instructor, but they are not always easy to achieve, especially if you are teaching a class of 100+ students.

What’s an Instructor to Do?

There are plenty of ideas out there in the literature. I did a quick search of an education database for “student engagement” and found over 5,000 entries! The problem is in knowing where to begin. So here’s my attempt to provide assistance: three steps for using assessment to create a more engaging course. These steps are based on the CR Conference discussions as well as my work consulting with faculty who are designing assessments for their online, hybrid, and web-enhanced courses.

Take a look at what you’re already doing.

You can incorporate questions about assessment in your end-of-the-course evaluations. Better yet, throughout the semester you can ask the students about how the course assessments are going. Psychology professor Melissa Birkett recently developed a survey that she’ll administer at mid-semester in her PSY 255 course, Introduction to Cognitive and Behavioral Neuroscience. The survey asks students about a new portfolio project she’s incorporated into the course. In addition to some scaled items, Melissa asks students to write short answers to items such as “One aspect of this project I enjoyed was…” “One aspect of this project I found challenging was…” and “I learned the most from this project by…” By administering the survey mid-semester, Melissa will be able to make small modifications before the course ends, if necessary, as well as plan for next semester’s portfolio project. For other methods of gathering students’ reactions to your course assessments, check out the chapter on Techniques for Assessing Learner Reactions to Instruction in Thomas A. Angelo and K. Patricia Cross’s Classroom Assessment Techniques: A Handbook for College Teachers.

Think about what you might do differently.

Which course assessment strategies are working for you, and which need some work? CR Conference participants provided lots of ideas for rethinking how you use assessment in your course. Nora Dunbar, Psychology lecturer and CR Conference participant, suggested thinking about incorporating collaborative work. If you’ve historically used individual assignments and assessments, consider including group/pair work. Students learn from sharing ideas. You, the instructor, will benefit, too. If the group, rather than each individual student, is responsible for a discussion post or final project, then you will have fewer final products to assess.Paula Garcia McAllister, Director of Northern Arizona University’s Institutional Review Board, suggested using short writing assignments in courses. If you’ve routinely used only mid-term and final multiple-choice exams to assess students, think about integrating several brief papers throughout the semester and using a rubric. Think about your criteria for assessing students, and then in the rubric, for each criterion describe the performance levels that correspond to the grade or points you will award. Up-front work in constructing a solid rubric leads to huge payoffs for you and your students. Your students will know exactly what kind of performance you expect from them, and you will spend less time assessing. The quality of students’ work is also likely to improve as the semester progresses. For a quick overview of how to make and use a rubric, take a look at Peter Gow’s slide show.

Some of these ideas are discussed in more depth in the Tuesday Tips on Teaching with Technology podcasts Assessment and Large Course Redesign with Wally Nolan and Sue Pieper and Pedagogy and Large Course Redesign with John Doherty and Wally Nolan. These podcasts are available as part of a podcast series (iTunes U link) on using technology to improve teaching and learning. You might also find two books from the Jossey-Bass Guides to Online Teaching and Learning series helpful: Rita-Marie Conrad and J. Ana Donaldson’s Engaging the Online Learner and Rena M. Palloff and Keith Pratt’s Assessing the Online Learner.

Consider whether technology could help.

Once you’ve decided what you’d like to change about the assessments in your course, think about how technology might assist you. Cathryn Ellis, Theatre professor and CR Conference participant, added Blackboard Vista quizzes before the start of her class and found that drop/fail/withdraw grades decreased and student understanding of the course material increased. Short quizzes administered online throughout the semester provide immediate and ongoing feedback and help students to learn. They also help you know how and where to focus instruction. Other instructors are experimenting with “clickers,” automatic response systems that can be used to promote collaboration and critical thinking among students. For more about clickers, see John Doherty and Wally Nolan’s blog post, Pedagogy and Large Course Redesign. You might also want to check out a couple of resource sites that provide information about how to implement clickers and the impact of clickers on student learning. For more about web tools that could be used for assessment, take a look at Educause Quarterly Magazine, Volume 2, Number 4, 2009. The entire issue is devoted to technology tools and student engagement.

After you’ve implemented a new assessment technique, you’ll need to ask students again how your course assessments are working for them. If you continue to follow the steps listed above, you will be able to improve your course over time. However, these ideas are only the beginning of a treatment for lack of student engagement in our courses. I’d like to hear about what you’re doing to promote student engagement in your classes. Have you tried any of these strategies or techniques in your classes? How have you engaged students through assessment? Feel free to share your ideas in the comments.

Resources

Brophy, J. (1983). Conceptualizing student motivation. Educational Psychologist, 18 (3), 200-215.

Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82 (1): 33-40.