Reflections on teaching and learning reflection

by Dr. Sue Pieper, Coordinator of Assessment

Cartoon drawing of male student reading a book and thinking about himself reading the bookI recently read A Life in Bits and Bytes: A Portrait of a College Student and Her Life With Digital Media. The article portrayed a college student who, like many of her generation, is immersed in digital media. Katie Davis interviewed Anna, the student, asking her about the use of digital media in her life, her goals for using various digital media, and what opportunities and drawbacks she experienced from her daily media use.

Anna’s most striking observation came near the end of the interviews. She told Davis that while she appreciated being constantly connected to information and friends through her computer and her phone, at the end of the day she felt like she’d been “everywhere and nowhere.” Davis concluded that Anna’s portrait “highlights the need for and value of nurturing youth’s reflective practices and providing them with spaces for sustained reflection and authentic connection.” Others agree, including David M. Levy in No time to think: Reflections on information technology and contemplative scholarship (.pdf). In our current world of more-faster-better, he asks, “Where does one get the time to look and to think?” More specifically, how can you, as a college instructor, give your students time and space to think?

We might be able to solve this dilemma by using the same technology that created the dilemma in the first place. For college instructors, this can mean using technology to encourage reflection. Tools such as blogs (.pdf), wikis (.pdf), digital storytelling (.pdf), podcasting (.pdf), and even microblogs (.pdf), such as Twitter (.pdf), allow students to reflect as individuals and as part of a larger learning community.

The blog, a personal online journal that is shared on the Web, has become an increasingly popular tool for promoting student reflection. Blogs provide a venue where students (and instructors) can reflect on and write about course concepts, post their thoughts and any related links and media, and receive feedback and commentary from each other. Most course management systems have a blog tool that can be used by instructors and students. Alternatively, a number of free blogging services are available, including Blogger (part of Google), Movable Type, and WordPress, among many others.

Drawing of Asian student reading blog on laptopHow can blogs be used to promote reflection? A recent study (.pdf) by Shih-Hsien Yang described the use of blogs among student teachers training to teach English as a Foreign Language in Taiwan. Students in two classes were required to post their thoughts on a blog following each class meeting as well as to voluntarily respond to their peers’ messages. The instructors teaching the classes commented on their students’ postings and asked questions to challenge their thinking. The authors found that all students were reflective in their comments and some went beyond description to demonstrate critical thinking about their teaching and learning experiences. They also found that all students considered the blog a useful tool for reflecting and communicating with each other.

No matter which technology tools and strategies you use to promote and support reflection in your class, it’s important to remember that reflection is most effective when it is thoughtfully designed and integrated into course activities and assignments. Jan Harrington and Ron Oliver illustrated effective design for reflection in their article Designing for Reflection in Online Courses (.pdf). In one example, the authors incorporated a reflective journal in a Graduate Certificate in Online Learning course. Students in the course were asked to play the role of a college instructor and redesign a unit that they were currently teaching face-to-face for online delivery. They were also asked refer to the pertinent literature and to keep a journal of their thoughts about the differences between face-to-face and online delivery, including the strengths and weaknesses of each delivery mode. Finally, students were asked to submit a plan for an online unit, their edited journal, and a short article from an instructor’s perspective on the process of redesigning a face-to-face course for online delivery. The reflective journal became an integral part of the students’ course redesign task—not just an add-on to the assignment.

Can college instructors provide students with time to think? Levy concluded his article by calling for those of us in higher education to lead the way. By carefully designing for reflection and choosing tools and strategies that support reflective practices in your classes, you can do just that.

What are you doing to promote reflection in your own classes? Which tools and strategies have been most successful? Please share your ideas in the comments.

Textbooks of the Near Future

by Larry MacPhee, Associate Director

Drawing of printed textbooksWhen I went to college, back in the 1980s, each of my new hardcover textbooks weighed over 5 pounds and cost over $100.00. Cheaper used and softcover texts weren’t yet readily available. Since that time, increasing numbers of students have been selling their textbooks back to the bookstore or other re-sellers in order to get a wad of cash to fund a keg-party or the next semester’s textbook purchases. The increased availability of used textbooks has driven publishers, they argue, to raise the prices of their new texts and, at least to my skeptical eye, to make numerous small changes to each edition and release these new editions at ever shorter intervals to try to reduce the usefulness of old editions. So, for decades, students and publishers have been locked in an “arms race” that hasn’t been particularly good for either side. That’s about to change.

Kindle logo (person sitting under a tree, reading)Enter, the model for a disruptive new relationship between students and textbook publishers. Amazon is the world’s largest bookseller and they now sell more eBooks than paper books. They deliver their eBooks via the Kindle, but also through the free Kindle reader app for Android phones, iPhones, and iPads because Amazon only cares that you buy their content; not what you read it on. I bet not too many college students own Kindles, but they sure like their smartphones! I recently asked a fairly typical group of over 100 university students how many of them owned “smartphones.” Almost every hand in the audience went up, so most students already have a mobile device capable of reading eTextbooks. I also asked them how many were currently using electronic textbooks. Not a single hand went up. In the business world, this is what people call an “opportunity.”

Student using a smartphoneThe big academic publishers in K-12 and higher-ed, including Wiley, Pearson, Cengage, Benjamin Cummings, Houghton Mifflin, MacMillan and all the rest, are ready to get into the game. They’ve been watching Amazon long enough now to see that it’s a winning strategy. According to the publishers, and I don’t doubt their numbers, about one third of textbooks purchased annually are used, not new. Each of those re-sales is lost profit for the publishers. But because of something called DRM, or “digital rights management,” students won’t be able to re-sell their eTexts. While there are plenty of ways in which eBooks might be superior to paper books, the big one for the publishers is DRM. With eBooks, the used textbook market is dead. It’s also possible that the publishers will profit from not having to print and distribute physical books, but at least some of those profits will be offset by the need to publish online editions, and maintain servers and a larger IT infrastructure. Publishers will tell you that students are going to love eTexts for the mobility, reduced weight, the ability to get corrections, updated content, and for the multimedia elements that make the eBook a richer learning experience. While both paper and electronic editions exist side by side, you can even expect the eText to be cheaper to drive customers into the new market. So students will like eContent, and publishers will profit from it. But convincing faculty, most of whom don’t particularly like technology or change, that eTextbooks are worth the effort will be a challenge.

Image showing ebook creation in Apple's iBooks Author toolFaculty control the textbook adoption process, and they remain somewhat skeptical that moving to eTexts is worth the effort. Publishers could try to pressure them by eliminating the paper edition, but that might drive an instructor to select a competitor’s product. They could use student demand, by making the eText cheaper than, and different from, the paper edition. They might even try to convince faculty with incentives like a free iPad, or by encouraging faculty to “build your own book” by assembling chapters of pre-built content. In the days of the printed text, especially in the K-12 market where California and Texas heavily influence content decisions, the publisher sometimes faced the challenge of trying to satisfy diverse customers with the same content. Now the “controversial” chapter on Evolution or Global Warming or the Big Bang or Birth Control Methods can be easily deleted or replaced, because the customer is always right! In higher ed, the ability to easily mix and match digital content may also appeal to instructors who want to customize their courses. Apple hopes faculty will start writing their own eBooks for iPad using their free tools. I don’t think the faculty will go willingly into this brave new world, but it’s probably going to happen whether they like it or not.

Designing Rubrics That Work

by Dr. Suzanne L. Pieper, Coordinator of Assessment

An essay by Vincent Tinto in Inside Higher Ed reminds us that college success is built “one class and one course at a time.” Tinto further points out that in successful college classrooms, students get frequent, high-quality feedback on their products and performances. One way instructors can provide that feedback is by using rubrics to assess student work. Rubrics are efficient for making instructors’ expectations explicit and promoting fairness and consistency.

Why don’t more instructors use rubrics? A major obstacle is the amount of time it takes to construct a good one. A tempting shortcut is to choose a rubric from the wide variety of those online and in print. But how do you know if the rubric you choose is a good one? And how do you know if it will work for you and your students? Three questions will help you to choose and use the best rubric for your class.

Photo of man in suit and tie peering through a magnifying glassWhat am I looking for?

Think about three to five criteria that you could use to assess student responses to a performance task. In What’s Wrong—and What’s Right—with Rubrics, W. James Popham says it’s tempting to describe all possible assessment criteria, but it’s best to keep your rubric brief. When designing or selecting a rubric, ask yourself, “What are the most important elements of this assignment that demonstrate student learning?”

Let’s imagine that you want students to be able to write a research paper. You find an online rubric that lists the following criteria for assessing a research paper:

You wonder if you want to assess twelve criteria. You also notice that some of the criteria aren’t distinct. What is “feel,” and how does it differ from “tone”? When you look at the intended learning outcomes for your course, and you scrutinize the research paper assignment you’ve given to your students, you discover that five criteria are most important:

  • Purpose
  • Organization
  • Content
  • Mechanics
  • Use of references

Now you have a manageable number of rubric criteria that will guide your students to improve their skills in writing research papers. An added bonus: you can use these same criteria for a variety of writing assignments in your course.

Photo of three rulers, one yellow, one blue, one redWhat is the possible range of student products/performances?

Rubrics need to accommodate the entire range of possible student responses, but how do you decide how many separate levels of performance you want to recognize in your rubric? The best way is to review actual student work. Start by sorting the work into upper range and lower range responses, and then further sort the work as needed. How many “piles” do you have? The number of piles should give you an idea of how many performance levels you will need in your rubric. Each performance level needs to be clearly distinct from the next so that there is no question about which level a particular piece of student work meets. Most rubrics allow three to five levels; if you include more levels, you might find it difficult to clearly distinguish the levels. You also want to think about how to label the performance levels. The labels should make clear the distinctions among levels but not discourage students. Here are some examples.

Photo of female student studying at a computer.Let’s say you want to assess discussion posts in your online course. You know what you’re looking for, and you’ve identified five levels of student performances from your review of prior student discussions. However, you aren’t sure how to describe the levels of student performance. Say you find a collection of performance level descriptions online, and you think one set of descriptions shows promise: accomplished, advancing, developing, beginning, no concept.

Ask yourself how you would distinguish between “beginning” and “developing.” Also think about how your students might react to being described as having “no concept.” You realize that the set of descriptions seems to apply to student development rather than to the students’ work, so you instead might settle on five different performance levels for online discussion: excellent, good, average, fair, and poor. These levels reflect the range of your students’ performances, make it possible to distinguish between performance levels, and are not discouraging to students.

How do I describe what I am looking for at every point in the range of student products/performances?

Now that you have identified what you are looking for and the possible range of student performances, you are ready for the final step in adopting or adapting your rubric: the descriptions. The descriptions are the “meat” of the rubric because they explicitly detail what a student needs to do to get a score at each scale point. They also provide instructors with clear guidelines for improving student learning. Descriptions need to be consistent, distinct, and written “in plain English” so students can understand them.

Generic example of clearly distinct levels
of performance
Top Level Middle Level Low Level
Do X, Y, and Z Do X and Y Do X

Photo of a male student giving a speechLet’s say you want to assess oral presentations in your course. You know what you’re looking for, you’ve identified and labeled three levels of student performance, but you’re not sure how to describe exactly what a student needs to do to get a score at each point in the scale. You find an oral communication rubric in a resource book that at first glance looks perfect! When you inspect the rubric more carefully, however, you notice that the descriptions don’t always focus on the same characteristics across performance levels. For example, you want to assess pacing, yet pacing is described as “paced for audience understanding” in the high performance level, “sometimes too fast or too slow” in the middle performance level, and not mentioned at all in the low performance level.

You revise the descriptions of pacing so that the characteristic is addressed at all performance levels, and that the value of the characteristic changes in a measurable way between adjacent levels. Drawing from an example given by Robin Tierney and Marielle Simon in What’s still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels, you modify the rubric so that at the high performance level, the student “always paces for audience understanding.” At the middle performance level, the student “sometimes paces for audience understanding.” At the low performance level, the student “doesn’t pace for audience understanding.” With a few more revisions, you can use your rubric to effectively assess your students’ oral presentations. Additionally, when you share the rubric with your students, they can use it to better understand your expectations and deliver higher quality presentations.

Example of clearly distinct levels
of performance
Excellent Acceptable Needs Improvement
Always paces for audience understanding Sometimes paces for audience understanding Doesn’t pace for audience understanding

What successes have you had with designing rubrics for your courses? What challenges have you encountered? Please share your comments.


Popham, W. J. (1997). What’s wrong—and what’s right—with rubrics. Educational Leadership, 55, 72-75.

Tierney, Robin & Marielle Simon (2004). What’s still wrong with rubrics: focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2).

We’re hiring, and here’s how to get the job

by Lorraine B. Elder and Larry MacPhee, Associate Directors

[UPDATE: This post was originally written in 2012, but the information still applies to our job openings whenever they occur.]

We have a job opening. This blog post offers tips about the RIGHT way to apply for a job with the e-Learning Center at Northern Arizona University (NAU) or, really, almost anywhere.

If you don’t meet the minimum qualifications, don’t bother applying

Whenever we have a job opening we get dozens of applications. Usually more than half of them don’t merit a second glance. Why? We mean it when we state the minimum qualifications for the job. If you don’t meet ’em, don’t apply. We know—you’re thinking it’s like the lottery. Can’t win if you don’t play, right? The part of the job description that trips up unqualified applicants is that line about “combination of experience, training, and/or education.” If you’ve got to sell us on that, your equivalent education or experience can’t be much of a stretch, and it needs to meet NAU’s definition of equivalency. If you don’t meet the minimum qualifications, we can’t legally hire you, so don’t waste your time or ours.

Photo of green characters on a black computer screen, reminiscent of the movie The Matrix

Photo by My Melting Brain on Flickr

The matrix isn’t just a movie

Our employment searches are a committee affair. The committee chair—usually the hiring manager—creates a matrix (an Excel spreadsheet) listing the job qualifications based on the posted job description. Some qualifications are weighted more than others, and if you are truly a qualified candidate, you’ll probably be able to figure out which ones. Be sure to address those in your cover letter and resume.

The search committee members score your application against the matrix, and the highest scoring applicants get considered for an interview. The first item we score applications on is whether you meet the minimum qualifications. If you don’t, your application immediately goes onto the reject pile.

Your cover letter

A cover letter is not required, but you’d be foolish not to write one, because the cover letter is where you can point out how well your qualifications line up with the “skills and abilities” section of the job description, which directly affects how well you score on the matrix. If you’re qualified and you write a strong cover letter, you greatly increase your chances of getting an interview. Use the cover letter to explain how your experience in peripherally related areas makes you a stronger applicant, and explain any noticeable gaps in your employment history. Don’t ramble in the letter, and do strike a balance between confidence and humility. Don’t address your cover letter to “Dear Sir.” That really irritates the women on the search committee.

Illustration of person sending correspondence from a computerYour resume

We like resumes and cover letters that make it easy for us to score your application, so be crystal clear about how your experience matches what we’re looking for. But don’t load up your letter and resume with buzz words if you can’t back up the lingo. If you’re bluffing, we’ll catch on soon enough.

Your resume can also sink you. Remove references to extinct programming languages and defunct applications that make you look dated. If the last version of Windows you’re familiar with is ’95, you’re done.

We do a lot of design work here. That means more than half the staff are Mac users, and many of us use PCs too. If your resume is dripping with contempt for Apple products, or if you are a Mac fanboy who hates Windows, we don’t want you. You need to be able to work with all kinds of people and all kinds of tools. If you make a clever joke about Google being more evil than they used to be, we won’t argue with you, but if you choose to snark, do it very carefully when applying for a job. You don’t know whose buttons you might be pushing.

We’re looking for experienced people, not newbies, so extensive previous employment is expected. That means it’s okay, even preferable, to send us a two-page resume, but fill the space wisely, with details that clearly illustrate how you meet the requirements of our job. You don’t have to be a master of every single bullet point in the job posting, but you’d better be pretty good at the majority of them. Don’t waste resume space on pointless objective statements. A generic objective is inane, and one that says your objective is to get hired for the job you’re applying for falls into the “Duh” category.

Drawing of a letter that says Dear Sir, I'm grate! For realz! Higher me!

Don’t do this

Proofread your letter and resume, and have someone else look them over for you. Embarrassing typos suggest to us that you’re not attentive to detail.

If there’s anything odd on your resume, someone on the committee will notice. For example, if none of your references is from a past supervisor, we will wonder why. If you didn’t explain gaps in your employment history, we will wonder whether you’re trying to hide a job that didn’t go well. That alone might not rule you out, but if you get an interview, we’re going to ask about it. If your work history consists of very short stints with different companies, we will wonder whether you get along well with others. The committee will be looking for a pattern of relevant work experience and a progression of responsibility. We will Google you.

Photo of the robot C-3PO from Star Wars

Photo by Andres Rueda on Flickr

Technical proficiency is required

If your technology prowess is on the level of successfully scanning your ramen noodles in the self-service line at the grocery store, you are not the person we seek. That doesn’t mean you need mad skillz on the order of having written your own Linux compiler when you were knee-high to C-3PO, and you don’t need to have developed a custom LMS that runs on your cell phone (although we’ll give you props if you have). But you do need to understand computing and the web in a hands-on way.

Being an uber geek isn’t enough

Sure, it’s cool if you know and love all gadgetry, and your resume is all alphabet soup (HTML, CSS, LMS, ADA, CS6, W3C, etc.), but if you can’t carry on an intelligent, unintimidating conversation with nontechnical people, and you don’t understand the culture and practices of higher education, you’re not the right one for us. You’ll be working directly with faculty, and the quickest way to turn them off is to start spouting technical jargon. Knowledge of teaching and learning is even more important than technical proficiency. The best applicants are those who know how and when—and when NOT— to apply technology in education. You’ve got to be able to teach faculty how to use software and hardware that they might be very vocally opposed to using, and you’ve got to do it with kindness and a smile. We want confidence and diplomacy, not arrogance, and we want people who thrive on patiently teaching and helping others.

Apply promptly and do it right

Our current job opening was posted without a closing date. However, as soon as the chair of the search committee thinks we have enough promising-looking applications, we’ll close the job posting with three days’ notice. If you’re interested in the position, don’t dawdle about applying. When you apply, follow the instructions on the Human Resources web site. You MUST fill out the application using HR’s system, even though it is tedious. Don’t email your resume directly to the department.

Photo of person waiting beside phone, hoping phone will ringDon’t be a pest and do be patient

We’re not opposed to the occasional polite, short inquiry—preferably by email—about some aspect of the job. But don’t bug us with daily or weekly emails or phone calls about when we’re going to interview you or tell you what is going on. Hiring at a university is a SLOW process. It will be many weeks before you hear something from us.

The least qualified applicants—those who don’t meet the minimum qualifications—will usually get a “Dear John” letter via email sooner than the other applicants because they get weeded out almost immediately after the committee starts reviewing applications. But that could still be two months or more after the job opening was first posted.

Once the matrix scoring is completed and the committee convenes to discuss the highest-scoring applications, which in itself can take weeks because of scheduling conflicts, we will invite the most promising applicants for an interview.

The applicants in the middle of the pack have to wait the longest before hearing something. We hold off on notifying them because we might eventually invite them for an interview if the highest-scoring applicants don’t pan out. Usually, though, the also-rans get their Dear John email only after we’ve made a job offer to another applicant, and that person has accepted the position.

Drawing depicting a job applicant and a three-person committee writing notesThe interview

We sometimes do the initial interviews in person, sometimes on the phone, sometimes by videoconference. If you can interview in person, do so; you will be better able to read the people in the room and adjust your responses accordingly. Second interviews are usually in person and typically are reserved for only the top one, two, or (rarely) three applicants. Again, that can take several weeks.

There are pros and cons to being either the first or last interviewee. If you’re first and you are excellent, you set a high bar for the remaining applicants, but your stellar performance might fade from the committee’s memory by the time of the last interview. If you’re last, you can leave a positive impression on the committee, but by then the committee might be so weary of the hiring process that you practically have to tap dance to get their attention.

Before the interview, do your research. That means digging through our web site (yes, we know it needs work, but we’ve been busy and short-handed), following us on Twitter, finding us on Facebook, reading our blog, etc. That will answer a lot of questions you might have.

On interview day, show up on time. Dress appropriately. We’re not a suit-and-tie kind of department, but holey jeans and a funny-but-NSFW t-shirt are not appropriate choices for an interview. Dress as if you respect us. You can wear your purple Crocs after we hire you. For an interview, we suggest “business casual,” but we won’t penalize you for going fancier. Skip the tux or ball gown, though.

The committee chair will tell you what to bring to the interview. In advance of your interview we’ll probably ask you to send us links to online examples of your work. We might ask you to prepare a short training session or some kind of demo on a topic of your choice.

Have some good questions to ask us, but be sure they aren’t questions you could’ve answered yourself if you’d done your research. A candidate who interviews us will earn our respect. Just as we need to decide whether you’re the right fit for us, you need to figure out whether our job fits your talents and whether you can embrace our quirks and foibles.

Show enthusiasm. If you think the job really is a good fit, make us believe you want the job. Not just a job, but our job. And even if you are thinking it, try not to tell the interviewers that what you really want is one of their jobs. That doesn’t go over well. Nor does arguing with the committee or correcting their grammar or belligerently challenging the accuracy of something said. (Yes, past applicants have exhibited all of these behaviors during interviews.)

We know you’re nervous when you’re being grilled by several people who can decide your employment fate, but try to relax and let your sense of humor come through. Unless you like puns. We’ve got one of those already.

Drawing of a dollar symbolThe salary is what it is

If we post a single figure for the salary, it means that’s either all we’ve got the budget for, or that figure represents internal salary equity, and there’s no negotiating room. If we post a salary range in the job description, and you can convince us you’re extremely well suited for the job, we will consider hiring at the high end of the range. Salaries at NAU are often below average.

The thank-you note

We appreciate thank-you notes, but we’re not gonna complain to your mom if you don’t write one. A thank-you note is an excellent way to send the committee any follow-up information you promised during the interview. We like emailed thank-you notes better than snail mail. We don’t like thank-you phone calls.

Drawing of mortarboard and diplomaAcademia

If your employment history is in a sector other than higher education, brace yourself for culture shock. People here are free-thinking, outspoken, informal, and nonhierarchically organized, but universities are also steeped in tradition, rife with bureaucracy, and slow to change. Practices that would be unthinkable or trivial in the corporate world can be entrenched or strangely significant here.

Internal, local, and foreign applicants

We sometimes get asked if our job openings are just for show because we already have an internal candidate in mind. The answer is no. If we think we have sufficiently qualified candidates within the department, we post positions as being open only to internal applicants. If the positions are posted as open recruitments, we really do want external applications, but of course we will also consider well-qualified candidates who apply from within NAU, and internal candidates can definitely have an advantage. If you are already an NAU employee, you will almost certainly be more familiar with many of the processes and tools used here, and you probably have already passed the university-required background check. If the committee wants someone who can get up to speed especially quickly, those factors can help, but they’re no guarantee. Being an internal candidate can work against you if the committee thinks an external perspective is especially valuable for the position or if you’ve received several negative performance appraisals. We do check those.

If you already live in Flagstaff, we know you won’t be scared off by the climate, the high cost of living here, or the need to buy or sell a house in a tough market. But we have hired many staff members from distant locations. We do generally want people who can work in our offices on the Flagstaff campus, as opposed to full-time telecommuting.

If you are not a U.S. citizen, and you don’t already have a green card, it is theoretically possible that you can get hired, but you must procure an H1-B visa, which means the committee has to write a justification stating that we were unable to find a suitably qualified candidate who is a citizen. In the current job market, that’s not likely. If you’ve got a green card already, you’re welcome to apply.

Drawing of a trophy labeled "The Best"What we really want

Overall, we want the person who has the best mix of skills and experience, as well as a great, collegial personality to fit into the department and to work successfully with our clientele. If you meet those criteria, you improve your odds of getting hired.

Helping Students to Succeed

by Dr. Sue Pieper, Coordinator of Assessment

With more students struggling academically, particularly in the first year, and fewer students persisting to graduation, many of us in higher education are asking how we can help our students to succeed. One way is to teach students to “think about their thinking.” In other words, we can teach them to develop metacognitive skills that will help them to become aware of their own thinking processes and use that self-awareness to regulate those processes. Researchers and practitioners agree that metacognition is critical to academic success.

In a previous blog post, I promised to provide some easy-to-implement strategies for teaching metacognition in face-to-face and online courses. When I reviewed the research and talked to colleagues about what they were doing to promote metacognition in their classes, a theme emerged: The best way to teach metacognition is to do it in conjunction with activities and assignments that are already a part of your class. Here are some strategies for teaching your students to think about their thinking when they take exams, listen to lectures, or work on writing assignments in your class.

Taking Exams

Drawing of student holding a test with a grade of FCollege students are often unaware of what they know and don’t know (Zabrucky and Bays, 2011). When taking exams, students frequently overestimate their level of understanding and readiness to take a test. First-year students in particular report that “looking over their notes” before an exam has worked well for them in the past (Ruban and Reis, 2006), and they are shocked when they receive exam scores that are lower than expected.

Karen Zabrucky and Rebecca Bays suggest that we can help students better understand what they know and don’t know by asking them to predict their exam scores right before they take an exam and then also estimate their scores right after taking an exam but before receiving their grades. Students can then compare their predictions and estimates with their actual exam scores. The authors also suggest that instructors ask students questions about how they studied for an exam and whether they felt they were adequately prepared to take the exam. These questions prompt students to reflect on both their level of preparation for an exam and the consequences of their level of preparation.

Listening to Lectures

Drawing of professor at lectern showing an empty speech bubbleShawn Nordell (2009) conducted research with students in a large introductory biology course and found that most students had difficulty recalling course knowledge. When students were asked to write down two or three of the main points discussed in a lecture and readings, most students had no response at all or could remember only a key word or phrase.

Marsha Lovett (2008) described a technique called “wrappers,” activities that wrap around a learning activity or assignment and can be used to foster students’ metacognitive skills, including the recall of course knowledge. Instructors can “wrap” a lecture by presenting tips on active listening before the lecture, having students write down the three key ideas from the lecture immediately after the lecture, and then giving students a list of the three key ideas from the lecture for students to self-check. Lovett found that over time the students’ three key ideas increasingly matched those of the instructor.

Working on Writing Assignments

Drawing of student preparing to writeOne way to prompt students to reflect on their writing is to provide them with questions for self-assessment. Before students begin to write, ask them to answer questions such as “What are my goals for this writing assignment?” or “What do I need to do to prepare to write?” You can also ask students to answer questions right after they write. Here are some suggested questions, adapted from a questionnaire used in a large-scale university writing assessment at Truman State University:

  • How do you feel about your finished writing sample?
  • How representative is this sample of your writing?
  • Describe your writing process.
  • What do you feel is especially strong about your writing sample?
  • What do you feel could be improved in your writing sample?

The Writing Place at Northwestern University offers good examples of self-assessment questions and a worksheet for students to use when evaluating their own writing.

All of these strategies help students to practice their metacognitive skills and grow as learners. Have you tried any of these strategies in your online or face-to-face class? What worked, and what didn’t work? What other methods have you used to encourage students to think about their thinking? Please comment on your experiences with teaching your students how to learn.


Lovett, M. C. (2008, May 5). EDUCAUSE Learning Initiative Events. Retrieved May 18, 2008, from Metacognition and Monitoring: Understanding and Improving Students’ Skills for Learning.

Nordell, S. E. (2009). Learning how to learn: A model for teaching students learning strategies. Bioscene: Journal of College Biology Teaching, 35 – 42.

Ruban, L., & Reis, S. M. (2006). “Patterns of Self-Regulation: Patterns of Self-Regulatory Strategy Use among Low-Achieving and High-Achieving University Students. Roeper Review, 148 – 156.

Zabrucky, K. M., & Bays, R. (2011). Helping students know what they know and do not know. College Teaching, 123.

Posters and Slides from the 2011 Blackboard World Conference

At the Blackboard World conference in July 2011, several of the e-Learning Center’s staff, along with other Northern Arizona University colleagues, gave poster sessions or presentations that were well received and generated a lot of interest among conference attendees. By request, we’re making the posters available in a downloadable size (17 x 11 inches, .pdf). Click on each small poster image below to see the larger .pdf version. The posters were created by the e-Learning Center’s Creative Design Group. PowerPoint slides (.pptx) are also available for one presentation.

Small image of larger PDF posterListening to Student Voices: Assessing and Responding to Students’ Experiences with Blackboard Learn

A poster session by Dr. Sue Pieper, Coordinator of Assessment,
and Don Carter, Director

Last year, as part of a pilot project in which Northern Arizona University investigated Blackboard Learn for possible adoption as the university’s learning management system, we surveyed our students, asking them about their experiences in using Blackboard Learn in the pilot courses. The results of that survey are summarized on the poster along with recommendations for Blackboard and for other institutions that are considering using Blackboard Learn.

Complete survey results are available in the full research report.

View the poster (17″ x 11″ .pdf).

Smaller image of larger PDF posterThe Pedagogical Opportunities of Mobile Technologies: iPads and Kindles in the Classroom

A poster session by Dr. John Doherty, Instructional Designer,
and Kevin Ketchner, Librarian, Cline Library

With mobile devices beginning to enter the classroom, students are only two finger-swipes away from the game Angry Birds Rio, or YouTube, or any number of other popular diversions. Ketchner and Doherty describe how to use mobile apps and tools in ways that engage students in class and encourage student interaction with peers, instructors, and content. They provide examples of appropriate and effective choices that can enhance learning.

View the poster (17″ x 11″ .pdf).

Smaller image of larger PDF poster(R)eflective Learning: The Experience of Journaling from Campus Edition to Bb Learn

A poster session by Dr. John Doherty, Instructional Designer,
and Kevin Ketchner, Librarian, Cline Library

This poster is organized around Dee Fink’s Creating Significant Learning Experiences (Jossey-Bass, 2003). Ketchner and Doherty deconstruct Fink’s model, presenting an application and example of how online journaling can enhance students’ critical
reflection, self-reflection, thinking, and metacognition. They contend that reflective learning through journaling leads to insight and significant learning experiences for students.

View the poster (17″ x 11″ .pdf).

From Vista 8 to Blackboard Learn 9.1— Lessons
Learned and Tips for Success

A presentation by Dan Stoffel, Assistant Director,
and Erin Shelley, Blackboard Learn System Administrator

Northern Arizona University has just made the transition from Blackboard Vista 8 (formerly WebCT Vista) to Blackboard Learn 9.1. The process of moving to a new learning management system includes the migration of nearly 2000 courses from the old system to the new. This presentation describes NAU’s course migration process and suggests strategies for other institutions who are following a similar path.

Download the PowerPoint slides (.pptx).

Pedagogy and Online Learning: Training Users on Backward Design

A presentation by Dr. John Doherty, Instructional Designer,
and Wally Nolan, Instructional Designer

The design and development of online content is sometimes too focused on the learning management system rather than on students’ learning. The backward design approach, described well in Understanding by Design by Grant Wiggins and Jay McTighe (Pearson, 2006), helps instructors begin with the end—learning—in mind. This presentation described self-paced online tutorials that guide instructors and course creators to align their learning objectives, assessments, and learning activities independent of a learning management system.

Teaching Students How to Learn

by Dr. Sue Pieper, Coordinator of Assessment

Photo of instructor standing in front of blackboard as four students watch.An Inside Higher Ed article titled “Can Students Learn to Learn?” piqued my long-standing interest in students and metacognition. As a teacher of writing for nearly twenty years, I have been intrigued by differences in the ways that students learned and have sought to understand how two students of similar ability could perform so differently in my English composition classes. Over many years of working with student writers, I noticed that most of them didn’t take the time to reflect on their writing. They weren’t asking basic questions: What worked well in my writing? What didn’t? What can I try next time? What I observed in my students was a lack of metacognitive skills.

What is metacogniton?

Strict definitions vary, but the common definition of metacognition is “thinking about thinking.” Many researchers believe that metacognition involves two steps: knowing one’s own cognitive processes and using that self-awareness to regulate those processes (Brown, 1981; Niemi, 2002; Shimamura, 2000). By comparing the behaviors of expert and novice learners, researchers (Butler, 1997; Lovett, 2008; Pintrich, 2000; Winne & Hadwin, 1998) have discovered that experts, unlike novices, routinely move through a self-regulation cycle of

  1. Photo of statue "The thinker" (a seated figure whose chin is propped on hand and elbow on knee)Planning
  2. Setting goals
  3. Applying strategies
  4. Monitoring
  5. Evaluating
  6. Adapting

By teaching these self-regulation strategies to our students, we can improve their learning.

The “Can Students Learn to Learn?” article reported on the efforts of members of the Associated Colleges of the Midwest (ACM) who are doing just that. One of the projects highlighted was led by Kristen Bonnie, an assistant professor of psychology at Beloit College. Bonnie described a simple technique for prompting metacognition in her students. On a multiple-choice test, she lets students select a few questions that they don’t want her to grade. She also requires them to give a reason for their selection, providing a list of reasons for students to choose from, for example “I don’t remember the material” or “I’m not confident in my answer.” Her goal is to make students stop and think more about why they don’t know an answer. The ACM projects are only two years old, but they are already showing promise in terms of improving student learning.

Photo of a female and male professor arguingThe comments following the article were even more interesting than the article itself. Many commenters were already using metacognitive strategies in their own classrooms or were eager to try these techniques. Other commenters disagreed with the idea of teaching students how to learn, asserting that the goal of university instructors is to teach their subject and that teaching metacognition skills is a waste of instructors’ time. Instead, they argued, students should come to college prepared and should be held accountable for their work.

One commenter asked, “What good is teaching if too many students are not actually grasping the material?” Her question is a good one. Students should already know how to learn, but many students don’t have these skills. We need to teach students how to learn, not only to help them to understand the content we teach, but also to give them the opportunity to practice and improve their metacognitive skills. Most importantly, we want them to develop a lifelong habit of thinking about their thinking.

Some higher education professionals are already working on this. One commenter from the Office of Faculty Development at California State University Channel Islands described an intriguing initiative, a “learning across the curriculum” movement, where faculty were encouraged to teach one thing about learning in every course. Ideally, teaching students how to learn should happen in all courses across the curriculum and in all modes of delivery, but teaching metacognition might be especially important in online courses. Macdonald (2004) argued that to be competent, e-learners need a self-directed approach to learning, in addition to basic information and communication technology skills, information literacy skills, and collaborative learning skills. In my next post, I’ll explore some easy-to-implement strategies for teaching metacognition, both face-to-face and online.

Are you teaching students how to learn in your courses? If so, what strategies are you using? Please comment on your experiences with teaching students to learn.


Brown, A. (1987). Metacognition, executive control, self control, and other mysterious mechanisms. In F. Weinert & R. Kluwe, Metacognition, motivation, and understanding (pp. 65-116). Hillsdale, N. J.: Erlbaum.

Butler, D. (1997). The roles of goal setting and self-monitoring in students’ self-regulated engagement of tasks. Paper presented at the annual meeting of the American Educational Research Association. Chicago, IL.

Lovett, M. C. (2008, May 5). Educause Learning Initiative Events. Retrieved May 18, 2008, from Metacognition and Monitoring: Understanding and Improving Students’ Skills for Learning

Macdonald, J. (2004). Developing competent e-learners: The role of assessment. Assessment and Evaluation in Higher Education, 215-226.

Neimi, H. (2002). Active learning—a cultural change needed in teacher education and schools. Teaching and Teacher Education, 763-780.

Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner, Handbook of self-regulation (pp. 451-502). San Diego, CA: Academic Press.

Shimamura, A. P. (2000). Toward a cognitive neuroscience of metacognition. Consciousness and Cognition, 313-323.

Winne, P., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. Hacker, J. Dunlosky, & A. Graesser, Metacognition in educational theory and practice (pp. 277-304). Mahweh, NJ: Erlbaum.