An essay by Vincent Tinto in Inside Higher Ed reminds us that college success is built “one class and one course at a time.” Tinto further points out that in successful college classrooms, students get frequent, high-quality feedback on their products and performances. One way instructors can provide that feedback is by using rubrics to assess student work. Rubrics are efficient for making instructors’ expectations explicit and promoting fairness and consistency.

Why don’t more instructors use rubrics? A major obstacle is the amount of time it takes to construct a good one. A tempting shortcut is to choose a rubric from the wide variety of those online and in print. But how do you know if the rubric you choose is a good one? And how do you know if it will work for you and your students? Three questions will help you to choose and use the best rubric for your class.

What am I looking for?

Think about three to five criteria that you could use to assess student responses to a performance task. In What’s Wrong—and What’s Right—with Rubrics, W. James Popham says it’s tempting to describe all possible assessment criteria, but it’s best to keep your rubric brief. When designing or selecting a rubric, ask yourself, “What are the most important elements of this assignment that demonstrate student learning?”

Let’s imagine that you want students to be able to write a research paper. You find an online rubric that lists the following criteria for assessing a research paper:

• Purpose
• Content
• Organization
• Feel
• Tone
• Sentence structure
• Word choice
• Mechanics
• Use of references
• Quality of references
• Use of most recent edition of MLA Handbook for Writers of Research Papers

You wonder if you want to assess twelve criteria. You also notice that some of the criteria aren’t distinct. What is “feel,” and how does it differ from “tone”? When you look at the intended learning outcomes for your course, and you scrutinize the research paper assignment you’ve given to your students, you discover that five criteria are most important:

• Purpose
• Organization
• Content
• Mechanics
• Use of references

Now you have a manageable number of rubric criteria that will guide your students to improve their skills in writing research papers. An added bonus: you can use these same criteria for a variety of writing assignments in your course.

What is the possible range of student products/performances?

Rubrics need to accommodate the entire range of possible student responses, but how do you decide how many separate levels of performance you want to recognize in your rubric? The best way is to review actual student work. Start by sorting the work into upper range and lower range responses, and then further sort the work as needed. How many “piles” do you have? The number of piles should give you an idea of how many performance levels you will need in your rubric. Each performance level needs to be clearly distinct from the next so that there is no question about which level a particular piece of student work meets. Most rubrics allow three to five levels; if you include more levels, you might find it difficult to clearly distinguish the levels. You also want to think about how to label the performance levels. The labels should make clear the distinctions among levels but not discourage students. Here are some examples.

Let’s say you want to assess discussion posts in your online course. You know what you’re looking for, and you’ve identified five levels of student performances from your review of prior student discussions. However, you aren’t sure how to describe the levels of student performance. Say you find a collection of performance level descriptions online, and you think one set of descriptions shows promise: accomplished, advancing, developing, beginning, no concept.

Ask yourself how you would distinguish between “beginning” and “developing.” Also think about how your students might react to being described as having “no concept.” You realize that the set of descriptions seems to apply to student development rather than to the students’ work, so you instead might settle on five different performance levels for online discussion: excellent, good, average, fair, and poor. These levels reflect the range of your students’ performances, make it possible to distinguish between performance levels, and are not discouraging to students.

How do I describe what I am looking for at every point in the range of student products/performances?

Now that you have identified what you are looking for and the possible range of student performances, you are ready for the final step in adopting or adapting your rubric: the descriptions. The descriptions are the “meat” of the rubric because they explicitly detail what a student needs to do to get a score at each scale point. They also provide instructors with clear guidelines for improving student learning. Descriptions need to be consistent, distinct, and written “in plain English” so students can understand them.

Let’s say you want to assess oral presentations in your course. You know what you’re looking for, you’ve identified and labeled three levels of student performance, but you’re not sure how to describe exactly what a student needs to do to get a score at each point in the scale. You find an oral communication rubric in a resource book that at first glance looks perfect! When you inspect the rubric more carefully, however, you notice that the descriptions don’t always focus on the same characteristics across performance levels. For example, you want to assess pacing, yet pacing is described as “paced for audience understanding” in the high performance level, “sometimes too fast or too slow” in the middle performance level, and not mentioned at all in the low performance level.

You revise the descriptions of pacing so that the characteristic is addressed at all performance levels, and that the value of the characteristic changes in a measurable way between adjacent levels. Drawing from an example given by Robin Tierney and Marielle Simon in What’s still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels, you modify the rubric so that at the high performance level, the student “always paces for audience understanding.” At the middle performance level, the student “sometimes paces for audience understanding.” At the low performance level, the student “doesn’t pace for audience understanding.” With a few more revisions, you can use your rubric to effectively assess your students’ oral presentations. Additionally, when you share the rubric with your students, they can use it to better understand your expectations and deliver higher quality presentations.

What successes have you had with designing rubrics for your courses? What challenges have you encountered? Please share your comments.

References

Popham, W. J. (1997). What’s wrong—and what’s right—with rubrics. Educational Leadership, 55, 72-75.

Tierney, Robin & Marielle Simon (2004). What’s still wrong with rubrics: focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2).

We’re looking for an instructional technologist. Details about the job are available at NAU’s Human Resources web site. [UPDATE: The job posting closes on January 23, 2012, at midnight Flagstaff time.]

This blog post offers tips about the RIGHT way to apply for a job with us or, really, almost anywhere.

If you don’t meet the minimum qualifications, don’t bother applying

Whenever we have a job opening we get dozens of applications. Usually more than half of them don’t merit a second glance. Why? We mean it when we state the minimum qualifications for the job. If you don’t meet ‘em, don’t apply. We know—you’re thinking it’s like the lottery. Can’t win if you don’t play, right? The part of the job description that trips up unqualified applicants is that line about “combination of experience, training, and/or education.” If you’ve got to sell us on that, your equivalent education or experience can’t be much of a stretch, and it needs to meet NAU’s definition of equivalency. If you don’t meet the minimum qualifications, we can’t legally hire you, so don’t waste your time or ours.

Photo by My Melting Brain on Flickr

The matrix isn’t just a movie

Our employment searches are a committee affair. The committee chair—usually the hiring manager—creates a matrix (an Excel spreadsheet) listing the job qualifications based on the posted job description. Some qualifications are weighted more than others, and if you are truly a qualified candidate, you’ll probably be able to figure out which ones. Be sure to address those in your cover letter and resume.

The search committee members score your application against the matrix, and the highest scoring applicants get considered for an interview. The first item we score applications on is whether you meet the minimum qualifications. If you don’t, your application immediately goes onto the reject pile.

Your cover letter

A cover letter is not required, but you’d be foolish not to write one, because the cover letter is where you can point out how well your qualifications line up with the “skills and abilities” section of the job description, which directly affects how well you score on the matrix. If you’re qualified and you write a strong cover letter, you greatly increase your chances of getting an interview. Use the cover letter to explain how your experience in peripherally related areas makes you a stronger applicant, and explain any noticeable gaps in your employment history. Don’t ramble in the letter, and do strike a balance between confidence and humility. Don’t address your cover letter to “Dear Sir.” That really irritates the women on the search committee.

Your resume

We like resumes and cover letters that make it easy for us to score your application, so be crystal clear about how your experience matches what we’re looking for. But don’t load up your letter and resume with buzz words if you can’t back up the lingo. If you’re bluffing, we’ll catch on soon enough.

Your resume can also sink you. Remove references to extinct programming languages and defunct applications that make you look dated. If the last version of Windows you’re familiar with is ’95, you’re done.

We do a lot of design work here. That means more than half the staff are Mac users, and many of us use PCs too. If your resume is dripping with contempt for Apple products, or if you are a Mac fanboy who hates Windows, we don’t want you. You need to be able to work with all kinds of people and all kinds of tools. If you make a clever joke about Google being more evil than they used to be, we won’t argue with you, but if you choose to snark, do it very carefully when applying for a job. You don’t know whose buttons you might be pushing.

We’re looking for experienced people, not newbies, so extensive previous employment is expected. That means it’s okay, even preferable, to send us a two-page resume, but fill the space wisely, with details that clearly illustrate how you meet the requirements of our job. You don’t have to be a master of every single bullet point in the job posting, but you’d better be pretty good at the majority of them. Don’t waste resume space on pointless objective statements. A generic objective is inane, and one that says your objective is to get hired for the job you’re applying for falls into the “Duh” category.

Don't do this

Proofread your letter and resume, and have someone else look them over for you. Embarrassing typos suggest to us that you’re not attentive to detail.

If there’s anything odd on your resume, someone on the committee will notice. For example, if none of your references is from a past supervisor, we will wonder why. If you didn’t explain gaps in your employment history, we will wonder whether you’re trying to hide a job that didn’t go well. That alone might not rule you out, but if you get an interview, we’re going to ask about it. If your work history consists of very short stints with different companies, we will wonder whether you get along well with others. The committee will be looking for a pattern of relevant work experience and a progression of responsibility. We will Google you.

Photo by Andres Rueda on Flickr

Technical proficiency is required

If your technology prowess is on the level of successfully scanning your ramen noodles in the self-service line at the grocery store, you are not the person we seek. That doesn’t mean you need mad skillz on the order of having written your own Linux compiler when you were knee-high to C-3PO, and you don’t need to have developed a custom LMS that runs on your cell phone (although we’ll give you props if you have). But you do need to understand computing and the web in a hands-on way.

Being an uber geek isn’t enough

Sure, it’s cool if you know and love all gadgetry, and your resume is all alphabet soup (HTML, CSS, LMS, ADA, CS5, W3C, etc.), but if you can’t carry on an intelligent, unintimidating conversation with nontechnical people, and you don’t understand the culture and practices of higher education, you’re not the right one for us. You’ll be working directly with faculty, and the quickest way to turn them off is to start spouting technical jargon. Knowledge of teaching and learning is even more important than technical proficiency. The best applicants are those who know how and when—and when NOT— to apply technology in education. You’ve got to be able to teach faculty how to use software and hardware that they might be very vocally opposed to using, and you’ve got to do it with kindness and a smile. We want confidence and diplomacy, not arrogance, and we want people who thrive on patiently teaching and helping others.

Apply promptly and do it right

Our current job opening was posted without a closing date. However, as soon as the chair of the search committee thinks we have enough promising-looking applications, we’ll close the job posting with three days’ notice. If you’re interested in the position, don’t dawdle about applying. When you apply, follow the instructions on the Human Resources web site. You MUST fill out the application using HR’s system, even though it is tedious. Don’t email your resume directly to the department.

Don’t be a pest and do be patient

We’re not opposed to the occasional polite, short inquiry—preferably by email—about some aspect of the job. But don’t bug us with daily or weekly emails or phone calls about when we’re going to interview you or tell you what is going on. Hiring at a university is a SLOW process. It will be many weeks before you hear something from us.

The least qualified applicants—those who don’t meet the minimum qualifications—will usually get a “Dear John” letter via email sooner than the other applicants because they get weeded out almost immediately after the committee starts reviewing applications. But that could still be two months or more after the job opening was first posted.

Once the matrix scoring is completed and the committee convenes to discuss the highest-scoring applications, which in itself can take weeks because of scheduling conflicts, we will invite the most promising applicants for an interview.

The applicants in the middle of the pack have to wait the longest before hearing something. We hold off on notifying them because we might eventually invite them for an interview if the highest-scoring applicants don’t pan out. Usually, though, the also-rans get their Dear John email only after we’ve made a job offer to another applicant, and that person has accepted the position.

The interview

We sometimes do the initial interviews in person, sometimes on the phone, sometimes by videoconference. If you can interview in person, do so; you will be better able to read the people in the room and adjust your responses accordingly. Second interviews are usually in person and typically are reserved for only the top one, two, or (rarely) three applicants. Again, that can take several weeks.

There are pros and cons to being either the first or last interviewee. If you’re first and you are excellent, you set a high bar for the remaining applicants, but your stellar performance might fade from the committee’s memory by the time of the last interview. If you’re last, you can leave a positive impression on the committee, but by then the committee might be so weary of the hiring process that you practically have to tap dance to get their attention.

Before the interview, do your research. That means digging through our web site (yes, we know it needs work, but we’ve been busy and short-handed), following us on Twitter, finding us on Facebook, reading our blog, etc. That will answer a lot of questions you might have.

On interview day, show up on time. Dress appropriately. We’re not a suit-and-tie kind of department, but holey jeans and a funny-but-NSFW t-shirt are not appropriate choices for an interview. Dress as if you respect us. You can wear your purple Crocs after we hire you. For an interview, we suggest “business casual,” but we won’t penalize you for going fancier. Skip the tux or ball gown, though.

The committee chair will tell you what to bring to the interview. In advance of your interview we’ll probably ask you to send us links to online examples of your work. We might ask you to prepare a short training session or some kind of demo on a topic of your choice.

Have some good questions to ask us, but be sure they aren’t questions you could’ve answered yourself if you’d done your research. A candidate who interviews us will earn our respect. Just as we need to decide whether you’re the right fit for us, you need to figure out whether our job fits your talents and whether you can embrace our quirks and foibles.

Show enthusiasm. If you think the job really is a good fit, make us believe you want the job. Not just a job, but our job. And even if you are thinking it, try not to tell the interviewers that what you really want is one of their jobs. That doesn’t go over well. Nor does arguing with the committee or correcting their grammar or challenging the accuracy of something said.

We know you’re nervous when you’re being grilled by several people who can decide your employment fate, but try to relax and let your sense of humor come through. Unless you like puns. We’ve got one of those already.

The salary is what it is

If we post a single figure for the salary, it means that’s either all we’ve got the budget for, or that figure represents internal salary equity, and there’s no negotiating room. If we post a salary range in the job description, and you can convince us you’re extremely well suited for the job, we will consider hiring at the high end of the range. Salaries at NAU are often below average.

The thank-you note

We appreciate thank-you notes, but we’re not gonna complain to your mom if you don’t write one. A thank-you note is an excellent way to send the committee any follow-up information you promised during the interview. We like emailed thank-you notes better than snail mail. We don’t like thank-you phone calls.

Academia

If your employment history is in a sector other than higher education, brace yourself for culture shock. People here are free-thinking, outspoken, informal, and nonhierarchically organized, but universities are also steeped in tradition, rife with bureaucracy, and slow to change. Practices that would be unthinkable or trivial in the corporate world can be entrenched or strangely significant here.

Internal, local, and foreign applicants

We sometimes get asked if our job openings are just for show because we already have an internal candidate in mind. The answer is no. If we think we have sufficiently qualified candidates within the department, we post positions as being open only to internal applicants. If the positions are posted as open recruitments, we really do want external applications, but of course we will also consider well-qualified candidates who apply from within NAU, and internal candidates can definitely have an advantage. If you are already an NAU employee, you will almost certainly be more familiar with many of the processes and tools used here, and you probably have already passed the university-required background check. If the committee wants someone who can get up to speed especially quickly, those factors can help, but they’re no guarantee. Being an internal candidate can work against you if the committee thinks an external perspective is especially valuable for the position or if you’ve received several negative performance appraisals. We do check those.

If you already live in Flagstaff, we know you won’t be scared off by the climate, the high cost of living here, or the need to buy or sell a house in a tough market. But we have hired many staff members from distant locations. We do generally want people who can work in our offices on the Flagstaff campus, as opposed to full-time telecommuting.

If you are not a U.S. citizen, and you don’t already have a green card, it is theoretically possible that you can get hired, but you must procure an H1-B visa, which means the committee has to write a justification stating that we were unable to find a suitably qualified candidate who is a citizen. In the current job market, that’s not likely. If you’ve got a green card, you’re fine.

What we really want

Overall, we want the person who has the best mix of skills and experience, as well as a great, collegial personality to fit into the department and to work successfully with our clientele. If you meet those criteria, you improve your odds of getting hired.

With more students struggling academically, particularly in the first year, and fewer students persisting to graduation, many of us in higher education are asking how we can help our students to succeed. One way is to teach students to “think about their thinking.” In other words, we can teach them to develop metacognitive skills that will help them to become aware of their own thinking processes and use that self-awareness to regulate those processes. Researchers and practitioners agree that metacognition is critical to academic success.

In a previous blog post, I promised to provide some easy-to-implement strategies for teaching metacognition in face-to-face and online courses. When I reviewed the research and talked to colleagues about what they were doing to promote metacognition in their classes, a theme emerged: The best way to teach metacognition is to do it in conjunction with activities and assignments that are already a part of your class. Here are some strategies for teaching your students to think about their thinking when they take exams, listen to lectures, or work on writing assignments in your class.

Taking Exams

College students are often unaware of what they know and don’t know (Zabrucky and Bays, 2011). When taking exams, students frequently overestimate their level of understanding and readiness to take a test. First-year students in particular report that “looking over their notes” before an exam has worked well for them in the past (Ruban and Reis, 2006), and they are shocked when they receive exam scores that are lower than expected.

Karen Zabrucky and Rebecca Bays suggest that we can help students better understand what they know and don’t know by asking them to predict their exam scores right before they take an exam and then also estimate their scores right after taking an exam but before receiving their grades. Students can then compare their predictions and estimates with their actual exam scores. The authors also suggest that instructors ask students questions about how they studied for an exam and whether they felt they were adequately prepared to take the exam. These questions prompt students to reflect on both their level of preparation for an exam and the consequences of their level of preparation.

Listening to Lectures

Shawn Nordell (2009) conducted research with students in a large introductory biology course and found that most students had difficulty recalling course knowledge. When students were asked to write down two or three of the main points discussed in a lecture and readings, most students had no response at all or could remember only a key word or phrase.

Marsha Lovett (2008) described a technique called “wrappers,” activities that wrap around a learning activity or assignment and can be used to foster students’ metacognitive skills, including the recall of course knowledge. Instructors can “wrap” a lecture by presenting tips on active listening before the lecture, having students write down the three key ideas from the lecture immediately after the lecture, and then giving students a list of the three key ideas from the lecture for students to self-check. Lovett found that over time the students’ three key ideas increasingly matched those of the instructor.

Working on Writing Assignments

One way to prompt students to reflect on their writing is to provide them with questions for self-assessment. Before students begin to write, ask them to answer questions such as “What are my goals for this writing assignment?” or “What do I need to do to prepare to write?” You can also ask students to answer questions right after they write. Here are some suggested questions, adapted from a questionnaire used in a large-scale university writing assessment at Truman State University:

• How do you feel about your finished writing sample?
• How representative is this sample of your writing?
• Describe your writing process.
• What do you feel is especially strong about your writing sample?
• What do you feel could be improved in your writing sample?

The Writing Place at Northwestern University offers good examples of self-assessment questions and a worksheet for students to use when evaluating their own writing.

All of these strategies help students to practice their metacognitive skills and grow as learners. Have you tried any of these strategies in your online or face-to-face class? What worked, and what didn’t work? What other methods have you used to encourage students to think about their thinking? Please comment on your experiences with teaching your students how to learn.

References

Lovett, M. C. (2008, May 5). EDUCAUSE Learning Initiative Events. Retrieved May 18, 2008, from Metacognition and Monitoring: Understanding and Improving Students’ Skills for Learning.

Nordell, S. E. (2009). Learning how to learn: A model for teaching students learning strategies. Bioscene: Journal of College Biology Teaching, 35 – 42.

Ruban, L., & Reis, S. M. (2006). “Patterns of Self-Regulation: Patterns of Self-Regulatory Strategy Use among Low-Achieving and High-Achieving University Students. Roeper Review, 148 – 156.

Zabrucky, K. M., & Bays, R. (2011). Helping students know what they know and do not know. College Teaching, 123.

Listening to Student Voices: Assessing and Responding to Students’ Experiences with Blackboard Learn

A poster session by Dr. Sue Pieper, Coordinator of Assessment,
and Don Carter, Director

Last year, as part of a pilot project in which Northern Arizona University investigated Blackboard Learn for possible adoption as the university’s learning management system, we surveyed our students, asking them about their experiences in using Blackboard Learn in the pilot courses. The results of that survey are summarized on the poster along with recommendations for Blackboard and for other institutions that are considering using Blackboard Learn.

Complete survey results are available in the full research report.

View the poster (17″ x 11″ .pdf).

The Pedagogical Opportunities of Mobile Technologies: iPads and Kindles in the Classroom

A poster session by Dr. John Doherty, Instructional Designer,
and Kevin Ketchner, Librarian, Cline Library

With mobile devices beginning to enter the classroom, students are only two finger-swipes away from the game Angry Birds Rio, or YouTube, or any number of other popular diversions. Ketchner and Doherty describe how to use mobile apps and tools in ways that engage students in class and encourage student interaction with peers, instructors, and content. They provide examples of appropriate and effective choices that can enhance learning.

View the poster (17″ x 11″ .pdf).

(R)eflective Learning: The Experience of Journaling from Campus Edition to Bb Learn

A poster session by Dr. John Doherty, Instructional Designer,
and Kevin Ketchner, Librarian, Cline Library

This poster is organized around Dee Fink’s Creating Significant Learning Experiences (Jossey-Bass, 2003). Ketchner and Doherty deconstruct Fink’s model, presenting an application and example of how online journaling can enhance students’ critical
reflection, self-reflection, thinking, and metacognition. They contend that reflective learning through journaling leads to insight and significant learning experiences for students.

View the poster (17″ x 11″ .pdf).

From Vista 8 to Blackboard Learn 9.1— Lessons
Learned and Tips for Success

A presentation by Dan Stoffel, Assistant Director,
and Erin Shelley, Blackboard Learn System Administrator

Northern Arizona University has just made the transition from Blackboard Vista 8 (formerly WebCT Vista) to Blackboard Learn 9.1. The process of moving to a new learning management system includes the migration of nearly 2000 courses from the old system to the new. This presentation describes NAU’s course migration process and suggests strategies for other institutions who are following a similar path.

Download the PowerPoint slides (.pptx).

Pedagogy and Online Learning: Training Users on Backward Design

A presentation by Dr. John Doherty, Instructional Designer,
and Wally Nolan, Instructional Designer

The design and development of online content is sometimes too focused on the learning management system rather than on students’ learning. The backward design approach, described well in Understanding by Design by Grant Wiggins and Jay McTighe (Pearson, 2006), helps instructors begin with the end—learning—in mind. This presentation described self-paced online tutorials that guide instructors and course creators to align their learning objectives, assessments, and learning activities independent of a learning management system.

An Inside Higher Ed article titled “Can Students Learn to Learn?” piqued my long-standing interest in students and metacognition. As a teacher of writing for nearly twenty years, I have been intrigued by differences in the ways that students learned and have sought to understand how two students of similar ability could perform so differently in my English composition classes. Over many years of working with student writers, I noticed that most of them didn’t take the time to reflect on their writing. They weren’t asking basic questions: What worked well in my writing? What didn’t? What can I try next time? What I observed in my students was a lack of metacognitive skills.

What is metacogniton?

Strict definitions vary, but the common definition of metacognition is “thinking about thinking.” Many researchers believe that metacognition involves two steps: knowing one’s own cognitive processes and using that self-awareness to regulate those processes (Brown, 1981; Niemi, 2002; Shimamura, 2000). By comparing the behaviors of expert and novice learners, researchers (Butler, 1997; Lovett, 2008; Pintrich, 2000; Winne & Hadwin, 1998) have discovered that experts, unlike novices, routinely move through a self-regulation cycle of

1. Planning
2. Setting goals
3. Applying strategies
4. Monitoring
5. Evaluating
6. Adapting

By teaching these self-regulation strategies to our students, we can improve their learning.

The “Can Students Learn to Learn?” article reported on the efforts of members of the Associated Colleges of the Midwest (ACM) who are doing just that. One of the projects highlighted was led by Kristen Bonnie, an assistant professor of psychology at Beloit College. Bonnie described a simple technique for prompting metacognition in her students. On a multiple-choice test, she lets students select a few questions that they don’t want her to grade. She also requires them to give a reason for their selection, providing a list of reasons for students to choose from, for example “I don’t remember the material” or “I’m not confident in my answer.” Her goal is to make students stop and think more about why they don’t know an answer. The ACM projects are only two years old, but they are already showing promise in terms of improving student learning.

The comments following the article were even more interesting than the article itself. Many commenters were already using metacognitive strategies in their own classrooms or were eager to try these techniques. Other commenters disagreed with the idea of teaching students how to learn, asserting that the goal of university instructors is to teach their subject and that teaching metacognition skills is a waste of instructors’ time. Instead, they argued, students should come to college prepared and should be held accountable for their work.

One commenter asked, “What good is teaching if too many students are not actually grasping the material?” Her question is a good one. Students should already know how to learn, but many students don’t have these skills. We need to teach students how to learn, not only to help them to understand the content we teach, but also to give them the opportunity to practice and improve their metacognitive skills. Most importantly, we want them to develop a lifelong habit of thinking about their thinking.

Some higher education professionals are already working on this. One commenter from the Office of Faculty Development at California State University Channel Islands described an intriguing initiative, a “learning across the curriculum” movement, where faculty were encouraged to teach one thing about learning in every course. Ideally, teaching students how to learn should happen in all courses across the curriculum and in all modes of delivery, but teaching metacognition might be especially important in online courses. Macdonald (2004) argued that to be competent, e-learners need a self-directed approach to learning, in addition to basic information and communication technology skills, information literacy skills, and collaborative learning skills. In my next post, I’ll explore some easy-to-implement strategies for teaching metacognition, both face-to-face and online.

Are you teaching students how to learn in your courses? If so, what strategies are you using? Please comment on your experiences with teaching students to learn.

References

Brown, A. (1987). Metacognition, executive control, self control, and other mysterious mechanisms. In F. Weinert & R. Kluwe, Metacognition, motivation, and understanding (pp. 65-116). Hillsdale, N. J.: Erlbaum.

Butler, D. (1997). The roles of goal setting and self-monitoring in students’ self-regulated engagement of tasks. Paper presented at the annual meeting of the American Educational Research Association. Chicago, IL.

Lovett, M. C. (2008, May 5). Educause Learning Initiative Events. Retrieved May 18, 2008, from Metacognition and Monitoring: Understanding and Improving Students’ Skills for Learning

Macdonald, J. (2004). Developing competent e-learners: The role of assessment. Assessment and Evaluation in Higher Education, 215-226.

Neimi, H. (2002). Active learning—a cultural change needed in teacher education and schools. Teaching and Teacher Education, 763-780.

Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner, Handbook of self-regulation (pp. 451-502). San Diego, CA: Academic Press.

Shimamura, A. P. (2000). Toward a cognitive neuroscience of metacognition. Consciousness and Cognition, 313-323.

Winne, P., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. Hacker, J. Dunlosky, & A. Graesser, Metacognition in educational theory and practice (pp. 277-304). Mahweh, NJ: Erlbaum.

The annual Horizon Report, published by the New Media Consortium (NMC), describes educational technologies that are ripe for adoption in the coming year, the next 2 to 3 years, and 4 to 5 years out. The 2011 Horizon Report is now out. The full report (.pdf) is available on the NMC site. The work that went into the report is documented in NMC’s Horizon Report wiki.

Dead on or dead wrong?

Past Horizon Reports have included some notable hits and misses, perhaps stemming from the biases and interests of the researchers, many of whom inhabit the leading edge of educational technology and are removed from the laggard community. For example, the 2005 Horizon Report said social networks would be adopted within education in four to five years. Social networks went mainstream long before that but have seen less adoption in education than expected, perhaps partly because until recently they weren’t well integrated with learning management systems, and partly because many educators often viewed—and still view—social networks with suspicion.

Another element of the 2005 report suggested that educational gaming would be widespread by 2008. The 2011 report says that game-based learning is still two to three years away, evidence that prognostication is an inexact business.

Still, it’s worth contemplating some of this year’s predictions.

Electronic books and mobile devices

These two technologies do seem to be obvious choices, and they’re not unrelated. For example, Amazon Kindles and Apple iPads are good devices for displaying ebooks, and both are popular with consumers (the National Federation of the Blind’s lawsuit against Arizona State University for its Kindle use notwithstanding). Last July, Amazon’s ebook sales outpaced hardcover sales, with 143 ebooks sold for every 100 hardcover books.

However, digital rights management still acts as a bottleneck for adoption of ebooks in education, especially for books published outside the U.S. For example, the Kindle edition of Jerusalem: The Biography, by Simon Sebag Montefiore, is receiving very good reviews in British publications but is not yet published in the United States and won’t be available here until at least November 2011, a year after its British publication. Fortunately, some publishers realize the value ebooks can bring to teaching and learning, and they are decoupling the purchases of ebooks and printed books. McGraw-Hill now allows students to buy or rent—for much lower cost—an ebook without forcing purchase of its physical counterpart.

A new report by Rob Reynolds of Xplana indicates projections for the digital textbook market over the next five years. Their projections agree with the Horizon Report regarding coming adoption of ebooks, but their time frame is more conservative.

The 2011 Horizon Report notes that some tablet devices, such as the Apple iPad and the Samsung Galaxy Tab offer enough compelling additional features to make ebooks “a potentially transformative technology” because they can now include rich media and supplemental material not possible in a printed book. Inkling’s edition of Raven Biology is cited as one example of a title that “brings the study of this science to life with detailed illustrations and animations, in-line keyword definitions, and interactive quizzes embedded in each chapter.”

On the mobile device front, the 2011 Horizon Report notes three converging points:

• Within the next year, Internet-capable devices will outnumber computers.
• In Japan, 75% of Internet users already choose mobile devices as their primary means of access.
• By 2015, 80% of people accessing the Internet will do so from a mobile device.

The Pew Internet & American Life Project has an interesting infographic showing rates of ownership of several types of mobile devices by age group. Tablets and ebook readers show comparatively low rates of ownership, but laptops, MP3 players, and cell phones show high rates of ownership across age groups. Only 9% of adults don’t own any of the devices, and 63% of those non-owners are age 66 or older. Take a look at the full-sized graphic for details.

Many educators are already challenged by the presence of mobile devices in classrooms, with some instructors issuing blanket bans on phones and tablets. We think that’s the wrong reaction. Instead, instructors should capitalize on the devices. Have students use mobiles to search for, create, and present content and to collaborate and interact with each other and with the instructor. For example, envision students examining 3D models of chemical structures while they work on lab experiments, or consider getting feedback from students by polling them and letting them respond from their mobile devices. Instructors who object to mobile devices in the classroom are missing a golden opportunity.

Augmented reality & game-based learning

Think about the benefit of using augmented reality applications on anthropology or geology field trips. Star Walk, a popular astronomy app by Vito Technology, puts a virtual planetarium on your phone or tablet, showing you a real-time view of the night sky with informative overlays. These kinds of applications have obvious potential in higher education, and their cost is often low, once you discount the initial expense of purchasing devices capable of using them. Many developers have already jumped on this bandwagon, so the choices should increase rapidly.

Northern Arizona University’s Cline Library augmented an exhibit, Route 66 in Arizona: Don’t Forget Winona!, with a Google Maps add-on that lets users look at photo archives of towns highlighted along the route. The map works in web browsers and on smartphones.

Game-based learning has spawned several conferences and studies (.pdf), a sure sign that the topic is gaining traction among educators. However, given the time and cost of developing high-quality games across the many academic disciplines, as well as possible bias against games among faculty, we suspect widespread adoption of this technology will take more than three years. Pioneering educators are gaming already, but the masses will follow more slowly.

Analytics

At the 2011 ELI Annual Meeting, David Wiley, Associate Professor of Instructional Psychology and Technology at Brigham Young University, defined learning analytics as a way “to harness the power of advances in data mining, interpretation, and modeling to improve understandings of teaching and learning, and to tailor education to individual students more effectively.” As an example, he described the development of strategic tutoring, which relies on data to help predict the need for proactive student support. You can view a video or download his slides for that presentation and others on analytics and openness in education.

Businesses have already capitalized on analytics. Think of the recommendations you get from Amazon or Netflix or the coupons you receive when you pay for your groceries. Given the current budget-slashing climate for higher education, analytics are a no-brainer, and savvy institutions are getting on board now, using the data they already have and not waiting five years. Any university seeking funding or even tuition dollars is going to have to rely on data to demonstrate the effectiveness of its programs. No convincing numbers? No dollars.

Northern Arizona University has already started using analytics in its Grade Performance Status (GPS) tool, which uses data on students’ academic performance, grades, attendance, and positive feedback to identify students who might be headed for trouble in their courses, and to prompt instructors to reach out to those students to assist them.

So what’s the immediate message of all this for educators? We think it’s get moving on getting mobile, look toward ebooks, and start analyzing your data now.

It’s official! During this year, Northern Arizona University will be moving to a new learning management system, Blackboard Learn. As you move your courses to the new system or use the new system for the first time, take advantage of the opportunity to rethink your courses, adjusting them as needed to incorporate a learner-centered approach in line with the university’s values.

Effective learner-centered courses are distinguished by five characteristics outlined by Maryellen Weimer in Learner-Centered Teaching: Five Key Changes to Practice (2002). According to Weimer, who also edits The Teaching Professor,  the following course elements change when teaching becomes learner-centered:

• The balance of power
• The function of content
• The role of the teacher
• The responsibility for learning
• The purposes and processes of evaluation

In a learner-centered course power is shared among teachers and students, resulting in a more engaging and motivating course environment. Content in a learner-centered course is used to promote students’ knowledge as well as their learning how to learn. The role of the teacher is to guide and facilitate student learning, with the students assuming the primary responsibility for learning. The purpose of evaluation includes not only grading, but also fostering improved student learning (Weimer, 2002).

Let’s consider some ways you can incorporate learner-centered teaching in your
own courses.

The Syllabus Quiz

The syllabus is an essential part of every course, providing information about learning outcomes, required textbooks and materials, a course calendar, and student and instructor responsibilities. We ask students to read the syllabus at the beginning of a course. Then we typically spend class time reviewing the syllabus, or we spend time answering multiple questions because students haven’t read the syllabus.

An alternative approach is to require students to pass a syllabus quiz, which can be set up as a brief true/false or multiple-choice online quiz that can be graded automatically by the learning management system. Many instructors have found that using a syllabus quiz as a “gateway” is effective. Students must pass the quiz with an acceptable score in order to gain access to the rest of the course. As a result, responsibility for reading and understanding the syllabus is placed on the students, not the teacher.

Self- and Peer-Evaluation of Group Work

Group work can be challenging in a face-to-face course, and effective group work is even more challenging in an online environment when instructors and students are at a distance.

If you want to improve group work in your course, consider involving students in not only completing the group assignments, but also in evaluating how they function as a group. First, teach them about the dynamics of group work and the various roles in a group. Then teach them how to assess their participation in their groups.

Rubrics — documents that describe an instructor’s grading criteria and levels of performance expected from students for each grade possible in an assignment — are very helpful for both instructors and students in assessing group work. An example of a team and leadership skills rubric, which assesses criteria such as group organization and coordination, is available from the e-Learning Center’s website.

When you create a rubric and share it with your students, you enable them to take responsibility for evaluating their own and their group members’ contributions to the group. Consequently, students learn content while also developing an awareness of how they work with others.

Student-Designed Exam Questions

Designing exams has traditionally been the purview of instructors, but what if we ask students to design an exam? Weimer offers examples, including a math instructor’s experience with giving students an end-of-the-course option to develop a final exam. She evaluated the assignment on criteria such as how well the exam questions corresponded with the intended student learning outcomes for the course, the solutions for the problems, and the point values assigned to the problems, taking into account their relative importance in the course content. The students told the instructor that they spent more time designing the exam than they would have spent studying for it.

Involving students in constructing exams and other course evaluations transforms the goal of assessment from just obtaining a grade to promoting student learning and development. Especially in an online environment, where students can sometimes feel isolated, designing exams, particularly as a group, can be an engaging and empowering course assignment.

Assess Your Courses

Are you incorporating learner-centered outcomes, assignments, and assessments? Are students assuming the primary responsibility for learning? Are you taking the role of guide and facilitator? If you think you could make some course improvements in these areas, try one of the ideas described here. For assistance in changing to a learner-centered approach or choosing the best learning management tools to accomplish your goals, contact us at the e-Learning Center. We’re here to help!

Perhaps the hardest part of owning an iPad is trying to avoid the addictive world of Angry Birds, the favorite game app of the new British Prime Minister. For just $5 you, too, can attack pigs with hard-headed flying birds.

That kind of distraction is exactly what many faculty worry about when we ponder the place of tools such as the iPad in the classroom. We fear that our students will be more engaged with their games or Twitter feeds than with our lectures. But the iPad, the iPhone, the Droid, and the flood of other new tablets and smart phones finding their way into our classrooms are not the cause of student inattention. These gadgets are just tools, and student engagement depends on good teaching whether or not a cool gadget is present. In Technological and Pedagogical Content Knowledge: A Framework for Teacher Knowledge (.pdf), Punya Mishra and Matthew J. Koehler note that we need to understand not only various technologies but also how the technologies affect our pedagogy.

The iPad is a good case in point. Since the iPad’s launch in April 2010 some educators have begun to examine its place in education. Our experience in using the iPad for teaching is that it is useful in two distinct ways, consumption and production, although it currently is more useful for the former than the latter.

Consumption

The iPad is very obviously a tool for consuming information and media. Reading publications such as the New York Times or watching movies through the Netflix app seems to be a natural fit for the device. In the classroom, we have been using iPads as a way to deliver information to our students.

For example, in a course on Reinventing King Arthur, John used the Kindle app on his iPad to give his students access to some of the course readings. In fact, he chose the course readings based on their availability in ebook format, and in one instance, John was able to bring back into his course a text that had been out of print for almost 20 years! In another case, students who chose to download the Kindle version of a new book were able to do so several days before others were able to order the printed version (although this might have been an error on Amazon’s part). The students used their own devices, mobile or otherwise, to read the ebooks, which was possible because the Kindle app is no longer limited to the Kindle device. The app can be used on PCs or Macs, desktops or laptops, iPads, iPhones, Droids, and iPod Touches. One of its best features is that you can read the same book on various devices, and the app will remember where you left off. You can start reading a book on your laptop and then later switch to reading on your phone without losing your place.

Kevin has been using the iPad for an Honors course on the cultural impact of comic books. Marvel, DC, and other publishers have iPad apps (iTunes link) that present comics in sharp, HD-like quality. When you can zoom in on details of comics in high definition, the panels take on an immersive quality.

The following panel is a screen capture of the Iron Man (2004) #1 book, available in the Marvel app.

Students in the course can download the movies Iron Man (2008) and V for Vendetta (2006) via the iTunes Store or other media sources. By examining both the comic serial and the movie versions of these stories, students see how the superhero movie genre has reinvented the comic.

Students can also use reference apps such as Wikipanion (Wikipedia) to find out details about characters in these stories (such as Guy Fawkes, the original fundamentalist terrorist who is burned in effigy in the UK every November 5th).

The ebooks John uses in his course include built-in dictionaries that make it convenient for students to look up unusual words, which is invaluable for texts based on older versions of the English language. The iPad apps can also read the texts aloud.

Here are some other apps that we’ve experimented with and have found useful for teaching.

App	Description
iAnnotate PDF ($9.99)	Syncs PDF documents between your desktop and your iPad. It also lets you highlight and underline and add commentary, which is great for our e-reserve readings.
CineXPlayer ($3.99)	Delivers Xvid movies to the iPad through iTunes. Includes subtitles and TV/VGA output. The advantage to this app is that it supports more than just iTunes native formats.
Netflix (Free)	The app is free, but you need a Netflix subscription to use it. You get access to the Netflix streaming library. John has had his students view the BBC TV series Merlin (2008) and the movies Camelot (1967) and Excalibur (1981).

Accessibility

The iPad is opening doors for some differently abled users. Its built-in text reader can give voice to anything downloaded to iBooks. Dragon Dictation is a free iPad app that uses voice recognition to create text email messages and even Facebook and Twitter updates. The iPad’s magnification and closed-captioning features also make content available for a wide range of people. The New York Times recently reported on how an iPad elicited reactions from a 7-year-old boy who has a severe motor neuron disease. The same article noted that autistic children are also engaging more with this tool than a regular computer.

Production

As a production tool—that is, for generating rather than just consuming content—the iPad has plenty of potential, but it still has lots of room for improvement. Much of this post was written in the Pages app on the iPad, using the on-screen keyboard. Or at least it was until we started getting annoyed when we typed too many extra n‘s and m‘s because those keys are uncomfortably close to the space bar for our not-so-nimble digits. So we just synced the iPads to our desktops and continued typing from there, using real keyboards.

The on-screen keyboard works well enough for some things: browsing the web, searching the App Store, typing things into Wikipedia. In our opinion, though, an external keyboard is an essential add-on if you want to use the iPad for serious writing.

In addition to Pages, the other two iWork apps—Numbers and Keynote— for the iPad are useful production tools . Each costs $9.99 from the iTunes Store. iWork will soon be available to the NAU campus through a site license, and it is a decent alternative to the MS Office applications. John developed some of his course documentation in Pages and Numbers.

Drawbacks

Some of the drawbacks to using these apps include lack of easy cloud syncing and inability to print directly from the iPad. That said, the MobileMe iDisk app does provide some access to the MobileMe cloud (which requires a subscription) or the beta of iWork.com. There are rumors of an about-to-be-released Google Apps app for the iPad and Android that will allow for editing of Google docs, spreadsheets, and presentations. Apple is undoubtedly working on updates to their products to fix these and other shortcomings, and they are also exploring cloud-based computing, but if you want to take the iPad plunge now, you should be aware of these limitations. Syncing the iPad to your computer also leaves a lot to be desired, because you need to go through iTunes to make it work at all. The advantages of using the iPad over a netbook as a mobile production tool are currently few.

Recently, we discovered two note-taking apps for the iPad that let you use your finger or a stylus. We tested Penultimate and WhiteNote in meetings and in the classroom. The former is pretty slick but simple. The latter needs some work, especially in how it uses the screen and the writing space. But it offers the bonus of letting you browse and import from the web, PDFs, and other formats, and it also uses cloud-based resources for saving and sharing PDFs via Google Docs.

The iPad and some smart phones are beginning to offer access to learning management systems (LMS), such as Blackboard Learn and Moodle. Mobile apps are available for both of these LMS, but their functionality is quite limited, although the Blackboard Mobile Learn app syndicates content from that LMS quite well. For the moment, though, both Blackboard Learn and Moodle are best accessed on the iPad through a browser.

Lessons Learned and In Progress

The iPad holds much promise for education, but it is still in its infancy. Missing but available soon with the impending release of iOS 4.2 release are the ability to multitask (that is, to run more than one app at a time), print from the iPad, and even output information to other devices through AirPlay. The ability to easily display the content of your iPad on an overhead projector is essential for educators, and we hope Apple addresses that shortcoming quickly. Apple’s prohibition of Adobe’s Flash on the iPad means that some of our preferred media sources, such as streaming films available through our university library, are not viewable on the device. However, the Skyfire browser, which converts Flash video to a format that works on Apple’s mobile devices, might soon solve that problem.

The most obvious lesson we learned is that students like using the iPads for classes. It wasn’t at all hard to get the students to use the iPads. It was hard to get the students to give them back.

You’ve probably heard about it. You might even be engaged in it right now.  It goes by many names, including hybrid and mixed-mode learning. It’s blended learning, and it’s coming our way.

I came to this conclusion after attending a recent two-day EDUCAUSE Learning Initiative webinar, Blended Learning: The 21st Century Learning Environment. Blended learning is a complex mix of face-to-face and online elements in a single course. After learning about and reflecting on the latest theory and research on blended learning, quality considerations, design and implementation issues, and practical applications, I came away with a few ideas.

Thoughtful design can promote student learning

I learned that blended learning, if designed thoughtfully, can promote student learning. As one webinar participant put it, “intentionality,” is necessary for the success of a blended learning course. Obviously, thoughtfulness or intentionality is important in designing any course, but it may be paramount in designing a blended course. Combining components of face-to-face and online teaching can be challenging, even for instructors who are experienced at teaching in both modes. So what does an instructor need to think about when designing a blended course?

This question is addressed in Ten Questions to Consider when Redesigning a Course for Hybrid Teaching and Learning. One question is particularly important:

As you think about learning objectives, which would be better achieved online and which would be best achieved face-to-face?

This question asks instructors to think carefully about how they and their students can best use their time in class and online.

Rethinking class time

Gerald Bergtrom

A striking example of rethinking teaching, learning, and time was provided by webinar presenter Gerald Bergtrom at the University of Wisconsin-Milwaukee. Bergtrom described how he transformed his face-to-face biology course to a blended learning course that provided content coverage and promoted student learning at the same time. In his original face-to-face course, he devoted all of his class time to lectures and in-class exams. In the blended course, he eliminated all lectures in the face-to-face meetings and replaced them with critical thinking and interactive exercises: muddiest point writings, clicker questions, and small group index card activities. Content was delivered online through a variety of modes, including text readings, PowerPoint lectures, discussions, papers, and quizzes. In this way, Bergtrom integrated and mutually reinforced face-to-face and online activities. Bergtrom found that, although creating a blended course required effort, he enjoyed teaching more and his students learned better in the new blended course.

The idea of rethinking how time in class and online can enhance student learning was also discussed in episodes #26 and #27 of Tuesday Tips on Teaching with Technology (iTunes U link), a podcast series produced by Northern Arizona University’s e-Learning Center. Wally Nolan and John Doherty explained how offering online learning activities before and after class and linking them to in-class activities can improve instructor and student productivity. They described a common instructor experience: students arriving in class without having read the assigned readings. To address that problem before class, instructors can have students complete an online quiz on the readings. During class, instructors can then devote their time to clarifying and enhancing student learning through discussion. After class, students can extend their in-class conversations through an online discussion or an activity that lets them apply what has been learned.

You can find other examples of effective blended learning designs in Blended Learning in Higher Education, written by two other webinar presenters, Randy Garrison at the University of Calgary and Norman Vaughan at Mount Royal University. Garrison and Vaughn organize their examples around three scenarios common to higher education: small class courses, large enrollment courses, and project-based courses. Each scenario has its own challenges that can be addressed through different blended learning designs.

Your thoughts?

What is your experience with blended learning? If you have taught a blended course, what has and hasn’t worked?  If you haven’t taught a blended course, what do you think will be important to consider in course design? Please share your thoughts in the comments.

Smoke plume from the Schultz Fire in Flagstaff, Arizona

Some people still think of Twitter as a tool for sending out 140-character messages about trivia such as what you ate for lunch, but during Flagstaff’s wildfire crisis, Twitter turned out to be one of the best sources for up-to-date information on the Hardy Fire (282 acres with 95% containment as I write this) and the Schultz Fire (currently 14,800 acres with 40% containment) as well as the Eagle Rock fire near Williams. Anyone could use the Twitter hashtags #flagstafffire, #hardyfire, #schultzfire, #schultz, #wildfires, or #flagstaff to find or tag information about the rapidly unfolding drama in our mountain town. Hashtags are short text identifiers preceded by the # character to indicate the topic of a message. Twitter messages are often called tweets.

One person in our department who was following Twitter was able to notify another about an impending neighborhood evacuation even before county officials had knocked on the evacuee’s door. Flagstaff’s mayor, Sara Presler, or @sarapresler in Twitter notation, used Twitter effectively to send out information about the status of the fires and to inform citizens about upcoming press conferences and public meetings related to the fires. Tweets from government organizations, individuals, and various groups got information out much more quickly and frequently than conventional media could. Of course the local newspaper and radio and television stations also used Twitter, and their tweets fleshed out the picture of the fire situation in advance of their regularly scheduled publications or broadcasts. Speaking of pictures, many Twitter users posted photos of the fires from their vantage points, which was an incredibly effective way to calm (or terrify) loved ones from afar. Tweets about the fires generated so much traffic on Twitter that the story was picked up by Mashable, a popular technology-oriented social media site.

Sunset view of smoke from the Schultz Fire enveloping the San Francisco Peaks in Flagstaff, Arizona

If your impression of Twitter is that it’s only an endless stream of inane chatter, think again. If you judiciously choose the people and groups to follow, Twitter can serve as your up-to-the minute personal newsfeed. Of course you need to view some tweets with a critical eye. Not everyone who who uses Twitter follows the journalistic standards and ethics we’ve come to expect from mainstream media, but many regular folks do a fine job of reporting the situation.

Here are some of the Twitter users who provided valuable information during the fires. Click the links to see their Twitter pages.

• @ArizonaDOT (Arizona Department of Transportation)
• @azds (Arizona Daily Sun newspaper)
• @AzEIN (Arizona Emergency Info Network)
• @AZFireInfo (Arizona Fire Info)
• @AZPubRadio (KNAU radio)
• @coconinocounty (Coconino County official information)
• @CoconinoNF (Coconino National Forest official information)
• @dsoltesz (Deborah Lee Soltesz, Flagstaff resident)
• @ENeitzel (Eric Neitzel, national public information officer from Show Low Fire Dept.)
• @inciweb (national incident information system)
• @FLAGscanner (live scanner feeds from police, fire, and EMS groups in Flagstaff)
• @naztoday (students at NAU’s School of Communication)
• @sarapresler (Flagstaff’s mayor)

Let us know in the comments about other good sources of information on Twitter.

If you’d like to learn more about how to use Twitter, either as a newsfeed or an educational tool, contact the e-Learning Center. Or follow us on Twitter (@nauelearning) or Facebook. In the meantime, hug a firefighter. They’ve worked hard to save our town and our forests.