eCoaching Tip 44 Planning Assessment at Course Beginnings

May 16 2007 Summer Tip #3 (Checked Nov 21 2019)

eCoaching Tip 44 Planning Assessment at Course Beginnings

This tip includes another fundamentals tip as part of our summer quick reminder series.

Fundamentals Tip – Start with Effective Learning Objectives/Performance Goals

This fundamental tip addresses the challenges of assessing learning online. While the assessment choices for online are still evolving, the fundamentals of quality assessment still apply.

Here are two basic assessment principles for your online course. A valid learning assessment begins with well thought-out learning outcomes and performance goals. The outcome or performance goal describes an observable event or work product that allows you, the instructor, to determine with confidence that a student has learned the targeted knowledge or developed the targeted skill. A learning outcome or performance goal planned at the beginning of a course should specify exactly what evidence will provide a reasonable basis for concluding that knowledge has been achieved. Here are a few examples of evidence-based outcomes and performance goals.

Develop a hurricane disaster relief plan for your particular region.
Compare and contrast the characteristics of major contemporary philosophies and their implications for leadership.
Assess non-organic chemical reactions using the periodic charts of the elements.

For more on objectives, performance goals and task models for assessment and enduring understandings, see eCoaching Tip 28: Designing Assessments and Tasks that Matter.

An advanced tip follows that focuses on developing an assessment plan for your online course that balances the need to assess objective core knowledge with significant work products.

Advanced Tip – Developing an Assessment Plan — A Focus on Work Products

When planning your online course, it is wise to step back and take a fresh look at how to assess your students’ learning. As always, a best practice is to design assessment that is (1) ongoing and that (2) includes multiple points of assessment.

The steps in designing a course usually proceeds as follows: (1) develop the course learning outcomes, (2) plan the learning units and activities, and finally (3) the assessment(s). This approach sometimes results in weak assessment designs that assess the learning activities rather than the learning outcomes.

Here is a way to strengthen your assessment.

Develop the learning outcomes/performance goals concurrently with designing the assessment(s). You might even begin with the performance goal first, answering the question, “What do learners want to be able to do — with confidence — once they have completed the course?”

You may find a worksheet such as the following helpful in mapping your assessments directly to each learning outcome.

Worksheet for Mapping Learning Outcomes, Strategies and Assessment Products

Learning Outcome /Performance Goal	Assessment Strategy	Assessment Product
Example #1:Learners will know the stages of implementing change in an organization.	Individual project — with the final product being a plan with audio/presentation elements	Learners will develop a communication plan for a company’s new strategic direction.
Example #2:Learners will develop a performance improvement plan for a business unit	Group Project with a presentation and written plan	Develop a performance improvement plan for your assigned unit. The plan should include all elements which have been discussed in our weekly course discussions. And so forth….

Now let’s take a look at two more assessment issues that need special attention in online courses: the assessment of online collaboration and the security of online assessments.

Assessing Online Collaboration

Assessment of collaborative work can be complex. The traditional method of providing one summative grade based on the quality of the demonstrable work product of a collaborative group ignores the multiple dimensions of a team activity. Relying solely on a group grade for assessment often contributes to the dissatisfaction of some learners participating in a collaborative process. Some participants feel that it rewards those who have not fully participated while not acknowledging the contribution of the learners who may have contributed the most to the final product. Adding peer assessment provides a more complete picture of the participation of each individual.

Peer assessment is an evaluation tool in which team members assess the contributions, skills and behaviors of each individual team member. Peers have a unique perspective of each other’s behavior that occur outside the purview of the instructor. This means the individual group members can more accurately assess each team member’s contributions. This is particularly true in online learning where the instructor does not have the opportunity to observe group interaction outside the course site. (Note: Collaborative, synchronous tools such as the live classroom that can archive meetings might be useful for faculty in the future.)

Here’s one example of a peer assessment that could be used for an online team project:

Team Member Evaluation Form

Using your best, objective and fair professional judgment, complete the following evaluation form concerning your team member’s performance on your team project. For questions 1 through 8 use the following meanings for the numbers:

1 = poor (Did practically nothing)
2 = fair (Did as little as possible)
3 = about right (About the right amount)
4 = good (Performance was better than average)
5 = excellent (Performance was super! Beyond the call of duty!)

1. The LEVEL of effort this team member gave toward the conference was…	1 2 3 4 5
2. The QUALITY of that effort was…	1 2 3 4 5
3. How much INPUT did this team member contribute to the team discussions?	1 2 3 4 5
4. How much INPUT did this team member contribute to the team’s plan?	1 2 3 4 5
5. How much INPUT did this team member contribute to the team’s presentation?	1 2 3 4 5
6. How would you rate this team member’s level of cooperation?	1 2 3 4 5
7. How would you rate this team member’s level of time on the project?	1 2 3 4 5
8. The level of POSITIVE impact this team member’s work had on the total project was…	1 2 3 4 5
9. The level of quality of the resources this team member contributed was…	1 2 3 4 5
10. How would you rate this team member’s completion of his/her role responsibilities?	1 2 3 4 5
11. This team member met team deadlines	Rarely Sometimes Mostly Always
12. Relative to what they were supposed to do, HOW would you rate this team member’s OVERALL work and contribution to this (project, conference, discussion, presentation)?	Well Somewhat Somewhat Well Below Below Above Above

Peer assessment can positively affect accountability and responsibility within groups. However, there still may be individuals who do not contribute to a project. Peer assessment cannot remedy the obstinate lurker. It is only a deterrent. But it can ensure that the lurker is not rewarded by receiving the same grade as team members who actually produce the work.

Dealing with the Issue of Security

One characteristic of online assessment that can be particularly unnerving is the issue of security. Reasons for concern include reduced visibility of learners who are online rather than face-to-face and the fewer natural interactions. This concern needs to be addressed from the perspective of learners as well as from that of instructors and institutions.

Learners are primarily concerned with the privacy of test answers and results, and the reliability of the assessment environment. A good practice is to have a back-up plan in the event the online environment fails at a critical point in the assessment. Some possibilities for a back-up plan might be the use of email or text messages. Or the assessment might be able to be changed and rescheduled.

From an instructor’s perspective, the primary concern surrounding security in the online environment is academic honesty. This is best addressed by clearly communicating expectations and policies in the course documents, such as the syllabus and assessment plans. Some faculty favor the use of timed assessments. These can be effective in ensuring that students prepare ahead of time for online exams. Some software tools prevent students from using electronic copy and paste of test material; other software prevents students from using a printer, which can also help in minimizing cheating.

Many online faculty now minimize their dependency on security by using the automated quizzing and grading features available in most course management systems for important factual knowledge, and relying on the more substantive work products such as papers, presentations, blogs, and other creative work for significant assessment purposes.

Just the Tip of the Iceberg — Going Deeper

One of my favorite resources on assessment is an article by Thomas Angelo (1999) titled “Doing assessment as if learning matters most.” In this article Angelo analyzes the question, “Why hasn’t all the talk about assessment led to more learning improvements?” One possible reason he suggests is that most assessment initiatives have been implemented without a clear vision of what “higher” or “deeper” learning is and just how the assessment strategies that we use can begin to promote such deeper or higher learning. This question of just what deeper learning or higher learning is and how to promote it could generate a lively discussion among faculty and is perhaps a discussion we should have with our students as well. A discussion forum asking our students that question for our particular course might be very worthwhile.

Angelo provides a list of 10 guidelines for guiding our assessment practices. For now, here is just one: Engage actively — intellectually and emotionally— in our learners’ academic work. This practice encourages us to get into and understand the mental models that our learners are grappling with and that we “grow with them as they grow with us.” This type of engagement with our learners is one way of implementing “cognitive presence” that we discussed in earlier tips.

Note: This e-coaching tip was derived from a chapter in Assessing Learners Online by Oosterhof, Conrad & Ely, Pearson Prentice Hall, 2008.

References

Angelo, T. A. (1999, May 1999). Doing assessment as if learning matters most. AAHE Bulletin (American Association for Higher Education. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.490.421.

Boettcher, J. V. (2007, 2012). eCoaching Tip 28: Designing Assessments that Matter to the Learners. Retrieved from http://designingforlearning.info/ecoachingtips/ecoaching-tip-28/

Note: These eCoaching tips were initially developed for faculty in the School of Leadership & Professional Advancement at Duquesne University in Pittsburgh, PA. This library of tips has been organized, expanded and updated in the second edition of the book, The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips (2016) coauthored with Rita-Marie Conrad. Judith can be reached judith followed by designingforlearning.org.