May 16 2007 Summer Tip #3
eCoaching Tip 44 Planning Assessment at Course Beginnings
Fundamentals Tip – Start with Effective Learning Objectives/Performance Goals
This tip includes another fundamentals tip as part of our summer quick reminder series.
This fundamental tip is on the challenges of assessing learning in the online environment. Assessing online learning is a contemporary challenge, often using new and unfamiliar technologies. So the practices of assessing online learning are still evolving.
Because the technologies associated with online assessment are relatively new, it may appear that basic assessment fundamentals are being transformed. However, most of the fundamentals of quality assessments are still highly relevant.
Here are two basic assessment principles to build on in your online course. A valid learning assessment begins with well thought-out learning outcomes and performance goals. The outcome or performance goal describes an observableevent or work product that allows you to observe with confidence that a student has learned the targeted knowledge or developed the targeted skill. A learning outcome or performance goal shouldspecifyexactly what evidence will provide a reasonable basis for concluding that knowledge has been achieved. Here are a few examples of evidence-based outcomes and performance goals.
- Develop a hurricane disaster relief plan for your particular region.
- Compare and contrast the characteristics of major contemporary philosophies and their implications for leadership.
- Assess non-organic chemical reactions using the periodic charts of the elements.
For more on objectives, performance goals and task models for assessment and “enduring understandings,” see E-Coaching Tip 28: Designing Assessments and Tasks that Matter.
An advanced tip follows that focuses on developing an assessment plan for your online course that balances the need to assess objective core knowledge with significant work products that you can be relatively certain are completed by the enrolled students.
Advanced Tip – Developing an Assessment Plan — A Focus on Work Products
When planning your online course, it is wise to step back and take a fresh look at how you plan on assessing your students’ learning. Given the special environment and tools of the online space, a best practice in assessing online learning is to design assessment that is ongoing and that includes multiple points of assessment.
The usual steps in designing any course generally proceed in this fashion: develop the course learning outcomes, then the learning units and activities, and finally the assessment(s). This development strategy sometimes results in weak assessment designs that assess the learning activities rather than the learning outcomes.
Here is a way to strengthen your assessment plans.
Develop the learning outcomes/performance goals concurrentlywith designing the assessment(s). You might even begin with the performance goal first, answering the question, “What do learners want to be able to do — with confidence — once they have completed the course?”
You may wish to use a worksheet such as the following to map your assessments directly to each learning outcome.
Worksheet for Mapping Learning Outcomes, Strategies and Assessment Products
|Learning Outcome /Performance Goal||Assessment Strategy||Assessment Product|
Learners will know the stages of implementing change in an organization.
|Individual project — with the final product being a plan with audio/presentation elements||Learners will develop a communication plan for a company’s new strategic direction.|
Learners will develop a performance improvement plan for a business unit
|Group Project with a presentation and written plan||Develop a performance improvement plan for your assigned unit. The plan should include all elements which have been discussed in our weekly course discussions. And so forth….|
Now let’s take a look at two more assessment issues that need special attention in online courses: the assessment of online collaboration and the security of online assessments.
Assessing Online Collaboration
Assessment of collaborative work can be complex. The traditional method of providing one summative grade based on the quality of the demonstrable work product of a collaborative group ignores the multiple dimensions of a team activity. Relying solely on a group grade for assessment often contributes to the dissatisfaction of some learners participating in a collaborative process. Some participants feel that it rewards those who have not fully participated while not acknowledging the contribution of the learners who may have contributed the most to the final product. By adding peer assessment, a more accurate picture of the participation of an individual can be determined.
Peer assessment is an evaluation method in which all team members assess the contributions, skills and behaviors of each individual team member as they relate to group work and project completion. Peers have a unique perspective of significant aspects of each other’s behavior that occur outside the purview of the instructor. This means the individual group members can more accurately assess each team member’s contributions. This is particularly true in an online learning environment where the instructor does not have the opportunity to observe group interaction outside the course site. (Note: New collaborative tools such as the live classroom tool that can archive meetings might be useful for faculty in the future.)
Here’s one example of a peer assessment that could be used for an online team project:
Team Member Evaluation Form
Using your best, objective and fair professional judgment, complete the following evaluation form concerning your team member’s performance on your team project. For questions 1 through 8 use the following meanings for the numbers:
1 = poor (Did practically nothing)
2 = fair (Did as little as possible)
3 = about right (About the right amount)
4 = good (Performance was better than average)
5 = excellent (Performance was super! Beyond the call of duty!)
|1. The LEVEL of effort this team member gave toward the conference was…||1 2 3 4 5|
|2. The QUALITY of that effort was…||1 2 3 4 5|
|3. How much INPUT did this team member contribute to the team discussions?||1 2 3 4 5|
|4. How much INPUT did this team member contribute to the team’s plan?||1 2 3 4 5|
|5. How much INPUT did this team member contribute to the team’s presentation?||1 2 3 4 5|
|6. How would you rate this team member’s level of cooperation?||1 2 3 4 5|
|7. How would you rate this team member’s level of time on the project?||1 2 3 4 5|
|8. The level of POSITIVE impact this team member’s work had on the total project was…||1 2 3 4 5|
|9. The level of quality of the resources this team member contributed was…||1 2 3 4 5|
|10. How would you rate this team member’s completion of his/her role responsibilities?||1 2 3 4 5|
|11. This team member met team deadlines||Rarely Sometimes Mostly Always|
|12. Relative to what they were supposed to do, HOW would you rate this team member’s OVERALL work and contribution to this (project, conference, discussion, presentation)?||Well Somewhat Somewhat Well
Below Below Above Above
Peer assessment can positively affect accountability and responsibility within groups. However, there still may be individuals who do not participate in an online project. Peer assessment cannot remedy the obstinate lurker. It is only a deterrent. But it can ensure that that the lurker is not rewarded by receiving the same grade as team members who actually produced the submitted work.
Dealing with the Issue of Security
One characteristic of online assessment that can be particularly unnerving when assessing online learners is the issue of security. Reasons for concern include reduced visibility of learners who are online rather than face-to-face and the fewer interactions that may result. This concern needs to be addressed both from the perspective of learners as well as from that of instructors and institutions.
Learners are primarily concerned with the privacy of test answers and results being maintained, and the reliability of the assessment environment. To help address these concerns, an instructor needs to be scrupulous about following obvious security practices, such as guarding his or her password and only providing access to assessment information on a need to know basis. Another good practice is to have a back-up plan in the event the online environment fails at a critical point in the assessment. Some possibilities for an effective back-up plan might be the use of email or fax. Or the assessment might be able to be changed and rescheduled.
From an instructor’s perspective, the primary concern surrounding security in the online environment is academic honesty. This is best addressed by clearly communicating expectations and policies in the course documents, such as the syllabus and assessment plans. Some online faculty favor the use of timed assessments. These can be effective in ensuring that students do prepare ahead of time for online exams. Some software tools prevent students from using electronic copy and paste of test material; other software prevents students from using a printer, which can also help in minimizing cheating.
Many online faculty now minimize their dependency on security by using the automated quizzing and grading features available in most CMS systems for important factual knowledge, and relying on the more substantive work products such as papers, presentations, blogs, and other creative work for assessment purposes.
Just the Tip of the Iceberg — Going Deeper
One of my favorite resources on assessment is an article by Thomas Angelo (1999) titled “Doing assessment as if learning matters most.” In this article Angelo analyzes the question, “Why hasn’t’ all the talk about assessment led to more learning improvements?” One possible reason he suggests is that most assessment initiatives have been implemented without a clear vision of what “higher” or “deeper” learning is and just how the assessment strategies that we use can begin to promote such deeper or higher learning. This question of just what is deeper learning or higher learning and how to promote it could generate a lively discussion among faculty and is perhaps a discussion we should have with our students as well. A discussion forum asking our students that question for our particular course might be very worthwhile.
Angelo provides a list of 10 guidelines for guiding our assessment practices. For now, here is just one: Engage actively — intellectually and emotionally— in our learners’ academic work.This practice encourages us to get into and understand the mental models that our learners are grappling with and that we “grow with them as they grow with us.” This type of engagement with our learners is one way of implementing “cognitive presence” that we discussed in earlier tips.
Note: This e-coaching tip was derived from a chapter in Assessing Learners Onlineby Oosterhof, Conrad & Ely, Pearson Prentice Hall, 2008.
Angelo, T. A. (1999). Doing assessment as if learning matters most. AAHE Bulletin (American Association for Higher Education)Retrieved September 16, 2010 from http://www.che.org.il/download/files/angelo.pdf.
E-Coaching Tip 28 (Fall, 2006) Designing Assessments and Tasks that Matter Retrieved September 16, 2010 from http://www.designingforlearning.info/services/writing/ecoach/tips/tip28.html
Note: These E-coaching tips were initially developed for faculty in the School of Leadership & Professional Advancement at Duquesne University in Pittsburgh, PA. This library of tips has been organized and updated through 2016 in the second edition of the book, The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips coauthored with Rita Marie Conrad. Judith can be reached judith followed by designingforlearning.org.