"What is the point of testing in Maritime Training"? Very few people know the correct answer to this question. This is a problem.

Feb 24, 2014, 4:42PM EST
"What is the point of testing in Maritime Training"? Very few people know the correct answer to this question. This is a problem.
Knowing why we test our maritime trainees is critical to designing an effective assessment program. However, the answer is not what most people think it is - resulting in critical training mistakes and the common use of ineffective assessment practices. This article answers the critically important, but rarely understood question “what are we actually trying to achieve with testing”?

Maritime Training: The full library of maritime training articles can be found here.

Blog Notifications: For notifications of new maritime training articles, please follow this blog.

Maritime Mentoring: International Maritime Mentoring Community - Find a Mentor, Be a Mentor


 

 

"What is the point of testing in Maritime Training"? Very few people know the correct answer to this question. This is a problem.

 

We talk a lot about assessment and how to get it right. But in order to make good decisions about testing, we have to understand why we test in the first place. And looking back over my past articles, I realize I have never directly addressed this simple question. Shame on me. This is a problem for two reasons:

 

  1. Because knowing the answer is critical to designing an effective assessment program, and
  2. because the answer is not what most people think it is - resulting in the common use of ineffective assessment practices.

 

As such, I have decided to interrupt my series on “where does learning content come from” (which I promise to return to shortly). I do so to write a short article to answer the question “what are we actually trying to achieve with testing”? Without understanding the answer, one cannot make important decisions about testing - such as the format of tests,  the need (or not) for supervising test takers, the length of exams, when (during the training process) should tests be delivered, and so on. A correct understanding of the role of testing informs all of these questions, yet very few people know the real answer.

 

So, let’s get to it.

 

This is the first of two articles covering a fundamental understanding of why we test and the implications of that knowledge. If you would like to be notified when the second article and subsequent articles are ready, and have not already done so, please feel free to sign up here for notifications.

 

WHAT ISN’T TESTING?

When asked why we test trainees, most will provide an answer along the lines of “to see if they know the required knowledge / can perform the desired skill”. This is a misleading answer, and it leads us to false conclusions about how we should test. It leads us to conclude that if someone masters a test, they are fully competent - possessing all the required knowledge and skills.

 

However, testing can never comprehensively assess all the required knowledge or the ability to perform a skill under all expected conditions. It can test whether some knowledge is known, or if some skill can be performed under specific and limited test conditions, but it can never assess the full breadth of knowledge or competencies required by a worker in the maritime industry - or that of almost any other industry.

 

Why do I say this?

First, I say this because it is not possible to devise a test which covers all required knowledge or all important skills under all conditions. More than that, it would be impossible for anyone even to imagine all conditions under which a skill might need to be performed, let alone devise a test to cover them all.

 

And secondly, even if you could list all the knowledge and skills which need to be mastered by a trainee to do a job throughout their lifetime (which you cannot), and knew how to devise a test to assess each one (which you do not), constructing a test to address them all would take years, and almost as long to write.

 

So testing can never tell us whether someone has all the required knowledge or can perform all the required skills under all conditions. It is not possible for a test to tell us whether someone is fully competent!

THEN WHY DO WE TEST?

Testing can never comprehensively assess all required knowledge and skills. Yet we test anyhow, and we do so for good reason. It is reasonable to ask, then, if we cannot assess all requirements, what does testing actually achieve?

 

A good explanation comes from a quote found in the Wikipedia article on the audit process (you can visit the page here). This is a process employed in many industries - most notably by auditors when assessing the accounting practices of a business. There is a direct parallel between the audit process and the form of testing applied in the maritime industry. The Wikipedia quote discusses the role of the financial audit as follows (the emphasis is mine):

 

“... The goal of an audit is to express an opinion of the person / organization / system (etc.) in question ... based on work done on a test basis. Due to constraints, an audit seeks to provide only reasonable assurance that the statements are free from material error. Hence, STATISTICAL SAMPLING is often adopted in audits. ...

 

Testing follows the same process. Due to the constraints of time and ability, we cannot test a trainee on every single requirement. In fact, most argue that we can only directly test a very small fraction. Therefore, our testing of trainees is an audit process of sorts, where instead of testing everything, we test only a subset (a “sampling”) of the knowledge and skills the trainee requires. We then extrapolate from the results of that sample to make assumptions about how well they understand everything else they need to know. In this way we either gain (or don’t gain, depending on the outcome) a “reasonable assurance” that the trainee possesses the required breadth of knowledge and skills - including those we have not assessed explicitly in the test.

 

That may seem like a pedantic distinction / definition, but it is not. And there are important implications.

IMPLICATIONS

The main implication comes from that fact that when we test, we are making assumptions about the candidate’s knowledge of those items not directly covered by the test. If testing is only an audit process or sampling as we have said above, and we want to use that sampling to draw conclusions about those items we did not explicitly test, then we had better be careful not to do anything which makes it impossible to draw those conclusions.

 

Said another way, I have always believed that the real value of testing is the incentive it creates for the learner to learn as well as they can and as much as they can. If they want to pass the test that they know is coming, and they have no idea what is going to be on that test (i.e. they don’t know what items will be “sampled”), then they have no choice but to study everything equally, as well as they possibly can. This is a great incentive to learn, but there is also an implication for our interpretation of test results.

 

If we assume, from the above, that the candidate learned everything equally well, then their exam performance is not only a reliable indicator of their knowledge and skills of those items on the exam, but also of those that were not present on the exam - everything else they need to know. This is the goal we are trying to achieve, and if we do achieve it, a reasonably constructed test will allow us to draw conclusions about not only those items explicitly tested, but also about those items not tested. This is very important since, as argued in the beginning, it is impossible to test everything - or even most things.

 

But we have to be very careful in how we deliver exams because it is easy to “break” that audit effect and render useless any conclusions about those items you did not test. Let’s think about this a bit.

 

First, to construct a meaningful “sampling”, it must be the case that the trainee (the subject of the audit) does not know which items are going to be on the test (sampled). Otherwise we cannot extrapolate exam results to items which were not on the exam because the trainee had no incentive to study those other items. Therefore, we must make sure that no two trainees receive the same test - they will tell one another what is on the exam and the incentive to study broadly (and our ability to extrapolate) is gone.

 

This is only one of many ways to “break” our ability to extrapolate test results to items not covered on the exam. Another is “open book” exams. An open book exam will provide you information about whether the candidate knew the answers to the questions on the exam (and may even teach him or her the answers in the process), but will provide no information about knowledge and skills not present on the exam. Having said that, open book exams do test the ability to successfully find information - so if that is your goal, they can be a good choice.

 

There are many other important and varied implications as well. Understanding these implications is critical for:

  • The creation of an effective remedial learning plan for those who fail an assessment (this is a big one - and is often done incorrectly).
  • The creation of our assessments
  • The delivery of our assessments

 

I promised in the opening that this article would be short and it looks like I failed. So I will leave those implications and others for the next article in two weeks time. If you would like to be notified when that article and subsequent articles are ready, and have not already done so, please feel free to sign up here for notifications.

CONCLUSION

Testing is a critical part of the training process. Sadly, it is really easy to get it wrong. However, it is also really easy to get it right - so long as there is a good understanding of what testing actually is. Knowing that testing is necessarily the process of sampling knowledge and skills with a view toward extrapolating those results to a general grasp of skills and knowledge, makes it easy to make intelligent choices when designing and delivering your testing program.

 

As I say, check back in two weeks for the conclusion of the implications to this fact. Until then, sail safely and thanks for reading!

 

 

# # #

 

About The Author:

Murray Goldberg is the founder and President of Marine Learning Systems (www.marinels.com), the creator of MarineLMS - the learning management system designed specifically for maritime industry training. Murray began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, the world’s first commercially successful LMS for higher education; serving 14 million students in 80 countries. Murray has won over a dozen University, National and International awards for teaching excellence and his pioneering contributions to the field of educational technology. Now, in Marine Learning Systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.

 

Maritime Training: The full library of maritime training articles can be found here.

Blog Notifications: For notifications of new maritime training articles, please follow this blog.

Maritime Mentoring: International Maritime Mentoring Community - Find a Mentor, Be a Mentor

 

 

 
Filed under: assessment, testing, training
Report abuse



Bookmark this page to:Add to Faves Add to MyAOL Add to Simpy Add to Delicious Add to Live Add to Digg Add to Newsvine Add to Reddit Add to Multiply Add to Blogmarks Add to Yahoo MyWeb Add to Slashdot Add to Mister Wong Add to Spurl Add to Furl Add to Link-a-Gogo Add to Yahoo Bookmarks Add to Twitter Add to Facebook Add to Diigo Add to Mixx Add to Segnalo Add to StumbleUpon Add to Magnolia Add to Ask Add to Backflip Add to Terchnorati Add to Google Bookmarks Add to MySpace

Comments
John Douglas
Murray,
Been wondering when you would get around to this one!! It is the most crucial of all the building blocks in competence training. I prefer the words competence checks than testing as it is a continuous process through the competence plan.
We say "If you don't know where you are going, any road will do"!!
The important point is to have a clear idea of what needs to be fulfilled to declare competence. That is in fact the "test". Then you build backwards in the normal fashion.
sounds simple but of course it is not.
A story on how it can go wrong.
A Japanese LNG shipping company trained its officers to international standards in internationally approved training centres but still wanted a double check by an independent centre. They failed!! The invesitgation showed the ta the Japanese trainers shortcut the testing!!!

When you next meet the Seagull gang give them my regards!
2/25/2014 2:34:55 AM
 
Murray Goldberg
Thanks John! I like "competence checks" as well - all commonly used words are loaded with implied meanings.

Interesting point about the Japanese LNG shipping company. This is a good example of what happens sometimes when the trainers are the same people who perform the assessment. In a proper test environment, these are two different people / parties. Lot's to say there. Maybe a topic for a future post (though I think I covered it last year).

Always great to get your input. Thanks for the note.
2/25/2014 11:58:36 AM
 
Murray Goldberg
Thanks John! I like "competence checks" as well - all commonly used words are loaded with implied meanings.

Interesting point about the Japanese LNG shipping company. This is a good example of what happens sometimes when the trainers are the same people who perform the assessment. In a proper test environment, these are two different people / parties. Lot's to say there. Maybe a topic for a future post (though I think I covered it last year).

Always great to get your input. Thanks for the note.
2/25/2014 11:58:40 AM