Is Simulator Training Worth It?

Feb 11, 2013, 2:51PM EST
Is Simulator Training Worth It?
For decades, simulation has been a part of maritime bridge and engine room training. But as with many safety initiatives, its effect is sometimes difficult to quantify. We know it has value, but it does come at a cost. Is the cost worth the value derived from simulator training? This article examines some recent research in an attempt to answer this question.


Blog Notifications: For notifications of new maritime training articles, please Follow this blog.

Maritime Mentoring: International Maritime Mentoring Community - Find a Mentor, Be a Mentor

Is Simulator Training Worth It?

Introduction

For decades, simulation has been a part of maritime bridge and engine room training. But as with many safety initiatives, its effect is sometimes difficult to quantify. We all know, both intuitively and empirically, that simulator training has value. It extends a trainee’s experience base in both typical and atypical scenarios. But while we all agree that simulator training is valuable, we also know that it comes at a cost. After all, simulators and the time spent on them are expensive. Is the cost worth the value derived from simulator training?

 

This article examines some recent research which attempts to answer this question. To receive notification of future maritime training articles, please sign up here if you have not already done so.

Cost vs. Benefit

One compelling argument applied to safety training in general is that the value of one life saved is greater than any cost - as long as it is affordable. Thus if we believe that simulator training has the potential to save one life, it is worth the costs associated with it and therefore no further analysis is necessary. But there are real problems with that line of reasoning. First, it does not provide us with any basis on which we can compare other safety initiatives. It may be that simulator training is indeed worthwhile, but that some other safety initiative can save more lives at a far lower cost. Unless we assess the costs and value of each we are unable to make informed decisions. Another issue is that without a cost benefit analysis, implementation decisions are sometimes more emotional than logical. After all, if it can be shown that simulator training actually saves money through a reduction in accident related costs or performance issues, then perhaps its use would be even more pervasive than it already is.

 

This is exactly the question addressed by a very interesting MET paper given by Professor Capt. Stephen Cross of the Maritime Institute Willem Barentsz (MIWB) in West Terschelling, The Netherlands. Prof. Capt. Cross’ paper, “Aspects of Simulation in Met - Improving Shipping Safety and Economy”, was presented at the IMLA 20 conference on maritime education and training in July, 2012 at the MIWB (where I had the good fortune to meet Capt. Cross). His paper presents a concrete view of the economic effects of simulator training. The results are compelling.

The Idea

Prof. Capt. Cross expressed the motivation of his study as follows:

If simulator training can improve safety of operations, this would result in fewer accidents, which in turn will save funds, which could be used to afford the additional training efforts.

 

 

Additionally if the amount of the increased costs of training is compared to the funds spent presently on damages from accidents, a simple cost benefit analysis could show if such training efforts are worthwhile”.

The Study

In order to answer this deceptively simple question, Cross needed to look at a wide array of information related to the desired objectives, the current conditions of MET and maritime operations, and then had to study (and sometimes project) the consequences of change. To give you some idea as to the complexity of the study,  Cross proceeded along the following path:

  1. First he determined what percentage of maritime accidents were attributable to human error.
  2. Next he determined what percentage of these accidents could be attributed to training shortcomings.
  3. Next he determined what percentages of competencies could be improved by simulator training.
  4. Next, he had to determine by how much the above competencies could be improved through simulator training.

 

Multiplying the various percentages together gave an estimate of the reduction in accidents through the use of simulator training. With that information, he could then look at the cost of simulator training in order to compare it to the cost savings through a reduced number of accidents. I summarize his analysis below. Please note that in the interest of space, only a portion of Cross’ analysis can be presented. I encourage you to read the paper for full details.

Finding the Percentages

  1. Human Error: Looking first at the percentage of accidents attributable to human error, Cross arrived at 80% based partly on the following:

 

“... the Norwegian DAMA database of accidents for the Safeco project (EU 4th FP, Safeco, 1996) whereby from 1981 to 1996 some 5400 accidents were included and some 1100 were analysed, the division of basic causes was 80% human factors and 20% technical factors.”

 

… Human Error: 80% of accidents

 

  1. Lack of Sufficient Training: Looking at how training influences accidents, Cross looked at a number of studies which evaluated the causes of accidents. Among them he cites the following three:

 

  1. “Wagenaar and Groeneweg (Wagenaar, et al, 1987) found 35% of the accidents were due to improper training and 46% due to bad habits, which could most likely be influenced by procedural training. This totals 81%.
  2. Inoue (Inoue, 1996) finds 55% of accidents to be collisions and 15% groundings. That means that implicitly 70% of the reviewed accidents could have been avoided by better trained personnel. Although possibly technical failure can also account for some of this kind of incidents.
  3. For the Safeco project (EU 4th FP Safeco, 1996), 41% of the 80% human error related accident causes, indicate lack of knowledge, skills and attitude, which could be improved by training. Additionally 37% of 80% is due to lack of operational procedures. This means that up to 33% + 30% = 63% of the investigated accidents could have been influenced and possibly partly avoided through relevant competence based and procedural training.

 

From these three studies it seems conservatively acceptable to say that from 65% upward of the investigated casualties has relevance to (lack of) sufficient training.

 

… Lack of Sufficient Training: 65% of accidents

 

  1. Simulator Training Applicability: Not all competencies needed for safe operations can be taught and practiced in a simulator. Thus the next step was to determine what percentage of competencies were, in fact, “teachable” via simulator training. According to Prof. Capt. Cross:

 

“Part of the required seafarer training can be done using simulator systems. In order to estimate which part, a count can be made of the number of competences or skills per function and level, as stated in STCW Code part A where simulators are indicated. This figure in relation to the total number of competences for that function and level gives a percentage of simulator applications. … The average of the counted percentages equals 58%.

 

… Simulator Training Applicability: 58% of competencies

 

  1. Competency Improvement Through Simulator Training: Finally, Cross needed to determine the level of improvement in performance that could be achieved through simulator training. To do so, the study provided simulator training to groups of mariners, both experienced and inexperienced. It then looked comprehensively at the outcomes of exercises for these groups over the time that they were involved in the training. In the end, both groups (experienced and inexperienced) benefitted significantly from Simulator training. In the words of Prof. Capt. Cross:

 

“Based on the … observations a calculated average performance improvement of 45% seems acceptable to be assumed [due to simulator training]”.

 

… Performance Improvement Due to Simulator Training: 45%

Putting the Numbers Together

Prof. Capt. Cross took the findings above, and then multiplied them together to arrive at a conservative estimate of the accident reduction possible via simulator training. The ultimate result, 14%, is shown in his table below:

 

 

Thus far, Cross’ analysis has estimated that through the appropriate application of simulator training, 14% of maritime accidents could be avoided. What does this mean for the economics of simulator training versus the cost of accidents?

So - Is it Worth It? The Economics of Simulator Training:

Prof. Capt. Cross indicates in his paper that there are many potential cost savings available through improved operations from simulator training, even when ignoring the potential for accidents. But to look at accident costs in particular, he cited the claims history of the International Oil Pollution Compensation Fund over the period of its existence. Even though the IOPC Funds claims represent a fraction of the cost of maritime accidents worldwide, they are well documented and thus provide a reliable source of information on accident costs. The results are impressive. According to Cross:

 

“Over the 28 year period of [IOPCF] observations used, at least 856 million $US have been claimed for accidents which in some way have a relationship to bridge, engine room or cargo handling procedures. …[A reduction of] 14% related to the simulator training course cost would allow for at least 376946 “average” student simulator courses to be afforded. As this figure is almost similar to the global officer population it means every officer could be afforded a simulator training course from the avoided accident claim costs of the IOPC Fund relevant accidents.

 

So - if the 14% accident reduction estimate is accurate, and it is applied to the relevant IOPC funded accidents, the cost saved could provide every officer in the world with a simulator training course. And since there are far more accidents (and their related costs) than are funded by the IOPC, the conclusion is that simulator training has the effect of both reducing costs and improving safety - a win-win.

Conclusion

Prof. Capt. Cross’ analysis is a compelling argument for simulation training as both a cost-saving measure and a safety improvement measure. Even if you find an argument with one or another of the numbers presented in his analysis (and he is clear that there are many factors which would influence the 14% he arrived at), one could argue that the “margin of safety” in the analysis is very large. That is, it seems unlikely that his assessment could be so far off as to make simulation training a net cost, as opposed to a net saving. And even if it were a net cost, as unlikely as that might be, we can go back to the original visceral argument: if one life is saved, the any affordable cost is one well spent.

 

# # #

 

About The Author:

Murray Goldberg is the founder and President of Marine Learning Systems (www.marinels.com), the creator of MarineLMS - the learning management system designed specifically for maritime industry training. Murray began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, the world’s first commercially successful LMS for higher education; serving 14 million students in 80 countries. Murray has won over a dozen University, National and International awards for teaching excellence and his pioneering contributions to the field of educational technology. Now, in Marine Learning Systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.

 

Blog Notifications: For notifications of new maritime training articles, please Follow this blog.

Maritime Mentoring: International Maritime Mentoring Community - Find a Mentor, Be a Mentor

 
Filed under: elearning, simulation, training
Report abuse



Bookmark this page to:Add to Faves Add to MyAOL Add to Simpy Add to Delicious Add to Live Add to Digg Add to Newsvine Add to Reddit Add to Multiply Add to Blogmarks Add to Yahoo MyWeb Add to Slashdot Add to Mister Wong Add to Spurl Add to Furl Add to Link-a-Gogo Add to Yahoo Bookmarks Add to Twitter Add to Facebook Add to Diigo Add to Mixx Add to Segnalo Add to StumbleUpon Add to Magnolia Add to Ask Add to Backflip Add to Terchnorati Add to Google Bookmarks Add to MySpace

Comments
Lou Vest
It would seem that the competencies that are subject to being improved by simulator training were also subject to being improved by on board training. It would seem that they would be. So by not giving any weight to onboard training the paper seems to assume that on board training is currently maximized and that any further improvement in these areas has to come from simulators.

I don't think that's true. I have been a pilot in a busy harbor for 26 years and I almost never see any training going on during transits. Occasionally a mate or captain will ask a few questions about how things are done or why I did something a certain way, but that is relatively seldom and not organized.

On the contrary, on many transits I see evidence that the captain has very little interaction with the crew other than to shout at them over the radio when docking.

I would say that 3/4 of what is taught in simulators could be taught onboard by captains and chief mates taking an active interest in the junior officers.

Obviously there are wonderful exceptions, but we're talking about the big statistical picture. I will make an exception for emergency procedures. Simulators are great for throwing a lot of emergencies at people in a short time, but that's another paper altogether. How many accidents are prevented by better emergency response - after your routine operations have got you in an emergency situation?
2/12/2013 11:13:03 AM
 
Murray Goldberg
Having read the whole paper thoroughly, I don't think the author considers any form of current training to be maximized and, I am sure, would most certainly agree that improvement is available in many ways. But the paper did indeed focus on simulator training and what its value is in terms of reducing accidents. And yes - your point of the overlap in simulator and onboard training is very well taken. This is why blended approaches are found to be most effective (better than either one in isolation).

Thanks for the great comments.
2/12/2013 11:17:52 AM
 
Tony Palenzuela
Kudos to Capt. Cross for his very informative analysis about simulators. As a deck officer, I have undergone several simulator training - Radar Observer course, Radar Simulator course, Automatic Radar Plotting Course, Ship Handling, Twin Screw, SSBT, ECDIS and I must say it helped me a lot that what I learned is really applied at work, knowing the basics
However, I saw the problem now that simulators and on board equipment are getting more complicated. If the simulator is not similar to the equipment that the ship officer will use on board then his training is not so effective. And there are so many equipment brands now and they are not exactly the same. That's why they require now the Type-Specific Training.
I know that in many simulator tests for students, they will not pass if they used a different simulator in a training course other than what they will use in the test.
I just hope that there will be only one type of simulator similar to the the equipment on the vessel. This can save lives.
2/12/2013 12:03:55 PM
 
Tony Palenzuela
Yes Murray, blended approaches are found to be most effective. But I believe the on board familiarization, related to simulated equipment or procedures, needs to be given emphasis.
2/14/2013 10:56:37 AM
 
Ed Enos
Great comments. I would add another perspective that we also see a lot of these days. Typical in the cruise industry today are officers training themselves or each other. A company hire a simulation facility and a group of officers from same company gather around and replicate their own ship organization with company policy and procedures. They need to engage with different people (at least) to better assess where they need improvement. A senior captain training with junior officers from within the same company is inevitably going to impart what he wants, how he likes it done, how he learned a procedure and will teach others the same. As a pilot, I frequently see seasoned masters doing something fundamentally wrong regarding a ship handling maneuver. But in a "de-brief" they explain to a junior officer how and why he did it that way...even though he is utterly wrong. But as you see, bad habits are perpetuated within a company or bridge team, because they are being "taught" bad habits.

Simulation is great, I advocate it's increased use. But as far as achieving the goal of reducing human error, that depends on HOW simulated exercises are being accomplished, WHO is teaching the course, and HOW is the education obtained being validated later on?
2/16/2013 2:38:00 AM
 
John Douglas
Hi Murrray, your Norwegian commentator again.
I have known Stephen for a number of years and we share many concerns on maritime competence and training so I am not surprised at this article. However I have a couple of comments. The dats he used is a little dated but the basic premise that 80% of accidents are contributed to by human factors remains true today.
Secondly costing human life is fraught with danger. I remember Richard Goss once stating; do you want the "hot" answer or the "cold and economists answer"??
The point is not only the life issue but the environmental consequence issue and I think Stephen did not touch this.
So for simulators. I prefere to use simulation as a far better way to refer to the use of virtual and augmented reality in am aritime context. Whilst landbased simulators are the main vehicle for training do not forget that simulatos are place on ships and even more exciting it is now possible now to put the ship in "simulation mode" so that checks on equipment and for the use of traning can be conducted without actually moving the ship!!
Also the major drawback with simulators has been the lack of inbuilt assessment but thankfully many manufacturers have included ongoing assessment against competence standards within various scenarios.
Today it is unthinkable not use them where risk is high. For example docking a 15000 teu container ship in cross winds, bringing a VLCC through a narrow depth limited channle or working offshore supply to a platform in a tidal regime. Simulators can do all of this without loss of life!
Finally there must be some standard setting for manufacturers and users of simulators to ensure quality. DNV has a standard and certification system for simulators, the only one in the world. Here endeth my contribution!!
2/16/2013 3:59:45 AM
 
Murray Goldberg
Ed - thanks for your comment. Yes - this is true for all training. Two (at least) links much each be strong in order for training to be effective: Training procedures and training "content". If the people providing the content or expertise in this case are teaching poor principles, then the whole system fails.

There have been interesting discussions about this in the past: the question of who is training the trainer - and is the trainer qualified to train. Very important questions.

Thanks again for commenting!
2/18/2013 6:36:51 PM
 
Murray Goldberg
John - it is always a pleasure to receive your comments. Well said. I have nothing to add - other than to say that in his full paper Stephen does indeed refer to other kinds of consequence (including environmental). But for the purpose of the results stayed with those that could be most directly measured. Even with the limited scope, the financial advantage still proved more than worthwhile. If you are interested in reading the full paper let me know and I will connect you with Stephen.

Best regards to my Norwegian friend!
2/18/2013 6:41:08 PM
 
Campbell Jenkins
If you look at the MAIB accident reports they also say human error BUT most are related to crew exhaustion.
That problem doesn't get fixed by training the crew.
It gets fixed by legislation
PS
DNV has a standard of hardware for simulators BUT not what you do when using them
2/20/2013 5:27:27 AM
 
Murray Goldberg
Campbell - I think your point is well taken that accident causality is a complex issue where there is always an interplay of contributing factors that end up creating the final result. All of these factors need attention. Taking this argument further, I suspect it is fair to say that crew exhaustion alone is likely never "the" sole accident cause. Training, policies, rest, experience, communication, equipment condition, etc - all contribute to most accidents, and I think Prof. Capt. Cross does a good job of reflecting that reality in his paper.

Thanks for the good comment.
2/20/2013 1:14:14 PM
 
Tony Palenzuela
You got it right Murray. Accidents are caused by many factors. This week alone, there are many casualties all over the world- collisions, groundings, fires, sinking, etc. and we can trace the root cause and they would come up with old ship, poor maintenance, poor operator’s support, defective equipment, crew doing multiple jobs, software that are very unfriendly to users, lack of information, poor training, etc.
2/26/2013 11:21:24 AM