• For Contributors +
• Journal Search +
Journal Search Engine
ISSN : 1229-3431(Print)
ISSN : 2287-3341(Online)
Journal of the Korean Society of Marine Environment and Safety Vol.24 No.3 pp.311-318
DOI : https://doi.org/10.7837/kosomes.2018.24.3.311

# A Study on the Effectiveness and Improvement of Simulation Training for Apprentice Officers

Myoung-ki Lee*, Young-Soo Park**, Weon-Jae Ha**
*Ocean Science and Technology school, Korea Maritime and Ocean University, Busan 49112, Korea
**College of Maritime Sciences, Korea Maritime and Ocean University, Busan 49112, Korea
Corresponding Author : youngsoo@kmou.ac.kr, 051-410-5085
20180312 20180424 20180529

## Abstract

In accordance with the stipulations of the STCW Convention, simulation training has been enforced in order to develop practical skills so as to prevent accidents by predetermining the risks in special marine environments. Simulation training is a useful way to acquire navigation abilities, and can continuously measure the ability of a trainee by applying an appropriate evaluation. However, the result of training is evaluated by the instructor’s subjective judgment without quantitative criteria. Therefore, this study aims to quantitatively evaluate the effectiveness of simulation training. For this purpose, evaluation items were derived by analyzing legal standards, earlier studies, and the current status of MET institutions. The simulations were then performed three times in the same scenarios and analyzed the results. As a result, it has been shown that the objectively analyzed ability to keep the route and to make safe passage with other vessel, as well as subjectively evaluated ability by the apprentice officer has been improved as training progressed. Through the evaluation of simulation training results, it can be derived that simulation education needs supplementation, and can be provided as a basic form of data to quantify the evaluation results of the simulation training in the future.

## 1 Introduction

A navigator needs to possess practical skills of navigation with the good use of navigational equipment, as well as overall knowledge of navigation. In particular, they should quickly and accurately make a decision in dangerous situations. Therefore, in accordance with the International Convention on Standards of Training Certification and Watchkeeping for Seafarers (STCW) II/1 and IMO Model Course 1.07, completion of Automatic Radar Plotting Aid (ARPA) training and radar simulation training are required (IMO, 2010). Domestically, in accordance with The Ship Personnel Act (http://www.law.go.kr, 2017), ship officers should have radar simulation training based on a ship handling simulator (5 days) as well as Automatic Radar Plotting Aid training (3 days) in order to have the necessary practical skills for predetermining the risks inherent in special marine environments, and preventing accidents.

Simulation training can be repeated in a safe and controllable situation, and enables instructors to provide proper feedback and evaluate the trainees in a useful learning method so they may acquire the practical skills of navigation (Nahlinder, 2010). By applying proper evaluation methods, it is possible to measure a trainee's ability continuously during simulation training, and observe the process of a trainee's competency improvement (Kobayashi, 2005).

Regarding the simulation training specifically, Karlsson (2011) suggested the items to be evaluated by an instructor at the time of briefing and debriefing, and Park (2016) classified and suggested the items to be evaluated depending on the basis of different roles during simulation training. However, the suggested items, including proper CPA maintenance and a CPA/TCPA setting, were mostly based on an instructor's subjective judgment. Shin et al. (2017) conducted a questionnaire survey on the improvement of equipment utilization ability, radar analysis ability, maneuvering ability and comprehensive ability and performed control analysis and proximity analysis of the simulation result. However, in the study, their improvement was evaluated by trainers' subjective judgment, and the analysis of the simulation result failed to take into account passage with other vessels.

Therefore, this study aims to analyze the effectiveness of simulation training in an objective way. For this purpose, the main items that help to evaluate the training effectiveness were drawn by analyzing legal standards on simulation training, previous studies, and the current simulation training evaluation status of MET (Maritime Education Training) institutions. While performing a 64-hour simulation training program, the result of each simulation after 4 hours, 24 hours, and 48 hours was evaluated and compared by derived evaluation items.

Through the evaluation of the simulation training results, it is possible to draw out the complementary point and improvement plan for simulation training, and to provide a basic data set in order to quantify the evaluation results of the simulation training.

## 2 Extraction of main items for simulation training

### 2.1 Survey on legal standards for simulation training

In order to find the evaluation criteria defined to prove one's navigational competency related to Radar/ARPA simulation training, legal standards for simulation training were surveyed.

Table 1 presents a comparison between international and Korean legal standards for simulation training.

Regarding international standards, STCW (International Convention on Standards of Training, Certification and Watchkeeping for Seafarers) Code includes navigational competency items as the minimum compulsory standards of ship officers in charge of navigation in a ship with more than 500 tons. In Part A / A-II / 1 of the Code, the items related to the use of Radar and Automatic Radar Plotting Aids (ARPA) for keeping safe navigation are described. Although the evaluation items related to information acquisition and analysis based on Radar/ARPA, actions to avoid collision, changes in a ship's course and speed, communication, and maneuvering signals are described, the evaluation criteria of the items are not defined. IMO Model Course 1.07 suggests the guidelines for training program on the basis of the minimum standards of STCW, and the necessity of evaluation is described, but evaluation items or criteria are not defined (IMO, 2017).

With regard to Korean legal standards, Radar/ARPA training contents and training hours are defined to prove an officer's qualifications in accordance with Article 2 of the Enforcement rule of the Ship Personnel Act. The curriculum of such training is defined in Article 14 of the Notice of Standard for Designation Education Institution by the Ministry of Oceans and Fisheries. However, items about training evaluation are not defined in the Ship Personnel Act and Notice of Standard for Designation Education Institution.

### 2.2 Survey of earlier studies on simulation training

To draw the main items for the evaluation of effectiveness of simulation training, the evaluation items of simulation training suggested or used in previous studies were surveyed.

Table 2 presents a comparison of evaluation items suggested or used in previous studies.

To evaluate trainees, Karlsson (2011) suggested evaluation items in the categories of simulation and debriefing. These items include CPA/TCPA limit setting and minimum 1 mile CPA maintenance. In his study, whether or not items were executed was evaluated.

Park (2016) classified roles into captain, officer, VTSO, and Q/M in simulation, and suggested different evaluation items depending on the roles. The items include proper CPA maintenance and proper radar use. The evaluation score ranged from 1 to 10 points.

Shin et al. (2017) conducted a subjective evaluation and simulation analysis. In the subjective evaluation, items such as Radar/ARPA utilization ability, radar analysis ability, and maneuvering ability were evaluated in the points scale ranging from 1 to 10. In simulation analysis, control and proximity were analyzed.

Given the analysis results of previous studies, it was found that Karlsson (2011) and Park (2016) commonly suggested RADAR use and maintaining proper CPA for safe passage as items for evaluation, and that Shin et al. (2017) evaluated simulation training through trainees' subjective evaluation and simulation based proximity and control analysis.

### 2.3 Survey of the status of MET institutions for simulation training

To draw the main evaluation items in order to analyze the effectiveness of simulation training, the evaluation items which are used in Korean MET institutions were analyzed in the categories of communication, safe passage with other vessels, implementation of the voyage plan, and usage of equipment.

Table 3 shows a comparison of simulation training evaluation items between Korean MET institutions.

In institution A, there are 6 evaluation items pertaining to communication, 3 items focused on safe passage with other vessels, 5 items on the implementation of the voyage plan, and 2 items about the usage of equipment. Basic points are given, and a final score is calculated by way of adjusting points on the basis of evaluation criteria.

In institution B, there are a total of 14 items: no evaluation item for communication, 7 items pertaining to safe passage with other vessels, 1 item on the implementation of the voyage plan, and 6 items on equipment usage.

In institution C, there are 2 evaluation items pertaining to communication, 7 items on safe passage with other vessels, and 1 item about the usage of equipment. The overall score is calculated depending on whether each item is executed properly.

The simulation evaluation items of each training institution were surveyed in the categories of communication, safe passage with other vessels, implementation of the voyage plan, and usage of equipment. It was confirmed that MET institutions mainly used items related to safe passage with other vessels.

In the items pertaining to safe passage with other vessels, collision avoidance action and CPA/TCPA, were commonly found; in the items about communication, reporting and communication were commonly included; in the items on implementation of the voyage plan, course check were commonly found; in the items about the usage of equipment, use of radar was frequently found.

### 2.4 Extraction of evaluation items for simulation training

To evaluate the effectiveness of simulation training, legal standards of simulation training, earlier studies, and the evaluation items of Korean MET institutions were surveyed. As a result, three evaluation items apprentice officers' subjective evaluation, cross track distance, and clearance distance with other vessels were each derived.

• (1) In terms of apprentice officers' subjective evaluation, their evaluation items on navigation and overall skills which were used in Shin et al. (2017) were scored in the categories of ship-handling ability and navigation ability. Fig. 1 shows an example of apprentice officers' subjective evaluation.

• (2) Cross track distance is the item that evaluates the ability to keep course line. Course keeping, as suggested by Park (2016), and position fixing, which is commonly used in Korean MET institutions, are both used to evaluate the ability to keep course line so as to quantify the cross track distance from a planned route. Fig. 2 illustrates the outline of cross track distance.(1)(2)

$C r o s s T r a c k D i s tan c e = sin θ · l$
(1)

$θ = C O S − 1 ( b 2 + l 2 − a 2 2 b l )$
(2)

• a : The distance from the previous way point to the current position (m)

• b : The distance from the previous way point to the next way point (m)

• l : The distance from the current position to the next way point (m)

• θ : The angle between b and l (°)

(3) In regards to the clearance distance with other vessels, the item of safe passage which is reflected via the item related to CPA which was suggested by both Karlsson (2011) and Park (2016) and is commonly used in Korean MET institutions was used to quantify the clearance distance with other vessels.(3)

$C l e a r a n c e D i s t a n c e = ( x o − x t ) 2 + ( y o − y t ) 2$
(3)

• (xo, yo) : The coordinates of own ship's position

• (xt, yt) : The coordinates of a target ship's position

## 3 Analysis on the result of simulation training

### 3.1 Experimental subjects and scenario setting

A 64-hour simulation training was conducted for 85 apprentice officers who had completed one year of onboard training. They were subjected to the same-scenario simulation after 4, 24 and 48 hours of training, and the results of the simulation were analyzed. The waterways in the scenario ware set to the Kanmon Straits, the most difficult one as reported by apprentice officers in simulation training (Shin et al., 2017). Kanmon Straits AIS data between 0800-0900 hours were used because it has the largest actual East-bound traffic volume in the Straits (Hiroaki et al., 2010). As a target ship type, a pure car carrier with good ship-handling performance was selected among vessels with less than 10m of draft which can pass the Kanmon Straits (UK Hydrographic Office, 2013).

Fig. 3 illustrates the waterways and the ship used in the scenario designed for the simulation training evaluation.

### 3.2 Analysis on the effect of each evaluation item

In order to analyze the effectiveness of simulation training, the results made after 4-hour, 24-hour, and 48-hour simulation training were analyzed in terms of the apprentice officers' subjective evaluation (ship-handling ability, navigation ability), cross track distance, and clearance distance with other vessels.

#### (1) Subjective evaluation of apprentice officers

At the end of each simulation, apprentice officers' subjective evaluation was performed to find their ship-handling ability and navigation ability. These apprentice officers played the role of a captain or an officer. Ship-handling ability refers to their ability to change course in a proper direction, and the navigation ability represents their ability of overall navigation skills in simulation. Fig. 4 shows graphs of the average self-assessed points of apprentice officers' derived from their subjective evaluations on their ship-handling ability and navigation ability.

As shown in Fig. 4, as the simulation training progresses, both ship-handling ability and navigation ability subjectively evaluated by apprentice officers gradually improved. In order to find out whether these changes were significant, a paired sample t-test was conducted. The result of this is presented in Table 4.

A paired sample t-test was also conducted for each simulation. As a result, ship-handling ability and navigation ability were shown to significantly differ between simulation 1 (after 4-hour training) and simulation 2 (after 24-hour training) with a p-value of .000. The mean increased between simulation 2 (after 24-hour training) and simulation 3 (after 48-hour training), but it was not significantly different due to having a p-value of more than 0.05. Given the analysis result, it was concluded that apprentice officer would subjectively feel improvements in their abilities after 52-hours of simulation training, and in particular that they would subjectively feel great improvements at the beginning of training.

#### (2) Cross track distance

The Kanmon Straits features a narrow width and a large passage amount so that there is a high risk of accidents. Therefore, leading lights are used to induce a vessel into a safe route. In this simulation, a vessel was supposed to steer a course of 141 degrees along the leading light Oseto No.1. To evaluate the apprentice officers' ability to keep course line, cross track distance was measured. For a deviation to the starboard, a ' ' sign was used; for a deviation to the port, a '+' sign was used. Fig. 5 illustrates the graphs of the average cross track distance for each simulation.

As shown in Fig. 5, as simulation training progressed, the average cross track distance reduced to 224.23 m, 197.36 m, and 90.09 m, corresponding to the order of the simulations. In particular, the difference between simulation 1 (after 4-hour training) and simulation 3 (after 48-hour training) was 134.14 m. Therefore, their ability to keep course line was greatly improved.

Table 5 shows the results of paired sample t-test which was conducted to find out if there was any significant differentiation in the average cross track distance between each simulation.

Significant variation was observed between simulation 1 (after 4-hour training) and simulation 2 (after 24-hour training), and between simulation 2 (after 24-hour training) and simulation 3 (after 48-hour training) with a p-value of .000. The standard deviations were also reduced. Therefore, as training progressed apprentice officers who had deviated from a route were seen to keep a route within a certain range.

#### (3) Clearance distance with other vessels

In order to evaluate safe passage with other vessels, the average clearance distance between an officer’s own vessel and other vessels was measured. In this case, to exclude distant vessels, only vessels passing within 1,200 m were surveyed. This value is calculated by setting the maximum distance of the Kanmon Strait to about 1000 m and the clearance to about 200 m.

Fig. 6 illustrates the average clearance distance with other vessels in each simulation.

The average clearance distance with other vessels was 792.15 m, 848.82 m, and 854.60 m, respectively. As simulation training progressed, navigation was made with a wider distance from other vessels. In particular, the clearance distance with other vessels between simulation 1 (after 4-hour training) and simulation 2 (after 24-hour training) increased by 56.67 m, and the clearance distance with other vessels between simulation 2 (after 24-hour training) and simulation 3 (after 48-hour training) slightly increased by 5.78 m. To find out if these changes were significant, a paired sample t-test was conducted on each simulation. The result is presented in Table 6.

This revealed a significant difference between simulation 1 (after 4-hour training) and simulation 2 (after 24-hour training) with a p-value of .000. No significant difference was found between simulation 2 (after 24-hour training) and simulation 3 (after 48-hour training) with a p-value of .877. In simulation 1 (after 4-hour training) and simulation 3 (after 48-hour training) with a p-value of .000, as simulation training progressed, apprentice officers' ability of safe passage with other vessels was improved. In the t-test, the ability had greatly been improved following the first 24 hours of training.

### 3.3 Comprehensive evaluation

To find the effectiveness of the simulation training, the apprentice officers' subjective evaluation, cross track distance, and clearance distance with other vessels were all analyzed. Fig. 7 comprehensively presents the results from the analysis on each evaluation item in graph form.

As shown in Fig. 7, the effect of training during a certain period of time was large in all areas except cross track distance, but the effect diminished after a certain period of time. This can be explained in comparison with the learning curve of pedagogy. This learning curve is a graphical representation of one's actions that change as a result of practice. At the beginning of learning, this curve is changed to a large degree. After a certain period, the learning curve remains unchanged in a sort of standstill state. Still later, the curve is also highly changed from this ‘recovery period’ (Education Research Institute Seoul National University, 2011).

To shorten the standstill state period, it is necessary to change a scenario or a training method after 24 hours. In case of cross track distance, the effect was small at the beginning of the training. This is because the apprentice officers' lack of steering ability and navigation ability lead to excessive collision avoidance action. Therefore, it is necessary to emphasize the importance of course-keeping during the beginning of training.

## 4 Conclusion

In accordance with legal standards, simulation training has been enforced in order to develop practical skills for the prevention of accidents by predetermining the risks present in a special marine environment. This study attempts to evaluate the effectiveness of simulation training, and to draw something to improve the training. The results are presented as follows:

• (1) In order to draw the main items for evaluation of the effectiveness of simulation training, legal standards of simulation training, previous studies, and the current status of MET institutions were surveyed. As a result, CPA/TCPA related items for safe passage with other vessels, and the item of position fixing, were found to be the primary evaluation items.

• (2) In order to evaluate the effectiveness of simulation training, simulations were performed in the same scenario after 4-hour training, 24-hour training, and 48-hour training periods in the 64-hour simulation training program. Cross track distance, clearance distance with other vessels, and apprentice officers' subjective evaluation (ship-handling ability, navigation ability) were used as evaluation items. As a result, it was found that not only were apprentice officers' ship-handling ability and navigation abilities improved, but their ability for course-keeping and their ability of safe passage with other vessels were improved as well.

• (3) Given the comprehensive analysis on the effectiveness of simulation training, the learning effect was still in standstill state after 24-hours of training. In order to shorten the state period and maximize the effect of training, it is necessary to change a scenario or training method. Given that the ability to keep course line was less improved than other abilities at the beginning of training, it is necessary to emphasize the importance placed on course-keeping.

Based on one scenario, this study used officers' subjective evaluation, cross track distance, and clearance distance with other vessels in order to evaluate the effectiveness of simulation training. In the future, more scenarios with various waterways and ship types, and more evaluation items including measurements of vessel speed and position fixing will be incorporated in order to come up with the criteria necessary to verify the objective effect of simulation training.

## Figure

Apprentice officer’s self evaluation score sheet.

Cross Track Distance.

Waterways and ship type.

Average comparison of navigator’s evaluation.

Average comparison of cross track distance.

Average comparison of clearance distance.

Comprehensive evaluation result.

## Table

Comparison of legal standards for simulation training

Comparison of evaluation items of earlier studies

Comparison of evaluation for MET institution

T-test result of apprentice officer’s evaluation

T-test result of cross track distance

T-test result of clearance distance

## Reference

1. Education Research Institute Seoul National University (2011) Glossary of education terms, http://terms.naver.com/list.nhn?cid=42126&categoryId=42126
2. S. Hiroaki , O. Mirai , U. Hideo , S. Masatoshi (2010) Marine Traffic Analysis of Three Major Bays Ship Handling Difficulty of Ship Equipped with AIS., Japan Institute of Navigation, Vol.Vol. 123 ; pp.13-19
3. IMO (2010) Manila Amendments to the Seafarer’s Training, Certification and Watchkeeping (STCW) Code 2010 Manila Amendments,, http://krcon.krs.co.kr/
4. IMO (2017) IMO Model Course 1.07 Radar Navigation at Operational Level.,
5. T. Karlsson (2011) The importance of structured briefings & debriefings for objective evaluation of ARPA simulator training, Ph.D. Goteborg: Chalmers university, pp. 31-41.,
6. H. Kobayashi (2005) Use of Simulators in Assessment, Learning, and Teaching of Mariners, WMU Journal of maritime affairs,, Vol.4 (1) ; pp.57-75
7. Korea Ministry of Government Legislation (2017) Ship Personnel Act, http://www.law.go.kr
8. Korea Ministry of Government Legislation (2017) Enforcement Decree of the Ship Personnel Act, http://www.law.go.kr
9. S. Nahlinder (2010) Flight Simulator Training : Assessing the potential. Linkoping: Linkoping studies in Science and Technology, Dissertation No. 1250.,
10. Y.S. Park (2016) A Study on the Standardization of Education Modules for ARPA/Radar Simulation., Journal of The Korean Society of Marine Environment & Safety, Vol.22 (6) ; pp.631-638
11. D.W. Shin , Y.S. Park , D.H. Kim (2017) A Study on the effect of ARPA/Radar Simulation Training., Korean Society of Marine Environment & Safety, Vol.23 (3) ; pp.294-300
12. UK Hydrographic office (2013) ADMIRALTY Sailing Directions Japan Pilot., UKHO, Vol.Vol. 3 ; pp.56