Harnessing technolgy and AI for seafarer safety

Credit: Shutterstock

Lloyd’s Register (LR) is championing the development of technologies that can detect the causal factors behind human error to improve safety

The established consensus is that 90% of ship accidents are caused by human error. In the broader sense, the remaining 10% too can be attributed to some human or other, especially regarding incidents caused by malfunctioning equipment. To combat this, the LR Foundation has embarked on a project to “assess the psychological and emotional health of crew” through a variety of new tools, making use of some of the latest technologies.

The Mission To Seafarers’ 2019 Seafarer Happiness Index, encompassing feedback from about 2,000 ship crews, showed that ‘Overall seafarer happiness’ had fallen from 6.31 out of 10 in the first quarter (Q1) of 2019 to 6.27 in Q2. Comprising questions around seafarers’ access to shore leave, wages, workload, family contact, and sense of community, it formed a new low for the survey.

More data, however, is needed to form a comprehensive picture, as the initiative only began in 2018. The survey question ‘How happy generally when at sea?’ generated a positive response, coming in at 6.21 in Q2, from 6.03 in Q1. Nevertheless, the averages were brought down by answers to questions ‘How happy with your workload?’ and ‘How happy about interaction with other crew on board?’. The former scored a low 5.73 in Q2, down from 5.99 in Q1, while the latter fell from the Q1 score of 6.95 to 6.85 in Q2.

Additional comments highlighted issues including female respondents reporting that they had experienced sexual harassment and even assault at the hands of their male colleagues; salaries were understood to be decreasing, particularly among junior officers; and others complained about shore leave, with one proclaiming it was “dead” and “never will it be seen again”.

It is no secret that the life of a seafarer is often stressful and challenging. While links between seafarer competence and a reduced likelihood of accidents are understood, those between seafarers’ mental health and accident risk are fuzzier, therefore, more challenging to address. But this traditionally ‘soft’ problem can have very hard consequences.

Fear, uncertainty, and doubt

In 2012, Costa Concordia cruise vessel was grounded on a rock below the surface off Isola del Giglio, Italy, then capsized, and partially sank. The accident claimed 32 lives, thanks in large part to the slow evacuation of the vessel, which took six hours rather than the legally mandated 30 minutes.

Ship’s captain Francesco Schettino, now serving a 16-year jail sentence, has been broadly condemned as a ‘coward’. However some, including seafarer rights campaigner Michael Lloyd, claimed that his actions better fit an altogether different ailment. Detailing the findings of his 148-page report on the disaster at a March seminar in London, Lloyd said, “The evidence overwhelmingly suggests that Captain Schettino was suffering from traumatic shock syndrome and was in a mental state of denial, and therefore, obviously incapable of taking any decisive action.”

Lloyd paints a picture of a bridge crew untrained for and unable to respond in the eventuality of a compromise in the command structure. “This was supported by video evidence of the chaos on the bridge,” the master mariner of 30 years said. “As this was recognised by two officers on the bridge, then this should also have been recognised by the staff captain who was also present. Therefore, it was his duty to assume command while the captain was in this state. The question must be then: why didn’t he?”

Lloyd said that training would be required in the future if crew were to respond more effectively to such an incident; the threat of a lapse in seafarer mental health should be guarded against with more effective training of fellow crew; if this had been the case, much of the tragedy might have been averted.

The LR project posits that this gap might be filled not by human empathy, instead by an algorithmically programmed counterpart. How might accidents be avoided in the future, if technology could detect a lapse in a crew member’s judgement instead?

Seeing into the soul

Based in Austin, Texas, US company Senseye develops software to assess emotional and psychological states, through incredibly sophisticated camera technology capable of reading and mapping tiny variations in the muscles of the human iris.

“If you have ever heard that the ‘eye is the window to the soul’ – that happens to be true,” said David Zakariaie, founder and CEO of Senseye, which is currently developing a shipboard system with the sponsorship of Pacific International Lines (PIL). “If the human brain is the world’s best command and control mechanism, we essentially need to build a wireless way to access the brain. That is essentially what we have built.

“Each iris is comprised of about 5,000 muscle fibres, linked, through the nervous system, to a different section of the brain. So, by using a video camera to look at the movements of these muscle fibres, we are able to look at most of the activity of the brain with the functionality of an MRI [magnetic resonance imaging], but without a giant USD5,000 machine you have to stick somebody inside of, we are able to use very small off-the-shelf cameras that cost about USD500.”

“Each muscle fibre is making its own movements,” Zakariaie added. “So, we are basically measuring the same thing scientists have for 50–60 years, instead of measuring [pupil dilation], we are actually measuring the small muscle fibres around it, which means we have 5,000 variables instead of one.

“One of the applications is an automated operational risk management platform we started to build in November 2019, after we won a challenge with LR. It is basically a camera and a screen – you go in front of it, it does a one-minute scan of your iris, and it gives you a red light or green light if you are safe or not. It measures three things: alcohol or drugs; fatigue; and what we call emotional and psychological risk factors, stress, depression, and schizophrenia. A green light means you are safe; if you get a red light it will tell you why. If it is alcohol or drugs, that is one set of consequences there; if you are tired then you will just be on a low-risk job for that shift.”

The tool is yet to be used to make a first-time judgement on a completely novel party, but it can compare future readings with a ‘baseline’ model for each seafarer. Over time, the technology will become more nuanced, and it will be able to detect subtler changes in mood and mental state.

Standing up straight

Another technology, Sensing Feeling, was developed as a way of reading human emotional states using CCTV camera footage, taking account of movements, habits, body language, and posture. In maritime, as part of a pilot study with Scorpio, it would theoretically be able to determine the emotional state of seafarers, or diagnose evidence of drug or alcohol abuse, to evaluate crews’ fitness for duty.

“We make advanced human behaviour and emotion-sensing products for business,” said Jag Minhas, CEO and founder. “We have designed them to be used in real-world spaces and real-world conditions, passively and unobtrusively. We work with retailers; trying to better understand how humans interact with lecturers in a university scenario; through to the safety application where we aim to better understand safety and how to detect high-risk behaviours around safety-critical assets in the transport and rail infrastructures. We have a strong focus on privacy and ethics – we want to be known as the company that does this in the most ethically correct way.

“In this initiative with the [LR] foundation, we are applying our technology to better understand decision-making on a ship through sensing, with the objective to ultimately reduce accidents in the industry, to ultimately analyse and predict accidents before they occur, and to provide a richer set of data for things like cause analysis.” Through its collaboration with Scorpio, it will incorporate the technology in the bridge on a vessel that will undertake several voyages, as well as in a training simulator. Data will be collected based on human behaviour and ancillary data generated by systems on the vessel.

Sensing Feeling’s main aim is to look for patterns of behaviour correlated with the behaviour on a vessel, and ultimately construct a risk model. The data captured will focus on how crew move, how they dwell within the bridge environment; looking for expressions of behaviour and emotion through body posture and gesture, motion of the body, animation, and arousal. Areas of particular interest are signs of fatigue, and stress, but also to collect data from the vessel itself, for example, weather, time of day, other contextual information about cargo, crew, operational history, and berthing or navigation procedures.

“The idea following this pilot is: can we construct a real-time voyage risk index, and what that can potentially be used for – interventions by shore-based staff; delivering better learning and development within seafarer training facilities, and could it potentially be used to influence human factors such as bridge and vessel design, or perhaps the way in which vessels navigate. And, potentially, using this technology in underwriting,” said Minhas.

Shouldering the blame

With the phenomenal innovation and pioneering technology enthusiasm on display – Zakariaie, for example, is just 22 years old – it is easy to forget that success in the maritime industry will depend not only on the bleeding-edge algorithms on which these technologies rely, but also the way shipping companies will react to this new information.

Asked about the developments at the LR Foundation Safety Accelerator, Katie Higginbottom of International Transport Workers’ Federation (ITF) Seafarer’s Trust said, “Technology as a tool to help individuals assess their physical state and make better judgements about appropriate action can be a positive thing.”

However, she cautioned that “using technology to make determinations about human states of mind or conditions is dangerous, with potential for manipulation and error that can be difficult to challenge”. But on a vessel with sparse manning and six-(or four) hour-on, six-hour-off rotation, as many have, crews might struggle to find an individual on their vessel unaffected by elements of stress or fatigue. In an industry with its own specially designed Bridge Navigation Watch Alarm System (BNWAS) tool – designed, every few minutes, to prod dozing watch officers awake by sounding alarms – it would be naive to imagine that nobody has ever considered putting a worn-out seafarer on the bridge.

ITF colleague Fabrizio Barcellona, head of actions unit, added, “After reading the Lloyd’s Register report, it is not clear what is the suggested support or rectification of the issues after technology discovers the problem. I would say that this technology could potentially highlight issues of which we all are aware – fatigue and stress – and promote directly and indirectly the supporters of extreme automation.”

Alcohol and drug tests

It is widely accepted that heightened emotional states, or high levels of stress or fatigue, can impair job performance to a similar extent as alcohol and drugs. On land, it is taken for granted that employees must consent to drug and alcohol checks before going on shift in many industries. In the absence of a human resources department at sea, replaced with a command structure with little room for taking the initiative – as the Costa Concordia disaster showed – then perhaps it is neglectful not to check for these as well?

“The overall goal of the accelerator is to understand how technology can be used to improve safety,” said Maurizio Pilu, LR’s vice-president for digital transformation. “In the case of Senseye, the challenge that PIL had was the additional methods for detecting fatigue. Seafarers who are drunk, or on drugs, and so on, are not as effective. This new technology has already tried with fighter jet pilots, and it’s far more accurate than existing methods.

“The privacy part is absolutely front and centre – this is part of the reason why it is a proof of concept,” Pilu assured SAS. “It is being trialled with seafarers who consented. This is part of the idea, to understand trade-offs between saving lives, privacy, and the cost of the technology.”

If seafarers are diagnosed by Senseye or Sensing Feeling as fatigued or stressed out, but are put on watch regardless, who will take responsibility if disaster strikes? History shows – in many cases beyond that of Costa Concordia – that it will be the crew, and not the shipowner, who take the fall. Perhaps these new technologies, developed with guidance and funding from LR Foundation, will change the scenario.