Network Hardware Issues Affecting the IT systems of Public Healthcare Institutions

NOTICE PAPER NO. 1314
NOTICE OF QUESTION FOR ORAL ANSWER

Name and Constituency of Member of ParliamentDr Tan Wu MengMP for Jurong GRC

Question No. 3284

To ask the Minister for Health in light of network hardware issues affecting the IT systems of public healthcare institutions on 27 August 2022 (a) how many institutions, apps and IT-dependent services are affected respectively; (b) how many patients are existing inpatients, patients admitted that day and outpatients scheduled to be seen that day respectively; (c) how many of such patients have experienced diversion or delayed care provision; and (d) how are the affected patients and healthcare workers supported.

NOTICE PAPER NO. 1316
NOTICE OF QUESTION FOR ORAL ANSWER
FOR THE SITTING OF PARLIAMENT ON 12 SEPTEMBER 2022

Name and Constituency of Member of Parliament
Dr Tan Wu Meng
MP for Jurong GRC

Question No. 3287

To ask the Minister for Health in respect of network hardware issues affecting the IT systems of public healthcare institutions on 27 August 2022 (a) whether a cyber attack has been ruled out; (b) what are the root causes of the outage; (c) whether existing redundancy measures are sufficient to maintain provision of healthcare services; (d) what lessons have been learned; and (e) what is being done to improve resilience against such incidents.

NOTICE PAPER NO. 1316
NOTICE OF QUESTION FOR ORAL ANSWER

Name and Constituency of Member of ParliamentDr Tan Wu MengMP for Jurong GRCQuestion No. 3288

To ask the Minister for Health (a) what are the budget, headcount and deliverables of the Integrated Health Information Systems (IHiS); (b) how do these benchmark against best-in-class international healthcare institutions and top technology-sector firms; and (c) what is being done to strengthen the capabilities of IHiS systems, processes and staff.

NOTICE PAPER NO. 1328
NOTICE OF QUESTION FOR ORAL ANSWER

Name and Constituency of Member of Parliament

Dr Wan Rizal
MP for Jurong GRC
Question No. 3337

To ask the Minister for Health with regard to the network hardware issues that caused a system outage at some polyclinics on 27 August 2022, what are the safeguards in place or that will be implemented to prevent disruptions in the future.

NOTICE PAPER NO. 1332
NOTICE OF QUESTION FOR ORAL ANSWER

Name and Constituency of Member of Parliament

Ms He Ting Ru

MP for Sengkang GRC

Question No. 3361

To ask the Minister for Health with regard to the system outage that occurred on 27 August 2022 at some polyclinics (a) how many polyclinics are affected; (b) whether any other institutions or facilities are affected and, if so, how many; (c) whether any systems apart from the patients appointment system are affected; (d) how many patients are affected; and (e) whether an update can be provided into the nature of the outage and the remedial measures taken.

Answer

Members have asked about the cause and impact of the IT systems outage at public healthcare institutions on 27^th August 2022. There was a related outage that occurred in the morning of 5^th September 2022, which I will also address.

From 7am on 27^th August 2022, the public healthcare monitoring systems detected IT network connectivity failures. The faults were rectified and the systems were restored by 10.45am on the same day.   In total, 26 IT applications were affected, including the electronic medical records, appointment, pharmacy and laboratory systems. 17 public healthcare institutions, including the acute hospitals, community hospitals and specialist outpatient clinics, and all the polyclinics were affected.

On 5^th September at about 10am, another fault occurred in the IT infrastructure. Some functionality was restored from 1pm on the same day, full functionality was restored by 6pm the next day. This outage affected eight public healthcare institutions and two out of three polyclinic groups. Due to the nature of the outage, the time to recovery of the system was longer, hence operations and services were switched to their back up infrastructure.

Both incidences caused a significant impact on operations. On both 27^th August and 5^th September, our affected public healthcare institutions activated downtime procedures and business continuity plans to keep operations running using alternative systems and in some cases manual documentation. These business continuity plans are exercised regularly, and staff were able to switch processes to sustain operations during the outage.  But they had to work doubly hard to keep healthcare operations running smoothly.

Patients experienced longer wait-times ranging up to 1 hour at the affected institutions. Some had their outpatient appointments rescheduled. There were delays in dispensing medications to patients.

Fortunately, there was no compromise to urgent care services across the institutions during the IT disruption. Nobody was turned away from the emergency departments, or denied urgent care. I would like to express my thanks to all the teams, nurses, admin staff, and clinicians for keeping the services running for our public healthcare system, and for keeping our patients safe. We are investigating into the incident with the manufacturer of the IT hardware to rectify any weaknesses in the system.  This is what we have found out so far.

The main cause of the outages were failures of hardware devices in our data centres.  Public healthcare IT infrastructure is housed in more than one data centres for resilience and redundancy. At each data centre, there are a few firewall zones.  Each firewall zone consists of multiple nodes – hardware devices which operate in tandem, so if one fails, the load of data traffic is managed by the other nodes in the cluster, so that service operation is uninterrupted.  This system had generally been working well until the recent outages.

On August 25^th a node failed, and the system resilience features kicked in and services were maintained. The same thing happened on the 26^th of August, when another node failed, but systems and services continued functioning.

On the 27^th of August, when the engineers tried to restore the two failed nodes, under the supervision of the manufacturer and following the manufacturer’s procedures, which had been used successfully in the past, the operation failed. It is this failed operation that caused the cluster of firewall nodes to malfunction and subsequently caused the outage. The engineers worked to reset the systems to the prior state without the function of the two affected nodes, and service was progressively restored.

The failure of the nodes was caused by bugs in the firmware of the devices.  They have since been identified by the manufacturer, CISCO, and the devices have been patched.

The outage on the 5^th of September was caused by the simultaneous failure of two further nodes, again from the same manufacturer, and of the same model. The way in which this failure occurred was noted to be different from the previous incident, it was assessed that it would take longer to restore operations, and hence the decision was made to switch operations to the backup systems. The root cause of why these two nodes failed is still under investigation.

There was a suggestion in one of questions that the failures may be due to the lack of manpower of IHiS.  IHiS has a headcount of 3,500, they have a lot to do and will always welcome more manpower, but lack of manpower is not the cause of the failures.

Cybersecurity specialists are monitoring our network and systems for threats in our public healthcare network round the clock.  When the network problems occurred, IHiS initiated an investigation and also alerted the Cyber Security Agency.  Based on the investigations thus far, there are no indications of compromise to the affected systems.

All the firewall hardware involved in the incidents are from the same established device manufacturer.  Fixes for some of the issues have been made available and have been deployed. For the others we continue to work with the manufacturer.

In the meantime, we have increased capacity in the network for more operational buffer to increase resilience.

I thank the members for their questions. We will continue to review our system design and architecture, and invest in capabilities and readiness to reduce disruptions. Disruptions like this can occur again in the future, and we continue to have ready backup systems, downtime procedures and manual processes. Once again Mr Speaker allow me to express my thanks to all the personnel of our public healthcare system who kept our patients safe and the services running and for the members of the public who were affected, their patience and understanding as we tried to cope and mitigate the circumstances as best as we could.