Did DBS CEOs Apology Appease You?
July 21, 2010
DBS C.E.O. Piyush Gupta Apologized But What Is Your Experience Of DBS
Rate Your Experience With DBS at http://singaporeserviceacademy.com/dbs-bank-ltd/
Dear DBS and POSB Customers
I am writing to personally apologise to you for the inconvenience caused by the sudden disruption in our banking and ATM services from 3 am to 10 am, Monday, 5 July 2010. You have every right to expect uninterrupted services 24/7, 365 days a year from us and I am sorry we have failed you on that count. This is the first time we have experienced a system outage and service disruption of such magnitude, so please allow me to explain what happened.
The outage last week was triggered during a routine repair job on a component within the disk storage subsystem connected to our mainframe. This component was emitting alert messages, indicating that there could be an intermittent problem. As our IT environment is highly resilient and as the banking system was still fully functional, the problem was classified as “low severity”.
A component replacement was scheduled for 3 am, a quiet period, which is standard operating procedure. Unfortunately, while IBM was conducting this routine replacement, under the guidance of their Asia Pacific team, a procedural error inadvertently triggered a malfunction in the multiple layers of systems redundancies, which led to the outage. The IBM Asia Pacific team is the central support unit for all IBM storage systems in the region, irrespective of whether the installation has been outsourced or is being managed in-house.
I am treating this matter with utmost priority and the full scale investigation that we initiated last week is still underway. This investigation is being done with the support of IBM’s labs in the U.S and their engineering teams in Asia. So far, we understand from IBM that an outdated procedure was used to carry out the repair. In short, a procedural error in what was to have been a routine maintenance operation subsequently caused a complete system outage.
We take full responsibility for this incident. The matter is obviously of grave concern to us and we are working closely with IBM to ensure that such lapses do not recur or cause such significant impact. In fact, 12 months ago DBS commenced work on a major two- year program to further strengthen the resiliency of our system and minimize the risk of service disruptions.
Please rest assured that all payments and transactions that were scheduled to be made on 5 July were completed. Nothing was held over and full data integrity was maintained at all times. When the system was down, our priority was to minimise customer inconvenience. We:
- allowed all cheque encashments up to $500. In fact, we encashed almost 1,700 cheques worth over $500K between 8.30am to 10am
- provided frequent situation updates to the media, posted updates on our website and also alerted our staff accordingly so that they could inform their customers
- contacted over 10,000 customers via phone/sms to inform them when services were restored
- kept our branches open for an additional two hours on the evening of 5 July
IBM informed us of the system outage at 3 am and a technical command function comprising DBS and IBM staff was activated by 3.40 am. A restart of the systems was initiated at 5.20 am. Following complications during the machine restart, at 6.30am, we activated our bankwide disaster recovery command centre. By 8.30 am, we knew that our branch and ATM systems could be restored from 10 am onwards and therefore did not need to invoke full disaster recovery measures. All other services were progressively restored through the morning and virtually everything was back on track by lunchtime.
On hindsight, our internal escalation process could have been more immediate. We could also have done more to mobilise broadcast channels to inform customers of the disruption in services first thing in the morning. Once again, please accept my apologies and know that we take full responsibility for this incident. There have been valuable lessons learned and I assure you that this matter will continue to remain a top priority for me. My colleagues and I are doing everything we can to prevent an incident of this scale from happening again.
Yours sincerely
Piyush
Comments
Got something to say?