IT functions like an electrical circuit, in which all the switches in a line must be off before current can flow, explains Tommy Gardner, chief technology officer at HP Federal. "Often, several failures must occur in succession for ... a dramatic loss of data to occur," he says.
A cascading series of failures can occur when important preventative steps are overlooked. “For example, continuity of operations may keep the IT team focused on a perceived threat,” says Gardner. “But if the team’s time and attention are diverted elsewhere, routine updates may not be completed.”
Often, errors can be traced back to poor system design. “If a system isn’t designed to properly handle errors in subsystems, then one error can cause a failure that can trigger a whole series of failures,” explains Tom Chisholm, principal engineer for learning solutions at software company Perforce.
“For example, if a web application doesn’t handle database connection errors correctly, a single error can cause the entire application to fail if other parts of the system aren’t resilient to that failure,” he explains. “Badly written database queries can also trigger a chain of failures. This can lead to deadlocks in the database, which in turn can el salvador mobile database to cascading failures in the front end.”
Preventive measures
Gardner believes that the best way to prevent a potential disaster is to create proactive defenses: “IT teams need to think about multiple problems at once, understand the limits and limitations of their system, and create structured protocols that are followed on an ongoing basis.” He advises training team members on IT best practices to prevent small user errors from escalating into bigger problems. And conducting independent software testing and scheduling regular code updates to ensure that both open source and proprietary software are free of vulnerabilities.
Chisholm also suggests testing software thoroughly before it goes live. “Find single points of failure in a controlled environment before you discover them in production,” he recommends.
Meanwhile, the best way to prevent or recover from cascading failures is to build fault tolerance into each subsystem and periodically test it for fault tolerance, Chisholm says.
He also recommends using monitoring tools to keep an eye on the health and performance of your system. “Be proactive in addressing any issues that arise,” Chisholm says. “Also, regularly reviewing logs and metrics can help you identify potential issues before they become serious.”
Break the chain