Chapter 10. Incident Response9410.5. Restoring and Recovering ResourcesWhile an incident response is in progress, the CERT team should be investigating while workingtoward data and system recovery. Unfortunately, it is the nature of the breach which dictates thecourse of recovery. Having backups or offline, redundant systems during this time is invaluable.To recover systems, the response team must bring any downed systems or applications back online,such as authentication servers, database servers, and any other production resources.Having production backup hardware ready for use is highly recommended, such as extra hard drives,hot-spare servers, and the like. Ready-made systems should have all production software loaded andready for immediate use. Only the most recent and pertinent data needs to be imported. This ready-made system should be kept isolated from the rest of the network. If a compromise occurs and thebackup system is a part of the network, then the purpose of having a backup system is defeated.System recovery can be a tedious process. In many instances there are two courses of action fromwhich to choose. Administrators can perform a clean re-installation of the operating system on eachaffected system followed by restoration of all applications and data. Alternatively, administrators canpatch the offending vulnerabilities and bring the affected system back into production.10.5.1. Reinstalling the SystemPerforming a clean re-installation ensures that the affected system is cleansed of any trojans,backdoors, or malicious processes. Re-installation also ensures that any data (if restored from atrusted backup source) is cleared of any malicious modifications. The drawback to total systemrecovery is the time involved in rebuilding systems from scratch. However, if there is a hot backupsystem available for use where the only action to take is to dump the most recent data, systemdowntime is greatly reduced.10.5.2. Patching the SystemPatching affected systems is a more dangerous course of action and should be undertaken with greatcaution. The problem with patching a system instead of reinstalling is determining whether or not agiven system is cleansed of trojans, security holes, and corrupted data. Most rootkits (programs orpackages that a cracker uses to gain root access to a system), trojan system commands, and shellenvironments are designed to hide malicious activities from cursory audits. If the patch approach istaken, only trusted binaries should be used (for example, from a mounted, read-only CD-ROM).10.6. Reporting the IncidentThe last part of the incident response plan is reporting the incident. The security team should takenotes as the response is happening and report all issues to organizations such as local and federalauthorities or multi-vendor software vulnerability portals, such as the Common Vulnerabilities andExposures site (CVE) at http://cve.mitre.org/3. Depending on the type of legal counsel an enterpriseemploys, a post-mortem analysis may be required. Even if it is not a functional requirement to acompromise analysis, a post-mortem can prove invaluable in helping to learn how a cracker thinks andhow the systems are structured so that future compromises can be prevented.3 http://cve.mitre.org