1208 | E-Series TeraScale Debugging and Diagnosticsw w w . d e l l . c o m | s u p p o r t . d e l l . c o m In a dual RPM system, the two RPMs send synchronization messages via inter-RPM communication(IRC). As described in the High Availability chapter, an RPM failover can be triggered by loss of theheartbeat (similar to a keepalive message) between the two RPMs. FTOS reports this condition via syslogmessages, as follows:FTOS automatically saves critical information, about the IRC failure, to NVRAM. Use the same three-stepprocedure to capture this file for analysis by Dell Force10.FTOS actually saves up to three persistent files depending upon the type of failure. When reporting anRPM failover triggered by a loss of the IPC or IRC heartbeats, look for failure records in the followingdirectories:— Application or kernel core dump RP in the CORE_DUMP_DIR— CP trace log file (look for a filename with the phrase “failure_trace”) in theTRACE_LOG_DIR— RP and/or CP sysinfo file in the CRASH_LOG_DIR, as explained aboveDebug commandsFTOS supports an extensive suite of debug commands for troubleshooting specific problems whileworking with Dell Force10 technical support staff. All debug commands are entered in privileged EXECmode. See the FTOS Command Reference for details.Hardware watchdog timerThe hardware watchdog command automatically reboots an FTOS switch/router with a singleRPM that is unresponsive. This is a last resort mechanism intended to prevent a manual powercycle.Message 7 RPM heartbeat report20:29:07: %RPM1-S:CP %IRC-4-IRC_WARNLINKDN: Keepalive packet 7 to peer RPM is lost20:29:07: %RPM1-S:CP %IRC-4-IRC_COMMDOWN: Link to peer RPM is down%RPM1-S:CP %RAM-4-MISSING_HB: Heartbeat lost with peer RPM. Auto failover on heart beat lost.%RPM1-S:CP %RAM-6-ELECTION_ROLE: RPM1 is transitioning to Primary RPM.Command Descriptionhardware watchdog Enable the hardware watchdog mechanism.
