> ## Documentation Index > Fetch the complete documentation index at: https://docs.embedder.com/llms.txt > Use this file to discover all available pages before exploring further. # Debugging and root-cause analysis > Reproduce a fault, separate hardware and firmware causes, and verify a correction with measurements and regression checks. Use Embedder to connect a failure in the application to evidence in source code, device documentation, board connections, and measured behavior. Root-cause analysis (RCA) should explain the mechanism that produced the failure and show why the correction addresses it. Embedder can inspect code and schematics, collect supported debugger and instrument observations, change firmware, and run repeatable checks. It can also help identify a wiring, power, or component problem and propose a correction. Physical inspection, rewiring, and component replacement still require someone at the bench. ## Define the failure and preserve the evidence Record the board and fitted-part revisions, firmware artifact and build settings, power setup, workload, expected result, and exact failure. Include reproduction steps and how often the fault occurs. If the board has never worked, start with [Rapid bring-up](/solutions/rapid-bring-up). ```text theme={"system"} The sensor stops responding after the radio starts. It happens in about one of ten runs on board revision B with the current release build. Read the existing serial history and inspect the sensor and radio startup code. Preserve the current target state. List the evidence we have and the next observation needed before changing firmware or resetting the board. ``` Start with captured logs and any existing debugger session. A new UART connection can change DTR/RTS; a normal debugger connection can program, reset, or halt the target. Review [UART connection behavior](/debug-mode/uart) and [debugger attachment precautions](/debug-mode/gdb#attach-to-a-running-target) before opening a new connection. Specify whether a halted fault must remain halted or a running target may be interrupted. A fresh [xSCOPE connection](/debug-mode/xscope) can launch firmware, and [RTT through a bridge](/debug-mode/rtt#use-rtt-through-a-bridge) requires a prior flash in the current session. Use captured history or confirmed reuse of an existing session when preserving a fault; do not reflash just to enable an observation path. Keep the matching debug artifact before rebuilding. Symbols from another build can give a misleading stack or source location. ## Prepare the investigation 1. Select the intended project with `/project` and confirm the active platform and peripherals. Provide the exact datasheet, errata, and [schematic revision](/core-concepts/schematics) for the populated board. 2. Check the build, flash, and observation commands in [EMBEDDER.md](/core-concepts/embedder-md). Identify the actual board, probe, port, or selected hardware bridge. 3. In VS Code, run **Embedder: Open Embedder Monitor**. Use its logs and capture views beside the source; the terminal also supports investigations and saved results. 4. Ask Embedder to check the needed instruments and complete their setup. Follow the [tool-specific guide](/debug-mode/overview#open-your-tool%E2%80%99s-guide) for channels, voltage limits, trigger support, and host dependencies. 5. Run `/debug` for debugger state inspection or bounded J-Trace instruction/coverage captures. Ordinary instrument measurements and hardware scripts can also run in Act mode. Ask for an investigation record such as `.embedder/hardware/cases/sensor-timeout/CASE.md`. Keep the symptom, reproduction, candidate causes, observations, and next test there, with paths to raw captures. Update it after each test instead of treating the first explanation as settled. ## Choose a test that separates the causes A missing response can result from firmware, a physical connection, the device's operating state, or the observation setup itself. First confirm the port, probe, artifact, decoder settings, and capture window. A missing or malformed capture is an incomplete observation. ```text theme={"system"} Compare the failing path with the active schematic and device requirements. For each plausible cause, state what evidence supports it, what result would contradict it, and the smallest test that distinguishes it from the alternatives. Start with observations that preserve the current failure. Do not patch several possible causes at once. ``` Use the observation suited to the question: * **Source, datasheets, and schematics:** check intended pin mapping, clocks, reset order, address selection, register semantics, and driver behavior. A schematic describes the design; it does not prove that the assembled board matches it. * **Serial logs and GDB:** inspect startup order, error paths, stack, variables, and available task information. Debugger and firmware support determine which state is available. * **Logic capture:** check actual digital transactions, address, acknowledgments, framing, and edge timing. Decoded traffic alone does not establish signal-voltage margins. * **Scope and power measurements:** check rail voltage, reset or enable signals, rise time, ringing, current, and workload transitions. Use a scope for short voltage transients; averaged power statistics can hide them. Set a bounded capture duration, physical channel map, trigger, and expected result. If two observations must describe the same event, use a common marker or trigger and verify their timing relationship. Independent logs and instrument captures do not necessarily share a clock. See [Combined observations](/debug-mode/combined-workflows). ## Work through hardware and software hypotheses Ask Embedder to compare the driver's initialization and transaction sequence with the exact device datasheet and schematic. Then capture the bus using a [supported logic analyzer](/debug-mode/logic-analyzer). ```text theme={"system"} Capture the failing I2C transaction using the channel map recorded for this bench. Save raw and decoded results. Compare the address, read/write direction, acknowledgments, and transaction order with the driver and fitted sensor. Report the first point where the observed sequence differs from the expected one. ``` If the expected pins never toggle, check whether the code reaches the transfer, the peripheral clock and pin configuration, and the routed connection. If the address or command is wrong on the wire, trace how the driver constructs it. If the transaction is as intended but the device does not acknowledge, check supply, reset/enable state, address straps, and signal levels before blaming the parser. Use an [oscilloscope](/debug-mode/oscilloscope) to measure the relevant signal at the device and compare rise time and levels with its requirements. Ask for a physical check when the suspected problem is a missing pull-up, open connection, or incorrect fitted part. Specify the board's limits before anyone changes wiring or components. Correct a demonstrated address, pin, or sequence error in firmware. For a demonstrated electrical problem, document the board correction and remeasure afterward. Reducing bus speed may improve a marginal connection, but that result alone does not establish the original cause or validate the required bus speed. Use a scope to capture VDD at the target and a known workload marker in the same acquisition. Choose ranges and a trigger from the board's supply and reset requirements, and arm before reproducing the load change. ```text theme={"system"} Prepare a bounded scope capture of target VDD and the recorded load-start marker. After I reproduce the reset, save that acquisition. Before any new reset or programming, check whether the matching reset-cause register can be read without losing evidence. Compare its flags with the voltage trace and boot log, and identify what remains uncertain. ``` [Register lookup](/debug-mode/registers) supplies definitions when available; a live value requires a separate debugger observation. Check access rules before reading: some status clears on read, and startup code may already have cleared reset flags. Avoid treating a reference reset value as a measured value. A captured voltage drop together with relevant reset flags supports a supply-related hypothesis; determine whether wiring, supply capacity, decoupling, or firmware load sequencing caused it. Stable voltage at the measured point does not rule out an unmeasured transient elsewhere. A watchdog flag identifies the reset mechanism, but the missed service can still result from a driver deadlock, long interrupt, or stalled peripheral. Ask Embedder to inspect the relevant scheduling and driver paths or propose the next electrical measurement. Apply a firmware sequencing fix or have the identified board issue corrected, then repeat the original load transition under the same supply conditions. Preserve the matching artifact, captured output, and any existing stopped state. Use [GDB](/debug-mode/gdb) to inspect the current frame, stack, variables, and available task state, then compare the result with locks, interrupt masking, DMA completion, and bounded waits in the source. ```text theme={"system"} Using the existing stopped debugger session and matching symbols, inspect the blocked transfer and its wait condition. Identify who should complete it and how errors or timeouts release the caller. Check the target-specific register access rules before reading peripheral status. Propose one test that distinguishes a software wait bug from a device that never completed. ``` Stepping and halting change timing. If the failure disappears, use a bounded run with minimal logging or a measured GPIO marker, and check the external transaction while firmware runs normally. Inspect code and host tests for a lost wake-up or unbounded wait; inspect the bus and peripheral state for the missing completion event. Use [trace](/debug-mode/coverage-and-trace) when execution order can answer the question. Compatible firmware instrumentation is required for event traces. Bounded J-Trace captures reset and run the target; streamed ETM repeatedly halts it. Preserve the original fault first and treat a later trace as a separate run, not proof of uninterrupted real-time behavior. Correct the demonstrated synchronization or error-handling defect and add a regression. A timeout can prevent an indefinite hang while leaving the reason for the missing device response unresolved; record that distinction. ## Apply a correction and test the explanation Ask Embedder for a focused correction tied to the evidence. For a larger change, use `/plan` to review affected files and tests, then `/act` to implement it. Run the relevant software checks and target build before flashing the confirmed board. For a hardware correction, have someone inspect or modify the identified connection or component, then record what changed. Embedder can coordinate supported instrument settings and measurements, but a proposed rework is not a completed repair. Change one factor at a time where possible. A comparison with a known-good firmware build on the same board, or the same build on another matching board, can help isolate the problem. Keep wiring, supply, workload, and measurement settings fixed, and record any differences that prevent a clean comparison. ```text theme={"system"} Apply the correction supported by the investigation, run the focused tests, and build the target. Repeat the original reproducer with the same workload and capture setup. Compare failure counts and the relevant bus or voltage measurement before and after. Report whether we verified a root-cause fix, a mitigation, or an unresolved hypothesis. ``` For intermittent faults, record failures and total attempts before and after under comparable conditions. Choose the run count or duration from the observed failure frequency and requirement; one successful run does not establish a fix. Do not reintroduce an unsafe fault just to demonstrate reversibility. ## Close the RCA with a regression Ask Embedder to finish the investigation record with: * The observed failure, affected board/build, trigger, and reproduction frequency. * The causal sequence, supporting source/document references and captures, and alternatives tested or still open. * The firmware or hardware correction actually applied, with before/after results and measurement limits. * A repeatable test, pass criteria, and any remaining mitigation or follow-up work. Use a [hardware script](/debug-mode/hardware-scripts) to repeat the supported bench sequence with bounded waits and cleanup. Keep [software regression tests](/solutions/automated-testing) for parsing, state transitions, timeouts, and recovery behavior. Re-run the original workload after changes to the board, driver, SDK, clocks, or supply setup. If the needed measurement or hardware is unavailable, leave the cause unresolved and state the next test that would distinguish it. A useful RCA can identify a missing observation without inventing a diagnosis.