Debugging and root-cause analysis

Use this workflow when firmware fails on hardware and you need to know why. Embedder connects the failure to evidence in your source code, device documentation, board connections, and measurements. A root-cause analysis (RCA) explains the mechanism that produced the failure and shows why the correction addresses it. Embedder can inspect code and schematics, take debugger and instrument observations on supported hardware, change firmware, and run repeatable checks. It can also help identify a wiring, power, or component problem and propose a fix. Physical inspection, rewiring, and component replacement still need someone at the bench.

Define the failure and preserve the evidence

Record the board and fitted-part revisions, the firmware file and build settings, the power setup, the workload, the expected result, and the exact failure. Include the reproduction steps and how often the fault occurs. If the board has never worked, start with Rapid bring-up.

The sensor stops responding after the radio starts. It happens in about one
of ten runs on board revision B with the current release build. Read the
existing serial history and inspect the sensor and radio startup code.
Preserve the current target state. List the evidence we have and the next
observation needed before changing firmware or resetting the board.

Start with the logs you already have and any existing debugger session. A new connection can change the target’s state:

Connection	What to expect
New UART connection	Asserts DTR and RTS, which can reset a board wired to use those lines. See UART.
Normal debugger connection	Can program, reset, or halt the target. See the attachment precautions.
New xSCOPE session	Can launch the firmware.
RTT through a bridge	Works only after Embedder flashes the firmware through that bridge in the current session.

Tell Embedder whether a halted fault must stay halted or whether a running target may be interrupted. Use captured history or an existing session to preserve a fault. Don’t reflash only to enable an observation path. Keep the firmware file and symbols that match the failing run before you rebuild. Symbols from another build can show a misleading stack or source location.

Prepare the investigation

Run /project to select the project, and confirm the active platform and peripherals. Provide the exact datasheet, errata, and schematic revision for the populated board.
Check the build, flash, and monitoring commands in EMBEDDER.md. Identify the actual board, probe, and port. If the board is connected to another machine, run /bridge and choose its remote bridge.
In VS Code, run Embedder: Open Embedder Monitor to keep serial output and captures beside your source. In the terminal, run /serial to show serial output beside the chat.
Ask Embedder to check the instruments you need and finish their setup, or connect a bench instrument from the Monitor. Follow the tool’s guide for channels, voltage limits, trigger support, and host software.
Choose the mode for the next step, using the table below.

Mode	Use it for
Plan (`/plan`)	Reviewing code, documents, schematics, and captured serial history without editing source, building, or flashing.
Act (`/act`)	Builds, tests, flashing, serial reads, instrument measurements, and hardware scripts.
Debug (`/debug`)	Everything Act does, plus GDB sessions, target-state inspection, and bounded instruction and coverage captures.

If another session is using a probe or instrument, run /hardware-queue to see the hardware workflow queue before you start a conflicting operation. Ask Embedder to keep a hardware case for the investigation, such as .embedder/hardware/cases/sensor-timeout/. The case folder holds the investigation’s scripts and a CASE.md record of the bug. Ask Embedder to keep the symptom, reproduction steps, hypotheses with their verdicts, findings, the next check, and paths to raw captures in that record. Update it after each test instead of treating the first explanation as settled.

Choose a test that separates the causes

A missing response can come from firmware, a physical connection, the device’s operating state, or the observation setup itself. First confirm the port, probe, firmware file, decoder settings, and capture window. A missing or malformed capture is an incomplete observation, not a result.

Compare the failing path with the active schematic and device requirements.
For each plausible cause, state what evidence supports it, what result would
contradict it, and the smallest test that distinguishes it from the
alternatives. Start with observations that preserve the current failure.
Do not patch several possible causes at once.

Choose the observation that answers the question:

Observation	Use it to check	Limits
Source, datasheets, and schematics	Intended pin mapping, clocks, reset order, address selection, register behavior, and driver logic.	A schematic describes the design. It doesn’t prove that the assembled board matches it.
Serial logs and GDB	Startup order, error paths, stack, variables, and available task information.	The debugger and firmware determine which state is available.
Logic analyzer	Digital transactions, addresses, acknowledgments, framing, and edge timing.	Decoded traffic doesn’t show signal-voltage margins.
Oscilloscope and power analyzer	Rail voltage, reset or enable signals, rise time, ringing, current, and workload transitions.	Averaged power statistics can hide short voltage transients. Use a scope for those.

For each capture, set a bounded duration, the physical channel map, a trigger, and the expected result. If two observations must describe the same event, use a common marker or trigger and check their timing relationship. Independent logs and instrument captures don’t necessarily share a clock. See Combine hardware observations.

Work through hardware and software hypotheses

A peripheral does not respond

Ask Embedder to compare the driver’s initialization and transaction sequence with the exact device datasheet and schematic. Then capture the bus with a supported logic analyzer.

Capture the failing I2C transaction using the channel map recorded for this
bench. Save raw and decoded results. Compare the address, read/write
direction, acknowledgments, and transaction order with the driver and the
fitted sensor. Report the first point where the observed sequence differs
from the expected one.

Read the result this way:

The expected pins never toggle: check whether the code reaches the transfer, then check the peripheral clock, the pin configuration, and the routed connection.
The address or command is wrong on the wire: trace how the driver builds it.
The transaction is correct but the device doesn’t acknowledge: check the supply, reset or enable state, address straps, and signal levels before you suspect the parser.

Use an oscilloscope to measure the signal at the device, and compare its rise time and levels with the device’s requirements. Ask for a physical check when the suspected problem is a missing pull-up, an open connection, or the wrong fitted part. State the board’s limits before anyone changes wiring or components.Correct a demonstrated address, pin, or sequence error in firmware. For a demonstrated electrical problem, document the board correction and measure again afterward. Reducing the bus speed may help a marginal connection, but that alone doesn’t identify the original cause or validate the required bus speed.

The board resets when a load starts

Capture VDD at the target and a known workload marker in the same scope acquisition. Choose ranges and a trigger from the board’s supply and reset requirements, and arm the scope before you reproduce the load change.

Prepare a bounded scope capture of target VDD and the recorded load-start
marker. After I reproduce the reset, save that acquisition. Before any new
reset or programming, check whether the matching reset-cause register can
be read without losing evidence. Compare its flags with the voltage trace
and boot log, and identify what remains uncertain.

Register lookup gives register definitions when they are available. Reading a live value takes a separate debugger observation. Check the access rules before reading: some status bits clear on read, and startup code may already have cleared the reset flags. Don’t treat a documented reset value as a measured value.A captured voltage drop together with relevant reset flags supports a supply-related cause. Then determine whether wiring, supply capacity, decoupling, or firmware load sequencing caused the drop. A stable voltage at the measured point doesn’t rule out an unmeasured transient elsewhere. A watchdog flag identifies the reset mechanism, but the missed service can still come from a driver deadlock, a long interrupt, or a stalled peripheral.Ask Embedder to inspect the relevant scheduling and driver paths, or to propose the next electrical measurement. Apply a firmware sequencing fix or have the identified board issue corrected. Then repeat the original load transition under the same supply conditions.

A hang or timing failure changes under the debugger

Keep the matching firmware file, the captured output, and any existing stopped state. In Debug mode, use GDB to inspect the current frame, stack, variables, and available task state. Compare the result with locks, interrupt masking, DMA completion, and bounded waits in the source.

Using the existing stopped debugger session and matching symbols, inspect
the blocked transfer and its wait condition. Identify who should complete
it and how errors or timeouts release the caller. Check the target-specific
register access rules before reading peripheral status. Propose one test
that distinguishes a software wait bug from a device that never completed.

Stepping and halting change timing. If the failure disappears under the debugger, use a bounded run with minimal logging or a measured GPIO marker, and check the external transaction while the firmware runs normally. Inspect the code and host tests for a lost wake-up or an unbounded wait. Inspect the bus and peripheral state for the missing completion event.Use trace when execution order can answer the question. Event traces need compatible firmware instrumentation. Bounded J-Trace captures reset and run the target, and streamed ETM halts it repeatedly. Preserve the original fault first, and treat a later trace as a separate run, not proof of uninterrupted real-time behavior.Correct the demonstrated synchronization or error-handling defect and add a regression test. A timeout can prevent an indefinite hang while leaving the reason for the missing device response unresolved. Record that distinction.

Apply a correction and test the explanation

Ask Embedder for a focused correction tied to the evidence. For a larger change, run /plan to review the affected files and tests, then approve the plan to implement it in Act mode. Run the relevant software checks and the target build before you flash the confirmed board. For a hardware correction, have someone inspect or modify the identified connection or component, then record what changed. Embedder can set up supported instruments and take measurements, but a proposed rework is not a completed repair. Change one factor at a time where possible. To isolate the problem, compare with a known-good firmware build on the same board, or the same build on another matching board. Keep the wiring, supply, workload, and measurement settings fixed, and record any difference that prevents a clean comparison.

Apply the correction supported by the investigation, run the focused tests,
and build the target. Repeat the original reproducer with the same workload
and capture setup. Compare failure counts and the relevant bus or voltage
measurement before and after. Report whether we verified a root-cause fix,
a mitigation, or an unresolved hypothesis.

For an intermittent fault, record failures and total attempts before and after the change under comparable conditions. Choose the run count or duration from the observed failure rate and the requirement. One successful run doesn’t establish a fix. For a long reproduction, ask Embedder to run the check as a background hardware script. Don’t reintroduce an unsafe fault only to show that the fix is reversible.

Close the RCA with a regression

Ask Embedder to finish the case record with:

The observed failure, the affected board and build, the trigger, and the reproduction rate.
The causal sequence with its supporting source, document references, and captures, plus the alternatives tested or still open.
The firmware or hardware correction actually applied, with before-and-after results and measurement limits.
A repeatable test, its pass criteria, and any remaining mitigation or follow-up work.

Use a hardware script to repeat the bench sequence with bounded waits and cleanup. Keep software regression tests for parsing, state transitions, timeouts, and recovery behavior. Re-run the original workload after changes to the board, driver, SDK, clocks, or supply setup. If a needed measurement or piece of hardware is unavailable, leave the cause unresolved and state the next test that would settle it. A useful RCA can name a missing observation without inventing a diagnosis.

Get started

Core concepts

Complex workflows

Monitor and debug

Project tools

Customize

Integrations

Automation

CLI

Reference

Debugging and root-cause analysis

Define the failure and preserve the evidence

Prepare the investigation

Choose a test that separates the causes

Work through hardware and software hypotheses

Apply a correction and test the explanation

Close the RCA with a regression

​Define the failure and preserve the evidence

​Prepare the investigation

​Choose a test that separates the causes

​Work through hardware and software hypotheses

​Apply a correction and test the explanation

​Close the RCA with a regression

Define the failure and preserve the evidence

Prepare the investigation

Choose a test that separates the causes

Work through hardware and software hypotheses

Apply a correction and test the explanation

Close the RCA with a regression