- Home
- Prep by Job Role
- Robotics Engineer
- Top 100 Q&A
Top 100 Robotics Engineer Interview Questions and Answers
The questions most likely to come up in your Robotics Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-1A colleague writes `T_base_camera` and uses it to convert a point measured by the camera into base coordinates. When is that right, and how would you make the mistake impossible to repeat?
The first thing I would pin down about transform direction and naming is what the machine physically does when the assumption is wrong.
A transform is directional. The convention that reads "the pose of the camera expressed in base" is exactly the matrix that maps a point in camera coordinates into base coordinates, and its inverse does the opposite. Both are valid conventions; only one can be in force in a codebase.
Concretely, write the convention down once, name every transform with both frames in a fixed order, and make composition check that the adjacent frames match rather than trusting the caller. A helper that takes source and target frame names and refuses an unmatched pair costs a few lines and removes the whole class of error.
The reason for that specificity is a failure I have seen: A pick cell composed a camera-to-tool transform in the wrong order and placed parts 214 mm from the target, which reads as a plausible calibration error rather than as an inverted matrix, so the team spent two days re-running calibration.
The same 214 mm error, two explanations.
| Hypothesis | Predicted residual | Varies with pose? | Matches log |
|---|---|---|---|
| calibration drift | 0.5-2 mm | slightly | no |
| inverted transform | 100-400 mm | strongly | yes |
| observed | 214 mm | strongly | inverted |
I would not consider it settled without evidence: Assert on a known fixture: a point at a measured location in the camera frame must map to its surveyed location in the base frame within the stated tolerance, and the test must fail if the transform is inverted.
A transform without a stated direction is not data, it is a guess.
Curated: · Written: · Reviewed:
Q&A-2Your transform lookup always uses the latest available value. What does that cost on a moving robot, and what would you use instead?
I would start transform timestamps from the measured envelope, not from the number the datasheet promises.
A transform tree is a time series, not a set of constants. Asking for the newest value from each link gives a chain that describes several different instants, which is a pose the robot never held.
Concretely, look the whole chain up at one stamp, the stamp of the measurement being transformed, and accept a bounded wait or an interpolation rather than the newest sample. Declare a maximum age and fail loudly past it instead of silently using stale geometry.
The reason for that specificity is a failure I have seen: A mobile base transformed lidar returns using the newest odometry rather than odometry at the scan time; at 1.2 m/s and 40 ms of skew the map smeared walls by about 48 mm and the localiser drifted rather than converged.
Skew cost at 1.2 m/s.
| Skew | Position error | Effect on map |
|---|---|---|
| 5 ms | 6 mm | within noise |
| 40 ms | 48 mm | walls smear |
| 120 ms | 144 mm | localiser diverges |
I would not consider it settled without evidence: Replay a recorded run with deliberately delayed odometry and confirm the transform layer refuses the lookup rather than returning a mixed-time chain.
Geometry is only correct at an instant, so the instant is part of the query.
Curated: · Written: · Reviewed:
Q&A-88A camera vendor's SDK and the robot controller disagree about which axis is up. How do you handle it once?
The 60-second answer. Pick one internal convention, write it down, and convert at exactly one place. I use the ROS REP-103 body frame: x forward, y left, z up, right-handed, metres, radians, quaternions w-first. The adapter sits at the driver boundary and is the only code that knows the vendor's convention. Everything downstream (pick, calibration verification, diagnostics, logging) sees the same frame, so a missed conversion becomes a missing adapter rather than a silent 90° error.
What actually decides the design. Which convention you pick matters less than that there is one. The things that bite are handedness, rotation order (intrinsic XYZ vs extrinsic ZYX), angle units, quaternion component order, and length units. Get any of those wrong and the pose is plausible but wrong.
The adapter. Say the vendor SDK reports a left-handed frame (x right, y up, z into the screen) against REP-103's right-handed frame. A left-handed frame cannot be reached by rotation alone; the mapping needs a reflection, so its determinant is −1, not +1.
# Python 3.11, numpy 1.26
import numpy as np
# vendor (x right, y up, z into screen, left-handed) -> REP-103 (x fwd, y left, z up)
R_v2i = np.array([[0., 0., 1.],
[-1., 0., 0.],
[0., 1., 0.]])
assert np.isclose(np.linalg.det(R_v2i), -1.0) # reflection: handedness flip
assert np.allclose(R_v2i @ R_v2i.T, np.eye(3)) # orthonormal
def to_body(p_vendor: np.ndarray) -> np.ndarray:
return R_v2i @ p_vendor
Trace. Vendor point (0.3, 0.4, 0.5) m:
- x_i = 0·0.3 + 0·0.4 + 1·0.5 = 0.5
- y_i = −1·0.3 + 0·0.4 + 0·0.5 = −0.3
- z_i = 0·0.3 + 1·0.4 + 0·0.5 = 0.4
→ (0.5, −0.3, 0.4) m: 0.5 m forward, 0.3 m right, 0.4 m up. Cross-check the axes: vendor x (right) maps to −y_i (right), vendor y (up) maps to z_i, vendor z (forward) maps to x_i. All three agree.
Failure modes and how each is caught.
| Failure | Symptom | Detection |
|---|---|---|
| Handedness flip applied as a rotation | mirrored pose, looks plausible | det(R) = +1 where −1 is expected |
| Rotation order (intrinsic XYZ vs extrinsic ZYX) | correct at axis-aligned poses, wrong at compound angles | fixture at 30° about each axis |
| Quaternion w-first vs w-last | wrong rotation, norm still 1 | fixture, not a norm check |
| mm vs m | 1000× scale error | magnitude assertion on a known fixture |
| Conversion in the pick path only | verification passes on unconverted data | one fixture pushed through every path |
Enforcement. The adapter is the only constructor of the internal pose type. In Rust that is a newtype BodyPose with a private field; in Python, a frozen dataclass with a private constructor plus a module-level to_body. A raw vendor pose then cannot be passed where a BodyPose is expected, so the type checker catches the missed path at review time instead of the robot catching it at runtime.
Cost. Nine multiplies and six adds per pose. At 30 Hz that is under 500 flops/s: irrelevant next to the camera's own latency, so the adapter is never the reason to skip this.
When not to. If you only ever consume one vendor and never mix frames, the adapter is still one function and one test; there is no case where scattering conversions beats it. The one real alternative is a middleware transform tree (tf2), which is fine, but it must be a single static transform, not a per-consumer conversion.
Curated: · Written: · Reviewed:
Q&A-85A function modifies a pose array and the caller's data changes too. What happened and how do you prevent a class of these?
You handed the function a view, not a copy. poses[1:3] allocates a new ndarray header and points it at the caller's buffer, so an in-place write through it lands in the caller's memory. NumPy never copies on basic slicing.
Which operations alias
| Operation | Result | Shares the caller's buffer |
|---|---|---|
a[2:5], a[:, 0], a[::2] | view | yes |
a.reshape(...), a.T, a.ravel() on a contiguous array | view | yes |
np.asarray(a) when a is already an ndarray | same object | yes |
a[[2,3,4]], a[mask] | copy | no |
a.copy(), np.array(a) | copy | no |
Fancy indexing and boolean masks copy because they cannot be expressed as a stride; basic slicing can, so it does not.
A trace
import numpy as np
def normalize_in_place(poses):
poses /= np.linalg.norm(poses, axis=1, keepdims=True)
waypoints = np.array([[3.0, 4.0, 0.0],
[1.0, 0.0, 0.0],
[0.0, 5.0, 0.0]])
window = waypoints[1:3] # view, shape (2, 3)
normalize_in_place(window)
print(waypoints)
window is rows 1 and 2 of waypoints. Row 1 has norm 1.0, row 2 has norm 5.0. poses /= ... is np.divide(poses, norms, out=poses), so it writes through the view. Output:
[[3. 4. 0.]
[1. 0. 0.]
[0. 1. 0.]]
The caller's waypoints changed. np.shares_memory(window, waypoints) returns True.
Prevention
Copy at the boundary where ownership is ambiguous, and return new arrays from anything other people call:
def normalize(poses):
poses = np.array(poses, dtype=np.float64, copy=True)
return poses / np.linalg.norm(poses, axis=1, keepdims=True)
For arrays you hand out, freeze them. result.flags.writeable = False makes any later in-place write raise ValueError at the point of the bug instead of silently corrupting a caller. The flag is per ndarray object, so setting it on a view leaves the base writeable.
Trade-offs and failure modes
Copying is not free. A million poses at 7 float64 values is 56 MB, and a copy per call doubles peak memory and adds a full pass over the data. Copy at API boundaries, not inside inner loops; inside a hot loop, document that the function writes in place and name it normalize_in_place.
The quieter failure is retention. A 7-element slice of a 56 MB array keeps the whole base buffer alive as long as the slice lives, because the view holds a reference to it. arr.base is not None tells you an array is a view; np.shares_memory(a, b) gives an exact answer when you need one. Copy small slices you intend to keep.
Catch it in tests
For every function that takes an array, snapshot the input and assert it is unchanged after the call:
before = waypoints.copy()
normalize(waypoints)
assert np.array_equal(waypoints, before)
That test fails the moment someone reintroduces an in-place write on a view. Checked against NumPy 2.1; the view and copy rules above are unchanged from 1.26.
Curated: · Written: · Reviewed:
Q&A-83A long chain of transforms accumulates error even in double precision. What would you do?
Core
First pin down what "chain" means, because the answer flips.
A chain of compositions of known rotations (kinematic chain, pose graph, scene graph) is not a double-precision problem. Composing n 3×3 rotations in double drifts the determinant by roughly sqrt(n)·1e-15 (worst case n·1e-15). At n = 10⁶ that is 1e-12: a scale error of 3e-13, which moves a point 2 m out by 6e-13 m. Invisible.
A chain of incremental updates from odometry or an IMU is not an arithmetic problem either: a gyro bias of 1e-4 rad/s integrated over 100 s is 0.01 rad, which at a 2 m lever arm is 20 mm. Seven orders of magnitude above double rounding.
So: find the real source first, then make the representation robust anyway.
Mechanism
SO(3) is closed under multiplication in exact arithmetic; in floating point it is not. Each 3×3 multiply perturbs entries by ~γ₃ = 3u ≈ 3.3e-16 (u = 2⁻⁵³). The determinant moves by tr(RᵀδR), order 1e-15 per composition. That perturbation is not zero-mean in the scale direction, so it accumulates: a random walk in practice, linear in the worst case. The matrix picks up scale and shear and stops being a rotation: RᵀR ≠ I, det ≠ 1.
What I'd do, in order
- Check the precision actually in the pipeline. The most common cause of visible drift is a float32 stage: a GPU shader, a CUDA kernel, a ROS message, an Eigen
Matrix3f. A determinant of 1.0004 after 400k compositions is the float32 signature (sqrt(4e5)·1e-6 ≈ 6e-4); double lands near 1 + 6e-13. - Compose from a canonical reference, not incrementally. Absolute poses in a world frame do not compound: each costs one composition regardless of chain length. Biggest win, and free.
- If you must chain, use a representation closed under composition: unit quaternions with renormalization, or so(3) with the exponential map.
- Re-orthonormalize on a schedule, not every step: project to SO(3) by SVD every k steps or when the residual crosses a threshold.
- Set the tolerance from the downstream requirement. A 2 m reach good to 0.1 mm is 5e-5 relative; double holds that for ~10¹⁰ links.
Artefacts
Double: u = 2⁻⁵³ ≈ 1.11e-16, γ₃ = 3u ≈ 3.3e-16, per-composition |δdet| ~ 1e-15, accumulating as sqrt(n)·1e-15 (random walk) up to n·1e-15 (worst case). float32: u₃₂ = 2⁻²⁴ ≈ 5.96e-8, per-composition ~1e-6.
| n | double δdet | float32 δdet | scale error at 2 m, double |
|---|---|---|---|
| 1e5 | 3e-13 | 3e-4 | 2e-13 m |
| 4e5 | 6e-13 | 6e-4 | 4e-13 m |
| 1e6 | 1e-12 | 1e-3 | 7e-13 m |
import numpy as np
def residual(R):
return np.linalg.norm(R.T @ R - np.eye(3), 'fro')
def project_so3(R):
U, _, Vt = np.linalg.svd(R)
D = np.diag([1.0, 1.0, np.linalg.det(U @ Vt)])
return U @ D @ Vt
Trace: for R already a rotation, SVD gives U = R, Vt = I (up to sign), det(U@Vt) = 1, D = I, so project_so3 returns R unchanged. For R with det = 1.0004, U and Vt are orthogonal, U@Vt is orthogonal with det ±1, D fixes the sign, and the result is the nearest rotation in Frobenius norm (the Kabsch/Wahba projection). residual is 0 for a true rotation and grows with drift.
Failure modes
- float32 hiding in the pipeline: assert dtype at every frame boundary; one
Matrix3freintroduces the drift. - Lossy renormalization: a fast inverse square root or truncated series in quaternion normalize reintroduces ~1e-3 error. Detect by comparing against exact normalize on a unit test vector.
- Non-rigid update: adding a delta matrix instead of multiplying, or lerping rotations, breaks closure immediately. Detect by asserting det ≈ 1 and residual < tol after every update.
- Frame confusion: composing in a non-inertial frame looks like drift. Detect by round-tripping a known pose through the canonical frame.
When not to
Renormalizing every step costs an SVD (~1 µs) per link: a second over 10⁶ links. Renormalize every k steps or on threshold. Quaternions renormalize cheaper (one sqrt on 4 numbers) but carry the q ≡ -q sign ambiguity, which matters for interpolation. Rotation vectors are singular at π. Dual quaternions handle rotation and translation with one renormalization but have thinner library support.
Curated: · Written: · Reviewed:
Q&A-86A collision check occasionally reports contact between objects 0.0000001 mm apart and occasionally misses touching ones. What is the fix?
The fix is to stop testing exact equality and instead classify the signed clearance against a tolerance taken from the physical uncertainty of the objects. Compute the clearance d = distance - (r1 + r2). If d > eps, the objects are separated; if d < -eps, they are penetrating; otherwise they are in contact. Inflating one object by eps and testing intersection is the same operation. A 0.0000001 mm gap is below any real positional uncertainty, so it should be reported as contact, not as a separated pair.
Exact equality on floating-point results of geometric computation is meaningless because the last bits carry accumulated rounding rather than information. Transforms, rotations, and mesh tessellation all introduce error. A predicate that asks for exact zero clearance will flip sign on rounding, which is why the same pair can be reported as touching and then as separated.
import math
def classify_spheres(c1, r1, c2, r2, eps):
"""eps is the physical uncertainty in the same units as the coordinates."""
dx = c1[0] - c2[0]
dy = c1[1] - c2[1]
dz = c1[2] - c2[2]
d = math.sqrt(dx*dx + dy*dy + dz*dz)
clearance = d - (r1 + r2)
if clearance > eps:
return "separated"
if clearance < -eps:
return "penetrating"
return "contact"
Trace with eps = 0.4 mm, r1 = r2 = 1.0 mm, and c1 = (0, 0, 0):
c2 = (2.0000001, 0, 0):d = 2.0000001,clearance = 0.0000001,|clearance| <= 0.4, so the result iscontact.c2 = (2.5, 0, 0):d = 2.5,clearance = 0.5 > 0.4, so the result isseparated.c2 = (1.5, 0, 0):d = 1.5,clearance = -0.5 < -0.4, so the result ispenetrating.
For mesh collision, the same idea applies through a signed distance function or a GJK/EPA query with a margin. Inflate one mesh by eps and test intersection, or compute the distance between the closest features and compare it to eps. For AABBs, expand one box by eps on each axis before the overlap test.
The tolerance must come from the manufacturing and calibration uncertainty of the objects being tested, not from the floating-point type. A tolerance of 1e-7 mm is the wrong scale for a robot arm with 0.4 mm of positional uncertainty. Too large a tolerance creates false contacts and ghost collisions; too small a tolerance misses touching pairs. Apply the same eps consistently to both sides of the comparison.
Alternatives have limits. Exact arithmetic predicates such as Shewchuk's adaptive orientation tests eliminate sign errors for exact inputs, but CAD and sensor inputs are already rounded, so they do not fix physical uncertainty. Continuous collision detection with swept volumes avoids tunneling for moving objects, but it still needs a tolerance for resting contact. For large coordinates, an absolute eps fails; scale it by the coordinate magnitude or the object size.
Concrete failure modes to check: using 1e-7 as the tolerance instead of the measured uncertainty; comparing a squared distance to an unsquared eps; using different tolerances in different axes; and not handling eps > r1 + r2, where the lower bound must be max(0, r1 + r2 - eps) rather than a negative number. Set eps from the uncertainty budget. If the application truly needs to resolve 0.0000001 mm, the inputs and the arithmetic must both be exact, which is a different system.
Curated: · Written: · Reviewed:
Q&A-101Walk me through the difference between forward and inverse kinematics, and why one of them is the hard problem.
The 90-second answer
Forward kinematics is a function: joint angles in, pose out. It is always defined, always single-valued, closed-form, and cheap: a chain of 4×4 multiplies. Inverse kinematics is an inverse problem: pose in, joint angles out. It can have zero solutions, one, or infinitely many, and for a general six-axis arm there is no closed form at all. That asymmetry is the whole question.
| Forward | Inverse | |
|---|---|---|
| Maps | q → T | T → q |
| Solutions | exactly one | 0, 1, or a continuum |
| Cost | ~µs, closed form | closed form only under Pieper's condition, else iterative |
| Fails when | never | target outside the workspace, or the Jacobian loses rank |
A concrete arm. Take a planar 2R with L1 = 0.4 m, L2 = 0.3 m. FK is trivial: x = L1 cos θ1 + L2 cos(θ1+θ2), y = L1 sin θ1 + L2 sin(θ1+θ2). IK inverts it:
import math
def ik_2r(x, y, L1=0.4, L2=0.3):
c2 = (x*x + y*y - L1*L1 - L2*L2) / (2*L1*L2)
if abs(c2) > 1.0:
return [] # outside the reachable annulus
s2 = math.sqrt(max(0.0, 1.0 - c2*c2))
out = []
for sign in (+1, -1): # elbow-down, elbow-up
t2 = math.atan2(sign*s2, c2)
t1 = math.atan2(y, x) - math.atan2(L2*math.sin(t2), L1 + L2*math.cos(t2))
out.append((t1, t2))
return out
For (x, y) = (0.5, 0.2): r² = 0.29, c2 = (0.29 − 0.16 − 0.09)/0.24 = 0.1667, so θ2 = ±80.4° and there are two answers. The reachable set is the annulus 0.1 m ≤ r ≤ 0.7 m; ask for r = 0.8 and you get an empty list. That is the first hard part: existence.
The second is branch selection. Both solutions are geometrically valid; only one is near where the arm currently is. A solver that returns the elbow-up answer while the arm is elbow-down produces a large joint-space jump, and the controller executes it: that is how an arm sweeps through a fixture on a "valid" IK result.
The third is singularities. The Jacobian determinant for this arm is L1 L2 sin θ2, zero at θ2 = 0 (fully extended) and θ2 = π (folded). At θ2 = 0.05 rad, det J = 0.12 × 0.05 = 0.006, so a 0.1 m/s Cartesian move demands roughly 17 rad/s at the joints. The pose is reachable; the velocity is not. Detect it with σ_min(J) or manipulability √(det(JJᵀ)) and either damp the step, Δq = Jᵀ(JJᵀ + λ²I)⁻¹e, or slow the path.
Six axes and up. A wrist-partitioned 6R (three consecutive axes intersecting at a point, or parallel) satisfies Pieper's condition and has a closed form with up to 8 solutions: 2 shoulder × 2 elbow × 2 wrist. A general 6R has no closed form; Raghavan and Roth bounded it at 16 solutions, and you solve it numerically with a Newton step on a se(3) pose error, seeded from the current configuration. A 7-DOF arm is redundant: the solution set is a 1-D self-motion manifold, and you pick within it by projecting a secondary objective (joint-limit avoidance, distance from a singularity) into the null space of J.
What I'd watch in production. Log σ_min(J) and the chosen branch per cycle. Branch flips and σ_min below threshold are the two signals that predict the failures you actually see: a wrist snapping around, or a move that stalls and then lurches.
Curated: · Written: · Reviewed: —
Q&A-3When would you store orientation as Euler angles rather than a quaternion, and what breaks if you average quaternions componentwise?
Core answer
Assume you're storing orientation for a system that also composes, interpolates or filters rotations, and you can pick the representation per use. Then store and compute in quaternions (or rotation matrices), and use Euler angles only where the axes are physically independent or a human/hardware consumer demands them: always with the sequence named.
Euler is genuinely the right storage when the three axes are separate actuators or separate limits: a pan-tilt head, a 3-axis gimbal, a robot wrist where each joint has its own encoder and range, with the operating envelope clear of the singularity. It's also right as an interchange format: URDF, CAD, animation keyframes, telemetry logs and config files expect three numbers plus a sequence, and converting at the boundary is cheaper than teaching every consumer quaternions. Three floats instead of four, and no normalization drift, is a real if minor win for large logs.
What breaks with Euler: gimbal lock. In a ZYX (yaw-pitch-roll) sequence, at pitch = ±90° yaw and roll become the same rotation: you lose a degree of freedom, the Jacobian goes singular, and a controller that differentiates the angles divides by zero. Sequence dependence is the second trap: ZYX and XYZ give different orientations for the same triple, so an Euler value without its sequence is meaningless. Third, interpolating Euler triples linearly doesn't follow the shortest path and can swing through the singularity, and the angles wrap at ±180°.
Componentwise quaternion averaging
Quaternions fix gimbal lock but introduce the double cover: q and −q are the same rotation. The normalized componentwise mean is the extrinsic (chordal) mean, it minimizes Σ‖q − q_i‖² over unit q, and it is not sign-invariant. If estimates straddle the cover, they cancel.
Worked trace, two estimates about the z axis. q1 = (1, 0, 0, 0) is 0°; q2 = (0.0872, 0, 0, 0.9962) is 170°. Componentwise mean = (0.5436, 0, 0, 0.4981), norm 0.7373, normalized (0.7373, 0, 0, 0.6756) → 2·acos(0.7373) = 85°, the correct midpoint. Now flip the sign of the second estimate: same rotation, q2′ = (−0.0872, 0, 0, −0.9962). Mean = (0.4564, 0, 0, −0.4981), norm 0.6756, normalized (0.6756, 0, 0, −0.7373) → 2·acos(0.6756) = 95° about −z, i.e. −95°. That is 180° from the correct 85° mean, from a sign flip that changed nothing physically.
Averaging methods
| Method | Computes | Sign-invariant | Cost |
|---|---|---|---|
| Normalized componentwise mean | extrinsic (chordal) mean | no | O(N), one normalize |
| Markley / largest eigenvector of Σ w_i q_i q_iᵀ | intrinsic (geodesic) mean | yes | O(N) to build, 4×4 eigen |
| Incremental slerp, weight 1/k | order-dependent approximation | yes if you slerp | O(N) slerps |
| Karcher mean (Riemannian gradient) | true intrinsic mean | yes | iterative |
Markley's method and the "eigenvector method" are the same computation: the largest eigenvector of the weighted scatter matrix Σ w_i q_i q_iᵀ. It's sign-invariant because q qᵀ = (−q)(−q)ᵀ, so the flipped pair above yields the identical matrix and recovers 85°. Use it when you need a defensible mean and N is small. Incremental slerp is cheaper to write and fine for streaming, but the result depends on sample order. Karcher iteration is the reference when you need the true intrinsic mean and can afford the loop.
Failure modes and detection
- Sign flip in a filter: the mean's norm drops well below 1 and the result points away from every input. Detect by asserting the mean's angular distance to each input is within the input spread; canonicalize sign (flip q_i so dot(q_i, q_ref) ≥ 0) before any componentwise step.
- Gimbal lock: pitch approaching ±90°, Jacobian determinant → 0. Detect by monitoring |pitch| and switching representation or clamping.
- Euler sequence mismatch: two subsystems disagree on ZYX vs XYZ. Detect with a round-trip test: convert to matrix and back, assert the matrix is unchanged.
Unit-test that averaging q and −q returns q, and that a 179° pair doesn't produce an arbitrary result.
Curated: · Written: · Reviewed:
Q&A-4The URDF parses, rviz looks right, and the arm still misses by 4 mm at reach. Where do you look?
Where I look
At the kinematic parameters the URDF asserts but nobody measured: link lengths, joint zero offsets, axis signs, the tool frame, and the base mounting, and I look at them through an independent measurement, not through rviz.
Before that, one check decides whether the URDF is even the right place to look. Command the same pose twenty times and look at the spread. If the spread itself is around 4 mm, the error is mechanical (backlash, compliance, thermal drift) and no model edit fixes it. If the mean is off by 4 mm and the spread is tens of microns, it is a parameter error, and the model is where it lives.
Assumptions I would state out loud: the controller is loading the same URDF I am reading; the 4 mm is measured at the tool, not at the flange; and rviz is drawing the model I gave it, which is a claim about the machine, not a measurement of it.
The five claims a URDF makes
Each can be wrong while the file stays valid XML:
- link lengths: the mesh and the URDF value can disagree
- joint zero offsets: encoder zero versus model zero
- joint axis signs and directions
- tool frame: TCP offset and rotation
- base mounting: base_link to world
rviz validates none of these against reality. A file that parses has been validated as XML, not as geometry.
How to separate them
Measure the tool pose at a spatially diverse set of configurations with an independent instrument (laser tracker, CMM, or a touch probe on a known artefact) rather than reading encoders back. Then fit all parameters at once, least squares over the pose residuals, instead of nudging one offset until the taught point matches. A single-pose touch-up absorbs the error at that pose and pushes it somewhere else.
Worked example, illustrative figures: a cell showed 4.1 mm systematic tool error. The URDF listed a link at 300 mm; the machined part measured 299.6 mm. The tool plate added 3.7 mm of standoff nobody had modelled. 0.4 + 3.7 = 4.1 mm.
| Pose | Before | After single-pose touch-up | After parameter fit |
|---|---|---|---|
| taught | 4.1 mm | 0.1 mm | 0.3 mm |
| far corner | 4.4 mm | 8.2 mm | 0.5 mm |
| holdout | 3.9 mm | 7.6 mm | 0.4 mm |
The touch-up wins at the pose it was tuned on and roughly doubles the error at the far corner. The parameter fit is slightly worse at the taught pose and better everywhere else, which is what you want when the arm has to work across the envelope.
Before calling it fixed
Report positional residual across at least twenty poses spanning the working envelope, plus a holdout set the fit never saw. If the holdout residual is not in the same band as the fit residual, the model is overfit to the calibration poses. And version the URDF with the calibration that produced it: a model without its calibration record is a claim with no provenance.
Curated: · Written: · Reviewed:
Q&A-5Your IK returns a valid solution and the arm swings through the fixture on the way. What was missing?
I would treat inverse kinematics branch selection as a claim about the physical world that has to survive a bench test.
A six-axis arm typically offers up to eight closed-form solutions for a reachable pose. Every one places the tool identically and sweeps a different volume, so choosing among them is a policy decision, not a numerical one.
Concretely, state the rule (nearest to current configuration, or a fixed branch) log the branch actually taken with each solve, and treat a branch change inside a taught path as an event that requires a fresh collision check rather than a routine solver result.
The reason for that specificity is a failure I have seen: A palletising program flipped from elbow-up to elbow-down at rank 34 of a 60-point path because the nearest-solution rule crossed a boundary; the elbow swept through a fixture that the taught path had never approached.
Same tool pose, different swept volume.
| Branch | Elbow height | Swept volume | Clears fixture |
|---|---|---|---|
| elbow-up | 1180 mm | 0.42 m³ | yes |
| elbow-down | 610 mm | 0.51 m³ | no |
| chosen at rank 34 | 610 mm | collision |
I would not consider it settled without evidence: Log the solution branch per waypoint and assert that a branch change never occurs inside a segment that was collision-checked as a unit.
The tool pose does not determine the arm, so the arm has to be chosen.
Curated: · Written: · Reviewed:
Q&A-6A Cartesian move slows to a crawl and then the wrist snaps around. What is happening and what would you change?
The useful question for Jacobian conditioning near singularity is what still holds at the edge of the workspace, cold, and under full payload.
The Jacobian is configuration dependent. Approaching a singularity, one Cartesian direction costs increasingly large joint rates, and a plain pseudoinverse will happily command them because the mathematics does not know about the motor.
Concretely, monitor the smallest singular value or the condition number along the path, apply damped least squares with a damping term that grows as conditioning degrades, cap joint rate independently of the solver, and refuse the segment rather than degrading it silently.
The reason for that specificity is a failure I have seen: A wrist-singular waypoint drove joint 4 to a commanded 480 deg/s against a 200 deg/s limit; the controller clipped it, the tool lagged the path by 9 mm, and the recovery motion looked to the operator like a fault rather than a limit.
Conditioning along one path.
| Waypoint | σ_min | Commanded J4 rate | Result |
|---|---|---|---|
| 10 | 0.21 | 34 deg/s | fine |
| 22 | 0.04 | 210 deg/s | at limit |
| 24 | 0.008 | 480 deg/s | clipped |
I would not consider it settled without evidence: Record the minimum singular value along every commissioned path and gate release on a declared floor rather than on the absence of complaints.
A singularity is a property of the pose, so it has to be detected before the move, not after.
Curated: · Written: · Reviewed:
Q&A-7A seven-axis arm keeps drifting into a joint limit over a long task even though every Cartesian point is reached. Why, and what would you add?
Why it drifts
A 7-axis arm has one degree of redundancy against a 6-DOF Cartesian task, so the Jacobian J (6×7) has a 1-D null space. The minimum-norm solve
q̇ = J⁺ẋ
has no null-space component in exact arithmetic. In practice the drift comes from three places: numerical error in the pseudo-inverse; solvers that are not strictly least-norm (iterative IK, damped least squares, or anything that seeds from the previous configuration and takes a step); and the fact that the same Cartesian point is reachable by a continuum of configurations, so the solver's pick depends on the seed and the previous configuration. Whatever the source, the null-space component is unconstrained and integrates into q every cycle. Cartesian accuracy never notices, because null-space motion by construction does not move the tool. The arm reaches every waypoint while the configuration walks.
Near a singularity you switch to damped least squares, J⁺ → Jᵀ(JJᵀ + λ²I)⁻¹. If the null-space projector is built from the undamped inverse, the "null-space" term leaks into task space and you get Cartesian error on top of the drift.
What to add
An explicit secondary objective projected through the null space:
q̇ = J⁺ẋ + N q̇₀, N = I − J⁺J
with joint-limit avoidance as q̇₀, a per-cycle bound on the null-space step, and a declared margin checked at every waypoint.
# Python 3.11, NumPy 1.26
import numpy as np
def step(q, xdot, J, q_min, q_max, k_lim=0.5, dq_null_max=0.02):
Jp = np.linalg.pinv(J) # 6x7
N = np.eye(7) - Jp @ J # null-space projector
mid, half = 0.5*(q_max+q_min), 0.5*(q_max-q_min)
qdot0 = k_lim * (-(q - mid) / half**2) # repulsive, zero at mid-range
dq = Jp @ xdot + np.clip(N @ qdot0, -dq_null_max, dq_null_max)
return q + dq
Clip only the null-space term; clipping the whole step would corrupt Cartesian tracking.
Why the margin check matters
If the unmanaged null-space step averages 0.05° per waypoint in joint 3, over 900 waypoints that is 45° of accumulated drift. Starting from a 45° margin:
| Waypoint | Margin, no null-space term | Margin, limit avoidance |
|---|---|---|
| 100 | 40° | 44° |
| 500 | 20° | 41° |
| 780 | 6° | 38° |
Illustrative numbers, not a measurement. The point is the shape: linear erosion versus a bounded margin. At 0.05°/waypoint the arm has 6° left at waypoint 780, and the next segment that needs joint 3 to move outward is unreachable: the cycle aborts mid-part, which is the symptom you are seeing.
Alternatives and trade-offs
- Posture bias toward mid-range: cheapest, no Jacobian work, but it fights the task near the workspace edge.
- Manipulability maximisation: keeps you away from singularities but can push joints toward limits.
- Strict task-priority hierarchy (successive projection) or a hierarchical QP: handles hard joint limits as inequalities rather than a soft repulsion, and resolves conflicts between objectives by priority. More compute, more tuning.
- Skip it entirely if the path is short or the arm is non-redundant: there is no null space to manage.
Failure modes and detection
- Limit hit mid-segment → unreachable, abort. Detect by pre-checking margin at every waypoint offline before the run.
- Damped-inverse mismatch → task-space error. Detect by logging Cartesian error per cycle.
- Wrist flip or self-collision from a drifted posture. Detect with a collision checker on the planned configuration, not just the tool pose.
- Drift is deterministic given the seed, so replay the trajectory offline and plot minimum joint margin over the whole task; alarm below the declared threshold.
Curated: · Written: · Reviewed:
Q&A-8Your velocity-level IK tracks a Cartesian path and the tool ends 3 mm off after thirty seconds. Where did the error come from?
The short answer. Assume a 250 Hz loop: 30 s is 7500 cycles, so the loop is losing about 0.4 µm per cycle. No single step is wrong: the error is a systematic residual the integrator adds up. Two families of cause produce that, and they scale differently: a residual that depends on the number of cycles (linearisation/integration) and one that depends on distance travelled (a frame or kinematic-model error). Separate them before touching the controller.
Why velocity IK drifts. Velocity-level IK solves J(q)q̇ = ẋ once per cycle and integrates q. That is a first-order method: the pose after the step is x + ẋΔt + O(Δt²), and the O(Δt²) term is dropped every cycle. Over a fixed duration the dropped terms accumulate to O(Δt), so halving the control period halves the drift. The residual points in a slowly varying direction, so it adds rather than cancels.
The other candidate: a fixed frame error. If the Jacobian or the commanded twist is expressed in a frame that is off by a constant rotation θ, the tool is displaced by roughly θ × distance travelled. 3 mm over a 300 mm seam is θ = 0.01 rad = 0.57°: a sub-degree mix-up between tool and flange frames, or a TCP offset a few tenths of a millimetre wrong, produces exactly this. This error does not care how long you take.
Separating them.
| Cause | Scales with | Test |
|---|---|---|
| Linearisation/integration residual | cycle count (∝ 1/Δt at fixed duration) | run the same path at 1 kHz instead of 250 Hz; drift should drop ~4× |
| Frame / DH / TCP error | distance travelled | run the same path at half speed; drift unchanged |
| Unachieved twist (DLS damping, joint velocity or limit clamp) | configuration, not time | log ‖J q̇_actual − ẋ_cmd‖; it spikes near the singularity or the limit |
The measurement that settles it. Log, every cycle, the pose IK commanded and the forward kinematics of the joints actually commanded. Sum the per-cycle residuals. If the sum is 3 mm, the integrator is the whole story. If the per-cycle residual is ~0 and the tool is still 3 mm off, the error is in the model or the frame, not the integration. Servo following error is bounded (a constant velocity lag gives a constant offset, not 30 s of growth) so it cannot explain a growing error.
Fix. Close the loop on pose, not velocity: recompute the pose error each cycle and add a proportional correction to the commanded twist.
# 250 Hz, 6-DOF arm, base-frame twist
K = 20.0 * np.eye(6) # 1/s; a 1 mm error adds 20 mm/s
q = q_current
for k in range(n):
J = jacobian(q) # 6x6, base frame
e = pose_error(path(k*dt), forward_kinematics(q))
xdot = path_velocity(k*dt) + K @ e
qdot = np.linalg.solve(J, xdot) # pinv + damping if redundant or near-singular
q = q + qdot * dt # Euler; RK4 or a full IK solve per cycle removes the truncation term
The feedforward term keeps the tool on the path; the K·e term absorbs the residual instead of integrating it. Saturate K·e so it cannot fight the path near a corner. If the drift is the linearisation term, a higher-order update or a full IK solve per cycle removes it outright.
Curated: · Written: · Reviewed:
Q&A-9A planner reports success and the executed path violates a joint limit at run time. How do you make those two agree?
The 90-second answer
The planner and the controller only agree when they check the same constraints on the same object. Two things break that. The constraint sets differ: the planner validates joint position limits and collision on the geometric path, while the controller enforces velocity, acceleration and torque limits on the time-parameterised trajectory. And the limits themselves come from different sources: a URDF the planner reads versus the safety-rated configuration the controller actually enforces.
So: generate the planner's limits from the controller's configuration, validate the time-parameterised trajectory rather than the path, and sample it at the controller's interpolation period, not at waypoints. Then run the result through the controller's own limit checker before release and treat any violation as a planning failure.
Where the mismatch comes from
| Source | Planner sees | Controller enforces | Symptom |
|---|---|---|---|
| Constraint type | position, collision | position, velocity, accel, torque | plan passes, motion is scaled or aborts |
| Limit source | URDF / hand-maintained config | safety-rated config, often lower | repeatable small overshoot on one joint |
| Sampling | waypoints | interpolated at servo rate | violation between waypoints |
| Interpolation space | Cartesian straight line | joint-space tracking | mid-segment joint excursion |
| Units / sign | rad, one convention | deg or inverted axis | gross, immediate |
Arithmetic
Hypothetical: joint 6 travels 0.30 rad and the time parameteriser assigns that segment 0.05 s. Average speed is 0.30 / 0.05 = 6.0 rad/s against a 4.4 rad/s controller limit. No profile shape rescues it: even a constant-velocity profile is 36% over. The controller clamps and stretches the segment to 0.30 / 4.4 = 0.068 s, so the tool reaches the pick point 18 ms late. The planner reported success because it never evaluated qd.
The check
# Python 3.11
LIMITS = {"q": 2.97, "qd": 4.4, "qdd": 20.0} # rad, rad/s, rad/s^2
def first_violation(samples, limits=LIMITS):
for t, q, qd, qdd in samples:
for name, value in (("q", q), ("qd", qd), ("qdd", qdd)):
if abs(value) > limits[name]:
return t, name, value, limits[name]
return None
Trace with samples = [(0.000, 0.10, 0.00, 0.00), (0.025, 0.25, 6.00, 0.00), (0.050, 0.40, 0.00, 0.00)]:
- t=0.000: q=0.10 ≤ 2.97, qd=0.00 ≤ 4.4, qdd=0.00 ≤ 20.0. Loop continues.
- t=0.025: q=0.25 ≤ 2.97 passes; qd=6.00 > 4.4, so the function returns
(0.025, 'qd', 6.0, 4.4).
The check fires at the midpoint, which is exactly where a waypoint-only check misses it.
Failure modes and detection
- Controller silently scales. Compare commanded timestamps against executed timestamps; a scale factor other than 1.0 means the plan was infeasible. Log it as a planning defect, not a controller event.
- Limit drift. The URDF and the controller config diverge after a firmware update or a tool change. Diff the two limit sets in CI and fail the build on mismatch.
- Sampling too coarse. Re-check at the controller's interpolation period rather than the planner's waypoint spacing.
- Torque. A trajectory can satisfy position, velocity and acceleration and still exceed torque at the actual payload. Run the controller's dynamics model with the real payload mass and CoG.
What not to do
Don't loosen the controller's limits to match the planner's. The controller's limits are the safety-rated ones; the planner's are a model. And don't lean on controller-side scaling as a safety net: it converts a planning failure into a silent timing failure, which is harder to catch downstream.
Curated: · Written: · Reviewed:
Q&A-10How do you establish a tool centre point, and why is a four-point pivot not enough on its own?
Core answer
A four-point pivot solves three of the tool's six numbers. It fixes the TCP translation, where the tip sits relative to the flange, by touching one fixed point from several orientations. It tells you nothing about the tool frame's rotation relative to the flange. So the procedure is: pivot for position, a separate orientation step against a known direction, and report the conditioning of the fit, not just its residual.
Why the pivot only gives position
Each pose i gives R_i · t + p_i = P, where R_i is the flange orientation, p_i the flange position, t the TCP offset in the flange frame, and P the fixed point. Rearranged, R_i · t − P = −p_i: linear in the six unknowns (t and P), three equations per pose. Two poses are enough in exact arithmetic; four gives redundancy and lets you reject a bad pose.
The catch is conditioning. If every pose rotates about roughly the same axis, the offset along that axis is unobservable: the fit returns a number, and the residual looks small, because the error lies in the direction the poses never excited. You need rotations about at least two non-parallel axes, and the wider the angular spread the better conditioned the fit.
Orientation is a separate problem. The pivot constrains the tip to a point; it never constrains the tool axis. That needs its own measurement: touch the tool against a known plane or direction, or use a fixture.
Alternatives, and when the pivot is enough
| Method | What it gives | Cost | Use when |
|---|---|---|---|
| 4+ point pivot, sharp tip | Position only | Free, in situ | Point-like tool, orientation set mechanically (kinematic holder, dowel pin), tolerance loose vs lever arm |
| Vendor auto-TCP wizard | Position only, more poses, automated pose selection | Free with controller | Same as above, when you want the pose set chosen for you |
| Laser / beam-break fixture | Position and orientation, full 6-DOF | Fixture plus its own calibration | Long tools, orientation matters, repeated calibration |
| Touch probe on a known artefact (sphere, cube, plane) | Position and orientation | Artefact | You have a calibrated artefact and no laser |
| Mechanical datum / repeatable holder | Neither: TCP defined by design | Holder cost | Frequent swaps, holder repeats to microns |
The pivot is good enough when the tool is a point and its orientation is fixed by the mount. It is not enough when the tool is long, when orientation matters (a camera, a gripper's approach axis), or when the tool is hand-mounted and can rotate in its holder.
Failure modes and how to catch them
- Clustered orientations: residual small, offset wrong along the unexcited axis. Catch it with the condition number or singular values of the fit matrix, not the residual.
- Contact force during teaching deflects the tip and biases the fit. Catch it by re-teaching with a lighter touch and comparing.
- Thermal drift between calibration and use. Catch it by re-verifying at operating temperature.
- A crash or tool change invalidates the TCP. Re-verify; don't assume.
- A single scalar residual hides the direction of the error. Report per-axis.
Worked example (illustrative figures)
A 180 mm tool, joint position-and-orientation fit from the same pose set:
| Orientation spread | Position residual | Axis error | Tip error at 180 mm |
|---|---|---|---|
| 22° | 0.8 mm | 3.4° | 10.7 mm |
| 90° | 0.3 mm | 0.4° | 1.3 mm |
| 150° | 0.2 mm | 0.2° | 0.6 mm |
Tip error is 180 mm × sin(axis error): 180 × sin 3.4° = 10.7 mm. At 22° spread the position residual looks acceptable while the tip is 10.7 mm out when the wrist flips. That is the argument for reporting both numbers and for spreading the poses.
What done looks like
Report the pivot residual and the orientation check separately, publish the condition number or angular spread of the pose set, and reject a fit whose spread is below a declared threshold. Re-verify after any tool change or crash.
Curated: · Written: · Reviewed:
Q&A-11Explain eye-in-hand versus eye-to-hand calibration and what each one leaves unconstrained.
Core answer
Hand-eye calibration recovers the rigid transform between a camera and a robot frame from paired robot poses and camera observations of a target. The two mounting choices solve for different unknowns and leave different things free.
Eye-in-hand: camera bolted to the wrist/flange. You solve the camera-to-flange transform X. What is left unconstrained: the camera-to-base transform, which you get only by composing X with the robot's forward kinematics at each pose. Inside X itself, the rotational part is constrained only by the rotations present in the motion set. Pure translations leave X's rotation unidentifiable, and because the lever arm between camera and flange is revealed only when the flange rotates, the translation of X is unidentifiable too.
Eye-to-hand: camera fixed in the cell, target held by the gripper. You solve the camera-to-base transform Z. What is left unconstrained: the gripper-to-target transform, unless the target is fixtured with known geometry or calibrated separately. The camera's pose is fixed, so Z holds across the whole workspace once solved, but nothing in the motion set re-observes the camera, so a bumped camera is undetectable from the data.
Mechanism
Eye-in-hand is the AX = XB problem: A and B are the relative motions of the flange and of the camera between two poses, X is the unknown camera-to-flange transform. Both A and B come from the same robot motion, so the equation constrains X only through the relative rotations. A translation-only set gives A and B with identity rotation and AX = XB degenerates.
Eye-to-hand with a target on the gripper is AX = ZB: X is gripper-to-target, Z is camera-to-base, both unknown. If the target geometry is known and rigidly fixtured, X is known and a single pose pair determines Z = A X B⁻¹.
Trade-offs
| Eye-in-hand | Eye-to-hand | |
|---|---|---|
| Unknown solved | camera-to-flange X | camera-to-base Z |
| Left unconstrained | camera-to-base (needs FK); X rotation and translation without rotation | gripper-to-target (needs known target) |
| Excitation needed | rotation about ≥2 axes | target swept through the workspace |
| After a bump | re-solve; motion is self-consistent | full re-solve; no motion self-check |
| Occlusion | camera moves with the tool, can see into tight spots | fixed view, occluded by the arm |
| Accuracy driver | flange kinematics + X | base kinematics + target geometry |
Failure modes
- Eye-in-hand, translation-only set: the solver returns a confident X with a wrong rotation. Detect it by reporting the rotational spread of the set and rejecting below a declared angular coverage.
- Eye-to-hand, unknown target offset: assume the target sits at the flange when it does not, and Z absorbs the error. Every grasp is offset by the same vector, so it reproduces and looks like a robot bug.
- Eye-to-hand, bumped camera: no motion re-observes the camera, so the error stays silent until a pick fails.
Artefact: why rotation is what makes the lever arm visible
Hypothetical: camera 120 mm from the flange. A 5° flange rotation moves the camera by 2·120·sin(2.5°) ≈ 10.5 mm. That displacement is the only signal that reveals the 120 mm offset. A translation-only set produces zero such displacement, so the offset is invisible to the solver no matter how many poses you add.
Validate on a holdout pose the fit never saw, not on the fit residual.
Curated: · Written: · Reviewed:
Q&A-12A datasheet quotes 0.02 mm repeatability. What does that let you promise a customer, and what does it not?
The short answer. 0.02 mm repeatability lets you promise that the arm comes back to a point you taught it, to within 0.02 mm, under the conditions the spec was measured at. It does not let you promise that the arm goes to a point you computed in CAD, or that the tool is where the controller thinks it is. Those are different numbers, and the second is usually an order of magnitude or more larger.
Repeatability is the spread of repeated visits to the same commanded pose. Accuracy is the distance between the commanded pose and the mean achieved pose in the world frame. ISO 9283 is where these terms are defined for industrial arms, and the repeatability figure is a statistical bound, mean plus a multiple of the standard deviation over repeated visits, not a hard limit on any single move. It is measured at one pose, at rated payload, at steady temperature, after warm-up, and often in one approach direction. Approach the same point from the opposite direction and backlash and compliance add to the spread.
| You can promise | You cannot promise |
|---|---|
| Return to a taught point within 0.02 mm | Absolute position in the world frame |
| That spread holds at rated payload, steady temperature, after warm-up | That it holds with a different tool mass or over a shift as the arm warms |
| Cycle-to-cycle consistency at that pose | Path accuracy between poses, or accuracy at the edge of the envelope |
Three ways to close the gap, and when each wins.
Kinematic calibration. Identify link lengths, joint offsets and the tool frame with a laser tracker or ballbar. On a mid-size six-axis arm this typically brings absolute accuracy into the 0.2–0.5 mm range. It costs a measurement pass, and it is invalidated by a crash or a tool change, so it needs re-running. Choose it when you have many CAD-generated points, a stable cell, and no camera.
Vision compensation. A camera finds the feature and the robot moves relative to what it sees. This takes absolute accuracy out of the loop entirely: what remains is repeatability plus vision noise. It costs fixturing that keeps the feature visible, controlled lighting, and a camera-to-robot calibration. Choose it when part-to-part variation dominates, or when the feature lives on the part rather than the fixture.
Re-teach. Touch the points up by hand. No capital cost, labour that scales linearly with point count. Choose it when there are few points, low mix, or a one-off build.
Repeatability alone is sufficient when the program is taught, the fixture is rigid and presents the part the same way every cycle, and the tolerance is looser than repeatability plus thermal drift. Then 0.02 mm is the whole story and you can quote it.
Failure modes to name. Thermal drift over a shift moves absolute accuracy by tenths of a millimetre while repeatability barely moves: you see it as a slow walk of the process, not as scatter. A payload change shifts joint deflection, so a spec measured at rated payload does not transfer to a lighter tool. Accuracy degrades toward the edge of the envelope faster than repeatability does. Gear wear and backlash show up in bidirectional repeatability first. A crash invalidates calibration while repeatability may survive it, which is exactly the case where the datasheet still looks fine and the cell does not.
Worked example (illustrative figures). An offline-programmed cell was scoped on 0.02 mm repeatability. Measured absolute accuracy at reach was 0.9 mm uncalibrated and 0.25 mm after kinematic calibration. The placement tolerance needed was ±0.1 mm. Uncalibrated misses by 9×; calibrated still misses by 2.5×. Vision compensation closes it: a 100 mm field of view on a 2000 px sensor is 0.05 mm/px, and centroiding to 0.1 px puts the vision term near 0.005 mm, leaving repeatability as the dominant error. Every CAD point needed manual touch-up, which erased the reason for programming offline in the first place.
Before promising any CAD-driven tolerance, measure absolute accuracy across the working envelope with an independent instrument (laser tracker, CMM, or a dial indicator against a known artefact) at operating temperature and payload. Promise from that number, not from the datasheet.
Curated: · Written: · Reviewed:
Q&A-13Position feedback says the joint is at target and the tool is 0.4 mm off under load. Explain.
The first thing I would pin down about backlash and compliance in the loop is what the machine physically does when the assumption is wrong.
Motor-side feedback measures the motor, not the tool. Gearbox backlash, belt stretch, and structural compliance sit between them, so the encoder is telling the truth about a quantity that is not the one you care about.
Concretely, characterise the deflection against load rather than assuming it, approach critical points from a consistent direction so backlash contributes a repeatable offset, and add load-side sensing where the tolerance demands it rather than tightening the control gains.
The reason for that specificity is a failure I have seen: A press-fit station held 0.02 mm on the encoders and produced a 0.41 mm tool deflection at 60 N of process force; raising the position gain made the loop oscillate without moving the tool any closer.
Deflection against process force.
| Force | Encoder error | Tool deflection |
|---|---|---|
| 0 N | 0.02 mm | 0.02 mm |
| 30 N | 0.02 mm | 0.21 mm |
| 60 N | 0.02 mm | 0.41 mm |
I would not consider it settled without evidence: Measure tool deflection against applied load across the envelope and publish the curve, rather than quoting the encoder resolution.
The encoder is honest about the motor and silent about the tool.
Curated: · Written: · Reviewed:
Q&A-14A newly fitted tool causes sag on some joints and not others, and the numbers in the payload dialog came from CAD. What would you do?
In the first two minutes
Sag on some joints and not others is the signature of a gravity-torque error, not a uniform mechanical fault. Gravity torque about each axis is m·g·(horizontal distance from that axis to the CoM), so the error grows with the moment arm: joints far from the flange and roughly perpendicular to gravity show it, near-flange poses don't. That points at the payload number, and the payload dialog is fed from CAD.
I'd still spend ten minutes ruling out the mechanical causes that produce the same pattern before touching the payload: loose tool mounting bolts, a brake not fully released, a gearbox on its way out, or an axis already sitting near its continuous torque limit. A dial indicator on the tool at a repeatable pose, read before and after adding a known mass at a known offset, separates the two. If the sag moves the way the added moment predicts, it's payload. If it doesn't, it's mechanical.
Then decide CAD vs identification on the numbers, not on principle
CAD is good enough when the tool is a rigid one-piece body, every mass is modelled (cables, hoses, fasteners, the workpiece), the required TCP accuracy is looser than the sag you measured, and the worst-case axis keeps a comfortable torque margin. It costs engineering time and no robot downtime.
Identification costs robot downtime: a few minutes of motion plus setup, call it 10–30 minutes of cell downtime per configuration: a clear envelope, and for the loaded case a fixture or a known reference mass. It also returns a lumped payload at the flange: it cannot separate tool from workpiece if you swap parts at runtime.
| CAD payload | Identified payload | |
|---|---|---|
| Good enough when | rigid tool, all masses modelled, loose TCP tolerance, healthy torque margin | axis near torque limit, tight TCP tolerance, as-built cabling |
| Cost | modelling time, no downtime | 10–30 min downtime per config, clear envelope, fixture for the loaded case |
| Blind spot | as-built cabling, hoses, workpiece | lumped flange payload; can't split tool from part |
| Error bar | unknown unless you weigh the tool | run it twice; the spread between runs is your error bar |
Deciding criteria I'd write down: torque margin at the worst-case pose, required TCP accuracy against measured sag, whether the payload changes at runtime, and whether this is one cell or fifty. One-off tool: identify. Fifty identical cells: model it properly once and push the CAD number.
Why the CAD number is usually the wrong one
A CAD payload listed 3.2 kg at 60 mm from the flange. The tool as built, with cabling and a workpiece, was 4.1 kg at 104 mm. The first moment the controller compensates went from 0.192 kg·m to 0.426 kg·m: 2.2× too small. Joint 5 sagged 0.6 mm at full extension; near-flange poses looked perfect.
| Quantity | CAD | Identified |
|---|---|---|
| mass | 3.2 kg | 4.1 kg |
| CoM offset from flange | 60 mm | 104 mm |
| first moment | 0.192 kg·m | 0.426 kg·m |
| sag at reach | — | 0.6 mm |
Back-of-envelope, assuming roughly 1 m from the joint 5 axis to the tool at that pose: the missing gravity torque is (0.426 − 0.192) × 9.81 ≈ 2.3 N·m. A 0.6 mm sag at 1 m is 0.6 mrad, so the axis is making up that torque through position error at an effective stiffness near 3.8 kN·m/rad. If the sag implies a stiffness far from the axis spec, stop and look for the mechanical cause instead.
Failure modes to watch
- An identification run aborted by collision limits or a safety fence gives a garbage result. Check the routine ran to completion and look at its residual.
- Identifying with the workpiece but running unloaded over-compensates; the tool drifts up.
- Friction and gearbox wear bias the fit. Run slow, in both directions, and repeat.
- A sign-off on a number nobody re-checks after the next tool change.
Compensation corrects the payload you declared, not the one you bolted on.
Curated: · Written: · Reviewed:
Q&A-19Switching from manual to automatic steps the output by 20 percent. What was not initialised?
The integral term was not initialised.
A PI or PID controller's output is a sum: a proportional term, an integral accumulator, and a derivative term. While the loop is in manual, the controller is not driving the valve, the operator is, so the integral accumulator stops tracking the delivered output and freezes at whatever value it last held. On the switch to auto the controller evaluates u = Kp·e + I + Kd·de/dt and writes it straight out. If I does not correspond to the output that was actually being delivered, the valve jumps by exactly that mismatch. The 20% step is the mismatch.
Nothing else in the loop changed at the instant of transfer, so the step size is the difference between the manual output and the controller's first auto output.
Worked example (PI, output in %):
- Kp = 2.0 %/%, integral accumulator I in %.
- At transfer: SP = 50%, PV = 50%, so e = 0.
- Manual output was 55%.
- The accumulator was left at 35% from earlier auto operation.
- First auto output = Kp·e + I = 2.0·0 + 35 = 35%.
- Step = 55 − 35 = 20%.
Initialise I to 55 instead and the first auto output is 55%: no step.
The fix is bumpless transfer: at the moment of transfer, set the integral accumulator so the first auto output equals the delivered output.
# PI, output in percent. I is the integral accumulator (already in %).
def to_auto(manual_out, sp, pv, Kp):
e = sp - pv
return manual_out - Kp * e # initialise I
def step(I, sp, pv, Kp, Ki, dt):
e = sp - pv
I += Ki * e * dt
return Kp * e + I
With manual_out=55, sp=pv=50, Kp=2.0: to_auto returns 55. The first step returns 2.0·0 + 55 = 55. No step.
Two variants worth knowing. Output tracking (back-calculation) does this continuously rather than only at transfer: while in manual, the accumulator is loaded every scan with u_manual − Kp·e, so it is always ready. This is what most DCS and PLC PID blocks do when you enable "bumpless transfer" or "output tracking". The one-shot initialisation above is the minimum.
The same initialisation is needed on every transition that hands the loop to a different controller: auto→cascade (the inner loop's integral must be initialised to the outer loop's demanded output), override/select blocks, and controller restart, where the accumulator comes back from non-volatile memory or zero. Derivative state matters too if D acts on setpoint: a setpoint change at transfer produces a derivative kick; derivative-on-measurement avoids it.
Failure mode and detection: the process sees the step as a real disturbance. The valve moves, flow or temperature moves, and the loop then corrects a deviation it created. Detect it by logging the controller output on both sides of every mode change and asserting |Δu| is below a declared threshold, in both directions, for each transition. A transfer the process can feel is a disturbance, not a transfer.
Curated: · Written: · Reviewed:
Q&A-16A temperature loop overshoots badly after a long cold start but behaves near setpoint. What is wrong and how would you fix it?
My answer to integral windup under saturation begins at the failure mode: if I cannot name how it hurts someone, I do not have a design.
While the actuator is saturated the loop is open, but the integrator keeps accumulating error as though its output still mattered. The stored term then has to be unwound before the actuator leaves saturation, which shows up as overshoot proportional to how long the saturation lasted.
Concretely, stop or unwind the integrator during saturation by clamping, conditional integration, or back-calculation from the difference between the requested and delivered output, and make the software limit match the real actuator limit so saturation is visible rather than hidden.
The reason for that specificity is a failure I have seen: An oven loop sat saturated for 380 seconds during warm-up; the integral term reached the equivalent of 140 percent output and drove a 31 degree overshoot past a 200 degree setpoint, which was outside the process window for the parts already inside.
Overshoot against saturation time.
| Saturated for | No anti-windup | With back-calculation |
|---|---|---|
| 30 s | 4°C | 1°C |
| 180 s | 18°C | 1°C |
| 380 s | 31°C | 2°C |
I would not consider it settled without evidence: Log the output before and after limiting alongside the integral term, and test the loop from a cold start rather than only from near setpoint.
An integrator that cannot tell it is saturated is storing a debt.
Curated: · Written: · Reviewed:
Q&A-17Adding derivative gain makes the actuator chatter. What are your options and what does each cost?
I would treat derivative action and measurement noise as a claim about the physical world that has to survive a bench test.
Derivative action responds to rate of change, and measurement noise is mostly rate of change. An unfiltered derivative multiplies the noise by the derivative gain, so the term that was meant to add damping instead adds actuator wear.
Concretely, filter the derivative with a first-order filter whose coefficient bounds the noise gain, take the derivative of the measurement rather than the error so a setpoint step does not produce a spike, and record the filter coefficient with the gains since two otherwise identical tunings behave differently without it.
The reason for that specificity is a failure I have seen: A servo with an unfiltered derivative and 12 bits of encoder noise commanded ±8 percent output at 90 Hz while standing still; the valve logged 2.1 million reversals in a week against a rated 5 million lifetime.
Noise gain against filter coefficient.
| Filter N | Output noise | Reversals per hour |
|---|---|---|
| none | ±8% | 12500 |
| 20 | ±1.6% | 900 |
| 8 | ±0.7% | 210 |
I would not consider it settled without evidence: Measure output noise amplitude and reversal count at steady state, not only step response, before accepting a derivative term.
Derivative gain amplifies whatever the sensor is doing, including nothing.
Curated: · Written: · Reviewed:
Q&A-20A conveyor-tracking robot lags a moving part by a constant distance. Would you raise the gain?
The judgement in feedforward versus feedback roles is which timestamps and frames are pinned, not which library is fashionable.
Feedback acts on error that has already appeared, so tracking a ramp with proportional-plus-integral action leaves a lag set by the loop gain. Feedforward acts on the known command or measured disturbance before the error develops, which is the only way to remove that lag without pushing the loop toward instability.
Concretely, add a velocity feedforward term derived from the commanded or measured conveyor speed, keep feedback for the model mismatch it is good at, and tune the two separately so an error in the feedforward model does not masquerade as a gain problem.
The reason for that specificity is a failure I have seen: A tracking application raised proportional gain from 12 to 40 to close a 6 mm lag; the lag fell to 2 mm and the axis began oscillating at 14 Hz whenever the conveyor jogged, which produced worse placement than the original lag.
Lag against conveyor speed.
| Speed | Feedback only | With feedforward |
|---|---|---|
| 0.1 m/s | 2 mm | 0.3 mm |
| 0.3 m/s | 6 mm | 0.4 mm |
| 0.5 m/s | 10 mm | 0.4 mm |
I would not consider it settled without evidence: Measure tracking error against conveyor speed with and without the feedforward term and confirm the residual is speed-independent.
Feedback corrects what has gone wrong; feedforward prevents what is about to.
Curated: · Written: · Reviewed:
Q&A-21When does a cascade loop help, and what makes one actively worse than a single loop?
Cascade control helps when a fast, measurable intermediate variable sits between your actuator and the slow variable you actually care about, and the disturbances you are fighting enter at that intermediate point. You close a fast inner loop on the intermediate variable and feed the outer controller's output in as the inner loop's setpoint. The inner loop then rejects those disturbances before they reach the outer variable, so the outer loop sees a cleaner, more linear process and can be tuned more aggressively than it could alone.
The benefit is conditional on separation of timescales. The inner loop must be substantially faster than the outer: a common rule of thumb is that the inner loop should settle at least 3-5x faster, and 5-10x is a safer target. Tune the inner loop first, then the outer. If that separation is not there, the two loops interact and you get oscillation that neither loop produces on its own.
Worked example (hypothetical figures)
Jacketed reactor temperature control. The outer variable is reactor temperature, dominated by the vessel's thermal mass, with a settling time around 12 minutes. The inner variable is jacket temperature, which responds in roughly 40 seconds. A step change in coolant supply pressure moves the jacket temperature immediately; without cascade the reactor temperature drifts for minutes before the outer loop reacts. With cascade, the jacket loop absorbs the pressure step in about 40 seconds and the reactor temperature barely moves. The ratio here is roughly 18x, comfortably inside the range where cascade pays off.
Now shrink the separation: if the jacket loop settled in 8 minutes against a 12-minute outer loop, the ratio is 1.5x. The outer loop's corrections arrive while the inner loop is still moving, the two chase each other, and the cascade can be less stable than the single loop it replaced.
Where cascade makes things worse
| Situation | Cascade | Single loop |
|---|---|---|
| Inner settles ≥5x faster than outer | rejects inner disturbances before they reach the outer variable | outer loop waits for the slow variable to move |
| Inner only ~1-2x faster | loops interact, oscillation | stable and simpler |
| Inner measurement noisy | noise amplified into the actuator | slow outer process filters it |
| Inner actuator saturates | outer loop winds up without anti-windup | no inner loop to saturate |
Failure modes and how to detect them
Interaction oscillation: compare the closed-loop response with the inner loop in manual versus auto. If the cascade is worse than the single loop, the separation is insufficient: retune or drop the inner loop.
Reset windup: the outer loop integrates while the inner actuator is saturated (valve fully open, for example). Detect it by watching the outer controller output pinned at its limit while the process variable does not move. Fix with anti-windup or back-calculation.
Noise amplification: a fast inner loop with high gain passes sensor noise straight to the valve. Detect it by counting valve reversals per hour; if they climb, filter the inner measurement or reduce inner gain.
Added instrumentation: the inner sensor is a new failure mode. If it drifts or fails, the cascade drives the wrong variable with confidence.
When not to use it
Skip cascade when there is no fast measurable intermediate variable, when the inner loop cannot be made faster than the outer, when the disturbances enter the outer process directly rather than through the inner, or when the inner measurement is too noisy for the actuator to tolerate. In those cases a single well-tuned loop is the better answer.
Curated: · Written: · Reviewed:
Q&A-18A loop has 400 ms of transport delay and a 600 ms dominant time constant. What tuning do you expect to achieve, and when do you change structure?
θ/τ = 400/600 = 0.67. PID still works here, but it is the last comfortable point on the curve: expect a closed loop two to three times slower than the open-loop time constant, and do not chase anything faster.
Concrete tuning, assuming unity plant gain (K = 1) and a first-order-plus-dead-time model:
| Rule | Kc | τI | τD | Character |
|---|---|---|---|---|
| SIMC, τc = θ | 0.75 | 0.6 s | — | ~0% overshoot, 95% settle ≈ 1.6 s |
| Ziegler–Nichols reaction curve | 1.8 | 0.8 s | 0.2 s | quarter-amplitude decay, ~25% first overshoot |
Commission with the SIMC row: Kc = τ/(2θ) = 0.75, τI = τ = 0.6 s, PI only. Derivative on a 400 ms dead time mostly amplifies measurement noise; if you need it, put it on the measurement behind a 0.1 s filter.
What you cannot get: closed-loop bandwidth is capped near 1/(2θ) ≈ 1.25 rad/s, about 0.2 Hz. A spec asking for 0.5 s settling on this loop is asking for something the delay forbids: the correction arrives 400 ms after the disturbance, so the loop is already 400 ms behind before it acts. The ZN row buys its speed by accepting a sustained oscillation; on a blending or pressure loop that is usually the wrong trade.
Change structure when θ/τ exceeds about 1, or when the required settling time is under roughly 2θ + τc. Past that point, gain only buys oscillation. Options, in order of preference: remove the delay (move the sensor closer to the actuator, cut the network hop, drop the anti-alias filter), then a Smith predictor, then MPC with an explicit delay model. A Smith predictor is exact only when the delay model matches the plant; with jittery transport delay it degrades to worse than PID, which is why MPC is the safer choice when θ varies.
I have watched this go wrong. A blending loop with θ/τ = 0.67 was tuned to a settling target using ZN numbers; it oscillated with a 2.4 s period, the operators put it in manual, and it stayed there for a year. The ratio was the warning and nobody read it.
Measure θ, do not infer it from the tune. Step the output and timestamp the input response. A network hop, a filter, and a slow front end all present as dead time and only some are removable: the 400 ms here might be 40 ms of hop plus 360 ms of filter, and cutting the filter is a structure change that costs nothing.
Record the measured θ/τ in the tuning record and require a structure decision above a declared threshold, rather than letting someone raise gain to hit a number.
Curated: · Written: · Reviewed:
Q&A-22A loop is well behaved at low flow and unstable at high flow. What would you do beyond retuning?
Two things make a loop behave that way, and they have different fixes. Either the process gain genuinely rises with flow, or the valve's installed characteristic makes loop gain rise with lift. Diagnose which before touching gains.
Assume a flow loop, PI control, and that the high-flow oscillation is a limit cycle at crossover rather than stiction or a mechanical fault.
If open-loop process gain varies by more than roughly 2:1 across the operating range, no single tune is right everywhere: a tune stable at high flow is sluggish at low flow, and one fast at low flow oscillates at high flow. Measure the gain curve, then schedule gains against a reliable scheduling variable, interpolating between breakpoints so the transition is bumpless.
A valve loop I worked on had 4.2x the process gain at 80% open versus 20%:
| Valve position | Process gain | Single tune result |
|---|---|---|
| 20% | 0.35 | sluggish |
| 50% | 0.9 | good |
| 80% | 1.47 | oscillates |
Gain scheduling is not the only fix, and often not the first one.
- Signal characterizer or equal-percentage trim. If the gain rise tracks valve lift, a linear-trim valve in a roughly constant-pressure-drop installation, a characterizer block or equal-percentage trim flattens loop gain with no scheduling logic and no scheduling variable to trust. Use this when the gain curve is monotonic and matches the valve's inherent characteristic.
- Dead time, not gain. If what changes with flow is transport delay rather than gain, scheduling gain does not fix phase lag. Detune to the worst case or change structure, e.g. feedforward from measured flow.
- Oversized valve. If the valve is oversized, the fix is a smaller trim. Scheduling around a bad valve keeps the bad valve.
When scheduling is the wrong fix: when the "gain change" is really stiction or hysteresis, when the scheduling variable is noisier than the disturbance it tracks, or when the operating point never actually moves, then tune once at that point.
Failure modes to watch: a noisy scheduling variable causes gain chatter; step interpolation bumps the output at a region boundary; a schedule can mask a sticking valve; and boundaries are where scheduling errors show up, so test at each breakpoint, not mid-region.
Before choosing between one tune and a schedule, measure open-loop process gain at several operating points and publish the curve.
Curated: · Written: · Reviewed:
Q&A-23A loop tuned on a 100 Hz controller behaves differently after a move to 500 Hz hardware with the same gains. Why?
The engineering content of sample rate and controller portability is the tolerance budget and the stop condition, not the algorithm name.
Integral and derivative terms are defined against time, and an implementation that accumulates per cycle rather than per elapsed second silently rescales them when the rate changes. The same numbers then mean five different things.
Concretely, scale integral and derivative by the measured elapsed interval rather than assuming a nominal one, record the design sample rate with the gains, and detect a missed deadline rather than integrating the same stale sample as though it were new.
The reason for that specificity is a failure I have seen: A port from 100 Hz to 500 Hz kept per-cycle accumulation; the effective integral gain rose fivefold and the axis overshot 22 percent on a move that had previously overshot 3 percent.
Same gains, two rates.
| Rate | Effective Ki | Overshoot |
|---|---|---|
| 100 Hz | 1.0x | 3% |
| 500 Hz, per cycle | 5.0x | 22% |
| 500 Hz, per second | 1.0x | 3% |
I would not consider it settled without evidence: Run the same step at two sample rates and require the response to match within a declared tolerance before accepting the port.
A gain without a time base is a number, not a tuning.
Curated: · Written: · Reviewed:
Q&A-15What do you do before the first automatic-mode enable on a new axis?
Before the first automatic enable I prove three things at the lowest energy the machine allows: feedback counts in the commanded direction, the motor drives in the commanded direction, and the protective chain stops motion without the controller's help. Only then do I close the loop.
Why it matters. A closed loop assumes negative feedback: a positive error produces actuation that reduces it. Reverse the encoder, the motor, or the sign convention in the configuration and the same loop becomes positive feedback. It then runs away at whatever rate the actuator can manage, and the controller's following-error fault typically trips only after the axis has already hit something.
The sign check, done in two isolated steps.
- Drive disabled, brake released, move the axis by hand a few millimetres in the + direction. Watch the raw encoder counts. They must increase. This tests the sensor alone, with no torque in the picture.
- Enable at minimum torque (say 5% of rated) and command a small + move at low velocity. The axis must move + and the counts must increase. This tests the motor and the sign convention together.
If either step disagrees, stop and fix wiring or configuration before going further.
Scaling check. Move a known distance and compare the encoder delta to the expected counts. A 20-bit encoder is 1,048,576 counts/rev; on a 10 mm/rev ball screw that is 104,857.6 counts/mm, so a commanded 10 mm should read about 1,048,576 counts. Half that means the gear ratio or resolution is wrong; the right magnitude with the wrong direction means the sign is flipped.
Limits armed before enable. Soft limits set inside the hard stops. Following-error window sized to the move, not left at the drive default. Velocity and torque limits at commissioning values. E-stop and protective stop tested with the drive enabled and the axis stationary, so you know the chain works before it has to.
First enable. Hand on the e-stop, smallest move the controller allows, watching the position trace and the following error together.
| Condition | Loop behaviour | Time to hard stop |
|---|---|---|
| correct sign | converges | n/a |
| reversed encoder | diverges | ~0.2 s |
| reversed motor | diverges | ~0.2 s |
The 0.2 s figure is from a rebuilt axis whose encoder cable was swapped during maintenance: the first automatic enable drove it to the hard stop in under 200 ms and bent the mechanism, with the following-error fault arriving after the impact.
Trade-off. A clean simulation proves the control law, not the wiring. Simulation and sign verification are different events, and only one of them is on the critical path to a safe first enable. Make the sign check an explicit, recorded commissioning step with a pass criterion, not something somebody remembers to do.
Curated: · Written: · Reviewed:
Q&A-28Would moving from a 17-bit to a 23-bit encoder improve your placement accuracy?
My answer to encoder resolution versus achievable precision begins at the failure mode: if I cannot name how it hurts someone, I do not have a design.
Resolution is the smallest change the device can represent. Accuracy is limited by whatever dominates the error budget, which on most machines is thermal growth, compliance, and kinematic parameter error rather than quantisation.
Concretely, build the error budget before buying resolution: list quantisation, bearing runout, thermal expansion, deflection under load, and calibration residual with measured magnitudes, and spend where the largest term is.
The reason for that specificity is a failure I have seen: An upgrade from 17-bit to 23-bit encoders cost a shift of downtime and moved placement error from 0.31 mm to 0.30 mm, because thermal growth across the 1.4 m frame contributed 0.24 mm and nobody had measured it.
Error budget at 1.4 m reach.
| Term | Contribution |
|---|---|
| thermal growth | 0.24 mm |
| compliance | 0.14 mm |
| kinematic residual | 0.11 mm |
| 17-bit quantisation | 0.01 mm |
I would not consider it settled without evidence: Publish the measured error budget with each term's magnitude before approving a resolution upgrade.
Resolution is cheap to quote and rarely the binding constraint.
Curated: · Written: · Reviewed:
Q&A-26Which sensor failures does a range check catch, and which does it miss?
The 60-second answer
A range check catches only faults whose signature leaves the configured band: open circuit and short circuit when the input rails or floats outside the band, saturation and clipping, gross mis-wiring, and any sensor that has failed hard to a rail. It misses every fault that produces a plausible number: stuck channels, slow drift, offset shifts inside the band, and increased noise. Those are the dangerous ones, because downstream logic treats them as valid.
Assumptions: the operating band is set strictly inside the sensing band, so a rail is distinguishable from a real reading, and the check runs on the digitised value.
Why range checks are structurally blind
A range check tests magnitude only. It has no memory and no statistics, so it cannot see a value that is wrong but stationary, and it cannot see a value that is right but noiseless. Three orthogonal checks cover the gap: range (magnitude), rate (first difference), variance (second moment over a window). Each catches a different signature, and none catches all of them.
| Failure signature | Range | Rate (Δ/Δt) | Variance (σ over window) |
|---|---|---|---|
| Open circuit, input rails to 0 V or full scale | catches | catches (step) | catches (σ→0) |
| Open circuit, input floats to mid-scale | misses | misses | catches |
| Short circuit to ground | catches | catches | catches |
| Stuck / frozen channel | misses | misses | catches |
| Slow drift (aging, fouling, thermal) | misses | misses | misses |
| Step offset shift inside the band | misses | catches | misses |
| Noise increase (loose shield, EMI) | misses | misses | catches |
| Intermittent dropout between samples | misses | misses | misses |
Worked trace: a stuck channel that passed everything
Setup: 4–20 mA transmitter, 0–100 °C, 250 Ω sense resistor, 16-bit ADC over 0–5 V.
- 4 mA × 250 Ω = 1.000 V → 0 °C; 20 mA × 250 Ω = 5.000 V → 100 °C
- Scale: 40 mV/°C. ADC LSB = 5 V / 65536 = 76.3 µV = 0.0019 °C
Open loop: the wire breaks, loop current falls to 0 mA, the input reads 0.000 V. Below the 1.000 V live-zero floor. The range check fires. This is the whole reason 4–20 mA uses a live zero: 0 mA is unambiguously a fault, not a valid reading.
Now the fault that got through. A firmware bug latches the last converted sample. The channel reads 42.6 °C for 90 minutes while the vessel climbs to 71 °C. Sixty samples at 1 Hz:
- Range: 42.6 ∈ [0, 100] → pass
- Rate: Δ = 0.0 °C/s against a 5 °C/s limit → pass
- Variance: mean 42.6, σ = 0.000 °C against a 0.01 °C floor → fires at t = 60 s
A live channel in the same rack shows σ ≈ 0.04 °C from transmitter noise plus ADC dither, so the threshold sits an order of magnitude below live noise and above a frozen channel. Detection latency equals the window length: 60 s here.
Failure modes of the checks themselves
A bare variance check false-positives on a genuinely isothermal process: a tank at equilibrium also has σ ≈ 0. Gate it on process state (only when the process is expected to move), or compare against a redundant channel, if two channels agree to 0.001 °C they are probably both stuck or cross-wired. A rate check false-positives on legitimate steps: valve slam, pump start, batch transition. And a range check set to the full sensor span catches nothing at all, because the operating band equals the sensing band.
Verification
Induce each declared fault physically (unplug the loop, short the input, freeze the value in firmware) and confirm the diagnostic fires. Reasoning that it would is not evidence.
Curated: · Written: · Reviewed:
Q&A-27A vibration signal shows a 3 Hz component that no mechanism can produce. What happened?
This is an area where a clean simulation and a correct handling of anti-alias filtering before sampling are not the same event.
Content above half the sample rate does not disappear when you sample; it folds into the band you are looking at and becomes indistinguishable from a real low-frequency signal. Digital filtering after conversion cannot separate them because the information is already gone.
Concretely, put an analogue filter ahead of the converter sized for the actual sample rate, sample fast enough for the genuine signal bandwidth, and verify with a swept input that out-of-band content is attenuated rather than folded.
The reason for that specificity is a failure I have seen: A 100 Hz accelerometer channel with no analogue filter aliased a 103 Hz bearing tone down to 3 Hz; a condition-monitoring alarm fired on a mechanism whose slowest natural frequency was 40 Hz.
Where 103 Hz lands at 100 Hz sampling.
| Input | Nyquist | Apparent frequency |
|---|---|---|
| 40 Hz | 50 Hz | 40 Hz |
| 103 Hz | 50 Hz | 3 Hz |
| 197 Hz | 50 Hz | 3 Hz |
I would not consider it settled without evidence: Sweep a known input across the band and above it, and confirm the recorded spectrum shows attenuation rather than mirrored content.
Aliasing is not noise, it is a wrong answer that looks like a signal.
Curated: · Written: · Reviewed:
Q&A-71An analogue sensor reads correctly on the bench and shows 50 Hz noise in the cell. What would you change?
First 60 seconds. 50 Hz is the mains frequency, so the coupling is almost certainly not radiated RF: it is magnetic induction into the signal loop, or a ground loop between two points that are both nominally "ground" but sit at different potentials. Before changing hardware I would confirm three things. Is 50 Hz the mains or the process? A vibration sensor on a 50 Hz machine is reading signal, not noise, and the fix is a different question. Does the hum disappear when the drives are off but the DAQ stays powered? That separates conducted/radiated pickup from a ground loop. Does it disappear when the sensor is unplugged and the input shorted at the connector? That tells you whether the loop is in the wiring or in the sensor's own return.
The distinction that decides the fix. A ground loop injects a common-mode voltage; a differential front end rejects it. Magnetic induction around the loop injects a series voltage in the signal path; no amount of CMRR helps, only shrinking the enclosed area does. 50 Hz with strong 150/250 Hz harmonics points at a ground loop or a mains transformer; a clean 50 Hz with no harmonics points at magnetic coupling from a mains-frequency current.
What I would change, in order of effect:
- Break the ground loop. Ground the signal reference at one point only. If the sensor is chassis-bonded at the machine and the DAQ is bonded at the cabinet, the shield or the return carries the difference current. Isolate the sensor with a loop-powered 4–20 mA transmitter or an isolated amplifier, and the loop current stops.
- Shrink the loop area. Twist the signal pair, run the return alongside the signal, keep the pair away from mains and drive cables, and cross other cables at 90°. Magnetic pickup is proportional to enclosed area, so twisting is often the single biggest win.
- Go differential with a twisted pair. CMRR is quoted at DC and degrades with frequency and with source-impedance imbalance: a few hundred ohms of imbalance in one leg can cost tens of dB of rejection at 50 Hz.
- Terminate the shield at one end, at the source end, to chassis. Both ends makes the shield the ground loop.
- Only then filter. A 50 Hz notch hides the coupling, rings, and does nothing for the 150 Hz harmonic. If you must do it digitally, average over an integer number of 20 ms mains periods: that nulls 50 Hz and every harmonic exactly.
Worked example. A 6 m single-ended type-K thermocouple run alongside a servo cable, measured with the drives energised. Type-K sensitivity is 41 µV/K, so the hum translates to apparent temperature like this:
| Wiring | Length | 50 Hz hum | Apparent ΔT |
|---|---|---|---|
| single-ended | 6 m | 40 mV | 976 K |
| differential, twisted | 6 m | 1.2 mV | 29 K |
| digital at sensor | 6 m | 0 | 0 K |
The bench test used a 300 mm lead, which is why it looked clean: the loop area and the common-mode path were both a fraction of the installed ones.
Verification. Measure with the drives energised and moving, at the production cable length and routing, using a battery-powered scope or a differential probe so the instrument does not add its own ground loop. The bench test measured a different cable.
Curated: · Written: · Reviewed:
Q&A-60A wrist force sensor reads 14 N with nothing touching the tool. What do you do before using the number?
Before letting it move at speed I would write down what a wrong result for force-torque sensor bias and tool weight looks like on the log.
A wrist sensor measures everything distal to it, which includes the tool's own weight projected through the current orientation, plus sensor bias and temperature drift. The contact force is what remains after those are removed.
Concretely, compensate tool weight using the identified mass and centre of mass at the current orientation, re-zero bias at a known no-contact pose rather than at power-on only, and monitor drift so a slow thermal shift does not become a phantom contact.
The reason for that specificity is a failure I have seen: An uncompensated 1.4 kg tool produced up to 14 N of apparent force as the wrist rotated; a 10 N contact threshold triggered on orientation alone and the robot reported contact in mid-air on every fourth approach.
Apparent force by wrist orientation, no contact.
| Orientation | Raw | Compensated |
|---|---|---|
| tool down | 14.0 N | 0.3 N |
| tool horizontal | 6.8 N | 0.2 N |
| tool up | -13.6 N | 0.4 N |
I would not consider it settled without evidence: Sweep the wrist through its orientation range with no contact and require the compensated reading to stay inside a declared band.
A force reading is a sum, and the contact is only one term.
Curated: · Written: · Reviewed:
Q&A-24How much does 10 ms of unaccounted camera latency cost a robot moving at 2 m/s, and what do you do about it?
20 mm. 10 ms at 2 m/s is 0.010 s × 2 m/s = 0.02 m. The error is systematic, always along the direction of travel, so no filter removes it. Only a time correction or a slower platform does.
Whether 20 mm matters is a tolerance question. On a 200 mm-wide grasp it is noise. On a pallet edge with 15 mm of fork clearance it is a mis-pick.
Mechanism: latency converts to distance through the platform's own motion, error = v × Δt, linear in both. Split it into bias (constant, compensable with an offset) and jitter (varies frame to frame, not compensable with an offset: you have to reduce it).
| Latency | Error at 2 m/s | Typical consequence |
|---|---|---|
| 5 ms | 10 mm | inside most grasp tolerances |
| 10 ms | 20 mm | marginal for tight insertion |
| 34 ms | 68 mm | mis-pick |
| 100 ms | 200 mm | collision risk |
What to do, and when each is the right call:
| Remedy | Fixes | Cost | Choose when |
|---|---|---|---|
| Hardware timestamping (capture interrupt, device clock, trigger line) | the unknown in the stamp | sensor/driver support, wiring | the sensor and host support it and the delay is stable |
| Software latency estimation (measure end-to-end, subtract a constant) | constant bias | no hardware change, but invalidated by any pipeline change | the pipeline is frozen and bias dominates |
| Slow the platform | scales error linearly with speed | cycle time | tolerance cannot be met otherwise and throughput allows |
| Widen tolerance / accept | nothing | none | 20 mm is inside the task tolerance |
Criteria, in order: measure first. If bias dominates and is stable, estimate it and subtract. If jitter dominates, timestamping plus shorter exposure and a hardware trigger. If neither gets you inside tolerance, slow down: halving speed halves the error.
The case for not compensating: if the driver already stamps at capture, subtracting again double-counts. The signature is an error that flips sign and roughly doubles after you "fix" it. And if the error is already inside tolerance, compensation adds a calibration that drifts and a new failure mode for no gain.
Measure end-to-end latency with a physical event visible to both the camera and an independent trigger (an LED strobed at a known time, or a target on a known-velocity axis) and compare the timestamp the pipeline reports against the trigger time. Do not trust the driver's stated figure.
# sketch: transform at sample time, not arrival time
T_world_target = lookup(cam_frame, t_capture) @ T_cam_target
# carry the stamp uncertainty instead of dropping it
sigma_pos = v * sigma_t
At 2 m/s a 1 ms stamp uncertainty is 2 mm of position uncertainty. That is the number to put next to the 20 mm.
Curated: · Written: · Reviewed:
Q&A-25Two sensors both report good data and the fused result is wrong. How do you establish whether time is the problem?
The 90-second answer
Time is the problem if the fused error scales with relative motion and vanishes when the platform is still. Sensor noise doesn't care whether you're turning; a clock offset does. So the test is: hold still, then move, and compare.
Three checks, cheapest first:
-
Static hold. Park the platform. If the fused estimate is clean when stationary and wrong only under motion, the error is motion-coupled: time skew or a calibration error, not noise. This costs nothing and eliminates most alternatives.
-
Lag search. Cross-correlate the two sensors' signals over a range of candidate lags and find the lag that minimises the residual. A clear minimum at nonzero lag is the skew.
-
Injected event. A sharp motion or a flash both sensors observe. This measures the offset directly and is the ground truth the lag search is checked against. Without it you are inferring.
Worked trace: lag search on a step
Both sensors sample at 100 Hz. A step (a sharp motion) enters sensor A at sample 100 and sensor B at sample 103: B's clock is 3 samples behind. For candidate lag k, compute Σ(A[n] − B[n+k])²:
| k | samples where A and B disagree | residual |
|---|---|---|
| 0 | 3 | 3.0 |
| 1 | 2 | 2.0 |
| 2 | 1 | 1.0 |
| 3 | 0 | 0.0 |
| 4 | 1 | 1.0 |
| 6 | 3 | 3.0 |
Argmin at k=3 → 30 ms of skew. The residual is V-shaped, so the minimum is unambiguous. If the residual is flat, the signals aren't correlated enough to estimate a lag: you need more excitation, not a better algorithm.
Labelled example: at 30 °/s yaw, an 18 ms offset pairs samples 0.54° apart in attitude (30 °/s × 0.018 s). If the yaw rate itself oscillates, the fused attitude oscillates at the yaw rate, which is why this gets misattributed to a sensor bias.
Once you know it's time: what to do, and what each costs
| Approach | Cost | Accuracy | Worth it when |
|---|---|---|---|
| Hardware sync: PTP/gPTP, shared trigger, PPS | PTP-capable NIC and switch, a grandmaster, days of integration | sub-µs to tens of µs | Residual budget under ~1 ms, or the platform can't be excited (static scenes, slow motion) |
| Software timestamping: driver/kernel arrival stamps | Nearly free; needs a monotonic clock and a low-jitter path | tens of µs idle, milliseconds under CPU contention | Single host, low-rate sensors, and you can log the jitter distribution |
| Estimation-based alignment: time offset as a filter state | One more state, needs excitation and observability, tuning | converges to ~ms with good motion; unobservable when static | Hardware sync isn't possible and the platform always moves; retrofitting an existing rig |
Failure modes
- Estimation-based alignment absorbs a real fault. If sensor B has a bias, the offset state drifts to compensate and the bias becomes invisible. Detect it by checking the estimated offset when the platform is static: a true clock offset stays put, a bias-compensating one wanders.
- Software timestamps measure arrival, not sample instant. Under CPU contention the gap between them jitters by milliseconds. Log both and look at the distribution, not the mean.
- PTP falls back silently. If the grandmaster drops, the clocks free-run and the offset grows without an alarm. Monitor the PTP offset and sync state, and alarm on it.
The reason to run the injected-event test even when the lag search looks clean: the lag search assumes both sensors are reporting the same signal. If one is subtly wrong (a scale error, a dropped frame) the lag search will find a lag that isn't there. The injected event doesn't care.
Curated: · Written: · Reviewed:
Q&A-87Perception runs at 10 Hz and control at 1 kHz. How do they share data?
The short answer. Perception publishes a complete, immutable snapshot and swaps it in with one atomic index exchange. The 1 kHz loop reads the newest snapshot without taking a lock and without waiting. The snapshot carries a timestamp so control can tell fresh from stale. Nothing on the fast side ever blocks on the slow side.
Why not a mutex. A mutex makes the control loop's worst-case latency a function of the perception thread's critical section and of whether that thread got preempted while holding it. That is priority inversion: the 1 kHz thread waits on the 10 Hz thread. Worked example (figures from one arm controller, not a general benchmark): perception holds the lock 1.4 ms while copying pose, covariance and a point-cloud header on a loaded CPU. The control loop's period is 1 ms, so it misses a deadline on every perception cycle: 10 missed deadlines per second. On a robot arm that surfaces as a 10 Hz vibration, not as an error message.
The handoff.
// C++17. Single writer (perception, 10 Hz), single reader (control, 1 kHz).
// std::atomic<uint32_t> is lock-free on x86-64 and ARM64.
struct Snapshot {
uint64_t t_ns; // CLOCK_MONOTONIC at capture
uint32_t seq; // increments per publish; catches skipped frames
Pose pose;
Cov6 cov;
};
class LatestSnapshot {
Snapshot buf_[3];
std::atomic<uint32_t> ready_{0}; // index of the newest complete snapshot
uint32_t writer_idx_ = 1; // writer-private
uint32_t reader_idx_ = 2; // reader-private
public:
// writer side (perception thread)
Snapshot& write_slot() { return buf_[writer_idx_]; }
void publish() {
// swap writer-private with ready; old ready becomes our scratch buffer
writer_idx_ = ready_.exchange(writer_idx_, std::memory_order_acq_rel);
}
// reader side (control thread); reference valid until the next acquire()
const Snapshot& acquire() {
reader_idx_ = ready_.exchange(reader_idx_, std::memory_order_acq_rel);
return buf_[reader_idx_];
}
};
Trace it. Start ready_=0, writer_idx_=1, reader_idx_=2. Writer fills buf_[1], publishes: ready_=1, writer_idx_=0. Reader acquires: ready_=2, reader_idx_=1, reads buf_[1]. Writer fills buf_[0], publishes: ready_=0, writer_idx_=2. Reader acquires: ready_=1, reader_idx_=0, reads buf_[0]. Each operation is a swap between one private slot and the shared ready_ slot, so {writer private, ready, reader private} is always a permutation of {0,1,2}: the writer never touches the buffer the reader is reading. Both exchanges are RMWs on the same atomic, so the release sequence carries the writer's buffer writes to the reader's acquire. If the writer hasn't published, the reader gets its own buffer back and re-reads the same snapshot; that is why the timestamp is not optional.
Choosing the mechanism.
| Mechanism | Reader blocks? | Torn read? | Delivers every frame? |
|---|---|---|---|
| mutex around a shared struct | yes, up to the writer's critical section | no | yes |
| unguarded shared struct | no | yes | yes |
| seqlock (seq counter, reader retries) | no | no | yes |
| triple buffer + atomic index swap | no | no | no: reader may see the same snapshot twice |
| SPSC ring buffer | no, unless full | no | yes, up to capacity |
Use the triple buffer for "latest state": pose, joint state, filtered attitude. Use an SPSC ring when the consumer must see every measurement: a logger, or an estimator that integrates each IMU sample. At 10 Hz into a 1 kHz consumer, a ring of 8 holds 800 ms of backlog. A seqlock is fine for small payloads and never blocks the writer, but the reader can retry indefinitely under a fast writer; at 10 Hz against 1 kHz that risk is negligible.
Failure modes and how they show up.
- Torn read. Pose from frame N with covariance from N+1. Detect with the
seqfield or a checksum inside the snapshot. - Use-after-free. If you swap raw pointers and delete the old buffer, the reader may still be inside it. The triple buffer never allocates. If you use RCU or epoch reclamation instead, the grace period must exceed the reader's longest critical section.
- Silent staleness. Perception dies and control keeps using the last snapshot forever. Check
now - t_nson every read; at 10 Hz one missed cycle is 200 ms of age. Hold the last command or go to a safe state past a threshold you pick. - Clock domain mismatch. Perception stamps with the sensor clock, control compares against its own, and the age comes out negative or absurd. Use one monotonic clock for both sides.
- Priority inversion. The mutex case. Detect it by measuring control-loop wakeup jitter, not by reading the code: jitter spikes that recur at exactly the perception rate are the tell.
Curated: · Written: · Reviewed:
Q&A-29Why does dead reckoning from an IMU alone fail, and how fast?
I would treat IMU bias and integration drift as a claim about the physical world that has to survive a bench test.
Accelerometer bias integrates twice into position, so a constant offset becomes an error growing with the square of time. Gyro bias integrates once into attitude, which then tilts the gravity projection and feeds the accelerometer error, so the two compound.
Concretely, estimate bias online with an aiding source: wheel odometry, GNSS, a visual fix, or a zero-velocity update when the platform is known to be stationary, and treat unaided integration as usable only across the interval where the growing error stays inside the tolerance.
The reason for that specificity is a failure I have seen: A 0.02 m/s² residual accelerometer bias produced 9.0 m of position error after 30 seconds of tunnel travel without aiding, which put the vehicle in the wrong lane on re-acquisition.
Unaided drift from 0.02 m/s² bias.
| Elapsed | Position error |
|---|---|
| 5 s | 0.25 m |
| 15 s | 2.3 m |
| 30 s | 9.0 m |
I would not consider it settled without evidence: Measure position error against unaided duration on the actual device and publish the curve that says how long the estimate is usable.
An IMU tells you about change, and change integrates.
Curated: · Written: · Reviewed:
Q&A-30Wheel odometry is accurate on the test floor and drifts badly in the warehouse. What changed?
What changed
Three things, and the third is the one that bites.
- Surface. The test floor is usually smooth epoxy or sealed concrete. Warehouse aisles are worn concrete with expansion joints, grit, cardboard, stretch wrap and the occasional oil patch. Rolling resistance and available friction both change, so the wheel slips a little more and the effective rolling radius changes with it.
- Load. Test runs are often unloaded or lightly loaded. A full pallet deflects the tyre, which shortens the effective rolling radius: the wheel turns the same amount but travels less. It also shifts normal force between the drive wheels, so left and right no longer share the same effective radius.
- Distance. This is what turns a small error into a miss. Odometry integrates. A scale error grows linearly with distance; a heading error grows with distance squared. Over a 5 m test path neither is visible. Over a 60 m aisle both are.
Mechanism
Differential drive: per wheel, Δs = r_eff · Δθ, and yaw from Δψ = (Δs_r − Δs_l) / b.
Three error terms fall out of that:
- Scale: r_eff is wrong, so along-track error grows linearly with distance.
- Yaw bias: r_eff differs left vs right, or b is wrong, so heading error grows linearly with distance and lateral error grows with distance squared.
- Slip: the encoder turns and the wheel doesn't. This is not zero-mean noise; it is biased in the direction of the manoeuvre, so it accumulates during hard acceleration and tight turns.
Worked example, numbers illustrative: b = 0.5 m, v = 1.5 m/s, and the left and right effective radii differ by 0.5% (one tyre 0.2 bar low, or more weight on one side). Yaw bias ω = v · (Δr/r) / b = 1.5 × 0.005 / 0.5 = 0.015 rad/s ≈ 0.86°/s. Over 40 s, 60 m at that speed, heading is off 0.6 rad ≈ 34°, and lateral error ≈ ½ · ω · t² = ½ × 0.015 × 1600 = 12 m. The robot is in the wrong aisle, not the wrong bay.
| Error source | Grows as | 5 m test path | 60 m aisle |
|---|---|---|---|
| Scale 0.5% | linear | 25 mm | 300 mm |
| Yaw bias 0.05°/s | quadratic | 5 mm | 0.7 m |
| Yaw bias 0.86°/s | quadratic | 83 mm | 12 m |
Telling them apart
- Bidirectional square path, driven both ways and averaged: cancels the slip bias and leaves scale and yaw bias. Run it loaded, on the warehouse floor.
- Straight-line test at constant speed: any heading change on a straight line is a yaw bias, which points at wheel radius mismatch or a wrong track width.
- Slip: compare wheel odometry against an IMU yaw rate or a scan match. A step in the residual during acceleration or a turn is slip, not noise. Motor current against expected torque confirms it.
What to do
Calibrate per-wheel effective radius in the loaded condition on the real floor, not the test floor. Fuse with an independent heading source: odometry alone cannot bound heading drift, because the error is quadratic in distance. And detect slip rather than averaging it out.
If you can only do one thing: measure the left and right effective radii loaded, on the warehouse floor. That single number is what turns a 12 m error into 300 mm.
Curated: · Written: · Reviewed:
Q&A-36Should a robot keep driving when its localisation covariance grows? What would you do with the number?
Yes, but slower, and only while the uncertainty still fits inside the free space you have. Covariance is not a stop signal by itself; it is the input to a speed limit. Stop when the uncertainty consumes the clearance you need to brake.
Assumptions I'd state before answering: 2D indoor AMR, map-frame pose from a lidar/odometry EKF or particle filter, a known obstacle map, and a braking model. If any of those is missing, the number can't be gated on.
The gate is arithmetic. Required clearance to stop from speed v is:
d(v) = v·t_lat + v²/(2a) + k·σ
where t_lat is sensing-to-brake latency, a is deceleration, σ is the 1σ position uncertainty along the direction of travel, and k is the safety factor (3 is a common choice). Pick the largest v such that d(v) is less than the distance to the nearest mapped obstacle along the braking path. If no positive v satisfies it, stop and re-localise.
Worked example: t_lat = 0.2 s, a = 2 m/s², k = 3, σ = 0.35 m.
- v = 1.5 m/s: d = 0.30 + 0.56 + 1.05 = 1.91 m
- v = 0.6 m/s: d = 0.12 + 0.09 + 1.05 = 1.26 m
- v = 0.2 m/s: d = 0.04 + 0.01 + 1.05 = 1.10 m
At σ = 0.35 m the uncertainty term dominates; slowing from 1.5 to 0.2 m/s buys 0.8 m, and the floor is 1.05 m. That is the point where re-localising beats creeping.
| 1σ uncertainty | Permitted speed | Required clearance d(v) |
|---|---|---|
| < 0.10 m | 1.5 m/s | ~1.0 m |
| 0.10–0.35 m | 0.6 m/s | ~1.3 m |
| > 0.35 m | re-localise | floor exceeds typical free space |
Failure mode: covariance is the filter's self-report, and it lies when the model is wrong. A scan matcher in a symmetric corridor can lock onto the wrong repeat and report small covariance while the pose drifts. Detect it with a consistency test: the normalized estimation error squared (NEES) for a 3-DoF pose should average 3 across runs if the filter is calibrated; if it averages 12, the filter is roughly 4× overconfident and the gate is unsafe. Compare reported σ against measured error from ground truth or a motion-capture run before trusting the table.
Alternatives: some teams gate on scan-match inlier count, or on time since last loop closure. Those are proxies; covariance is better because it fuses odometry, scan matching and IMU into one continuous number, but only if the consistency test passes.
What I'd do with the number, in order: convert to 1σ along travel (or take the largest eigenvalue of the position block), compute d(v), pick the largest safe v, log σ, v and the margin, and alarm when σ crosses the re-localise bound. An uncertainty nobody reads is a comment, not a safeguard.
Curated: · Written: · Reviewed:
Q&A-90A square fiducial's estimated pose flips between two orientations at long range. What causes it?
What a strong candidate says first
A square planar fiducial has a two-fold pose ambiguity, and it is geometric, not a detector bug. Four coplanar corners give a projection that barely changes when you flip the marker about its in-plane axis: one pose tilts the top edge toward the camera, the other tilts it away, and both land on almost the same pixels. The only thing separating them is perspective foreshortening, the trapezoid distortion of the square, and that cue shrinks as the marker gets smaller in the image. At long range the marker covers a handful of pixels, the cue drops below corner-detection noise, and the solver returns whichever of the two poses has marginally lower reprojection error. That choice is not stable frame to frame, so the estimate flips.
Why it behaves that way
PnP over four coplanar points has a residual surface with two shallow minima rather than one. The gap between them scales roughly with the apparent size of the marker, so the first fix is to make the perspective cue measurable again: a larger marker, a closer standoff, more sensor resolution, or a longer focal length. A second, independent fix is to break the planarity: mount several markers rigidly on one plate and solve them jointly. The combined geometry is no longer a single plane, the two-fold ambiguity of each marker is resolved by the others, and the joint minimum is unique.
What to do about it
- Increase apparent marker size until the residual gap clears the noise floor.
- Use a rigid multi-marker target and solve the whole set in one PnP.
- Reject, don't average: compute both candidate solutions and compare their residuals. If the ratio falls inside a declared margin, drop the detection rather than taking the marginally better pose.
- Temporal smoothing alone does not fix it. The flip is a genuine bimodal measurement, and a filter will lag or oscillate across it.
Failure mode and how you see it
The flip reverses the estimated heading, so a controller steering on marker pose commands the opposite direction on those frames: to an operator it looks like a random jerk. Detect it by logging the residual ratio between the two candidate poses per detection and counting flips per second; a healthy detection sits well above the margin.
Worked example
On one rig, a single 60 mm marker at 2.4 m subtended about 22 px and flipped on roughly one frame in eight. Scaling the marker to 150 mm at the same range put it at 55 px and dropped the flip rate to 0.4%; four 60 mm markers on a rigid plate, still 22 px each, eliminated it. The figures are illustrative of the trend rather than a spec: the point is that the flip rate tracks apparent size and rigidity, not the solver.
Curated: · Written: · Reviewed:
Q&A-33A checkerboard calibration reports 0.2 pixel reprojection error and vision-guided placement is still poor at the edges. What went wrong?
0.2 px reprojection error is the residual of the fit on the images you gave it. It is not a bound on error anywhere else in the frame. Reprojection error is normally reported as an RMS over all detected corners, so a set that sat in the middle of the sensor at one distance barely constrains the distortion terms there, and the model extrapolates badly toward the corners. Placement error at the edge is that undistortion error times the mm-per-pixel scale at the working distance.
Confirm it in an afternoon: hold out a third of the images, fit on the rest, and plot residual against distance from the principal point. A flat fitted curve with a holdout curve that lifts at the edges means coverage, not fit quality.
A set that filled the central third of the frame gave 0.2 px residual and 3.1 px of undistortion error at the corners. At that working distance the scale was about 0.77 mm/px, so 3.1 px was 2.4 mm of placement error on parts presented near the edge of the field.
| Region | Fitted residual | Holdout error |
|---|---|---|
| centre | 0.18 px | 0.2 px |
| mid | 0.21 px | 0.9 px |
| corner | 0.24 px | 3.1 px |
Four fixes, each wrong in a specific situation.
Recalibrate with full-frame coverage. Target fills the frame at every pose, distance and tilt varied across the working range, holdout images kept out of the fit. This is the default and usually the right one. It fails when the cell cannot present the board at the corners, if the arm cannot reach, or the target is too small for the working distance, you cannot buy coverage.
Change the distortion model. The 5-parameter model (k1, k2, p1, p2, k3) is fine for moderate fields of view; wide lenses often need the rational model (k1..k6) or a fisheye/equidistant model. In OpenCV 4.x that is CALIB_RATIONAL_MODEL on calibrateCamera, or fisheye::calibrate for the equidistant model. More parameters fit the same data better, which is the trap: accept the change only if holdout error at the corners drops.
Split the field. Two calibrations, or a local homography per region, removes the extrapolation. It introduces a seam and a rule for which model owns a point near the boundary, so it is worth it only when the lens is genuinely non-parametric at the edge and coverage is impossible.
Restrict the workspace. Keep placements inside the region where holdout error is under tolerance. Cheapest, and it costs field of view: you may need to move the camera back or add a second station.
Rule out the non-calibration causes first. Extrinsic or hand-eye error gives placement error that is roughly uniform across the frame, not one that grows with radius. A warped target, paper on a plate that is not flat, biases the fit. Motion blur and rolling shutter smear the corners. Thermal drift moves the extrinsic between calibration and production.
Report holdout error at the worst point in the working volume, not the mean residual.
Curated: · Written: · Reviewed:
Q&A-89Adding a second camera did not improve pose estimation. Why might that be?
The short answer
A second camera adds constraints, not information. If its viewpoint is close to the first one's, it re-answers the question the first camera already answered and leaves the weak direction weak. Three causes, in the order I usually find them:
- Baseline too small for the working distance. The new rays are nearly parallel to the old ones, so depth stays ill-conditioned.
- Correlated errors. Same bracket, same clock, same lighting, same occluder: fusion shrinks the reported covariance without shrinking the real error.
- The second stream never reaches the estimator, or reaches it with uncalibrated extrinsics or a bad timestamp.
Why geometry, not count
Triangulation depth uncertainty is
σ_z ≈ z² · σ_d / (f · B)
with z the range, σ_d the disparity noise in pixels, f the focal length in pixels, B the baseline. The gain is linear in B and quadratic in z, so a small baseline at long range buys almost nothing.
Worked example: z = 3 m, f = 800 px, σ_d = 0.2 px, and a monocular depth head that already gives 12 mm at this range:
B = 0.12 m:σ_z = 9 × 0.2 / (800 × 0.12) = 18.8 mm. Worse than the monocular prior.B = 0.9 m:σ_z = 9 × 0.2 / (800 × 0.9) = 2.5 mm. 7.5× better.
Fusing by inverse variance against the 12 mm prior:
| Axis | One camera | +2nd at 120 mm | +2nd at 900 mm |
|---|---|---|---|
| lateral | 0.9 mm | 0.7 mm | 0.6 mm |
| vertical | 1.0 mm | 0.8 mm | 0.7 mm |
| depth | 12 mm | 10 mm | 2.4 mm |
At 120 mm the depth column barely moves, because the new measurement is noisier than what you already had. Lateral and vertical were never the problem.
Why the metric lied
Pooled reprojection residual is dominated by the lateral and vertical axes, which were already well constrained. A 4% residual improvement with depth uncertainty stuck at ~11 mm is exactly what a small-baseline pair looks like. The residual is blind to the axis you care about.
Failure modes and how to catch them
- Correlated noise. Both cameras on one bracket share vibration and calibration drift; both behind the same IR filter or window share glare; both blinded by the same specular highlight or the same occluder fail together. Fusion assuming independence shrinks covariance by ~√2 while true error does not. Detect by comparing the fused covariance against empirical error on a held-out set, if empirical ≫ predicted, the independence assumption is wrong.
- Extrinsics. A 0.5° rotation error across a 900 mm baseline is ~8 mm of position error at the sensor, which the estimator reads as depth bias. Re-calibrate and check the residual on a known target.
- Time sync. Rolling shutter plus a 5 ms offset at 1 m/s is 5 mm of apparent motion, again read as depth. Verify with a moving target and a hardware trigger.
- Not actually fused. Wrong topic, dropped frames, or a monocular-trained model whose fusion layer weights the second stream near zero.
The test that settles it
Occlude the second camera and re-run. If the estimate does not degrade, it was never contributing. Then report per-axis uncertainty, or the smallest eigenvalue of the information matrix Jᵀ Σ⁻¹ J, before and after, not a pooled residual.
Curated: · Written: · Reviewed:
Q&A-32Why does a stereo depth camera get worse with distance faster than a time-of-flight one, and what does that mean for bin picking?
The short answer. Stereo recovers depth by triangulation, Z = f·b/d, so a fixed disparity error σ_d produces a depth error σ_Z = Z²·σ_d/(f·b): quadratic in range. Time-of-flight recovers depth from the phase or timing of modulated light; in the shot-noise-limited regime received power falls as 1/Z², phase noise grows as Z, and depth error grows roughly linearly. Double the distance and stereo error quadruples while ToF error only doubles. For bin picking that makes mounting height and baseline the design levers, and the grasp tolerance the budget: a depth spec quoted at one range says nothing about the range you will actually mount at.
Why the exponents differ. Differentiating Z = f·b/d gives dZ/dd = -Z²/(f·b), so stereo error is set entirely by how well the matcher localises a feature in the two images. That σ_d is a floor, not a constant: it degrades with weak texture, motion blur and specular highlights, so real stereo error grows faster than the quadratic on hard parts. ToF error is set by phase noise: σ_Z = (c/4πf_mod)·σ_φ. Ambient light adds a range-independent noise floor, and the unambiguous range is c/(2f_mod): 1.5 m at 100 MHz, so anything past that needs multiple modulation frequencies.
Illustrative arithmetic (hypothetical, stated assumptions): stereo with f = 1100 px, b = 60 mm, σ_d = 0.2 px; ToF at f_mod = 100 MHz with σ_φ = 0.01 rad at 1 m, scaling linearly with range.
| Range | Stereo σ_Z | ToF σ_Z |
|---|---|---|
| 0.5 m | 0.8 mm | 1.2 mm |
| 1.0 m | 3.0 mm | 2.4 mm |
| 1.4 m | 5.9 mm | 3.3 mm |
| 2.0 m | 12.1 mm | 4.8 mm |
The crossover sits near 0.8 m. Below it stereo is the better sensor; above it ToF pulls away, and by 2 m it is roughly 2.5× more accurate. The 2 m ToF row assumes the phase is unwrapped.
Failure modes, which are not the same as noise. Passive stereo on shiny metal returns no disparity at all (holes, not Gaussian error) and specular highlights produce gross outliers metres off. Left-right consistency checks and disparity confidence maps detect both. ToF on a concave bin gets multipath: light bounces off the walls and arrives with extra path length, biasing depth positively by centimetres, worst in the corners. Mixed pixels at part edges give flying pixels. A flat target at known range, or a multi-frequency capture, exposes the multipath bias.
What it means for bin picking.
- Budget
σ_Z ≤ tolerance/3. A parallel-jaw gripper with 2 mm clearance per side needs sub-millimetre depth; a suction cup tolerates several millimetres. - The worst case is the bottom of the bin, not the rim. A 0.5 m deep bin with the camera at 1.4 m puts the deepest pick at 1.9 m: about 11 mm of stereo error on the numbers above.
- Mounting height is the strongest lever: halving it quarters stereo error. Baseline is the second: doubling
bhalves error but worsens occlusion shadowing near the bin walls and raises the minimum working distance. - If the parts are shiny or the bin is deep, passive stereo is the wrong tool regardless of range. Active stereo or structured light projects texture and restores
σ_d; multi-frequency ToF rejects multipath. - Validate by sweeping a target through the actual working range with the bin in place, on the actual parts, and publishing the curve next to the mounting-height decision.
A depth accuracy figure without a range is half a specification.
Curated: · Written: · Reviewed:
Q&A-31A lidar-based safety function misses a dark object at a range where it detects a white one. Is that a fault?
Not a fault, unless the specification promised a range it cannot hold.
A lidar safety function that misses a dark object at a range where it sees a white one is behaving exactly as the physics dictates. Time-of-flight lidar fires a pulse and thresholds the returned energy. Returned energy scales with target reflectivity and falls as 1/R². A dark object returns a small fraction of what a white one does, so the range at which the return crosses the detector threshold is shorter. The sensor is not broken; the pair (sensor, target) simply has a shorter range.
It becomes a fault at the point where the safety function's specification, protective field, or safety distance was derived from a range measured on a cooperative target. Then the fault is in the specification and the validation, not the optics.
Measured example from a safety scanner, instrumented run, same unit, same day:
| Target reflectivity | Detection range |
|---|---|
| 90% | 4.1 m |
| 18% | 3.4 m |
| 1.8% | 2.3 m |
The 1.8% body is the low-reflectivity test body safety standards specify for exactly this reason. The curve is flatter than pure √ρ would predict (threshold effects, ambient light and pulse integration all bend it) which is why you measure it rather than model it.
Now the arithmetic that matters. The protective field had been laid out on the 4 m figure. For the dark object the real detection range is 2.3 m, so 1.7 m of the assumed field is unmonitored. At a 1.5 m/s approach that is 1.13 s of travel with no detection. If the safety distance calculation assumed 4 m of detection, the machine stops short of the object only by luck.
Trade-offs, because "always derate" is not free:
- Radar fusion. Radar returns from metal and from many dark dielectrics that lidar loses, and it is not blinded by the same conditions. Cost: coarser angular resolution, more clutter, and a second sensor with its own failure modes to validate.
- Retroreflective targets on the object. Cheap and effective, but only if you control every object that can enter the field. You usually do not.
- Conservative derating. Specify the field against the low-reflectivity body, shorten the field, or cap speed. This is the safe default and it costs throughput.
- Higher-power or multi-echo lidar. Buys range on dark targets, but you re-measure the curve; you do not assume it.
When the conservative choice is wrong: if the cell is genuinely closed and every object that can enter is verified high-reflectivity, derating the field throws away throughput for a hazard that cannot occur. That is a real engineering call, and it needs the verification written down, not assumed.
Failure modes and how you catch them:
- Datasheet max range used as the field radius. Detect by comparing the field layout document against the measured reflectivity curve.
- Validation with a white board or a hi-vis vest. Hi-vis is retroreflective and returns far more than dark clothing or skin: a validation that passes with a vest can fail with a person in a dark coat. Detect by running the standard low-reflectivity test body across the whole field, at the worst-case angle, and logging return intensity per target.
- Ambient light or rain degrading the dark-target return further. Detect by repeating the run under the worst conditions the cell sees.
So: not a fault in the sensor. A fault in any document that claims a single range number without a reflectivity qualifier.
Curated: · Written: · Reviewed:
Q&A-34Detection accuracy falls when line speed rises even though the model is unchanged. What is the first thing to check?
First thing to check: the exposure time actually used per frame against the line speed. I.e. how many pixels of motion blur the sensor integrated. Not the model, not the confidence threshold, not the training set. If the model is unchanged and accuracy tracks speed, the input distribution changed, and the cheapest explanation is that each frame now smears the part across more pixels.
The 60-second version
Blur length in pixels is set by three numbers you already have:
blur_px = (line_speed_m_per_s * exposure_s) / pixel_footprint_m_per_px
Pull the exposure from the frame metadata (not the configured value: auto-exposure will lengthen it when the part spends less time under the light), pull the speed from the encoder, and compute blur. If blur scales linearly with speed and recall falls with it, you have your answer before touching the model.
Why it breaks the detector
A detector trained on sharp edges has learned features at a specific spatial scale. Motion blur is a low-pass filter along the direction of travel: it attenuates exactly the high-frequency content the first conv layers key on. Once blur approaches the feature size, the defect's edge contrast collapses and the response drops below threshold. Retraining on the sharp set cannot recover information the sensor never captured: the blur is baked into the pixels.
Worked example (illustrative figures)
A 0.5 mm/px footprint, exposure fixed at 4 ms, 12 px (6 mm) minimum defect:
| Line speed | Motion | Blur | Recall |
|---|---|---|---|
| 0.4 m/s | 1.6 mm | 3.2 px | 0.97 |
| 0.7 m/s | 2.8 mm | 5.6 px | 0.88 |
| 1.1 m/s | 4.4 mm | 8.8 px | 0.71 |
At 1.1 m/s the blur is 73% of the defect: the edge is gone. A common design rule is to keep blur at or below ~⅓ of the smallest feature dimension; treat that as a starting point and confirm it empirically on your own parts.
How to confirm in ten minutes
- Compute predicted blur per frame from encoder speed and exposure metadata; plot it against per-frame recall. A monotonic relationship is the signature.
- Capture a slanted-edge target at the same exposure and measure MTF50: a direct, model-free sharpness number.
- Re-run the validation set captured at production line speed, not on a stationary rig. Report blur in pixels next to recall so the two are never separated again.
If blur is ruled out
Then check, in order: frame rate aliasing (parts moving more than a fraction of a feature between frames), trigger/timestamp skew between encoder and camera, rolling-shutter skew on fast parts, and compression or ISP sharpening that changed with the new frame rate. Each has a distinct signature: aliasing shows as periodic misses, timestamp skew as a constant spatial offset that grows with speed.
The trade-off you are about to make
Shortening exposure costs light. Halving exposure to halve blur costs one stop; you pay it back with more illumination, a strobe, or gain, and gain adds noise that also hurts recall. A global-shutter sensor removes skew but not blur. The camera and the conveyor are one system: pick exposure from the blur budget, then size lighting to hold SNR at that exposure.
Curated: · Written: · Reviewed:
Q&A-91A robot picks from a moving conveyor and placement error grows with belt speed. What is the likely cause?
I would settle conveyor tracking and encoder coupling against an instrumented run on real hardware before trusting the model.
Tracking requires the robot to know belt position at the same instants it knows its own. An encoder read on a different clock, at a different rate, or through a slow path introduces a phase error that scales directly with belt speed.
Concretely, latch belt position in the same time base as the robot's own state, ideally in the drive itself, and validate the coupling by picking at several speeds and checking that the residual does not trend.
The reason for that specificity is a failure I have seen: A belt encoder polled over a 20 ms application cycle gave a phase error that put placement 6 mm out at 0.3 m/s and 20 mm out at 1.0 m/s; the fix was hardware latching, not a better vision model.
Placement error against belt speed.
| Speed | Polled encoder | Latched encoder |
|---|---|---|
| 0.3 m/s | 6 mm | 0.4 mm |
| 0.6 m/s | 12 mm | 0.4 mm |
| 1.0 m/s | 20 mm | 0.5 mm |
I would not consider it settled without evidence: Plot placement error against belt speed and require a flat relationship rather than a low value at one speed.
A residual that grows with speed is a timing problem, not an accuracy one.
Curated: · Written: · Reviewed:
Q&A-35A SLAM map looks locally clean and has a duplicated corridor. What does that tell you?
The engineering content of SLAM loop closure and map consistency is the tolerance budget and the stop condition, not the algorithm name.
Local odometry is accurate over short intervals and accumulates error over long ones. Loop closure is what converts a locally consistent trajectory into a globally consistent map, and without an accepted closure the same place appears twice.
Concretely, instrument closure rate and rejection reason, tune place recognition against the environment's actual self-similarity, and treat a map with duplicated structure as unusable rather than as cosmetically imperfect, because the planner will route through the duplicate.
The reason for that specificity is a failure I have seen: A warehouse map closed no loops in a 140 m aisle because every bay looked alike to the descriptor; the corridor appeared twice, 2.1 m apart, and the planner routed a robot into a rack it believed was free space.
Closure outcome in a self-similar aisle.
| Descriptor | Closures found | Duplicate offset |
|---|---|---|
| geometric only | 0 | 2.1 m |
| geometric + intensity | 4 | 0.1 m |
| with fiducials | 9 | 0.02 m |
I would not consider it settled without evidence: Report loop-closure count and rejection reasons per run, and check the map against surveyed control points rather than against how it looks.
Locally clean and globally wrong is the normal failure of odometry.
Curated: · Written: · Reviewed:
Q&A-37A map that worked for six months starts producing localisation failures. Nothing on the robot changed.
The short answer
"Nothing on the robot changed" is the assumption I'd test first, but the likelier story is that the robot is fine and the map is stale. A map is a snapshot of a scene that keeps moving: racking gets reconfigured, pallets stack in new places, seasonal light changes what the lidar returns, and a reflective floor or a new glass partition shifts the geometry. Localisation degrades gracefully, match quality drops long before it fails, so the failure looks sudden when it isn't.
Clarify, then commit
I'd ask which stack is running (AMCL on a 2D occupancy grid, or a 3D scan-matching pipeline), whether it fails everywhere or in one zone, and whether it fails at the same place every time. Zone-localised failure points at the environment; global failure points at odometry, time sync, or a sensor. Then I'd answer: the map's validity has a lifetime, and the fix is to measure match quality continuously and version the map.
Mechanism
AMCL and scan matchers score a candidate pose by how well the current scan aligns with the map. Two numbers carry the signal: the inlier count (how many scan points land on occupied cells) and the residual (mean distance from a scan point to the nearest occupied cell). As the environment drifts, inliers fall and residual rises. The filter still converges, it just converges to a worse pose, until the correction is too weak to overcome odometry drift and the pose jumps or diverges.
Worked example (hypothetical figures)
A racking reconfiguration moved roughly 40% of the observable structure in one aisle. Trending inlier count per zone:
| Week | Inliers (aisle 3) | Residual | Localisation |
|---|---|---|---|
| 0 | 780 | 0.04 m | fine |
| 1 | 460 | 0.09 m | fine |
| 2 | 210 | 0.21 m | fails |
The failure at week 2 was visible at week 1. Alarm on the trend, not the failure.
What I'd actually do
Log inlier count and residual per zone every cycle, and threshold the rate of change, not just the absolute value. A zone that loses 30% of its inliers over a week is a map problem even while localisation still nominally succeeds.
Trade-offs
- Static map with versioned updates: predictable, reviewable, rollback-able. Cost is a human in the loop and a lag between the world changing and the map changing.
- Lifelong/continuous SLAM: adapts automatically. Cost is silent drift accumulation and a bad update that's hard to attribute or roll back.
- Hybrid: continuous monitoring, gated updates with a review step.
Failure modes
- Silent degradation: localisation "succeeds" with a bad pose until it doesn't. Detected by trending residual, not by success/failure flags.
- Bad update: a map update that bakes in a transient (a parked forklift) makes things worse. Detected by A/B on a held-out log; mitigated by versioning and rollback.
- Zone-localised blind spots: one aisle degrades while the fleet average looks fine. Detected by per-zone metrics.
Curated: · Written: · Reviewed:
Q&A-39Lidar says the aisle is clear and the camera says it is blocked. What should the robot do?
Stop, and hold. Treat the aisle as occupied: command a controlled stop (or drop to a crawl if already below ~0.3 m/s), refuse to plan through that cell, and resume only when the disagreement clears or a third observation breaks the tie. The asymmetry decides it: a false "blocked" costs a few seconds of availability, a false "clear" costs a collision. You never resolve a safety-relevant disagreement in favour of the permissive reading.
Assumptions: both sensors are nominally healthy, extrinsics are calibrated, and the robot is at aisle speed (~1.5 m/s). At that speed 200 ms of decision latency is 30 cm of travel, so the stop has to land inside one control cycle (~50 ms); you cannot wait for a multi-frame filter before slowing.
Why they disagree: the mechanism picks the recovery:
- Lidar returns nothing on black, matte, or specular surfaces (black pallets, shrink wrap, glass doors, polished floor). A "clear" lidar reading over a real obstacle is the dangerous case.
- Camera false positives come from reflections, shadows, hi-vis vests, or a stale extrinsic that projects the aisle edge into the path.
- A genuinely empty aisle with a spurious camera detection is the benign case, but you cannot tell it from the dangerous one at decision time.
Alternatives, and when each is wrong:
| Policy | Behaviour | Fails when |
|---|---|---|
| Conservative OR (stop on either) | safe, simple | camera FP rate ~1/1000 frames at 30 Hz → a stop every ~33 s; availability collapses |
| Confidence-weighted / Bayesian | smooth, no hard stops | needs calibrated per-sensor likelihoods; uncalibrated confidence is worse than a hard rule |
| Temporal consistency (N frames) | filters single-frame noise | adds latency; at 1.5 m/s you must slow first, then filter |
| Third modality / re-observe from new pose | resolves geometry-dependent causes | costs a manoeuvre; not always space to move |
Conservative arbitration is itself wrong in a narrow one-way aisle where a false stop deadlocks the fleet, or where the "blocked" reading is a known map-anchored artifact (a reflective floor strip at a fixed location): there you want a mapped exception, not a blanket rule.
Failure modes to detect:
- Silent preference: one sensor's disagreements discarded → undetected degradation. Count disagreements per sensor pair and alarm when the rate departs from baseline.
- Deadlock: stop-rate per aisle rises. Watch stops per 100 m.
- Calibration drift: disagreements cluster in one region of the field of view. Histogram them spatially.
Arbitration, checked against Python 3.11:
STOP, SLOW, GO = "stop", "slow", "go"
def arbitrate(lidar_occ, cam_occ, disagree_frames, speed_mps):
if lidar_occ and cam_occ:
return STOP
if lidar_occ != cam_occ:
if disagree_frames >= 3: # ~100 ms at 30 Hz
return STOP # persistent -> treat as blocked
return SLOW if speed_mps > 0.3 else STOP
return GO
Trace, lidar clear / camera blocked at 1.5 m/s: frame 1 → disagree_frames=1 → SLOW; frame 2 → SLOW; frame 3 → disagree_frames=3 → STOP. The robot has shed speed before the persistent verdict lands, so the stop distance stays inside the aisle.
Curated: · Written: · Reviewed:
Q&A-102Lidar and camera have disagreed about the same obstacle three times this week, always in the same aisle. How do you decide which one is wrong, and what does the robot do until you know?
First 90 seconds. The robot does not get to adjudicate. If either sensor claims an obstacle inside the braking envelope, the space is occupied and the robot slows or stops; the disagreement gets resolved offline from logs, not in the next control cycle. The reason is arithmetic. At 1.5 m/s with 300 ms of reaction and 1.0 m/s² of deceleration you need 1.5²/(2·1.0) + 1.5·0.3 = 1.125 + 0.45 ≈ 1.6 m to stop. A camera claiming an obstacle at 1.2 m while the lidar says clear is not a debate you can finish in the distance you have left.
Why "trust the lidar" is the wrong answer. Neither sensor is a reference. They fail in uncorrelated ways, which is the whole reason you carry both.
| Sensor | Characteristic failure | What it looks like on the wire |
|---|---|---|
| 905 nm ToF lidar | Low-reflectivity targets: matte black stretch wrap, dark rubber, fine mesh | Returns fall under threshold; the cell reads free |
| Lidar | Rain, fog, dust, grazing returns off a polished floor | Sparse spurious points, or a phantom plane at floor level |
| Camera | Exposure, shadow, specular reflection on the floor | A confident segmentation mask where nothing is |
| Stereo camera | No metric guarantee; range error grows roughly with z²/(f·b) | Obstacle position off by tens of cm at 3 m |
So the disagreement is information about the world, not a vote to be won. A single-sensor policy also has a correlated-failure problem: one environmental condition takes out the whole perception chain at once.
The policy: asymmetric authority. The safety-rated scanner owns the protective field, typically specified to PL d under ISO 13849-1, and the camera is not in that chain and cannot be. Give the camera veto power in one direction only: it may declare space occupied, never free. Occupancy-grid fusion with per-cell log-odds is fine for planning and for the "is this really an obstacle" question, but the safety function stays a deterministic OR of the two, evaluated against the conservative envelope.
DISAGREE_MS = 200 # sustained disagreement before a controlled stop
SLOW_FACTOR = 0.3
def step(lidar_occ, cam_occ, cam_health, dist_m, envelope_m, dt_ms):
occupied = lidar_occ or (cam_occ and cam_health == OK)
if not occupied:
disagree_ms = 0
return NOMINAL
disagree_ms += dt_ms
if dist_m < envelope_m or disagree_ms >= DISAGREE_MS:
return CONTROLLED_STOP # not e-stop; the safety chain is untouched
return SLOW # 30% speed, envelope still satisfied
Fallback when confidence is low. Define low confidence per sensor, not as a feeling: lidar return intensity below the target's calibrated threshold, a camera detection present in fewer than 3 of the last 5 frames, or a projection residual between the two streams above 0.3 m. On low confidence: controlled stop in lane, hold, request remote assist, dump both raw streams against a common clock. Not an e-stop: that belongs to the safety chain and costs a manual reset. Not an indefinite stop either; a robot that halts on every single-frame disagreement has no availability. Hence the hysteresis: one frame, slow down; 200 ms sustained, stop.
The recurring aisle. Three times in the same place is systematic, not noise. Replay both streams on a common clock and check in this order. First, time sync: project the camera detection into the lidar frame and look at the residual, if it grows with range, you have an extrinsics problem, not an obstacle. Second, the target itself: matte black wrap and glass are the usual lidar misses, a specular floor reflection the usual camera false positive. Third, exposure and the aisle's lighting. Fix the calibration or the sensor placement; do not widen the threshold to make the symptom disappear.
Curated: · Written: · Reviewed: —
Q&A-76Why is dynamic allocation discouraged in a real-time control loop when the allocator is fast on average?
My answer to memory allocation in the control path begins at the failure mode: if I cannot name how it hurts someone, I do not have a design.
The allocator's average is irrelevant to a deadline. Fragmentation, a coalescing pass, or a page fault produces a tail that arrives rarely and lands unpredictably, and a loop is judged on that tail.
Concretely, allocate at initialisation, use fixed-size pools and preallocated buffers in the loop, lock pages into memory where the platform supports it, and assert on allocation from the real-time thread so a regression is caught at the point it is introduced.
The reason for that specificity is a failure I have seen: A trajectory buffer that grew on demand triggered a 2.8 ms allocation once every few thousand cycles; the loop deadline was 1 ms and the resulting jerk was audible before it was measurable.
Allocation cost distribution.
| Percentile | Allocation | Loop budget |
|---|---|---|
| 50th | 0.9 µs | 1000 µs |
| 99th | 41 µs | 1000 µs |
| 99.99th | 2800 µs | exceeded |
I would not consider it settled without evidence: Instrument the real-time thread to fail on any allocation, rather than reasoning that the allocation is small.
Deadlines care about the worst allocation, not the average one.
Curated: · Written: · Reviewed:
Q&A-40A control loop averages 400 µs on a general-purpose kernel and occasionally misses its 1 ms deadline. Why is the average not the point?
My answer to real-time determinism versus average speed begins at the failure mode: if I cannot name how it hurts someone, I do not have a design.
A real-time system is judged on its worst case, not its mean. A loop that meets its deadline 99.9 percent of the time misses it roughly once a second at 1 kHz, and the controller has no way to make up the lost interval.
Concretely, measure the distribution including the tail, isolate the control thread from the sources of jitter (scheduling, interrupts, power management, shared cache) and detect a missed deadline explicitly so the loop can respond rather than integrating a stale sample as new.
The reason for that specificity is a failure I have seen: A 1 kHz loop with a 400 µs mean showed a 2.3 ms 99.99th percentile caused by a power-management transition; the axis logged a following-error spike every few minutes that the mean execution time never revealed.
Execution time distribution at 1 kHz.
| Percentile | Duration | Deadline |
|---|---|---|
| 50th | 0.40 ms | met |
| 99.9th | 0.95 ms | met |
| 99.99th | 2.30 ms | missed |
I would not consider it settled without evidence: Publish the execution-time distribution to the 99.99th percentile under production load, not the average under an idle system.
Real-time is a promise about the worst cycle.
Curated: · Written: · Reviewed:
Q&A-41A high-priority control task misses deadlines only when a logging task runs. Explain the mechanism.
The 90-second answer. "Only when logging runs" says the control task's worst-case time is a function of the logger. Three mechanisms produce that, and they need different fixes.
- The control task does the logging itself. If the loop calls a logging function that writes synchronously (to a file, a socket, a UART) the control task's own execution time now includes that I/O. Nothing is inverted; the control task is doing unbounded work. Discriminator: its on-CPU time grows when logging is enabled.
- Shared-resource blocking. The logger holds a lock the control task needs: log-buffer mutex, allocator, I/O queue. With logging off the lock is uncontended and the control task never blocks; with logging on it can wait for the whole time the logger holds it, and longer if a medium-priority task preempts the logger. Discriminator: its off-CPU wait grows, and the trace names the holder's priority.
- Interference. The logger burns CPU, interrupts, or memory bandwidth, or writes with interrupts disabled. Discriminator: execution time grows even with no shared lock.
Priority inversion is case 2's worst form, and it needs three tasks. H (control) blocks on a mutex held by L (logger); M (network handler, another flush, GC) preempts L. H now waits for M's whole run, not for L's critical section: unbounded, and neither H nor L looks wrong in isolation. Priority inheritance donates H's priority to L so M cannot preempt, bounding H's wait by L's critical section. It does not help when L is blocked on I/O while holding the lock: donation hands L CPU it cannot use.
Worked trace (hypothetical: 1 ms control period, 300 µs worst-case compute, SCHED_FIFO priorities 90/50/10).
| t (ms) | Control (90) | Logger (10) | Network (50) | Control state |
|---|---|---|---|---|
| 0.00 | compute | — | — | running |
| 0.30 | log_event(): lock M | holds M, runnable | — | blocked on M |
| 0.35 | — | preempted | wakes, runs 1.45 ms | blocked |
| 1.80 | — | finishes CS (0.05 ms), releases M | — | blocked |
| 1.85 | resumes, done at 2.10 | — | — | deadline 1.00 missed by 1.10 ms |
With PTHREAD_PRIO_INHERIT, at 0.30 the logger inherits 90, the network task cannot preempt, the logger releases M at 0.35, and control finishes at 0.65: deadline met. Worst-case response goes from 0.30 + 1.50 = 1.80 ms to 0.30 + 0.05 = 0.35 ms.
Keep logging off the control path. The control task writes a fixed-size record into a preallocated single-producer/single-consumer ring and returns; a low-priority logger drains it. No lock, no allocation, no I/O in the control path.
/* C11, checked with gcc 13 and clang 17 on x86-64 and aarch64 */
#define LOG_CAP 1024u /* power of two */
typedef struct { uint64_t t_ns; uint32_t code; int32_t arg; } log_rec;
static log_rec buf[LOG_CAP];
static _Atomic uint32_t head; /* producer: control task */
static uint32_t tail; /* consumer: logger task */
/* Called from the control loop. Never blocks, never takes a lock. */
bool log_try(uint64_t t_ns, uint32_t code, int32_t arg) {
uint32_t h = atomic_load_explicit(&head, memory_order_relaxed);
uint32_t next = (h + 1u) & (LOG_CAP - 1u);
if (next == atomic_load_explicit(&tail, memory_order_acquire))
return false; /* full: drop and count, never wait */
buf[h] = (log_rec){ t_ns, code, arg };
atomic_store_explicit(&head, next, memory_order_release);
return true;
}
Failure modes and how to catch them. PI does not bound blocking when the holder blocks on I/O or on a second lock (chained blocking). A ceiling protocol deadlocks if the ceiling is set below the highest priority of any task that takes the lock. Mixing PI and non-PI mutexes over the same lock graph reintroduces inversion. A lock-free ring drops records on overflow: fine for diagnostics, not for audit logs; count the drops. Detect with perf sched latency or an sched_switch trace recording the holder's priority at each block, and by comparing the control task's on-CPU versus off-CPU time with logging on and off. Priority says what runs; a lock quietly overrides it.
Curated: · Written: · Reviewed:
Q&A-75Debug logging is enabled on a robot controller and the control loop starts missing deadlines. What is the mechanism?
The mechanism: enabling debug logging puts an unbounded-latency operation on the thread that has a deadline. The loop's worst-case execution time becomes the worst-case latency of the logging path, and that latency is set by things the loop doesn't control: the console, the storage device, the scheduler, and whoever else holds the log lock. Average cost barely moves; the tail explodes, and a deadline miss is a tail event.
A log call is a chain, not one operation: format, allocate, lock, copy, flush. Each link is cheap on average and pathological at the tail.
- Formatting.
snprintfwalks the format string and converts each argument. Integer and pointer conversions are microseconds; float is the expensive case, and%fon a soft-float target is where microseconds become hundreds of microseconds. This cost lands on the loop thread. - Allocation. If the record is heap-allocated, the loop takes the allocator lock and, on the slow path, extends the heap and faults in pages.
- Locking. A shared mutex between the loop and a writer thread is a priority-inversion site. If the writer holds it and a medium-priority thread preempts the writer, the loop blocks for as long as that thread runs: unbounded, and unrelated to log volume.
- Flush.
write()to a console or file. This is where the numbers get brutal.
Arithmetic for a serial console at 115200 8N1: 10 bits per byte, so 11,520 bytes/s, or 86.8 µs per byte. A 120-byte log line costs 10.4 ms of wire time. On a 1 kHz loop (1 ms period) that is a 10× overrun for a single line; one line every ten iterations already consumes 104% of the budget. Nothing else has to go wrong.
Worked example (illustrative figures, 1 kHz loop, logging to a file on eMMC):
| Design | p50 | p99.9 | 1 ms deadline |
|---|---|---|---|
| format + write in loop | 0.6 ms | 3.2 ms | missed |
| binary record to SPSC ring, writer thread | 0.41 ms | 0.52 ms | met |
| logging compiled out | 0.40 ms | 0.48 ms | met |
The p50 barely moves between rows; the p99.9 is the whole story.
Failure modes and how they show up:
- Rotation and fsync. Closing, renaming and fsyncing a log file on eMMC can block for tens to hundreds of milliseconds. Misses cluster at rotation boundaries rather than tracking log volume.
- A ring that blocks. Bounding the ring is not enough if the producer waits when it's full: that reintroduces the writer's latency on the loop thread. The producer must drop and count.
- Priority inversion. Misses that correlate with other threads' activity, not with log volume, point at the lock.
- Cache pollution. Formatting touches buffers and evicts the loop's working set. This shows as a uniform rise in p50, not a tail.
Detection: toggle a GPIO at loop entry and exit and histogram the pulse widths, or read a cycle counter and histogram the deltas. Then correlate the outliers against rotation events, log volume, and other thread activity: that tells you which link in the chain is the one.
The fix is to keep the loop's contribution bounded: preallocated binary records into a single-producer/single-consumer ring, no allocation, no formatting, no fsync on the loop thread; a low-priority writer formats and writes; the ring drops and counts when full. Measure with logging at production verbosity before calling it done.
Curated: · Written: · Reviewed:
Q&A-47A control node subscribes to a sensor topic and occasionally acts on old data. What in the middleware configuration would you look at?
What I'd check first
Assume ROS 2 (DDS) pub-sub, a sensor publishing at 125 Hz (8 ms period), and a control node running at 50 Hz. Acting on old data is almost always the subscriber's queue, not the transport.
Three QoS settings on the subscription:
- History + Depth.
KEEP_LASTwith depth 1 means the reader holds only the newest sample, so a slow callback drops the backlog.KEEP_LASTwith depth 50 means the reader drains 50 samples in order and the callback acts on a sample 50 periods old.KEEP_ALLis worse. - Reliability.
RELIABLEretransmits, which is right for commands and wrong for a 125 Hz sensor feed: a retransmit of an already-stale sample just adds to the backlog. UseBEST_EFFORTfor sensor streams. - Topic and callback separation. If the control subscription shares a topic, a callback group, or an executor thread with high-rate telemetry, the control callback waits behind telemetry. Separate topics, separate callback groups.
Why depth is the whole story
The middleware buffers; it has no notion of "old." Age at use ≈ depth × period + callback latency. Measured on a 125 Hz feed:
| Depth | Age at use | Under a 0.4 s stall |
|---|---|---|
| 1 (keep-last) | ~12 ms | drops the backlog, acts on the newest sample |
| 10 | ~80 ms | lags 10 samples |
| 50 | ~400 ms | acts on a sample from before the stall |
The failure this prevents
A grasp node subscribed with depth 50. A 0.4 s GC pause stalled the callback; on resume it drained the backlog and executed a grasp against a pose from before the part moved, closing the gripper on empty air. Depth 1 would have dropped the 49 stale samples and acted on the current pose.
How to detect it
Instrument message age at the point of use, now - header.stamp inside the callback, and alarm on age, not on message rate. Rate stays healthy while age climbs; that divergence is the signature. A depth-1 keep-last subscription plus an age alarm is the configuration; the alarm is what tells you the callback itself is too slow.
Curated: · Written: · Reviewed:
Q&A-48Monitoring shows the sensor topic publishing at its full rate and the robot still behaves as though the data is late. What would you measure instead?
Measure the age of the data at the point of use, not the rate at which it is published. Rate is a liveness signal: the publisher is alive and the transport is delivering. It says nothing about how old the content is when the controller reads it. A buffer, a batch, a republisher, or a slow consumer callback all preserve rate while adding delay.
Three things to instrument:
- Capture timestamp in the message. The driver stamps the sample when the sensor produced it, not when the node published it. If the stamp is taken at publish time, driver readout, bus transfer and serialization are invisible.
- Age at consumption. At the consumer,
age = now_consumer − header.stamp. Record the distribution (p50, p95, p99, max) not the mean. A mean of 20 ms with a p99 of 300 ms is a robot that stutters. - Backlog per hop. Queue depth and drain rate at each node between sensor and consumer. Age grows when arrival rate exceeds service rate; depth is the leading indicator, age is the lagging one.
Clocks: age is only meaningful if the two clocks agree. Run PTP or chrony, monitor the offset, and treat an offset above your age budget as a measurement failure rather than a data point. On a single board with a shared monotonic clock this is free; across machines it is not.
Why rate survives buffering: a queue of depth N at rate f adds up to N/f seconds of age. A 6-deep buffer at 30 Hz is 200 ms of age with the rate unchanged at every hop. Nothing in the rate metric moves.
# rclpy, ROS 2 Humble
import rclpy
from rclpy.node import Node
class AgeMonitor(Node):
def __init__(self):
super().__init__('age_monitor')
self.create_subscription(LaserScan, '/scan', self.on_scan, 10)
self.samples = []
def on_scan(self, msg):
now_ns = self.get_clock().now().nanoseconds
stamp_ns = msg.header.stamp.sec * 10**9 + msg.header.stamp.nanosec
self.samples.append((now_ns - stamp_ns) / 1e6) # ms
| Stage | Rate | Content age |
|---|---|---|
| camera capture | 30 Hz | 0 ms |
| after 6-frame buffer | 30 Hz | 200 ms |
| at controller | 30 Hz | 210 ms |
Worked example, hypothetical figures.
Traps:
- Measuring rate at the publisher. The publisher's rate is fine while the subscriber's callback sits behind a slow executor. Measure at the consumer.
- KEEP_LAST depth 1. This drops old samples instead of queueing them, so age stays low and the consumer's rate drops. Watching rate alone sends you after a transport problem that does not exist. Watch both.
- Averaging. Tail age is what breaks control loops.
- Timestamping at publish. Hides the driver.
Detection and gating: alert on p99 age crossing the control period, and on queue depth trending up. Gate behaviour on age, if p99 age exceeds the budget, degrade (slow the robot, drop to a safe mode) rather than trusting the rate dashboard.
Trade-off: age gating costs availability. A burst of stale data stops the robot. That is usually right for a control loop, but a logging or mapping consumer may prefer to accept stale data and mark it. Decide per consumer, not per topic.
Curated: · Written: · Reviewed:
Q&A-49The system works when started by hand and fails intermittently on boot. What is the usual cause?
Core answer. The usual cause is a startup race: a consumer process reads state (a parameter, a transform, a device handle, a socket) before the process that provides it is ready, and the read has a default path that lets the consumer keep going with wrong state instead of failing. Starting by hand hides it because the seconds you spend typing are the synchronization the boot path never had.
Why it is intermittent: the launcher (systemd, a launch file, an init script) starts units in parallel, and ordering directives like After= only guarantee the provider process has been spawned, not that it has finished initializing. Whether the consumer's first read lands before or after the provider publishes depends on disk, CPU contention and network, so it is a coin flip weighted by how close the two are. A node that reads a parameter, subscribes to a transform, or opens a device before the provider exists will sometimes succeed and sometimes fail: exactly this profile.
What it looks like. A calibration node read its extrinsics parameter 40 ms before the parameter server had loaded it, fell back to an identity transform, and the cell ran a shift placing parts against an unrotated camera frame about one boot in seven.
| Order | Extrinsics loaded at read | Result |
|---|---|---|
| params first | yes | correct |
| node first, waits | yes | correct |
| node first, defaults | no | identity frame |
The fix: a bounded wait, not a sleep and not a default.
def wait_for_transform(parent, child, timeout_s=5.0, poll_s=0.05):
deadline = time.monotonic() + timeout_s
while True:
tf = tf_buffer.lookup(parent, child) # None until published
if tf is not None:
return tf
if time.monotonic() >= deadline:
raise StartupError(f"{parent}->{child} not published in {timeout_s}s")
time.sleep(poll_s)
Trace it: the provider publishes at t=1.80 s. The consumer polls at 0.00, 0.05, … and returns at 1.80 s. If the provider never publishes, it raises at 5.00 s instead of proceeding. The old code did one read at 0.04 s, got None, and used identity.
Trade-offs. A fixed sleep 2 is the common wrong answer: it works until the machine is slower, then fails again with no signal. Retry-with-bound fails loudly and is the minimum. The stronger fix is a readiness protocol (Type=notify with sd_notify, or a health endpoint) because it makes "ready" a fact the provider asserts rather than a duration the consumer guesses. The cost is more boot coupling, a slower start, and you must make init idempotent so a retry does not double-register a subscription or re-open a device.
Failure modes and how to detect them. Silent default: log the value actually read at startup and assert it is not the sentinel. Device enumeration: /dev/ttyUSB0 renumbers across boots. Use by-id paths. Clock: NTP not yet synced makes TLS certs read as "not yet valid". Detection is the same for all of them: force adverse ordering in test, start dependencies late deliberately, and require the system to refuse rather than default. A silent default at startup is a fault that waits for a reboot.
Curated: · Written: · Reviewed:
Q&A-50Two identical cells behave differently and both are running the same software version. Where do you look?
What I'd say in the first two minutes
"Identical" is the claim I'd test before anything else. Same software version, same model, same recipe, but not the same history. So I'd assume the gap is real and reproducible, then search from cheapest and most likely to most expensive:
- Reproduce it. Run the same operation on both cells with the same part, same operator, same conditions. If the gap moves or disappears, it's measurement or input, not the cell.
- Compare effective configuration, not files. Gains, limits, calibration tables, tool offsets: the values the controller actually loaded at startup, hashed.
- Compare physical state. Backlash, belt tension, coupling slip, sensor zero, tooling wear, fixture location.
- Compare environment and services. Temperature, air pressure, power quality, network timing.
- Compare inputs. Material lot, upstream variance, part presentation.
Config drift sits at step 2 because it's the one thing a version number doesn't cover. A release pins the code; gains, limits, calibration and tool data live outside it and are exactly the numbers a commissioning engineer touches on site.
Why config first, and when it isn't
A config hash is cheap and decisive when it differs: two cells, same release, different hash, you're done. But a matching hash doesn't clear the cell: it only moves you down the list, and that's the trap. Mechanical wear and sensor calibration drift produce the same symptom and won't show in any hash. So the hash is a filter, not a verdict.
I've seen the config case: two cells on the same release differed by a 0.6 factor on one axis gain, applied eighteen months earlier. The slower cell had been accepted as normal; the difference surfaced only in a cycle-time study.
| Cell | Software | Config hash | Cycle time |
|---|---|---|---|
| A | 4.2.1 | 9c31 | 8.4 s |
| B | 4.2.1 | 7ae0 | 11.9 s |
B is 42% slower: about 3.5 s per cycle. That's a factor, not a calibration nudge, which is why a single parameter is a plausible cause and why "same version" told us nothing.
Failure modes to watch
- Hashing the config file instead of the effective values. Runtime overrides and recipe-applied parameters won't appear, and the hashes match while the cells don't.
- Calibration stored outside the config system. Sensor zero and tool offsets drift and never enter the hash.
- Drift inside tolerance. Each parameter is within spec, but the combination isn't: no single diff flags it.
- A real mechanical cause masked by a config difference. Fix the config, the gap shrinks but doesn't close; you stop looking too early.
What settles it
Emit the effective configuration hash at startup and compare it across cells. If it differs, diff the values. If it matches, the cause is physical, environmental or input, and the ordered search above is how you find which. The running configuration determines behaviour, but it isn't the only thing that does.
Curated: · Written: · Reviewed:
Q&A-84Where does Python belong in a robot system, and where does it not?
The 90-second answer. Python belongs above the deadline, not inside it. Put it where a late result degrades behaviour instead of breaking the machine: mission sequencing, planning, perception glue, calibration, offline analysis, test harnesses, HMI. Keep the servo loop, the safety path and the motor bus in C/C++ (or a certified PLC) on a PREEMPT_RT kernel: mainline since Linux 6.12. The dividing line is not fast versus slow; it is whether the worst case is bounded. CPython's is not.
Why the worst case is unbounded. Three mechanisms, all of which show up as tail latency rather than average latency:
- The GIL.
sys.getswitchinterval()defaults to0.005s. A CPU-bound Python thread can hold the GIL that long before another thread runs, so a 1 kHz loop (1 ms period) can miss five periods from the switch interval alone. Dropping it to 1 ms narrows the window; it does not bound it. - Cyclic GC. CPython's generational collector runs when the gen-0 allocation counter crosses its threshold,
gc.get_threshold()defaults to(700, 10, 10), and a collection walks live objects. Reference counting is roughly deterministic; the cycle collector is not. - Allocation and page faults. Every Python object is a heap allocation. A
mallocin the loop, or a first-touch page fault, has no bound.
Free-threaded CPython (PEP 703, opt-in and experimental in 3.13/3.14) removes the GIL. It does not give you a bounded worst case, so it does not move Python into the hard-real-time slot.
The boundary pattern. Preallocated shared memory, one writer, no lock the RT thread can block on, and the RT thread never calls into Python.
// servo_loop.cpp — 1 kHz, PREEMPT_RT, no Python on this thread
struct Cmd { double q[7]; };
static Cmd slot[2]; // preallocated, never freed
static std::atomic<uint32_t> seq{0}; // seqlock: odd = writer active
void tick() {
uint32_t s0 = seq.load(std::memory_order_acquire);
if (s0 & 1u) return; // planner mid-write: hold last command
Cmd c = slot[(s0 >> 1) & 1u]; // 56-byte copy, no allocation
if (seq.load(std::memory_order_acquire) != s0) return; // torn read
servo_write(c.q);
}
The planner writes the inactive slot, then publishes with a release store: a few lines of C, or an atomic store through a shim, because Python cannot express the fence itself. It never blocks the RT thread, and the RT thread never waits on it.
Where Python is genuinely fine. A 20 Hz navigation controller with a 50 ms budget, MoveIt2 planning (hundreds of ms), calibration, log analysis, simulation scripting, ML inference orchestration with the math in C/CUDA. Even there, bound the tail: call gc.freeze() after startup so long-lived objects leave the gen-0 set, gc.disable() on the RT-adjacent thread, preallocate buffers, and never log from the loop.
Failure modes and how you see them.
| Failure | Symptom | Detection |
|---|---|---|
| GC pause | multi-ms stall, uncorrelated with load | gc.callbacks timing; loop overrun counter |
| GIL convoy | spikes at multiples of the switch interval | py-spy dump taken during a spike |
| Priority inversion | RT thread blocked on a Python-held mutex | cyclictest max, preemptirqsoff trace |
| Allocation in loop | latency grows with uptime | minor/major fault counters, latency histogram |
Arithmetic, illustrative. A 200 Hz node has a 5 ms period. A 34 ms GC pause misses six deadlines; at 1 m/s the arm travels 34 mm before the next command lands. That is the argument in one line: the mean is irrelevant, the tail is the specification.
Curated: · Written: · Reviewed:
Q&A-77When would you implement a control law in fixed point, and what is the risk?
What I'd say first
Assumptions I'd state before answering: the loop runs on a Cortex-M0/M0+/M3-class core with no FPU, the sample rate is fixed and the deadline is hard, and the sensor is 12–16 bit. Under those, fixed point is the default, not a fallback.
The trigger is arithmetic cost and determinism, not precision. On a core without an FPU, every float add or multiply becomes a call into the soft-float library: tens of cycles plus call overhead, register pressure, and a result that depends on compiler flags and library version. A 32-bit integer multiply-accumulate is single-cycle on M3/M4. If the loop has to finish inside a 100 µs ISR alongside other work, that difference decides the design.
The second trigger is reproducibility. Fixed point gives bit-identical results across builds and toolchains; float does not, once the compiler is free to contract a*b+c into an FMA or reassociate. If every unit has to produce the same actuator command, or you have to defend the behaviour in a certification review, fixed point removes a whole class of argument.
What fixed point actually costs you
A Q15 value is an int16_t with an implicit scale of 2⁻¹⁵: range [−1, 1), resolution 3.05e-5. A Q15 × Q15 product is Q30 and needs an int32 to hold it, then a right shift by 15 to return to Q15. Every intermediate in the chain has to be sized against the worst case, and the accumulator is where people get it wrong: a Q15 integrator state in an int16_t has no headroom at all.
When not to
If the part has an FPU (Cortex-M4F, M7), single-precision add and multiply are single-cycle and the analysis burden buys you nothing: write it in float. Same if the signal spans decades of dynamic range, if the law needs sqrt/atan2/matrix work, or if the loop is still being tuned. Fixed point is a commitment you make once the structure is settled.
The risk, worked
Take a 1 kHz current loop, Q15 PI, integrator increment 344 LSB per cycle while the error is saturated (Ki·e ≈ 0.0105 in Q15). The accumulator starts at zero.
| Cycle | t (ms) | acc before add | acc after add |
|---|---|---|---|
| 94 | 94 | 32336 | 32680 |
| 95 | 95 | 32680 | −32512 |
95 × 344 = 32680, still inside range. 96 × 344 = 33024, which wraps to 33024 − 65536 = −32512. The command goes from full forward to full reverse inside one 1 ms cycle. Nothing in the code signalled it; the only symptom is a mechanical bang and a current spike.
Two fixes, and they fix different halves. Saturating the add (__SSAT on Cortex-M, or clamp before store) turns the reversal into a stuck integrator: the loop stays wound up and recovers slowly instead of inverting. That is a better failure but still a failure, so you also need anti-windup: stop integrating when the output is saturated, or back-calculate from the difference between commanded and applied output.
How I'd know it's fixed
Assert on accumulator magnitude in debug builds and log peak |acc| over a run; if it ever exceeds roughly 0.8 of full scale, the scaling is wrong. Drive the input to both extremes and hold it there for minutes, not seconds: the wrap above needs 95 ms of saturation, but a slower integrator with a smaller Ki can take minutes, and a bench test that only exercises nominal inputs will never see it. Check coefficient quantization separately: rounding a continuous-time gain into Q15 moves the closed-loop poles, which shows up as a limit cycle at the LSB, not as an overflow.
Fixed point trades a hardware requirement for an analysis requirement, and the analysis is the part that gets skipped.
Curated: · Written: · Reviewed:
Q&A-78What can you meaningfully test for a robot without a robot?
Meaningful means the test can fail for a reason that matters on the robot. That splits motion code in two.
Pure computation (kinematics, limit logic, state machines, message encode/decode) is a function of numbers, so it can be tested exhaustively off-robot and the result transfers. Anything that depends on the physical plant (contact, friction, thermal drift, loop timing under load, cable flex) cannot be tested without the plant, and a mock of it only re-encodes the assumptions you already had.
So test the computable layer hard, and put the physical claims behind hardware-in-the-loop tests that are explicitly separate, not mocked.
Why the split falls where it does
A mock is a model of the interface, not of the machine. It answers "did the caller send what I expected", which is a statement about the caller. It cannot answer "does the controller accept it", because the mock's acceptance rule is the one you wrote.
The failure mode is specific. The suite goes green and the robot rejects everything. A representative example: 340 tests against a mocked motion interface, all passing, while the real controller rejected every command because joint values were sent in degrees against a radians interface. The mock accepted whatever it was given. The tests were not wrong about the caller; they were silent about the machine.
What each layer can actually prove
| Property | Unit | Sim | Hardware | Why |
|---|---|---|---|---|
| IK round trip | yes | yes | yes | pure arithmetic |
| limit clamping at boundary | yes | yes | yes | pure comparison |
| state machine recovery | yes | yes | yes | pure transition table |
| contact stability | no | partly | yes | depends on friction, compliance, unmodeled dynamics |
| control loop jitter | no | no | yes | depends on scheduler, bus, thermal state |
Sim sits in the middle. It catches gross contact errors, a grasp that misses by 5 cm, but not stability margins, because the contact model is the thing under test.
What to actually write
Round-trip properties against the real codec and the real limit module, not a stub of them:
# Python 3.12, pytest
import math, pytest
from robot.kin import forward, inverse
from robot.limits import clamp_joints, JointLimitError
def test_ik_fk_round_trip():
for pose in sample_poses(): # poses inside the reachable set
assert forward(inverse(pose)) == pytest.approx(pose, abs=1e-6)
def test_clamp_pins_the_documented_boundary():
lim = math.radians(180.0)
assert clamp_joints([lim]) == [lim] # inclusive at the limit
with pytest.raises(JointLimitError):
clamp_joints([lim + 1e-3])
The second test is worth more than it looks. The boundary convention (inclusive or exclusive, and the units) is exactly what drifts between the planner and the controller, and it is cheap to pin.
Then a hardware suite that runs before release, covering contact, payload and timing, kept separate so a red hardware run is never confused with a red unit run.
A mock validates the caller and asserts nothing about the machine.
Curated: · Written: · Reviewed:
Q&A-80You need to test candidate poses against a 40 m by 40 m workspace at 5 cm resolution. What do you store?
Store a dense 800×800 occupancy grid.
40 m ÷ 0.05 m = 800 cells per axis, so 800 × 800 = 640,000 cells. That fits in a flat array and indexes in constant time, so there is no reason to reach for anything sparse at this size.
W = H = 800 # 40 m / 0.05 m
CELL = 0.05 # metres per cell
grid = bytearray(W * H) # 640 kB, one byte per cell
def idx(x, y): # x, y in metres
return int(y / CELL) * W + int(x / CELL)
def occupied(x, y):
return grid[idx(x, y)] != 0
Bit-packed it is 80 kB (640,000 bits); one byte per cell it is 640 kB. Both fit in L2 on a server core, so the choice is about what you store per cell, not size: bytes if you want a value (log-odds, height, clearance), bits if it is strictly free/occupied.
Testing a pose means walking the cells its footprint covers. A 0.5 m × 0.5 m footprint at 5 cm is 10 × 10 = 100 reads spanning a couple of cache lines. If you test thousands of poses per planning cycle, precompute a Euclidean distance transform over the grid once (O(640k)) and then a disc-shaped footprint of radius r is collision-free iff the transform at the pose centre exceeds r: one lookup per pose.
Dense stops being right when you go finer or 3-D. At 1 cm over the same 40 m cube you would have 4000³ = 6.4×10¹⁰ cells: 64 GB at one byte each, 8 GB bit-packed. That is where a hash map keyed on (x,y,z) or an octree earns its keep, because real workspaces are a few percent occupied.
The failure mode is switching to sparse when the map is dense. Illustrative order-of-magnitude figures for this 640,000-cell map (not a benchmark: measure your own):
| Occupancy | Dense (byte/cell) | Octree | Lookup |
|---|---|---|---|
| 60% | 640 kB | ~2 MB | ~900 ns |
| 5% | 640 kB | ~90 kB | ~700 ns |
| 0.5% | 640 kB | ~12 kB | ~650 ns |
At 60% occupancy the octree's node overhead exceeds the dense array and every lookup pays pointer chasing; the tell is a lookup-latency histogram that shifts right after the change, not a memory graph.
So: measure occupancy fraction and per-query cost on your real maps. Above a few percent occupancy in 2-D at this resolution, dense wins and you should stop looking.
Curated: · Written: · Reviewed:
Q&A-79What belongs in a hardware-in-the-loop rig and what does putting it there cost you?
The 90-second answer. A HIL rig keeps the controller real and simulates the plant. Everything on the controller side is the production article: the board, the release firmware build, its scheduler, its I/O drivers, the bus transceivers, the harness. Everything on the plant side (motor, gearbox, hydraulics, thermal mass, the vehicle) is a model on a real-time target that closes the loop with the controller at a fixed rate. You buy reproducibility and safe fault injection; you pay with a fidelity gap you have to characterise, and with a plant model that costs more to build and maintain than the rig hardware does.
What goes in, and what each choice costs.
| Layer | Real or modelled | Why | What it costs you |
|---|---|---|---|
| Controller + firmware | real, unmodified release build | timing and driver bugs only appear on the real target | one rig per controller variant; firmware flashes serialise across teams |
| Loop rate and scheduler | real | jitter and overrun behaviour is the thing under test | the real-time target must hold the rate: 1 kHz with <100 µs jitter is a hardware spec, not a software one |
| Bus (CAN, EtherCAT) | real transceivers, real frame timing | arbitration, bus load, error frames | needs the production harness or an equivalent with the same stub lengths and terminations |
| Sensor feedback | modelled, unless the sensor's own dynamics matter | cheap, repeatable, injectable | this is where rigs lie: see below |
| Actuator load | emulated (electronic load, motor emulator) | avoids a real dyno | the emulator's electrical model diverges at saturation and at thermal limits |
| Fault injection | simulated: short-to-ground, open circuit, bus dropout | faults you cannot safely stage on a machine | injected faults are cleaner than real ones; intermittent faults are hard to reproduce faithfully |
Where the rig lies. Worked example, figures illustrative. The rig's encoder model published a fresh position every 1 ms, exactly on the loop boundary. The production encoder updates asynchronously, so the controller sometimes read a sample up to one loop old, and the velocity it derived from consecutive samples spiked. Following error tripped at 0.3° of lag on hardware; the rig never exceeded 0.05°. Detection is a histogram, not a pass/fail: log the age of the last encoder sample on both rig and machine. If the rig's sample-age histogram is a spike at 0 and the machine's is spread across 0–1 ms, the rig cannot see this class of failure, and no amount of test cases will change that.
Costs beyond the build. Illustrative figures for a mid-size rig: the real-time target, I/O cards, load emulation and harness land around $40k–$80k for a PC-based x86 target with EtherCAT I/O, or $150k–$400k for a commercial target with motor emulation. The plant model is usually the larger line item: three to six engineer-months for a validated motor, drivetrain and thermal model, against four to eight weeks to stand the hardware up. Then it keeps costing: every plant change (new gearbox ratio, new battery pack) needs re-parameterisation, and a stale model passes silently. Add scheduling contention, because one rig serves several teams.
When not to use it. If the failure you are chasing lives in the plant (contact force, thermal drift, EMI on a long cable run) a HIL rig is the wrong instrument. Use a dyno or a real machine with instrumentation, and keep HIL for controller logic, timing, bus behaviour and fault response. The release argument is: list the quantities the rig models and their fidelity, and hold a hardware acceptance test for the ones it does not.
Curated: · Written: · Reviewed:
Q&A-93What does a meaningful acceptance test for a cell contain beyond running the cycle?
A cycle that runs proves the happy path once. Acceptance has to establish capability across the variation the cell will actually see and behaviour at the boundaries, because those decide whether it still works in month three.
Four things belong in the test beyond the cycle itself.
A capability run, not a pass/fail run. Run a defined number of consecutive cycles at production takt with production parts spanning the tolerance range, and report a capability index against the placement tolerance rather than a count of successes. Twenty cycles is a demonstration. The standard error of a sample standard deviation is roughly σ/√(2(n−1)), so at n=20 the σ estimate carries a 95% interval of about ±32%; at n=100 it narrows to about ±14%. Capability numbers from a 20-cycle run are noise.
Take a placement tolerance of ±0.5 mm, a measured mean offset of +0.08 mm, and σ = 0.12 mm over 100 cycles. Then Cpk = min(0.5 − 0.08, 0.5 + 0.08) / (3 × 0.12) = 0.42 / 0.36 = 1.17. That is below the 1.33 most automotive customers require, so the cell fails acceptance even though every individual cycle passed.
Fault injection with recovery. Deliberately trigger each fault the cell is specified to handle: e-stop, guard open, vacuum loss, part-present sensor failure, gripper slip, PLC comms timeout, power loss mid-cycle, and verify three things per fault: the cell reaches a safe state, the fault is annunciated with the right code, and it recovers without scrapping the part in hand or losing position. A cell that stops safely but needs a manual re-home is a different acceptance outcome from one that resumes on its own.
A validated measurement system. If gauge R&R contributes more than 10% of the tolerance, the capability number is measuring the gauge, not the cell. Under AIAG rules a contribution above 30% makes the measurement system unacceptable and voids the capability study.
A recorded baseline. Log the cycle-time distribution (p50 and p99, not the mean), placement data, and fault counts, so month-three drift has something to compare against.
| Test | Cycles | Establishes |
|---|---|---|
| demo run | 20 | it works once |
| capability run | 100+ | rate and Cpk against tolerance |
| capability + fault injection | 100+ | recovery behaviour and safe states |
I have seen a cell accepted on 20 consecutive good cycles from a single part lot. Production capability against the placement tolerance was well below what the spec implied, and scrap settled at 4%.
Curated: · Written: · Reviewed:
Q&A-81Why is A-star usually preferred to Dijkstra for a navigation graph, and when does the heuristic hurt?
Assumptions. Static graph, non-negative edge costs, one source and one target, and a heuristic h(n) that is cheap to evaluate. If any of those fail, the comparison changes: noted at the end.
The answer. A* is Dijkstra with the frontier ordered by f(n) = g(n) + h(n) instead of g(n). With an admissible h, one that never exceeds the true remaining cost, it returns the same optimal path while expanding fewer nodes, because the heuristic pulls the frontier toward the target instead of growing a uniform ball around the source. That is the whole reason it is preferred: identical answer, less work. Everything else is about how much less.
How much less, exactly. On a 4-connected unit-cost grid with h = Manhattan distance, f(n) ≥ D for every node by the triangle inequality, with equality exactly on the nodes of optimal paths. So A* expands those and nothing else: 101 nodes for a goal 100 steps away. Dijkstra expands every node with g(n) ≤ D: the diamond |x| + |y| ≤ 100, which holds 2D² + 2D + 1 = 20,201 nodes. A 200× difference from one line of code. Obstacles and an imperfect h widen A*'s region, but the order of magnitude survives.
When the heuristic hurts. Four distinct failures:
- Inadmissible. If
hoverestimates anywhere, A* can settle the goal at a non-optimalg. Weighted A* (f = g + w·h,w > 1) is the deliberate version: it returns a path of cost at mostw ×optimal. State the bound; do not call the result optimal. - Admissible but weak. The common case. A heuristic that assumes the fastest possible speed everywhere underestimates badly on a slow route and prunes almost nothing.
- Inconsistent.
h(n) > c(n, n') + h(n'). With a closed set, A* may settle a node at a non-optimalgand never reopen it: suboptimal output even thoughhis admissible. Either use a consistenthor allow re-expansion. - Expensive. If evaluating
hcosts more than the expansions it saves, a landmark lookup per node on a small graph, A* is slower than Dijkstra despite expanding fewer nodes.
Worked example (illustrative figures). City routing, cost = travel time in seconds, edge cost = length_m / speed_mps, speeds 5–30 m/s, one source–target pair, 38.2 s optimal.
| Frontier order | Nodes expanded | Path cost (s) | Optimal |
|---|---|---|---|
g only (Dijkstra) | 41,200 | 38.2 | yes |
g + straight_line_m / 30 | 37,900 | 38.2 | yes |
g + ALT landmark bound | 6,100 | 38.2 | yes |
g + 1.5 × (straight_line_m / 30) | 2,300 | 41.0 | no |
Row 2 is admissible, 30 m/s is the fastest speed on the map, but the route runs on 5 m/s streets, so h underestimates by up to 6× and buys 8% fewer expansions. Row 3 uses the same A* loop with a better lower bound and buys 85%. The heuristic, not the algorithm, is doing the work. Row 4 is 7.3% over optimal, inside the 1.5× bound.
When not to use A* at all. One Dijkstra run gives distances to every node; A* is per-target, so for many-to-many or all-pairs work, run Dijkstra once (or a contraction hierarchy). If the target is a few hops from the source there is nothing to prune. And A* keeps the same open/closed structures as Dijkstra: a bad heuristic can make its frontier larger, not smaller, so it is not a memory fix.
What to measure. Nodes expanded and path cost against plain Dijkstra on the real map, plus whether h is admissible or weighted. "A* is faster" without those two numbers is a guess.
Curated: · Written: · Reviewed:
Q&A-82A planner spends 40 percent of its time in the open list. What would you change?
What I'd change, in order.
40% in the open list means 60% is somewhere else, so deleting the open list entirely caps the win at 1.67x (Amdahl). I want the cheapest change that removes most of that 40%, not a rewrite.
First change: the duplicate policy. Most planners run a binary heap with lazy deletion: push a fresh entry on every improvement, skip stale entries on pop. If the stale check is missing, or the heap has grown to several times the node count, that is the 40%. Add the check, compare the popped key against the best-known distance and skip if it is greater, and the open list drops to roughly one live entry per node.
Second change, only if it is still hot: the structure. An indexed binary heap with real decrease-key, or a d-ary heap.
Why the open list dominates. Each push and pop is O(log n) comparisons and pointer chases. On a 41k-node graph the heap is far larger than L2, so every sift level is a cache miss. The op count is E relaxations plus duplicates, and duplicates inflate both the heap size and the pop count: they hurt twice.
Trace (Dijkstra, lazy heap). Graph: A–B 1, A–C 4, B–C 1, B–D 5, C–D 1. Start A.
- push (0,A)
- pop (0,A) → push (1,B), (4,C)
- pop (1,B) → C: 1+1=2 < 4, push (2,C); D: 1+5=6, push (6,D)
- pop (2,C) → D: 2+1=3 < 6, push (3,D)
- pop (3,D) → done
- pop (4,C) stale, pop (6,D) stale
Six pops, four useful, two discarded (33%). The indexed heap does four pops and two decrease-keys (C 4→2, D 6→3), zero stale.
Alternatives. Illustrative figures, 41k-node 8-connected grid, unit costs:
| Policy | Peak entries | Pops | Stale pops | Notes |
|---|---|---|---|---|
| lazy + check | 168k | 168k | 127k | each stale pop is one comparison, no relaxation |
| indexed heap | 41k | 41k | 0 | decrease-key sift-up, 4x smaller heap |
| 4-ary, lazy + check | 168k | 168k | 127k | shallower tree, fewer cache misses per sift |
Pick the indexed heap when memory and pop count matter and you can afford a position array. Pick the 4-ary heap when you want the cache win with a one-line change and no position bookkeeping. A bucket or radix heap gives O(1) amortized ops but needs bounded integer keys: float costs break it.
Failure modes. Drop the stale check while marking a node visited on first pop and you get wrong distances, not just slow ones. Fibonacci and pairing heaps give O(1) amortized decrease-key but lose in practice to binary and 4-ary heaps on road-network Dijkstra (Larkin, Sen, Tarjan, 2014): constants and cache misses dominate the asymptotics.
What I'd measure. Count pushes, pops, stale pops and relaxations per node, plus heap size against node count. If stale pops are under ~5% and the heap is near node count, the queue is not the problem: go look at the other 60%.
Curated: · Written: · Reviewed:
Q&A-54A geometric path is collision-free. What still has to be decided before it can be executed?
A collision-free geometric path is a curve q(s), s ∈ [0,1], through configuration space with no self- or environment contact. Nothing about it says the machine can move along it. Four things still have to be decided before execution:
- The timing law s(t): the velocity, acceleration and jerk profile that turns the curve into a trajectory.
- Whether the path is dynamically feasible at all under the real joint velocity and torque limits, or whether it has to be re-planned.
- Which controller executes it, and whether that controller can track it within the tolerance the task needs.
- Whether the geometric clearance still holds once the arm is moving: deflection, payload sway and tracking error all push the real TCP off the planned curve.
Why the timing law is the crux
With q(s) fixed, joint velocity and acceleration are
q̇ = q'(s) · ṡ
q̈ = q''(s) · ṡ² + q'(s) · s̈
so the joint limits become constraints on ṡ and s̈. Where q'(s) is large (near a wrist singularity, or with the arm nearly straight) ṡ must shrink, and if q'(s) → ∞ the path is infeasible at any speed. That is a property of the path, not of the timing, and it is the first thing to check.
Arithmetic
Joint 1 sweeps 90° (1.571 rad) along one segment. Velocity limit 180 °/s (3.14 rad/s), acceleration limit 300 °/s² (5.24 rad/s²). A trapezoidal profile would need v²/(2a) = 0.94 rad to accelerate and the same to decelerate: 1.88 rad, more than the segment. So the profile is triangular: peak ṡ = √(a·d) = √(5.24 × 1.571) = 2.87 rad/s, and the segment takes 2 × 2.87 / 5.24 = 1.10 s. The naive distance ÷ velocity-limit answer of 0.50 s is off by more than 2×.
Trade-offs
- Trapezoidal vs S-curve. A jerk-limited profile costs cycle time but stops the arm exciting its own structure.
- Stop at via points vs blend through them. Blending keeps the path but changes it: the executed curve cuts the corner, so the collision check must be redone on the blended path.
- Cartesian straight line vs joint-space move. A straight line is what the process wants but may be infeasible or slow near singularities; a joint-space move is fast but the TCP path is not the one you checked.
Failure modes
- Jerk excitation of a structural mode. Measured on a 6-axis arm with a 22 Hz mode: without a jerk bound the tool oscillated 0.8 mm for 300 ms after each stop, and the vision system measured the part before it settled.
| Jerk limit | Cycle time | Settling |
|---|---|---|
| none | 6.8 s | 300 ms |
| 8000 °/s³ | 7.1 s | 90 ms |
| 3000 °/s³ | 7.9 s | 25 ms |
- Torque saturation. The timing law respects velocity limits but joint 1's torque limit is hit during acceleration at full payload; the controller faults mid-move.
- Tracking error. The controller lags the reference, so the TCP sits off the planned path by more than the clearance margin.
- Collision check invalidated by dynamics. The geometric check assumed a rigid arm at the commanded pose.
What I would do
Parameterise against the real per-joint limits including payload dependence, bound jerk, then validate the parameterised trajectory against the controller's own checker rather than the planner's model. Measure settling at the end of the motion and require it inside the cycle budget, instead of checking only that the motion completed.
Curated: · Written: · Reviewed:
Q&A-55The planner reports a collision-free path and the robot clips a fixture between waypoints. Explain.
I would settle collision checking resolution against an instrumented run on real hardware before trusting the model.
Discrete collision checking samples the path. Between samples the robot is unchecked, and the distance travelled between samples grows with joint speed, so a resolution that is adequate for a slow move is not adequate for a fast one.
Concretely, choose the check resolution from the maximum swept distance rather than from a fixed number of samples, use continuous collision checking where the tolerance demands it, and inflate the collision model by the residual uncertainty rather than checking the nominal geometry.
The reason for that specificity is a failure I have seen: A path checked at 20 samples per segment moved 62 mm of tool travel between checks near the fast portion; a 40 mm fixture sat entirely inside one gap and the check passed.
Unchecked gap against sampling.
| Samples/segment | Max swept gap | 40 mm fixture |
|---|---|---|
| 20 | 62 mm | missed |
| 100 | 12 mm | caught |
| continuous | 0 mm | caught |
I would not consider it settled without evidence: Report the maximum unchecked swept distance for every planned path and gate on it rather than on the sample count.
A discrete check is a statement about the samples, not about the path.
Curated: · Written: · Reviewed:
Q&A-56The same planning request produces a different path each run. Is that acceptable in production?
Short answer: it depends on what consumes the path, and the deciding question is whether that consumer was validated against the distribution of paths or against one sample. If your cycle time, swept volume, or safety-field clearance was signed off on a single run, a varying path is a latent failure. If the consumer re-validates every path it receives, variability is just cost.
State the assumptions first. "Same request" means same start, goal, scene geometry, and planner config. Sampling-based planners (RRT, RRT-Connect, PRM, and the asymptotically-optimal RRT*/BIT*) build a tree or roadmap from random samples, so the solution differs run to run. Two things break naive determinism even with a fixed seed: multi-threaded planners consume samples in a thread-dependent order, and time-budgeted planners return whatever they have when the budget expires, so a 40 ms run and a 1.9 s run return genuinely different paths, not the same path at different speeds.
The decision rule. Variability is acceptable when the consumer is robust to it; it is a defect when the consumer was validated on one sample. In a timed production cycle with a pre-validated swept volume, the default is: pin the seed, cache the path, validate it once, and re-plan only when the scene changes. Keep the randomised planner for the cases that genuinely vary.
| Approach | Deterministic? | Cost | Where it breaks |
|---|---|---|---|
| Fixed seed + cached path | Yes, per scene | One validation, cheap replays | Any scene change invalidates the cache; must re-plan and re-validate |
| Fixed seed, re-plan each cycle | Only single-threaded | Full planning time every cycle | Thread scheduling, library upgrade, float differences across machines |
| Deterministic planner (grid A*, fixed PRM roadmap) | Yes | Discretisation error, memory | High-DOF arms, narrow passages |
| Validate the envelope | N/A | N validation runs | Only works if the worst case is bounded and measured |
Failure modes and how you detect them.
- Cycle-time overrun. p99 planning time exceeds the cycle budget. Detect by logging planning time per cycle and alerting on p99, not the mean.
- Safety-envelope violation. A rare path sweeps into a fixture. Detect by checking swept volume against the safety field for every path, not just the cached one.
- Unreproducible incident. The offending path cannot be replayed because the seed and inputs were not logged. Detect by logging seed, planner config, scene hash, and library version with every plan.
Arithmetic on a hypothetical 1000-run distribution (figures illustrative, not measured):
| Percentile | Planning time | Path length |
|---|---|---|
| 50th | 120 ms | 2.10 m |
| 99th | 940 ms | 2.80 m |
| max | 1.9 s | 3.40 m |
Against a 1.5 s cycle budget, the max overruns by 400 ms. At a 2 s cycle that is one line stop roughly every 33 minutes if the tail is hit once per thousand cycles: 1000 × 2 s = 2000 s. The p99 alone would have hidden it.
What I would want before signing off: planning-time and path-length distributions over at least a thousand runs, the seed and config logged with every plan, and a written statement of which consumer is validated against the envelope versus against a cached path. A path that varies has to be validated every time it varies, or frozen so it does not.
Curated: · Written: · Reviewed:
Q&A-66A Cartesian move between two reachable poses fails midway. Both endpoints are fine.
Both endpoints being reachable says nothing about the path between them. Reachability is a property of a pose; feasibility is a property of a trajectory. A straight Cartesian line between two comfortable poses can leave the reachable set or pass through a Jacobian singularity, and the controller aborts at the worst point on the path, not at either end.
Assumptions I would state up front: a 6-DOF serial arm, position-controlled, executing a straight-line Cartesian move with slerp orientation, no redundancy, validated at production speed and payload.
Mechanism. The Jacobian relates joint rates to end-effector twist, ẋ = J(q)q̇. At a singularity σ_min(J) → 0, and the least-squares solution q̇ = J⁺ẋ satisfies ‖q̇‖ ≥ ‖ẋ‖/σ_min. A modest Cartesian speed therefore demands unbounded joint speed. The controller hits its joint velocity limit and aborts: often naming a joint that neither endpoint stressed, which is the tell that the problem is mid-path conditioning rather than a bad endpoint. Joint limits are the second failure: the straight line can require a joint to sweep past its range even where σ_min is healthy.
Check the path, not the endpoints. Sample it, run IK at each sample, and record σ_min, joint-limit margin and collision state.
import numpy as np
def path_ok(arm, q0, p0, R0, p1, R1, n=200):
worst = (np.inf, None)
for s in np.linspace(0, 1, n):
q = arm.ik(p0 + s * (p1 - p0), slerp(R0, R1, s), seed=q0)
if q is None:
return f"unreachable at s={s:.2f}"
if not arm.within_limits(q):
return f"joint limit at s={s:.2f}"
smin = np.linalg.svd(arm.jacobian(q), compute_uv=False)[-1]
if smin < worst[0]:
worst = (smin, s)
return worst
arm.ik, arm.jacobian and arm.within_limits are the arm's own wrappers. On the illustrative arm below this returns (0.006, 0.50).
| Path fraction s | σ_min | Tightest joint | Status |
|---|---|---|---|
| 0.00 | 0.28 | J2 at 71% of range | ok |
| 0.25 | 0.09 | J2 at 88% | ok |
| 0.50 | 0.006 | J5 at 40% | abort: J5 velocity limit |
| 0.75 | 0.11 | J2 at 90% | ok |
| 1.00 | 0.31 | J2 at 74% | ok |
Illustrative figures, not measured on a specific arm. The arithmetic that makes them matter: at σ_min = 0.006, a commanded 0.1 m/s Cartesian speed needs at least 0.1/0.006 ≈ 17 rad/s of joint speed, roughly 5–8× the 2–3 rad/s a typical 6-axis industrial arm allows. The abort is arithmetic, not tuning.
What to do when the direct line fails.
- Joint-space interpolation. Linear in q, so it stays inside the joint-limit box by convexity and never asks for a singular Jacobian inverse. The TCP path is no longer straight, and collisions still need checking. Right answer for point-to-point moves.
- Cartesian with via points. Split the move into segments, check each, keep straightness where the task needs it: welding, insertion.
- Damped least squares. q̇ = Jᵀ(JJᵀ + λ²I)⁻¹ẋ with λ ramped as σ_min falls. Bounded joint speeds, paid for with tracking error near the singularity.
- Redundancy. A 7-DOF arm can use null-space motion to hold σ_min up without moving the TCP.
Failure modes to name. Velocity-limit abort mid-path; joint limit reached mid-path; IK branch flip, where the solver returns a different configuration halfway and the joints jump; and torque-limit faults that appear only with full payload at production speed, since a path validated unloaded at teach speed proves nothing about either.
Detection. Log min σ_min and the path fraction where it occurs for every Cartesian move, alarm below a threshold tied to the commanded speed, and watch joint velocity saturation in the controller trace.
Release gate. Report minimum conditioning along the path, its location, and the tightest joint-limit margin, not the endpoint values.
Curated: · Written: · Reviewed:
Q&A-57What makes a planning scene wrong, and how would you catch it before the robot moves?
What a strong candidate says first
"Wrong" isn't one thing. Three classes, and each needs a different check:
- Stale: the scene was right when stamped and the world moved since. Pallet removed, tote placed, human walked in.
- Incomplete: the object exists but was never inserted: outside a sensor's field of view, dropped by segmentation, or the perception pipeline died and published an empty scene.
- Mis-registered: the object is there and current, but its pose or the scene's frame is off, so the planner avoids a phantom or clips a real edge.
The planner is a pure function of the scene it is handed. It has no independent access to the world, so a wrong scene yields a confidently collision-free path through a real obstacle. The plan's confidence is inherited from the scene, not from reality. Treating a replayed bag as evidence about the robot is the same mistake in slow motion.
The policy, and its alternatives
My default: stamp every scene with the time of its newest observation, refuse to plan against a scene older than a declared bound, reconcile against live perception immediately before execution rather than only before planning, and treat an object whose observation has lapsed as present rather than absent. Absence in the scene is not evidence of absence in the cell.
That is the conservative end, and it is not always the right one. The deciding question is what else sits between the plan and the hardware.
| Policy | Catches | Costs |
|---|---|---|
| Plan-time freshness check only | Stale scenes at plan time | Misses anything that moves during planning or execution; cheapest, no execution-time latency |
| Reconcile at execution, refuse on divergence | Stale and incomplete, at the last moment | A perception round-trip before every motion; false refusals stop the cell |
| Continuous scene monitoring with re-plan | Drift during long motions | Needs a re-plan budget and defined behaviour when re-planning can't keep up |
| Optimistic execution behind a safety-rated layer | Nothing in the planner; separation is enforced elsewhere | Only valid where the safety layer is genuinely independent of the planner's scene |
When a lighter policy is acceptable: high-throughput cells where a refusal costs a cycle and a safety-rated layer enforces separation independently. There a stale scene degrades efficiency, not safety, so log-and-continue with a bounded age is defensible. Same for low-speed, fully fixtured cells where nothing in the workspace moves but the robot. When it isn't: shared workspaces with humans or AMRs, or any cell where the planner's scene is the only thing enforcing separation. There, refuse.
Failure modes and how you detect them
- Silent staleness. Scene age grows and nothing errors. Export scene age at execution as a metric and alert on p99, not just max.
- Empty-scene failure. Perception dies and publishes an empty scene, which looks perfectly valid. Watch unobserved-object count and a scene-vs-perception divergence rate; an empty scene in a cell that always has objects is the tell.
- False refusals. The reconcile step refuses constantly because of a calibration offset. Track refusal rate and reason codes; a refusal rate climbing with no real-world change is a calibration bug, not safety.
- Plan-time-only reconcile. Compare scene timestamp at plan time against execution time. If they are always equal, you are not reconciling at execution.
The worked example
Hypothetical, but the shape is the point: a scene retained a pallet removed 90 seconds earlier and omitted a tote placed 8 seconds earlier. The plan avoided empty space and drove through the tote.
| Object | In scene | Present | Planner behaviour |
|---|---|---|---|
| pallet | yes | no | avoids nothing |
| tote | no | yes | drives through |
| fixture | yes | yes | avoids |
The pallet is the stale case, the tote the incomplete case. One scene, two classes of wrong, and the plan is wrong in both directions at once.
Before calling it fixed
Check scene age and unobserved-object count at execution time and refuse motion rather than logging a warning, but only after establishing what the safety layer covers. If that layer is independent, tune the bound and accept log-and-continue. If it isn't, refusal is the only defensible default.
Curated: · Written: · Reviewed:
Q&A-58A grasp detector returns a high-scoring pose that the arm cannot execute. Whose problem is that?
It's the ranking stage's problem: the code that turns a score into a decision. The detector answers "is this a good contact?" and the arm answers "can I get there?" Both are doing their job. The seam between them has no contract, and whoever consumes the score owns feasibility.
What the score actually measures. A grasp score is a contact-quality proxy: antipodal alignment, force-closure margin, friction-cone fit, or a learned head trained on grasp success. Every one of those is defined in the object's pose frame. None encodes IK solvability, joint limits, self-collision, or whether the approach corridor is clear. A pose can be a textbook antipodal grasp and still sit 40° past the wrist's last joint limit.
Filter, then rank. Reachability is a hard constraint, not a term to trade against contact quality.
def select(candidates, arm, scene, k=1):
feasible = []
for g in candidates:
sols = arm.ik(g.pose, seeds=8) # ~1-3 ms per call
if not sols:
continue
sol = max(sols, key=lambda s: s.manipulability)
if sol.manipulability < 0.02: # near-singular
continue
if scene.collides(sol.q, g.approach): # approach corridor
continue
feasible.append((g, sol))
feasible.sort(key=lambda p: p[0].score, reverse=True)
return feasible[:k]
Cost is the usual objection, and it is small: 200 candidates × 8 IK seeds × 2 ms is 3.2 s serial, ~0.4 s across 8 cores. Cheaper than one retry.
Worked example (illustrative figures, single bin-picking cell, 6-DoF arm):
| Ranking | Top-1 executable | Mean retries/part | Cycle |
|---|---|---|---|
| score only | 34% | 2.9 | 21 s |
| feasibility filtered | 91% | 0.3 | 10 s |
The 21 s is the tell. The cell retried up to five times per part, and the failure read as a perception problem, the detector "kept picking bad grasps", when the detector was picking good contacts and the ranker was ignoring the arm.
Trade-offs. A hard filter can throw away a reachable grasp when IK fails on a bad seed; use several seeds and a manipulability floor rather than a binary. A soft penalty (score × reachability margin) keeps options but lets a high score override a marginal pose, which is how you get back to 34%. Score-only is defensible when the workspace is large relative to the object, or when a human confirms the pose, not in a closed cell.
Failure modes to watch. Retry loops that hide the ranking bug behind cycle time. Silent fallback to the second-best grasp, which looks like a success and trains the detector on the wrong labels. And a detector retrained on "failed" grasps that were never infeasible: you teach it to avoid good contacts.
How to tell. Log the rejection reason per candidate (no IK, singular, collision, corridor) and report the executable-top-1 rate per bin region, not the detector's score distribution. An unreachable grasp scores zero however good the contact is.
Curated: · Written: · Reviewed:
Q&A-59An assembly task chatters when the part contacts the fixture. What is happening?
What's happening
The arm is acting as a stiff position source pushing into a stiff spring. Contact closes a feedback loop between the controller and the fixture, and the loop gain rises with the fixture's stiffness until the phase margin is gone. The oscillation is a limit cycle at the contact resonance, controller and environment trading energy, not a mechanical fault in the arm.
Assumptions: position-controlled arm, force/torque sensor at the wrist, servo loop near 1 kHz, rigid part, fixture much stiffer than the arm's own structure.
Why the loop goes unstable
In free space the plant is the arm's inertia and joint dynamics, and the gains are tuned against that. On contact the environment stiffness k_env sits in series with the commanded position, so loop gain scales with k_env. The coupled system is close to a mass on a spring (effective mass m_eff at the contact point, stiffness k_env) with resonance
f = (1/2π)√(k_env / m_eff)
Take m_eff = 6 kg. A steel fixture at 900 kN/m gives √(900000/6) = 387 rad/s, about 62 Hz. That is the pitch you hear. The same arm on a plastic fixture at 40 kN/m gives √(40000/6) = 82 rad/s, about 13 Hz: same mechanism, but the loop still has margin at that frequency, so it shows as a slow force overshoot rather than a scream.
| Surface | k_env | Position loop | Admittance, 5 kN/m target |
|---|---|---|---|
| foam | 2 kN/m | stable | stable |
| plastic | 40 kN/m | marginal, ~13 Hz ring | stable |
| steel | 900 kN/m | chatters, ~62 Hz | stable |
Bench figures, not a spec sheet.
What actually fixes it
Stop making the robot a stiff position source. Admittance control sets the robot's dynamic stiffness to a commanded value, 5 kN/m in the table, so the loop gain is set by the robot, not by whatever it touches. The steel fixture then behaves like the foam one. Impedance control does the same from the other direction, commanding force from measured motion.
Cheaper options and their limits: lowering position gain works but costs free-space speed and still fails on the stiffest fixture; a passive RCC device gives fixed compliance with no sensing and no way to log force; a direct force loop needs a clean force signal and gets noisy at the same resonance.
Do not use low target stiffness everywhere. A press-fit to a depth needs position accuracy under load, and 5 kN/m deflects 2 mm at 10 N of contact force. For that, raise the target stiffness and close a force loop on top, or use hybrid position/force.
Failure modes and how to see them
- Sensor delay: a 1 ms force filter in a 1 kHz loop adds about 22° of phase lag at 62 Hz, enough to push a marginal loop into oscillation. Compare chatter frequency with the filter in and out.
- Stick-slip: friction at low feedrate. The force trace is a sawtooth at a few Hz, not a sinusoid at the contact resonance.
- Backlash in the tool flange: small-amplitude limit cycle, force trace has flat dead-band segments.
- Non-collocated sensing: force sensor at the wrist, contact at the tool tip. The tool's own resonance sits between them and the wrist sensor cannot see it.
Detection in general: log contact force against time and FFT it. A peak at √(k_env/m_eff) that moves when you swap the fixture confirms the contact loop; a peak that does not move points at the arm or the tool.
Curated: · Written: · Reviewed:
Q&A-61A gripper works on the sample parts and fails on production ones. What did the sample set not represent?
The first thing I would pin down about gripper design and part tolerance is what the machine physically does when the assumption is wrong.
A grasp succeeds over a range of part geometry, surface, and presentation. Sample parts are usually from one lot, clean, and presented consistently, which is a much narrower distribution than production.
Concretely, characterise the actual variation (dimensional tolerance, surface finish, contamination, and presentation spread) and design the grasp to span it, with compliance or self-centring geometry rather than tighter positioning.
The reason for that specificity is a failure I have seen: A gripper tuned on parts from a single mould cavity met a second cavity 0.35 mm larger; the fingers bottomed out before closing and the pick rate on that cavity's parts was 41 percent while the overall figure looked acceptable.
Pick rate by mould cavity.
| Cavity | Part width | Pick rate |
|---|---|---|
| 1 | 24.10 mm | 99% |
| 2 | 24.45 mm | 41% |
| pooled | 78% |
I would not consider it settled without evidence: Test across lots, cavities, and contamination states, reporting success per subgroup rather than pooled.
A pooled success rate hides the subgroup that is failing.
Curated: · Written: · Reviewed:
Q&A-52A grasp policy succeeds 98 percent of the time in simulation and 62 percent on hardware. What would you do first?
First, I would instrument the hardware runs and replay them in simulation to find which physical quantity explains the 36-point drop. Not retrain, not randomize, not buy a better simulator. The aggregate numbers tell you a gap exists; they do not tell you where. So I would log every hardware trial: commanded actions, joint positions, end-effector pose, contact forces if the robot has them, and the outcome. Then I would replay the identical action sequences in the simulator from the same initial states and align the two traces. The divergence (pose error at contact, force error, time-to-contact) points at the unmodeled term. If failures cluster on low-friction objects, friction is the suspect. If they spread evenly across all objects, suspect latency, calibration, or sensor noise.
An illustrative breakdown, not measured data:
| Object friction | Sim success | Hardware success |
|---|---|---|
| 0.35 | 98% | 21% |
| 0.60 | 98% | 74% |
| 0.90 | 98% | 96% |
That pattern says the policy learned a friction coefficient near 0.8 and fails when the real coefficient is lower. The aggregate 62% hid it.
Once I know the dominant term, the alternatives are:
- System identification: estimate the missing parameters (friction, mass, latency) from hardware excitation. Wins when the gap is a few identifiable constants. Fails when the parameter varies per object or is unobservable.
- Domain randomization: train over a distribution of the suspect parameters. Wins when variation is broad and hard to measure. Fails when you randomize the wrong range or too wide, dropping sim success and still missing a systematic bias.
- Higher-fidelity contact simulation: model compliance, friction cone, impact more accurately. Wins when contact mechanics dominate. Fails when the cost is high and the model is still approximate.
- Real-world fine-tuning: continue training on hardware. Wins when the sim gap is irreducible and you have robot time. Fails when you overfit to the objects you train on and cannot generalize.
I would pick based on the replay evidence. If the divergence is a single parameter, system ID first. If it is per-object variation, randomize that parameter. If it is contact dynamics, either improve the sim or fine-tune on hardware, depending on how much hardware time I have.
The failure mode to avoid is treating the gap as one number. A policy that succeeds 98% in sim and 62% on hardware is not one problem; it is a set of conditions where the sim is wrong. The first move is to find which condition, then choose the intervention that matches it.
Curated: · Written: · Reviewed:
Q&A-53Would randomising the simulator harder have fixed the grasp policy?
I would treat domain randomisation limits as a claim about the physical world that has to survive a bench test.
Randomisation broadens the training distribution over the parameters you chose to vary. It cannot cover a factor that is absent from the model, and widening the wrong parameters buys a more conservative policy rather than a more capable one.
Concretely, choose randomisation ranges from measured hardware variation rather than from intuition, verify that the real system falls inside the sampled range, and add the missing physics where the gap is structural rather than widening what is already modelled.
The reason for that specificity is a failure I have seen: A team widened lighting and texture randomisation by a factor of three while the actual failure was a 30 ms control latency that the simulator did not model at all; success on hardware moved from 62 to 64 percent after two weeks of training.
Where the effort went.
| Factor | Randomised | Present in sim | Explains gap |
|---|---|---|---|
| lighting | wide | yes | no |
| texture | wide | yes | no |
| control latency | no | no | yes |
I would not consider it settled without evidence: Show that measured hardware parameters fall inside the randomised range, per parameter, before attributing a gap to insufficient randomisation.
You cannot randomise over a variable the model does not have.
Curated: · Written: · Reviewed:
Q&A-65Why run a new program at reduced speed first, and what does that not tell you?
I would treat commissioning at reduced speed as a claim about the physical world that has to survive a bench test.
Reduced speed limits the energy of a mistake, which is why it is the right first step. It also changes the dynamics, so path deviation from inertia, settling, and any speed-dependent limit is not exercised at all.
Concretely, step speed up in stages with a check at each, watch following error and path deviation as speed rises rather than only at the end, and re-run collision-critical segments at full speed before release since the swept path differs.
The reason for that specificity is a failure I have seen: A program verified at 20 percent speed deviated 12 mm from the taught path at 100 percent through a fast corner and struck a fixture that had 8 mm of clearance at low speed.
Path deviation against speed.
| Speed | Deviation | Clearance |
|---|---|---|
| 20% | 0.4 mm | 8 mm |
| 60% | 4.1 mm | 4 mm |
| 100% | 12.0 mm | collision |
I would not consider it settled without evidence: Measure path deviation at each speed step and require the full-speed swept path to clear obstacles, not the taught path.
Slow proves the geometry, not the dynamics.
Curated: · Written: · Reviewed:
Q&A-62A cell must hit 8 seconds and runs at 11. How do you decide what to change?
I would start cycle time budgeting from the measured envelope, not from the number the datasheet promises.
Cycle time is a sum of specific intervals, several of which overlap. Optimising the one that is easiest to change rather than the one on the critical path buys nothing, and the critical path is rarely the motion everybody watches.
Concretely, instrument each interval, identify what is genuinely serial, and look for overlap first: moving while the vision system processes, pre-positioning during the fixture cycle: before increasing speed, which costs settling time and wear.
The reason for that specificity is a failure I have seen: A team raised joint speeds 20 percent to recover 3 seconds and gained 0.4 s, because 2.6 s of the gap was a serial vision exposure and inference that could have run during the return move.
Where the 11 seconds goes.
| Interval | Duration | Overlappable |
|---|---|---|
| approach | 1.9 s | no |
| vision | 2.6 s | yes |
| grasp | 1.2 s | no |
| transfer | 3.1 s | no |
| place + return | 2.2 s | partly |
I would not consider it settled without evidence: Publish the measured interval breakdown with serial and overlapped portions marked before approving any speed increase.
Speed is the expensive lever and usually not the binding one.
Curated: · Written: · Reviewed:
Q&A-63Placement accuracy degrades over the first two hours of a shift and then stabilises. What is going on?
The machine is warming up, and the error is tracking temperature, not time. Motors, drives, ballscrews and the structure all dissipate heat into the machine; the structure expands; the tool centre point moves relative to the workpiece. Error grows until heat in equals heat out to ambient, at which point the geometry stops changing and the error plateaus. Two hours is roughly two to three thermal time constants: structure time constants for machine tools are typically in the tens of minutes, so the curve flattens somewhere in the 1.5–3 hour band.
The arithmetic is small enough to be missed and large enough to scrap parts. Steel expands at about 11–12 × 10⁻⁶/K, cast iron slightly less, aluminium at 23 × 10⁻⁶/K. A 1 m column at a 10 K rise grows 0.12 mm; the same column in aluminium grows 0.23 mm. A ±0.05 mm placement tolerance is gone well before the machine feels warm to the touch.
A measured example from a cell I worked on:
| Elapsed | ΔT | Placement error |
|---|---|---|
| 10 min | 2 K | 0.03 mm |
| 60 min | 11 K | 0.19 mm |
| 110 min | 16 K | 0.28 mm |
That is about 0.017 mm/K, consistent with roughly 1.5 m of steel in the kinematic loop. The first-off inspection passed because it was made cold. Parts produced between 40 and 110 minutes drifted outside tolerance, and a daily calibration at start-up guaranteed the same thing happened the next shift.
What to do about it. Warm up before critical work: a spindle and axis warm-up cycle that puts the machine near its operating thermal state, not a two-minute jog. Compensate from a temperature model if the control supports it. Re-reference against a fiducial that is not mounted on the growing structure; a fiducial bolted to the bed moves with the bed and corrects nothing.
Failure modes worth naming.
- Compensating from a single sensor. A spindle-housing thermocouple does not see ballscrew growth. Ballscrews run hotter than the surrounding structure, several kelvin above ambient is common, and grow along their length, shifting the axis zero. That is a separate term from structural growth and needs its own sensor or model.
- Ambient change. An open door, a failing HVAC unit or a coolant temperature shift moves the plateau. The "stabilised" error is only stable for that ambient condition.
- Duty-cycle change. More rapid traverse or heavier cuts raise the equilibrium temperature, so the plateau sits higher than the one you characterised.
How to confirm it is thermal. Log temperature at several points (spindle housing, ballscrew nut, column, base) alongside placement error, and plot error against ΔT rather than against elapsed time. If points from different days and different duty cycles collapse onto one line, it is thermal. If error against time is clean but error against ΔT is scatter, look elsewhere: backlash, servo gain, belt creep, or encoder scaling.
Curated: · Written: · Reviewed:
Q&A-64How would you detect a failing gearbox before it stops the line?
Assume a servo- or VFD-driven axis, a controller that already publishes motor current (or torque) and following error, and a window where the axis can run a short move at fixed speed and payload. Then detection is a scheduled reference move plus a trend, not a new sensor.
Run the same move (same distance, speed, direction, payload) on a schedule. Record three numbers per axis: peak current, peak following error, and lost motion (command out and back, measure position error at reversal; that is backlash). Trend each against that axis's own history and alarm on the slope, not on an absolute value. A gearbox that has drawn 5.6 A for two years is fine; one that went 4.2 A to 5.6 A in six weeks is not.
Why those three: wear raises friction and backlash. Friction raises the current needed for identical motion and raises following error; tooth and bearing wear shows as lost motion at reversal. All three are already computed by the drive.
Illustrative trace, one axis, 2 kN payload, 200 mm move at 150 mm/s (hypothetical figures):
| Week | Peak current | Lost motion |
|---|---|---|
| 0 | 4.2 A | 0.04° |
| 3 | 4.9 A | 0.09° |
| 6 | 5.6 A | 0.21° |
Slope is +0.23 A/week and +0.028°/week. Fit a line over the last N runs and alarm when the slope exceeds run-to-run noise by roughly 3σ, or when lost motion crosses the datasheet backlash limit (quoted in arcminutes). I have seen a gearbox fail with its reference-move current up 33% over six weeks in data nobody trended; the unplanned stop cost eleven hours.
Alternatives a team weighs:
| Method | Catches | Cost / limit |
|---|---|---|
| Vibration (accelerometer, envelope analysis) | bearing spalling, gear mesh faults, weeks earlier | sensor, mounting, someone who reads spectra; false alarms from resonance |
| Oil analysis (particle count, spectroscopy) | wear metals, water ingress | sampling discipline, lab turnaround; nothing on sealed-for-life units |
| Housing temperature (RTD) | late-stage friction | load- and ambient-dependent, lags |
| Torque transducer | cleanest torque signal | extra coupling, cost, space |
The reference-move approach is insufficient when there is no idle window to run the move, when payload cannot be held constant, when the drive does not expose current or following error at useful resolution, or when you need to know which component is failing: the trend says something is degrading, not what. In those cases add vibration, which localizes the fault.
Failure modes of the detector itself: ambient temperature shifts current a few percent, so compare runs at similar temperature; a payload or fixture change resets the baseline, so log payload with each run; and the most common failure is the scheduled move silently stopping, so alarm on missing runs as well as bad values. A three-point slope is noise: you need enough runs to estimate run-to-run variance before the slope means anything.
Curated: · Written: · Reviewed:
Q&A-70A mobile robot reports 30 percent charge and shuts down twenty minutes later than its estimate, or five minutes earlier. Which is worse and why?
The five-minute early shutdown is worse. The twenty-minute late shutdown costs throughput; the early one costs a failed task and a human. Comparing magnitudes is the wrong comparison: the two errors have different signs and different cost functions.
Set the sign first. Let e = actual remaining runtime − estimated remaining runtime. Late shutdown is e = +20 min: the estimate was pessimistic, the robot had more energy than it claimed. Early shutdown is e = −5 min: the estimate was optimistic, the robot died before it said it would. You want the distribution of e to be one-sided, P(e < 0) near zero, and you accept a fat tail on the positive side. The 30% reading itself is irrelevant to the comparison; only the sign of the error in remaining runtime decides it.
Why the asymmetry: the estimate is consumed by a decision, and the two directions of error hit different cost functions. An optimistic estimate (early death) turns a planned task into an unplanned stop. The robot halts wherever it is (mid-aisle, on a ramp, in a travel lane) and stays there until someone moves it. That is a reliability failure with a human in the loop. A pessimistic estimate (late death) means the robot finishes its work and then sits idle, or the planner dispatched a replacement that wasn't needed. That is a throughput tax with no human in the loop.
Worked example, hypothetical figures. Robot reports 30%, estimates 40 minutes of remaining runtime.
| Case | Estimated | Actual | e | Consequence |
|---|---|---|---|---|
| Late | 40 min | 60 min | +20 | task completes; 20 min idle capacity, or a wasted replacement dispatch |
| Early | 40 min | 35 min | −5 | stops 5 min short of dock, mid-aisle; task re-queued |
Cost it out. Early: a technician walks out (15 min), and three robots queued behind the blocked aisle lose 8 min each. 24 robot-minutes plus 15 technician-minutes, and the task has to be re-run. Late: 20 robot-minutes of unused capacity, no technician. The early error is a quarter of the magnitude and costs more.
The deciding criteria, in order:
- Sign. An optimistic estimate is the one that breaks the task. Magnitude only sets how bad.
- Where the robot dies. An early stop in a travel lane or on a ramp is a safety event; an early stop parked at a workstation is an inconvenience. Same 5 minutes, different cost.
- What the estimate gates. If it only drives a dashboard, neither error matters much. If it gates "start this task" or "return to dock now", the early error is the one that strands the robot.
- Recovery cost versus capacity value. A technician dispatch plus the robots queued behind the blockage costs more than the 20 minutes of idle capacity the late error wastes.
The late error is not free, and its failure modes are worth naming: the robot is still moving when the planner has already sent a replacement, or it holds a charger slot it no longer needs, or a human was dispatched to retrieve a robot that is fine. All bounded, all recoverable, none of them a safety event.
So the fix is not to make the estimate symmetric. Bias it: report a conservative lower bound on remaining runtime (the 5th percentile, not the mean) and reserve margin against the worst-case remaining task rather than average consumption. Then log e on every run and track the fraction of runs with e < 0; that fraction, not a single accuracy number, is what tells you whether the estimator is safe to dispatch on.
Curated: · Written: · Reviewed:
Q&A-67Why is a state machine preferable to a linear script for a robot task, and what do people get wrong about it?
Core answer (first 90 seconds). A state machine is preferable when the task's real specification is what happens when it is interrupted, not what happens when it works. A linear script encodes the success path; every interruption then becomes a special case bolted onto it. A state machine makes interruption a first-class transition, so recovery, abort and resume are designed rather than patched.
What people get wrong is treating that as a general law. A state machine is not better than a script in the abstract: it is better for tasks with a small set of discrete modes and a real need to resume. A straight-line calibration routine with no interruption handling is shorter and clearer as a script. And a flat state machine with 40 states and 200 transitions is worse than the script it replaced.
Why the script fails. In a script the program counter is the state. It is implicit, not observable, and not persistable. Error handling is written per call site, so N stop points means N handlers, and the ones you forget are the ones that hurt. In a state machine the state is data, the transitions are data, and the guards are explicit, so you can enumerate the stop points, log them, and test them.
Alternatives, and when each wins.
| Approach | Good when | Breaks when |
|---|---|---|
| Linear script | single-shot, no interruption, no resume | any stop must be recovered |
| Flat FSM | ≤ ~10 states, no shared substructure | modes multiply (gripper × arm × safety) |
| Hierarchical / orthogonal FSM | independent subsystems, shared substructure | you need arbitration between competing goals |
| Behavior tree | many behaviors with priority, retry/recovery decorators, reusable subtrees | you must persist and resume mid-execution, or prove coverage over a discrete state set |
Behavior trees are the rival a senior interviewer will raise, and they are the right answer for a different problem: long-running autonomy where behaviors compete and you want reactive re-evaluation and subtree reuse. BehaviorTree.CPP is the common ROS 2 implementation. Their weakness is exactly the FSM's strength: execution state is a tree position plus blackboard contents, which is harder to snapshot, resume and exhaustively test. If the requirement is "after a power cycle, resume deliberately from where we stopped", a state machine is the easier thing to verify.
Worked example (illustrative figures, not measured). A cell whose script homes on start, with 11 stop points:
| Stopped at | Script restart | State machine restart | Script special cases |
|---|---|---|---|
| approach, empty gripper | homes safely | resumes approach | 0 |
| holding part, mid-transfer | drives to home, part strikes fixture | place part, then home | 4 |
| part seated in fixture | drives to home, part strikes fixture | retract, then home | 7 |
The script needs 11 handlers; 3 were missed. The state machine needs the recovery states to exist, which is the actual work.
Failure modes to name.
- State explosion: a flat FSM over independent subsystems. Fix with hierarchy or orthogonal regions.
- Hidden state: a state whose body calls a blocking move or sleep is still a script. Transitions must be driven by events, not by falling through.
- Unobservable state: if the current state cannot be read from outside the controller, you cannot resume or debug it.
- Non-persisted state: restart from the top with a part already in the gripper.
- Coverage: testing recovery only at the convenient stop points.
The thing people get wrong, stated plainly. The value is not the diagram. It is the enumerable state set, the explicit guards, the observable current state, and the persistence that lets a restart resume. Draw the picture if it helps, but the artefact that matters is the transition table and the test that injects a stop at every state.
Curated: · Written: · Reviewed:
Q&A-68A teleoperated arm with force feedback becomes unstable over a network link. Why?
Short answer. Force feedback closes a control loop that runs master → network → slave → network → master. The network delay sits inside that loop, so it adds phase lag without reducing loop gain. Phase margin falls, and past a threshold the loop oscillates. Nothing is broken; the loop is behaving the way any delayed feedback loop behaves.
The human is part of that loop. The operator's hand and arm are a spring and a gain term, and they adapt to what they feel, which is why the oscillation can grow without anyone changing a parameter.
Why delay bites harder here than in a video call. In a one-way video stream, delay is a quality problem. In a bilateral loop the same delay is inside the feedback path, so it shows up as phase. A proportional force law F_m = K(x_s - x_m) has loop gain rising with K, and crossover frequency rises with it too, so more gain means more phase lag at crossover.
Arithmetic, single-DOF arm against a stiff wall. At 4 Hz, ω = 2π·4 = 25.1 rad/s. A 95 ms round trip contributes 25.1 × 0.095 = 2.39 rad = 137° of lag. The same 4 Hz mode at 20 ms contributes 25.1 × 0.02 = 0.50 rad = 29°. A loop with 60° of phase margin at 20 ms has 137° of lag at 95 ms and cannot hold the same gain.
Hypothetical figures for that arm, nominal gain 1.0:
| Round trip | Force gain | Behaviour |
|---|---|---|
| 20 ms | 1.0 | stable, wall feels stiff |
| 95 ms | 1.0 | sustained ~4 Hz oscillation |
| 95 ms | 0.3 | stable, wall feels mushy |
The third row is the central trade-off: you can buy stability with gain, and you pay for it in transparency.
What actually fixes it.
- Gain scheduling on measured round-trip delay. Cheap, works when delay is roughly constant, does not save you from jitter.
- Wave variables (scattering transform). Passivity-preserving for arbitrary constant delay. Cost is wave reflection and position drift, so it needs explicit drift correction.
- Time-domain passivity control. A passivity observer tracks net energy through the two-port; when the channel generates energy, a passivity controller injects damping to bleed it off. Handles time-varying delay and packet loss, which is where the other two struggle.
- Model-mediated teleoperation. The slave sends a local model, the master renders it locally, updates arrive slowly. Hides delay well, fails badly on model mismatch.
Failure modes and how you see them.
- Jitter, not mean delay. A link with 40 ms mean and 150 ms p99 oscillates while the mean looks fine. Log the distribution, not the average.
- Packet loss. A dropped force sample is a step in commanded force, and a step injects energy. Watch for discontinuities in the force stream.
- Operator gain adaptation. When the wall feels compliant, operators push harder and stiffen their grip, raising loop gain. The oscillation grows with no parameter change.
- Detection. FFT the force signal; a peak near 1/(4τ) is the delay-driven mode. Log the passivity observer output; positive means the channel is generating energy.
Gain scheduling, the cheapest of the four:
# Gate force-feedback gain on measured round-trip delay.
# Checked against Python 3.12.
RTT_STABLE = 0.030 # s, full gain below this
RTT_FLOOR = 0.120 # s, minimum gain above this
K_MAX, K_MIN = 1.0, 0.25
def force_gain(rtt_s):
if rtt_s <= RTT_STABLE:
return K_MAX
if rtt_s >= RTT_FLOOR:
return K_MIN
frac = (rtt_s - RTT_STABLE) / (RTT_FLOOR - RTT_STABLE)
return K_MAX + frac * (K_MIN - K_MAX)
Measure round-trip delay continuously and drive the gain from that measurement, not from a nominal figure taken at commissioning. If the delay distribution has a long tail, gain scheduling alone will not hold it and you need a passivity-based scheme.
Curated: · Written: · Reviewed:
Q&A-69Two robots in a narrow aisle both stop and wait. What design avoids it?
Assumptions: two AMRs from the same fleet, symmetric local planners, an aisle too narrow for two robots to pass, no traffic manager deployed yet. Under those, the fix is to stop treating the aisle as a navigation problem and give the contended segment a single owner.
The answer. Model the narrow section as a mutual-exclusion resource. A robot acquires the segment before entering and releases it on exit. Break the symmetry with a fixed ordering (lower robot ID wins, or direction of travel wins) so one robot proceeds and the other waits deliberately instead of both yielding. Add a wait-for detector: if a robot holds the segment and another has been blocked past a threshold, raise an alarm.
Why local avoidance can't do it. A local planner is a reactive controller: it resolves instantaneous geometry, not global ownership. Two symmetric controllers form a wait-for cycle (A waits for space B occupies, B waits for space A occupies) and neither sees an invalid plan, so neither reports a fault. The conflict is global; no local rule breaks it.
Alternatives, and when each wins.
| Design | Best when | Cost |
|---|---|---|
| Segment allocation (owner + ordering) | Layout fixed, small-to-medium fleet | Serializes the segment; throughput drops |
| One-way aisles | Layout allows a loop; head-on is the main conflict | Longer routes; needs a return path |
| Centralized traffic manager | Many robots, dense intersections | Single point of failure; added latency |
| Physical redesign (widen, passing bay) | Capex available | Removes the constraint entirely |
Allocation is the wrong choice on a wide aisle where two robots pass anyway, the lock adds latency for nothing, and on a segment carrying so much traffic that a single owner becomes the bottleneck; there, one-way routing or a traffic manager is better.
Failure modes and detection.
- Multi-segment deadlock: A holds segment 1 and waits for 2, B holds 2 and waits for 1. Detect with wait-for graph cycle detection, not a per-robot timeout.
- Stale lock: a robot dies holding the segment. Use a lease with a TTL and heartbeat so the lock expires.
- Silent mutual wait: the case that started this. A timer on the blocked robot, reported to the fleet manager, catches it.
Worked trace. Segment S is 8 m; robot speed 1 m/s, so traversal is 8 s. Robots 7 and 12 approach S. Both request it; ordering rule says lower ID wins. 7 acquires, 12 blocks with a wait timer. 7 traverses in 8 s and releases. 12 acquires and traverses. If 12's wait exceeds 30 s, the fleet manager alarms. Throughput ceiling is one traversal per 8 s, 450/hour, which is the number to check against demand before committing to allocation.
A 1.6 m aisle with 1.1 m footprints is the case that forces this: two units each yielded, both stopped, and the aisle stayed blocked until an operator intervened, with neither robot reporting a fault. Test that width explicitly.
Curated: · Written: · Reviewed:
Q&A-38A safety-rated scanner covers the approach and a person is still struck by the load. How is that possible?
I would start sensor field of view and occlusion from the measured envelope, not from the number the datasheet promises.
A protective field covers the volume the sensor can see. A load carried ahead of the sensor, a low object below the scan plane, or a person behind a pallet occupies space the field never observed, and the sensor reports clear because it is clear where it looked.
Concretely, analyse the swept volume of the robot including its load against the observed volume, add sensing or restrict motion where they differ, and treat a single scan plane as covering a plane rather than a space.
The reason for that specificity is a failure I have seen: A tugger scanned at 200 mm above floor level and carried a 1.2 m overhanging load; a person seated on a low pallet at 400 mm was outside the scan plane and inside the swept volume of the load.
Covered versus swept volume.
| Height | Scanned | Swept by load |
|---|---|---|
| 200 mm | yes | yes |
| 400 mm | no | yes |
| 1200 mm | no | yes |
I would not consider it settled without evidence: Overlay the measured protective field on the swept volume of the robot with its largest load and show the uncovered region explicitly.
A clear field means the sensor saw nothing, not that there is nothing.
Curated: · Written: · Reviewed:
Q&A-44The control software already limits speed near a person. Why add a separate safety function?
Short answer. The control limit is a performance function: it keeps the machine productive and reduces how often the hazardous situation arises. The protective function is a protective one: it has to work when the control limit has already failed. A protective function that shares sensing, logic, power or configuration with the thing it protects shares that thing's faults, so one defect removes both the motion plan and the limit on it. Keep both, but make the protective one independent, and prove it by fault injection rather than by watching it behave.
Why the control limit cannot be the protection. Two reasons. First, common-cause failure: a limit computed by the same software, from the same encoder, on the same power rail as the motion plan is not a second layer, it is the same layer counted twice. Second, demand rate. A control limit that works most of the time keeps the protective function in low demand, which is the assumption its PL or SIL calculation rests on. Remove the control limit and you have changed the demand rate the safety function was rated for.
What independence means in practice. Separate sensor, separate logic solver, separate output path, separate power, and diversity where you can get it. ISO 13849-1 Annex F scores exactly these measures (separation, diversity, design and application experience, diagnostics) because they are what breaks common cause.
| Element | Control limit | Protective function | Independent? |
|---|---|---|---|
| Speed sensing | encoder on motor shaft | second encoder on output shaft | yes |
| Scale and units | control config | safety-rated memory, separate tool | yes |
| Logic | motion controller | safety PLC or drive STO | yes |
| Output | torque command | two-channel STO | yes |
| Power | 24 V control rail | separate rail, own fuse | yes |
| Firmware | motion stack | safety firmware | yes |
Measuring on the output shaft rather than the motor shaft is not decoration: it catches a broken coupling or a wrong gear ratio that the motor-side encoder cannot see.
A worked failure. A cell runs with a 250 mm/s limit. Control and safety both read the same encoder and both apply a scale of 100 counts/mm held in a shared config file. During commissioning someone sets it to 50 counts/mm. The controller now commands 500 mm/s while computing 250; the safety function computes 250 from the same counts and does not trip. Nothing in either channel is broken: the shared constant is. Fix: the safety function reads the second encoder and holds its own constant in safety-rated memory. The same edit now makes the two channels disagree, and the safety function trips at 250 mm/s as measured on the output shaft.
Trade-offs. A second encoder, a safety PLC and separate wiring cost money and panel space. Set the safety limit above the control limit (control 250, safety 300) or normal operation trips constantly. If the safety function stops the axis rather than limiting it, expect cycle-time loss on every trip. And do not add a separate function where the control limit already lives in a rated channel: then you have one protective function, not two, and a second unrated one adds failure modes without adding coverage.
Failure modes and how they are detected. A shared 24 V rail browns out and drops both channels. PROFIsafe sharing a cable with control traffic means one network fault takes both. A shared clock lets the safety function evaluate stale data. A loose encoder bracket moves both sensors together. Detect these with the Annex F common-cause checklist, an FMEA over the shared elements, and a proof test that deliberately faults the control channel (write the wrong constant, corrupt the control scale) and confirms the protective channel still trips. Watching the machine run correctly at 250 mm/s proves nothing about the safety function.
Curated: · Written: · Reviewed:
Q&A-45A robot is described as collaborative. What does that entitle you to assume about the installation?
A "collaborative robot" is a claim about the arm, not about the cell. What it entitles you to assume is narrow: the robot was designed to ISO 10218-1 and carries the safety functions that make collaborative operation possible. Safety-rated monitored stop, hand guiding, speed and separation monitoring, and power-and-force-limiting capability. It does not entitle you to assume the installation is safe, that any force or pressure limit is met, or that the end effector is covered. Those come from the application risk assessment the integrator performs under ISO 10218-2, against the actual tool, payload, speed and reachable body regions. The robot's declaration of conformity covers the robot; the cell needs its own, from the integrator.
The word describes an application, not a product. The same arm is collaborative in one cell and not in another, and a conventional industrial robot with the right safety functions can be used in a collaborative application. (ISO 10218-1:2025 and -2:2025 now carry the power-and-force-limiting detail that used to sit in ISO/TS 15066:2016.)
The four methods each entitle you to assume something different:
| Method | What you may assume |
|---|---|
| Safety-rated monitored stop | The robot stops before you enter the shared space, and the stop is safety-rated (PL d, Cat 3), not that the robot is slow |
| Hand guiding | An enabling device and reduced speed while guiding |
| Speed and separation monitoring | The protective separation distance is maintained dynamically |
| Power and force limiting | Contact limits per body region are respected: for the tool, payload and speed the assessment covered |
Only the last one is about contact, and it is the one most often misread.
Worked example. A cell passes its assessment with the bare flange: a 110 N contact spreads over 900 mm², giving 0.12 N/mm², inside the pressure criterion the assessment applied. The integrator then fits a probe with a 12 mm tip. Same 110 N, now over 113 mm²: 0.97 N/mm², eight times the flange figure. Pressure is force over area, so shrinking the contact multiplies it. The force check still passes (110 N is under the 140 N quasi-static limit for the hand in ISO/TS 15066 Annex A); the pressure check is what the tool broke.
| Contact surface | Area | Pressure at 110 N |
|---|---|---|
| Bare flange | 900 mm² | 0.12 N/mm² |
| 12 mm probe tip | 113 mm² | 0.97 N/mm² |
| Padded plate | 2400 mm² | 0.05 N/mm² |
Failure modes worth naming, and how you catch them:
- Tool or workpiece change after the assessment. Catch it by re-running the measurement whenever the end effector changes.
- Payload increase: more moving mass means more energy in a transient contact. Catch it by validating at the maximum payload and speed the application permits.
- Speed override raised above the validated value. Catch it by capping the override in the safety configuration, not in the application program.
- Collaborative mode as a software setting that can be switched off. Catch it by verifying the safety functions independently of the application program.
I would not treat any of this as settled without measurement: force and pressure taken with the fitted tool on an instrumented body model, at the maximum speed the application permits, per ISO/TS 15066 Annex A.
Collaborative describes an installation, and the tool is part of it.
Curated: · Written: · Reviewed:
Q&A-46How does a speed-and-separation scheme decide how fast the robot may move?
Speed is the output of the scheme, not an input. The controller computes a protective separation distance, compares it against the measured separation, and permits only the speed whose stopping distance still fits inside the gap. Ask "how fast may the robot move" and the real answer is: as fast as the distance budget allows, recomputed every cycle.
The protective separation distance (ISO/TS 15066) is a sum of terms:
- Human approach distance: how far the person travels before the robot has stopped. At the 1.6 m/s whole-body approach speed of ISO 13855, a 300 ms total reaction-and-stop time gives 480 mm.
- Robot stopping distance: travel from the instant the stop is commanded to standstill, at the current speed and payload.
- Intrusion and uncertainty allowances: sensor resolution, position uncertainty of the human, and the robot's own positional uncertainty.
The permitted speed is whatever value keeps that sum at or below the measured separation. Slow the robot and the stopping term shrinks, so the budget closes; speed it up and the budget opens. That is the entire control law: there is no separate speed setting to tune.
Worked budget at 1.2 m/s tool speed:
| Term | Unloaded | 12 kg payload |
|---|---|---|
| human travel | 480 mm | 480 mm |
| robot stopping | 180 mm | 310 mm |
| uncertainty | 120 mm | 120 mm |
| required separation | 780 mm | 910 mm |
The failure this exposes is a stopping distance measured unloaded. A cell validated with an empty arm needed 180 mm to stop; with a 12 kg payload the same arm needed 310 mm, so the required separation rose from 780 mm to 910 mm and the cell was 130 mm short at every point of the cycle. Nothing in the sensor or the visualiser changed: only the mass the brakes had to arrest.
That failure has a signature you can watch for: the gap closes monotonically as payload increases, so a cell that passes unloaded validation and trips on loaded cycles is the tell. The fix is procedural, not algorithmic: measure stopping distance physically at the maximum speed and maximum payload combination and put that figure into the budget, then recompute whenever payload, speed, or tooling changes.
Two things worth stating as assumptions. The 1.6 m/s human approach speed is the ISO 13855 default for whole-body movement; a slower, deliberate approach in a constrained cell can justify a smaller figure, but you have to defend it. And the 300 ms reaction-and-stop time is a measured end-to-end number, not a datasheet value: it includes sensor latency, the safety controller, and the drive's own response. If you cannot measure it, you cannot claim the budget.
Curated: · Written: · Reviewed:
Q&A-43An emergency stop cuts power to everything. When is that the wrong design?
Assumption: "emergency stop" here means a Category 0 stop per IEC 60204-1. Immediate removal of power to the machine actuators. That is the right default when the hazard is created by motion or stored energy. It is the wrong design whenever the hazard is created by the loss of an active force, because then de-energising is itself the initiating event.
Three cases where cutting everything is wrong:
- Vertical axis with a power-off brake. The brake is spring-applied, so it holds without power, but only after it engages, and engagement takes tens of milliseconds. If the stop also drops servo torque, the axis free-falls in the gap.
- Hot process with forced cooling. Removing power stops the pump; the melt or the exothermic reaction keeps generating heat. Safe state is "cooling maintained", not "everything off".
- Magnetic gripper holding a part over a person or a conveyor. De-energising releases the part. Safe state is "part retained": hold the coil energised, or use a permanent-magnet gripper that fails closed.
The design rule: name the safe state per hazard first, then decide which energies must stay available to hold it. That usually means a Category 1 stop (controlled stop with power maintained to the holding function, then power removed once the axis is at rest) rather than Category 0, plus an independently powered path for the holding function so the E-stop can still remove drive power.
Worked example: 60 kg vertical fixture on a ballscrew, spring-applied power-off brake:
| Event | Time | Fixture drop |
|---|---|---|
| E-stop asserted | 0 ms | 0 mm |
| servo torque removed | 5 ms | 0 mm |
| brake torque applied | 65 ms | 18 mm |
Between 5 ms and 65 ms the axis is in free fall: ½ · 9.81 m/s² · (0.060 s)² = 18 mm. Ignoring friction and screw inertia, so treat that as the upper bound. The brake is not the problem: the 60 ms gap between torque-off and brake engagement is. Fixes: engage the brake before removing servo torque (sequence the stop, don't just drop the contactor), or add a counterbalance that holds the load without power.
Failure modes and how you catch them:
- Brake wear or contamination stretches engagement time. Detect by measuring engagement time on a schedule, not by reading the datasheet.
- Common-cause power loss, if the holding function shares the supply the E-stop cuts, the design fails exactly when it is needed. Detect by tracing the supply tree, then proving it by pulling the main breaker.
- Sequencing assumed but not enforced: the schematic shows brake-before-torque-off, the PLC scan order does not. Detect by measuring.
Verify by inducing the stop under the worst case: full load, at height, at temperature, with the brake at its wear limit. Measure the actual event sequence with a scope or the drive's trace buffer. A stop that is safe at rest proves nothing.
Curated: · Written: · Reviewed:
Q&A-42A watchdog fires and the robot restarts into the same fault. What is missing from the design?
The useful question for watchdog design is what still holds at the edge of the workspace, cold, and under full payload.
A watchdog proves that something is still executing. It does not prove the system is doing the right thing, and a restart that returns to the same state converts a fault into a loop rather than into recovery.
Concretely, kick the watchdog from a point that only executes when the real work completed, not from a timer, record the state that preceded the reset in non-volatile storage, and define what a repeated reset does: safe state and hold, rather than an eleventh attempt.
The reason for that specificity is a failure I have seen: A watchdog kicked from a timer callback kept a hung control loop alive for 40 minutes because the timer was fine; the axis held its last command throughout.
Kick source and what it detects.
| Kick from | Detects hang | Detects wrong output |
|---|---|---|
| timer callback | no | no |
| loop completion | yes | no |
| completion + plausibility | yes | partly |
I would not consider it settled without evidence: Test the watchdog by hanging the actual work, not by stopping the process, and confirm both the reset and the recorded cause.
A watchdog that a hung system can still feed is decoration.
Curated: · Written: · Reviewed:
Q&A-51A robot stopped unexpectedly last week and the logs do not explain it. What should have been recorded?
Assumptions: a 6-axis industrial arm, protective stop requested by a safety PLC or by a drive, servo drives on a 1 ms fieldbus. A cobot or an AMR has the same shape with different channels.
The answer in 90 seconds
Record at the control loop, not at the application. Keep a rolling ring buffer of commanded and measured state, mode, safety inputs and drive fault words at 1 kHz, and freeze a window (4 s before the stop, 1 s after) to non-volatile storage the instant any protective stop is requested. The safety layer's decision goes in the same record, on the same clock, naming the safety function that tripped.
A 1 Hz application log cannot resolve a 40 ms following-error spike, and a following-error spike is exactly what stops a robot. The record has to exist before the incident; nothing you can enable afterwards reconstructs it.
What each channel buys you
| Channel | Rate | What it decides |
|---|---|---|
| Commanded position/velocity/torque, 6 axes | 1 kHz | what the trajectory asked for |
| Measured position/velocity, following error | 1 kHz | the deviation that crossed the limit |
| Motor current / torque | 1 kHz | mechanical jam vs. command error |
| Mode, program pointer, motion segment ID | on change + 1 kHz snapshot | which instruction was executing |
| Safety inputs (guard, E-stop channel A/B, enabling device), safety PLC state word | 1 kHz, timestamped | the input that requested the stop |
| Drive status/fault word, fieldbus working counter, DC bus voltage | 1 kHz | drive trip vs. bus loss |
| STO / SS1 status | on change | whether power was removed, and when |
The freeze mechanism
/* control task, 1 kHz; C11, gcc 13, x86-64 */
typedef struct {
uint64_t t_cycles; /* monotonic servo-cycle counter */
float cmd[6], meas[6], torque[6], ferr[6];
uint16_t mode, safety_in;
uint32_t fault;
} sample_t; /* 112 bytes */
static sample_t ring[4096]; /* 4.096 s at 1 kHz */
static _Atomic uint32_t head;
static _Atomic int frozen;
void control_tick(void) {
uint32_t h = atomic_load_explicit(&head, memory_order_relaxed);
fill_sample(&ring[h & 4095u]); /* no malloc, no locks, no syscalls */
atomic_store_explicit(&head, h + 1, memory_order_release);
if (stop_requested() && !atomic_exchange(&frozen, 1))
signal_logger(h); /* logger thread copies the window */
}
The control task only bumps an index and posts the head value. A separate thread copies the last 4096 samples and keeps copying for 1 s, then writes to flash. Nothing in the control path blocks, so the log does not change the timing it is recording.
Storage arithmetic
112 bytes/sample × 1000 samples/s = 112 kB/s ≈ 400 MB/h if you kept everything. You don't. The 4 s ring is 448 kB; the frozen 5 s window is 560 kB. A 10 Hz decimated long trace alongside it costs 1.1 kB/s ≈ 4 MB/h and covers slow faults (bus degradation, rising DC bus temperature) that develop over minutes and never appear in a 4 s window.
Failure modes, and how you catch them
- Stop category 0 removes drive power before the controller flushes. The last samples die in RAM. Fix: drives latch their own pre-trigger trace in NVRAM, or the controller persists during the stop ramp. Detect: pull the E-stop and check the frozen window ends at the stop, not 200 ms before it.
- Clock skew between controller and safety PLC. Two clocks give you the wrong causal order. Fix: one monotonic time base (PTP, or a shared sync pulse into both). Detect: trigger two events inside one cycle and check they order correctly.
- Logging in the control path adds jitter. Fix: lock-free ring, no allocation, no syscalls in the control task. Detect: measure loop jitter with logging on and off.
- The safety PLC logs "stop requested" with no function ID. You cannot tell a guard opening from an SLS violation. Record the safety function identifier and its input states.
Validate before you need it
At commissioning and after every firmware change, inject a stop (open the guard, drop the safety fieldbus, command an out-of-limit move) and confirm the frozen window holds the preceding 4 s at 1 kHz, all channels present, one consistent clock. If that check fails, the next unexplained stop stays unexplained.
Curated: · Written: · Reviewed:
Q&A-74A cloud service sends motion commands to a robot. What must the robot enforce locally?
I would start remote commands and local authority from the measured envelope, not from the number the datasheet promises.
A remote command arrives across a link that can be delayed, replayed, or reordered, from a system whose state may be stale. Local protection has to remain the final authority because it is the only layer that knows what the machine is currently doing.
Concretely, authenticate and bound every command, reject stale or out-of-order ones by sequence and freshness, clamp against local limits regardless of what was requested, and keep protective functions outside the path that the remote system can influence.
The reason for that specificity is a failure I have seen: A queued command replayed after a 40 second network stall told a robot to move to a pose that had been safe when issued; the fixture had since been loaded and the arm drove into it, with the command perfectly valid on arrival.
Command validity checks.
| Check | Catches replay | Catches stale | Catches over-limit |
|---|---|---|---|
| authentication | no | no | no |
| sequence number | yes | no | no |
| freshness window | yes | yes | no |
| local clamp | no | no | yes |
I would not consider it settled without evidence: Test with deliberately delayed, replayed, and reordered commands, and confirm the robot rejects rather than executes them.
The robot is the only party that knows where it is now.
Curated: · Written: · Reviewed:
Q&A-95One of three cameras fails mid-shift. Should the cell stop?
Short answer: stop, unless a degraded mode was defined, validated, and is currently in force. The camera count is not the decision variable. The tolerance budget of the work in front of the cell is. If nobody wrote down what two cameras still guarantee, the cell is running outside its validated envelope while the HMI says "running".
What a strong candidate says first (60–120 s)
Ask two things before touching the cell. First: were the three cameras arranged for redundancy or for coverage? Redundancy means any pair can still triangulate, so losing one costs accuracy, not capability. Coverage means each camera watches a different face of the part, so losing one leaves a blind face that no amount of re-tuning recovers. Second: what does the open work order actually need? Compare the remaining pair's achievable accuracy against the placement tolerance for the variants in the queue. If it clears, continue at a reduced rate with the degraded mode entered explicitly and logged. If it doesn't, stop and let a human decide.
Redundancy vs coverage
| Design | Camera roles | Lose one | Verdict |
|---|---|---|---|
| Redundant stereo | 3 overlapping views, any pair triangulates | 2 views remain; depth error grows with the shorter baseline | continue if tolerance holds |
| Coverage | cam A top face, cam B side, cam C lead inspection | blind face | stop |
| Hybrid | 2 for pose, 1 for inspection | pose survives, inspection lost | continue pose, gate inspection |
Arithmetic (illustrative numbers)
Stereo depth error scales as σ_z ≈ z²·σ_d / (f·b). At 800 mm standoff with f = 1000 px and disparity noise σ_d = 0.5 px: a 400 mm baseline pair gives ±0.8 mm; the remaining pair at a 150 mm baseline gives ±2.1 mm. If the variant's placement tolerance is ±1 mm, the degraded pair is out of budget for that variant and in budget for a looser one. The stop condition is therefore per-variant, not per-cell.
Stop condition as a predicate
Write it down as: continue iff the remaining sensors resolve position to ±1 mm and orientation to ±1° for every variant in the open work order, and the mode has been entered explicitly. Anything else, stop. The operator should never be the one computing this at 03:00.
Failure modes and how to detect them
- Silent degradation. Cell keeps running at full rate, HMI shows normal. Detect with a first-article check on mode entry plus a per-part pose verification against the expected placement.
- Calibration coupling. The remaining pair's extrinsics were calibrated against the lost camera. Re-verify hand-eye calibration on mode entry; don't assume it survives.
- Variant drift. The degraded pair works for variant A and fails for variant B. Gate by work order, not by "the cell looks fine".
- No expiry. A degraded mode with no time or part budget becomes the new normal. Bound it in parts or minutes and alarm when it expires.
Before saying "continue"
A recorded test of each single-camera failure, with the measured accuracy of the remaining pair and the list of variants it covers. Without that, the honest answer to "should the cell stop?" is yes.
Curated: · Written: · Reviewed:
Q&A-94An operator jogs an axis expecting tool-frame motion and the arm moves in base frame. Whose fault is that?
Whose fault: the system's, by default.
A jog command is a velocity vector plus a frame. The operator supplies the vector by holding a button; the machine supplies the frame. If the machine's active frame and the operator's assumed frame differ, the machine executes a command nobody gave. That is a mode-disclosure failure in the controller or the pendant, not operator error. The operator cannot be expected to hold hidden state the machine never showed them at the moment they pressed the button.
The one case that shifts blame: the active frame was unambiguously on screen at the point of command, the operator read it, and jogged anyway. Even then the fix is the design that let a stale frame persist.
Mechanism. The controller takes the commanded Cartesian velocity, expressed in the active frame, rotates it into base frame, and feeds it to the inverse Jacobian:
q̇ = J⁻¹(q) · R_frame_in_base · v_cmd
The frame is a rotation matrix, not a label. Change the frame and the same button press produces a different base-frame direction. A wrist roll changes the tool frame without any operator action.
Trace. Tool pointing up (tool Z = +Z_base), wrist rolled 90° about tool Z. Operator holds "forward" (+X_tool) for 0.8 s at 50 mm/s.
# Python 3.11, numpy 1.26
import numpy as np
def to_base(v_tool, R_tool_in_base):
return R_tool_in_base @ v_tool
def Rz(deg):
c, s = np.cos(np.radians(deg)), np.sin(np.radians(deg))
return np.array([[c, -s, 0.0],
[s, c, 0.0],
[0.0, 0.0, 1.0]])
v = np.array([0.05, 0.0, 0.0]) # 50 mm/s along +X_tool
print(to_base(v, Rz(0))) # wrist not rolled
print(to_base(v, Rz(90))) # wrist rolled 90 deg about tool Z
Rz(0) @ v = [0.05, 0, 0] → +X_base. Rz(90) @ v = [0, 0.05, 0] → +Y_base. Same button, 0.8 s at 50 mm/s = 40 mm, but along +Y_base instead of +X_base: 40 mm lateral to what the operator intended.
| Active frame | Wrist roll | Command | Base-frame motion |
|---|---|---|---|
| base | any | +X | +X_base |
| tool | 0° | +X_tool | +X_base |
| tool | 90° | +X_tool | +Y_base |
Alternatives and when to use them. Base-frame-only jogging removes the ambiguity entirely and is right for gross positioning and for operators who jog rarely. Tool-frame jogging is what you want for approach and retract moves along the tool axis. Per-jog explicit frame selection is safest but adds a step to every move. Most cells land on: tool frame for fine moves, base for gross, with the active frame latched to a physical enabling device so it cannot change mid-jog.
Failure modes and detection. Retained frame across sessions or after a program stop; frame changed by a mode switch or by a tool change; wrist roll silently redefining tool X/Y. Detect by logging the active frame and the resolved base-frame velocity with every jog command, and asserting the frame at command time rather than at screen-render time. If the log shows the frame the operator saw differs from the frame the controller used, you have the bug.
Curated: · Written: · Reviewed:
Q&A-103Parts placed by the robot land a few millimetres off with a consistent rotation error. Walk me through how you'd find out whether that's hand-eye calibration, the TCP, or the kinematics, and how you'd re-calibrate and prove it's fixed?
A consistent offset is systematic, so it's a calibration error, not noise. Three things can produce it: the hand-eye transform between camera and robot, the tool TCP, and the kinematic parameters. The way to separate them is to see how the error depends on the robot's pose and the tool orientation.
First, does the offset rotate with the tool? Place the same part at two different tool orientations. If the offset rotates with the tool, it's a tool-frame error: TCP or hand-eye. If it stays fixed in base frame, it's more likely a base-frame or kinematic error.
Second, does the offset change with arm configuration? Approach the same placement pose from elbow-up and elbow-down. If the offset changes, it's kinematics. If it doesn't, it's TCP or hand-eye.
Third, isolate pick from place. Pick a part from a fixture and place it with the robot: any error is in the place path. Pick with the camera and place into a fixture: any error is in the pick. If the error only appears when the camera is in the loop, it's hand-eye.
| Error source | Offset in base frame | Offset in tool frame | Changes with arm config? |
|---|---|---|---|
| Hand-eye rotation | rotates with tool | constant | no |
| TCP position | rotates with tool | constant | no |
| TCP orientation | grows with distance from TCP | rotates about TCP | no |
| Kinematic params | varies | varies | yes |
Hand-eye and TCP position both give a constant tool-frame offset. To separate them, change the grasp offset: a hand-eye error changes the placement error when you grasp at a different point; a TCP error does not.
Re-calibration. Hand-eye: use a target and move the robot to 15–20 poses with rotation diversity. Pure translation gives a degenerate AX=XB. Solve with Tsai-Lenz or Park-Martin. Check the residual: reprojection error and the spread of the solved transform across subsets. If the rotation spread is above ~0.1°, the poses are not diverse enough. TCP: four-point pivot gives position only; orientation needs a separate method, such as touching a fixed point at different orientations. Kinematics: if the error changes with configuration, you need a laser tracker to measure the actual TCP at many poses and fit the DH/URDF parameters. That is a bigger job, often done by the manufacturer.
Verification: after re-calibration, place a part and measure with an independent method. A CMM, a calibrated vision system, or a gauge. Check the residual over the workspace, not just at one point. Measure both position and rotation. For a typical 6-axis arm, under 0.2 mm and 0.05° is a reasonable target, but it depends on the robot.
Arithmetic: a 0.5° hand-eye rotation error at 200 mm from the camera gives 200 × sin(0.5°) ≈ 1.7 mm. That is the few millimetres you are seeing. A 0.1° error gives 0.35 mm.
import numpy as np
from scipy.spatial.transform import Rotation as R
def hand_eye_residual(T_tool_cam, T_base_tool, T_cam_target):
preds = [Tb @ T_tool_cam @ Tc for Tb, Tc in zip(T_base_tool, T_cam_target)]
pos = np.array([T[:3, 3] for T in preds])
pos_err = np.linalg.norm(pos - pos.mean(axis=0), axis=1).max()
rots = R.from_matrix([T[:3, :3] for T in preds])
rot_err = (rots[0].inv() * rots).magnitude().max()
return pos_err, np.degrees(rot_err)
Curated: · Written: · Reviewed: —
Q&A-104You're asked to sign off a robotic cell before it goes into production. Walk me through the risk assessment: what hazards you enumerate, what standards frame it, and what it changes about the design.
I'd start by saying what the assessment is for: it is not a document, it is the loop that decides the cell's geometry, speed and control architecture. ISO 12100 gives the method: identify hazards, estimate and evaluate risk, reduce it, iterate, record residual risk. ISO 10218-1 covers the robot, 10218-2 the cell and its integration; ISO/TS 15066:2016 supplies the power-and-force-limiting limits for collaborative operation. The 2025 editions of 10218-1 and -2 were published in February 2025 and absorb much of the collaborative material, so I'd confirm which edition the customer spec and the notified body call out before quoting clause numbers.
Then I enumerate hazards by energy source and by task phase, not by component. The table is what I'd actually build:
| Energy / phase | Hazard | Typical control |
|---|---|---|
| Stored mechanical | Arm falls on power loss; spring or pneumatic release | Brake rating, counterbalance, dump valve, gravity-safe pose |
| Kinetic | Unexpected motion, trapping between tool and fixture | Fence and interlock, speed-and-separation, PFL limits |
| Tooling | Sharp edge, hot surface, ejected part, gripper opens on loss of air | Rounded covers, guarded nozzle, spring-close gripper, retention |
| Electrical / process | Arc, laser, EMC, vacuum, chemical | Enclosure, interlocks, PPE, extraction |
| Human factors | Reach-over, reach-through, cleaning, teach pendant in cell | Physical distance, height, key exchange, mode selection |
| Software / control | Mode confusion, stale sensor data, watchdog, muting stuck | Safety-rated logic, dual channel, time-limited muting, manual reset |
The phase column matters: load/unload, recovery from a jam, and cleaning are where people actually get hurt, and each has a different exposure.
The assessment then changes the design, and the arithmetic is where it bites. For a scanner guarding a whole-body approach, ISO 13855 gives S = K × T + C, with K = 1600 mm/s, T the total response time (scanner + safety relay + drive stop), and C = 850 mm for a vertical detection plane. If the scanner responds in 30 ms, the relay in 20 ms and the robot stops in 250 ms, T = 0.30 s, so S = 1600 × 0.30 + 850 = 1330 mm. If the cell is only 1.0 m deep, you cannot fit the scanner, so you change the design: cut the stop time with a bigger brake or a lower maximum speed, use a higher-resolution detection capability to shrink C, or fence the approach and guard the load opening instead. That is the assessment doing its job.
For a collaborative cell the same loop runs against ISO/TS 15066 Annex A. Hand and finger quasi-static contact is limited to 140 N, transient to 280 N, with a separate pressure limit in N/cm² that is usually the binding one at a pinch point. If a measured or simulated contact exceeds it, the fix is design: reduce payload and speed, round the tool, add a compliant cover, or move the pinch point out of the reachable space.
Failure modes I'd name and test: muting that stays muted because a part jammed the sensor (time-limit it and add a muting lamp); a scanner with a blind spot behind its own post; a teach pendant in the cell while someone else selects automatic; a safety function implemented in a standard PLC on one channel; a scanner on a network with no watchdog, so stale data reads as clear. Validation is by measurement, not by reading the drawing: stopping-time measurement, force and pressure measurement for PFL, and a functional test of every interlock including a deliberate bypass attempt. Residual risk and the validation records go in the technical file.
Curated: · Written: · Reviewed: —
Q&A-96Two designs have the same reliability figure. What else determines line availability?
Availability is MTBF/(MTBF+MTTR). Two designs with the same reliability figure have the same MTBF, so everything left sits on the repair side: how fast a failure is detected, how fast the failed unit is out and a good one in, and whether the line keeps producing while that happens.
Three things decide it:
- Detection time. MTTR starts when the failure is noticed, not when it happens. A camera that silently drops frames until a downstream check trips adds that latency to every event. Instrument the failure directly.
- Repair vs replace. If restoring service needs recalibration, alignment, or a specialist, MTTR is set by the procedure, not the part. Keyed mounting, stored calibration, and pre-aligned spares turn a 3.5-hour job into a 12-minute one.
- Spares and logistics. A 10-minute swap with no spare on site is a two-day outage. Hold spares for long-lead items and kit them so the swap needs no shopping.
Worked example: one camera on a line, 6-month failure interval (2 failures/year):
| Design | Interval | Repair time | Annual downtime |
|---|---|---|---|
| Recalibrate on replacement | 6 months | 3.5 h | 7.0 h |
| Keyed mount, stored extrinsics | 6 months | 12 min | 24 min |
Arithmetic: 2 × 3.5 h = 7.0 h/year → availability 1 − 7/8760 = 99.920%. 2 × 0.2 h = 0.4 h/year → 1 − 0.4/8760 = 99.9954%. Same reliability figure, roughly 17× the downtime.
Failure modes to watch:
- MTTR estimated from the assembly drawing rather than timed. A real replacement includes finding the tool, the spare, and the person.
- Spares held but not calibrated: the swap happens, then a three-hour calibration follows.
- Redundancy assumed to cover it. Failover helps only if the standby is healthy and the switch is automatic; a manual switch is downtime.
Verification: time an actual replacement during commissioning, with the real spare and the real technician, and record it as an acceptance figure. Re-time it after the first unplanned failure: that number is the one that shows up in availability.
Curated: · Written: · Reviewed:
Q&A-97Which datasheet numbers should you distrust for your application, and what do you measure instead?
Core answer. Distrust every datasheet figure whose test conditions differ from your cell, and measure the ones your tolerance depends on. In practice that is four numbers: rated payload, repeatability, tool-tip speed, and cycle time. Rated payload is quoted at the flange with the centre of mass at the flange face; repeatability is quoted per ISO 9283 at moderate reach, stable temperature, and one approach direction; speed is per joint, not at the tool; cycle time is a best-case path with no settling or I/O. Each can be true and none of them describe your application.
Assumptions I would state first. Tool mass and centre-of-mass offset, the reach band you actually use, ambient temperature swing, and the tolerance your process needs. If your tolerance is loose relative to the spec, the datasheet is enough. If it is within a factor of two of the spec, measure.
Which numbers, and why.
- Rated payload: the load chart is a moment limit, not a mass limit. A 10 kg rating with the CoM at the flange face collapses as the CoM moves out. Illustrative shape of a typical chart:
| CoM offset from flange | Permitted payload |
|---|---|
| 50 mm | 10.0 kg |
| 120 mm | 7.8 kg |
| 180 mm | 6.4 kg |
- Repeatability vs accuracy: repeatability is the spread of repeated visits to the same point; accuracy is how close that point is to commanded. Datasheets quote repeatability because it is small and flattering. Your process usually needs accuracy, which is larger and rarely quoted.
- Speed: per-joint maximum is not tool-tip speed. Tool speed depends on path and configuration; near singularities and at full extension it drops.
- Cycle time: quoted on an optimal path with no settling, no I/O, no vision.
When trusting the datasheet is fine. Early sizing, vendor shortlisting, and any cell whose tolerance is comfortably wider than the spec: a pick-and-place with ±5 mm tolerance against ±0.1 mm repeatability does not need a tracker. Measurement is mandatory when the tolerance is close to the spec, when the tool CoM is far from the flange, when cycle time is a contractual number, or when safety depends on the figure.
Cost of measuring. A day of cell time, a laser tracker or dial indicator, and a load cell. Against a mis-specified cell that fails on torque limits during ramp-up, that is cheap.
Failure mode and detection. The one I have seen: a cell specified on the 10 kg rating with the tool CoM at 180 mm, where the chart permits 6.4 kg. The arm hit joint torque limits on every acceleration once the part was added. Detected as servo torque saturation and following error on the scope, and as cycle time creeping past target. The fix was a lighter tool, not a bigger robot.
What I would measure instead. Payload margin: hang the real tool and part, command worst-case acceleration, watch torque and following error. Accuracy: command a grid of points across the working envelope at operating temperature, measure with a tracker. Tool speed: time the actual path. Cycle time: run the real program with I/O and settling.
Curated: · Written: · Reviewed:
Q&A-99What makes a robotics incident report useful six months later?
A robotics incident report is useful six months later when a reader who was not there can re-derive the conclusion from what is attached to it, without asking anyone who was. That is the bar. The rest of this is how to hit it, and when you can settle for less.
A software postmortem template is not enough because the machine's behaviour is not the software's decision. A planner can command a correct trajectory and the arm can still collide, because the controller, the calibration, or the physical scene differed from what the code assumed. So the report has to carry what the machine actually did, measured joint positions and commanded setpoints over time, not just the log line saying a motion was issued.
What to attach, and the trade-offs.
Full-rate motion capture is the strongest evidence and the most expensive to keep. A 6-axis arm sampled at 1 kHz across 8 channels is 8,000 samples/s; at 4 bytes each that is 32 KB/s, about 2.6 GB/day per cell if you retain everything. Most fleets do not. The credible alternatives, strongest first:
- Full-rate joint and command traces, retained for the incident window only: minutes to hours, not forever.
- A downsampled pose trace (10–100 Hz) plus the full-rate ring buffer frozen at the moment of the fault.
- Event-triggered capture: a rolling buffer frozen on a fault or e-stop, everything else discarded.
Retention cost and privacy pull against completeness. Motion data can reveal operator behaviour and, in a shared workspace, human movement; some sites treat it as personal data and cap retention. That is a real constraint, not an excuse: freeze the window around the incident and keep that, rather than keeping nothing.
When lighter evidence suffices: a near-miss with no contact, no human in the cell, and a fault that reproduces in simulation does not need the full record. A config hash, the fault code, and the sim reproduction are enough. Reserve the heavy evidence for contact events, human-adjacent events, and anything that will not reproduce.
Two things the report must separate: the cause and the detection gap. The cause is why the machine did the wrong thing. The detection gap is why nobody caught it sooner. They have different fixes: one changes the controller or the scene, the other changes a monitor or an interlock, and collapsing them into one paragraph is how a report ends up with no actionable change.
The table below is the checklist I would apply. Each row is a question a later reader will ask; the middle column is the artefact that answers it; the last column is what you lose if that artefact is absent.
| What the report must answer | Where the evidence comes from | If it is missing |
|---|---|---|
| What the controller commanded | frozen command trace | the conclusion is a hypothesis |
| What the machine measured | frozen joint/pose trace | cannot separate command from execution |
| What configuration was live | config hash + calibration record | cannot rule out a config change |
| What would have prevented it | analysis tied to the rows above | the report describes, it does not fix |
Hypothetical example: a report attributed a collision to operator error with no motion record. Eight months later the same collision recurred on another cell in the same 12-cell fleet. With no record, the real cause, a scene that still contained a fixture removed 90 seconds earlier, took three days to find from scratch.
Whether to require the motion record as a gate is a policy call. The strict version: no report accepted as complete without the frozen motion record and the effective config: costs storage and review time. The lighter version, require them only for contact or human-adjacent events, is what most fleets can actually sustain. Pick one and write it down, because the failure mode is a report that looks complete and cannot be re-derived.
Curated: · Written: · Reviewed:
Q&A-100When would you advise against automating a task?
The 90-second answer
I'd advise against automating when the lifetime cost of owning the automation exceeds the work it removes, when the task isn't stable enough to automate yet, or when the failure mode is silent and the blast radius is production. Writing the script is the cheap part; the maintenance, credentials, alerting and on-call surface are what you're actually signing up for.
Four cases where I say no:
- Frequency is too low to amortise. A task run twice a year that takes ten minutes is twenty minutes of annual toil. A script that takes four hours to write and two hours a year to keep working is a net loss. The break-even is arithmetic, not taste.
- The task is still changing. If the process is being redesigned next quarter, automating it now means paying to build it and paying again to rebuild it. Automate the stable core, leave the churning edges manual.
- The exceptions need judgment. Automation is fine when the happy path is 95%+ of cases. When a third of runs need a human decision, you've automated the easy third and made the hard two-thirds slower, because now someone has to detect the exception, unwind the partial automation, and finish by hand.
- Nobody will own it. An unowned cron job with no alerting is a liability that fails silently and is discovered during an incident. If there's no team on the hook for it, don't ship it.
Break-even, worked
Illustrative figures for a ten-minute task:
| Runs/year | Annual saving | Build + annual upkeep | Net |
|---|---|---|---|
| 2 | 0.33 h | 4 h + 2 h | −5.7 h |
| 52 | 8.7 h | 4 h + 2 h | +2.7 h |
| 250 | 41.7 h | 4 h + 2 h | +35.7 h |
The middle row is the one people get wrong. Weekly feels frequent enough to automate, and it barely clears the bar once upkeep is counted.
The failure mode that decides it
Silent failure is the reason I'm conservative about unattended automation. This is a real pattern:
#!/usr/bin/env bash
# nightly: archive yesterday's logs
aws s3 cp /var/log/app/ s3://logs-bucket/$(date +%F)/ --recursive \
| tee -a /var/log/archive.log
echo "archive complete"
Trace it with expired credentials. aws exits 255. The pipeline's exit status is the last command's, so it's tee's status: 0. The script prints archive complete and exits 0. The scheduler records success, no alert fires, and the logs are never archived: you find out the day you need them. set -euo pipefail at the top turns that into a non-zero exit and a page.
That's the general shape: automation that reports success while doing nothing is worse than no automation, because it removes the human who would have noticed.
What I'd do instead
Document the task as a runbook and automate the detection rather than the action: alert when the condition the task addresses occurs, and let a human run the documented steps. That gets most of the safety with none of the ownership. Revisit once the task has run unchanged for a few months and the exception rate is measured rather than assumed.
I'd also check whether the task is a symptom. If you're automating a nightly cleanup of files a bug keeps creating, fix the bug. Automating a workaround locks the workaround in.
Detection, not vibes
Before committing, measure the actual exception rate over a representative period, count the runs, and check that someone is named as owner with an alert that fires on failure. If any of those three is missing, the honest answer is no.
Curated: · Written: · Reviewed:
Q&A-72An IP65 robot fails in a washdown food plant. Was the rating wrong?
Probably not wrong: it was answering a different question. IP65 is a low-pressure fresh-water jet test. A food-plant washdown is usually high-pressure, hot, and caustic, and the failure is normally at the weakest element in the washdown path, not the arm.
What IP65 actually certifies. IEC 60529: the first digit 6 is dust-tight; the second digit 5 is "protected against water jets". A 6.3 mm nozzle at 12.5 L/min, held 2.5–3 m away, for at least 3 minutes. IPX6 raises the flow to 100 L/min through a 12.5 mm nozzle. The washdown rating is IP69K (ISO 20653 / DIN 40050-9): 80 °C water at 80–100 bar, sprayed at close range from four angles, 30 s each.
The gap is not incremental. From Bernoulli, jet speed ≈ √(2P/ρ):
- IPX5: 12.5 L/min through a 6.3 mm nozzle → 6.7 m/s
- IP69K: 80 bar → √(2 × 8×10⁶ / 1000) ≈ 126 m/s
Nineteen times the velocity, and stagnation pressure goes as v². Nothing about IP65 prepares a seal for that.
Two things no IP rating covers. Chemistry: IPX5/6 use fresh water and IPX9K uses hot water. Neither uses caustic or acid, so seal-material compatibility with your cleaning agent is a separate qualification. And the rating belongs to the enclosure, not the assembly: glands, connectors, breathers and tooling each carry their own.
Where it fails.
| Element | Rating | Survives the procedure |
|---|---|---|
| arm | IP65 | yes |
| tool connector | IP54 | no |
| cable gland | IP67 | yes |
IP54 is dust-protected and splash-only: no jet protection at all. The arm being undamaged while the connector corrodes is the expected signature. Note the gland: IP67 is an immersion test, not a jet test, so confirm it also carries an IPX6 or IPX9K rating rather than assuming one from the other.
Failure modes and how I'd detect them.
- Connector or gland ingress → pin corrosion → insulation resistance falls. A megger reading below roughly 1 MΩ, or a VFD ground-fault trip, is the tell.
- Thermal cycling: hot wash then cool-down creates a vacuum inside the enclosure and pulls water past lip seals and through breathers. Detect with a humidity indicator inside the enclosure, not by eye.
- Seal degradation: caustic at 60 °C attacks elastomers; compression set and hardening show up before visible cracking. A durometer reading on a scheduled inspection beats "looks fine".
What I'd do. Get the actual cleaning procedure (pressure, temperature, chemistry, nozzle distance, dwell) and rate every component in the washdown path against it, not against the standard's test. Then run the real procedure on a spare assembly and megger it afterwards. IP69K plus a chemically compatible seal is the usual answer; IP65 alone is not.
One trade-off worth stating: IP69K does not imply IP67 or IP68. They are separate tests. If the enclosure also sits in pooled water, you need both, and over-specifying IP69K on every part is expensive when only the washdown-path components need it.
Curated: · Written: · Reviewed:
Q&A-73How do you update firmware on a robot fleet without creating a fleet-wide outage?
The 90-second answer
Clarify first: fleet size, how many hardware revisions are in it, whether the update touches the safety controller or only the application layer, and whether robots can be parked. Those change the answer more than the update tooling does.
Assumptions I'd state out loud: robots are battery-powered and mobile, so a bad image is a physical recovery; the bootloader can select between two image slots; and the fleet manager can see per-robot health.
Then: never push fleet-wide. Write the new image to the inactive slot, reboot into it as a trial, gate the commit on a health check that exercises the robot's actual job, and let the bootloader revert automatically if that check doesn't pass. Roll out in cohorts, smallest first, and halt the whole rollout on the first cohort failure rather than letting it continue.
Mechanism
A/B slots with a trial boot. U-Boot's bootcount / bootlimit / altbootcmd is the classic implementation: each boot increments bootcount; once it exceeds bootlimit, the bootloader runs altbootcmd instead of bootcmd, which resets the counter and boots the other slot. The new image has to clear bootcount itself, and only after it has proven healthy. That inversion is the whole trick: the image that just booted is the one that has to earn the right to stay. Mender, RAUC and SWUpdate all build on this pattern; Android's A/B uses max_boot_attempts for the same reason.
The health check has to be about the mission, not the process. "systemd says the service is up" is not health. For a mobile robot: can it localize, is the safety-controller heartbeat arriving, do the motor controllers enumerate, can it accept and complete a short mission in a safe area. Any of those failing means don't clear bootcount.
Order cohorts by hardware revision first, then by size. The revision split matters because the failure mode is usually revision-specific.
Artefact: the trial-boot gate (pseudocode)
def on_boot():
if bootloader.active_slot_is_new():
if health_check(timeout_s=120):
bootloader.clear_bootcount() # commit: this slot is now permanent
fleet_manager.report("healthy", committed=True)
# else: leave bootcount alone; the bootloader will revert
else:
fleet_manager.report("healthy", committed=False)
def health_check(timeout_s):
deadline = now() + timeout_s
while now() < deadline:
if not safety_controller.heartbeat_ok(): continue
if not motors.enumerated(): continue
if not localization.converged(): continue
if mission.run_short_test(): return True
return False
Trace it. bootcount is a single U-Boot environment variable, not per-slot. The new slot boots with bootcount=1; suppose localization never converges, so health_check returns False after 120 s and we return without clearing. The robot reboots (watchdog, or an operator power-cycles it) → bootcount=2 → fails again → bootcount=3 → exceeds bootlimit=3 → U-Boot runs altbootcmd, which resets bootcount=0 and boots the old slot → the old slot takes the else branch and reports healthy. Net result: the robot is back on the known-good image with no site visit.
Rollout stages (200-robot fleet, illustrative)
| Stage | Robots | Cumulative | Halt condition |
|---|---|---|---|
| canary | 1 | 1 | any health-check failure |
| revision sweep | 1 per hardware revision (4) | 5 | any failure |
| 5% | 10 | 15 | any revert, or >1% failure |
| 25% | 50 | 65 | >0.5% failure |
| remainder | 135 | 200 | — |
Failure modes and how you catch them
- Revision-specific bootloader assumption. The rollback path was never exercised on the older board. Detection: the revision sweep cohort must contain every revision, and you deliberately fail a health check on each one in staging to watch the revert happen before the first production push.
- Shallow health check. The robot boots, reports healthy, commits, then can't localize in the aisle. Detection: the check runs a real mission, and you track post-commit incident rate per version, not just boot success.
- Rollout controller stalls on a network partition. Robots can't report, the controller waits forever, and the fleet sits half-updated. Detection: a local timeout on the robot that reverts on its own if it can't reach the fleet manager within N minutes of a trial boot.
- Power loss mid-flash. A/B protects the running slot; a single-slot design does not. Detection: refuse to start a flash below a battery threshold and require the robot to be docked.
- Flapping rollback. An intermittent bug bounces a robot between slots. Detection: count reverts per robot; two reverts on the same unit halts the rollout and pages a human.
Trade-offs
A/B costs a second copy of the image in flash and needs bootloader support; on a small MCU that is real money. Single-slot plus a recovery partition is cheaper, but recovery is a site visit. Automatic rollback is not a substitute for a canary: it bounds the damage per robot, not the number of robots damaged. And if the update changes the fleet protocol, teach the fleet manager both versions first, or you will partition your own fleet.
An update you cannot undo remotely is a site visit waiting to be scheduled.
Curated: · Written: · Reviewed:
Q&A-92A gantry rings after every move. What determines whether you fix it in mechanics or in control?
Measure the axis first, then decide on the ratio of the lowest structural mode to the bandwidth the motion profile actually needs. Control can avoid exciting a mode; only mechanics can move it. If the mode sits inside, or within roughly a factor of two of, the required bandwidth, no controller will save you and the fix is mechanical. If it sits well above, three to five times the bandwidth, a notch or input shaping is cheaper and faster to deploy.
Assumptions that matter: one dominant mode, roughly linear behaviour near the operating point, and a bandwidth defined by the frequency content of the commanded profile, not by the servo loop's crossover alone.
Why control cannot move a mode: the natural frequency is set by stiffness and mass, sqrt(k/m). A notch filter removes gain at the mode but adds phase lag that grows as you approach the notch frequency, and that lag comes straight out of your phase margin. Input shaping is delay-based: a zero-vibration shaper takes half the period of the mode to complete, so a 14 Hz mode costs about 36 ms of added delay on every move. A real cycle-time hit on a fast pick-and-place. Neither changes where the mode is.
Damping decides how much either remedy buys you. A mode with a damping ratio around 0.02 rings for many cycles and shaping helps a lot; a mode already at 0.1 or above may not need anything.
Mechanics: adding stiffness or mass raises the frequency, but mass costs acceleration and motor sizing, and stiffening a gantry usually means more material, more cost, and a heavier axis. That trade is worth it only when the mode is in the way.
| Mode | Required bandwidth | Remedy |
|---|---|---|
| 60 Hz | 9 Hz | notch |
| 14 Hz | 9 Hz | stiffen |
| 14 Hz | 3 Hz | input shaping |
Failure mode I have seen: a team notched a 14 Hz mode on an axis that needed 9 Hz of bandwidth. The ringing disappeared, but the notch's phase lag at 9 Hz ate the margin, the axis could no longer track the profile, and cycle time rose 18 percent. Detection is straightforward: measure the frequency response function of the axis, swept sine or impact hammer, and plot the mode against the required bandwidth before choosing. Then verify with tracking error during the actual profile, not just a step response.
The decision rule: mode well above bandwidth and stable, notch or input shaping. Mode near or inside bandwidth, stiffen. Mode unstable or drifting, fix the mechanics regardless, because a controller tuned around a moving mode is a controller that will need retuning.
Curated: · Written: · Reviewed:
Q&A-98A cell handles four product variants and errors cluster in the first ten minutes after a changeover. Why?
The 60-second answer
The clustering is two effects stacked, and you have to separate them before you fix anything.
At the instant of changeover the cell runs on state nobody has verified: tooling, fixture, program offsets, part presentation, process parameters. Any one of them can still be in the previous variant's state. And the first ten minutes is also when the cell is inspected hardest: first-article checks, an operator watching, sometimes 100% inspection, dropping to sampling after. So part of the cluster is real defects and part is detection.
What makes the errors stop tells you which cause you have. Stop at a physical intervention → state mismatch. Stop after a fixed part count with no intervention → carryover or flush. Stop because nobody is looking → detection bias.
Mechanism
Four causes produce a first-ten-minutes cluster, and they look identical in a defect Pareto:
- State mismatch. A gripper, fixture pin, feeder lane or program offset left in the previous variant's state. Errors start at part 1 and continue at a constant rate until someone changes something physical. This is the one that does not self-correct.
- Parameter carryover. Thermal and mechanical state lags the recipe. A dispenser nozzle or weld head that ran variant A's duty cycle is at the wrong temperature for the first parts; adhesive viscosity, weld penetration and torque all drift back over a few minutes. Error magnitude decays with part index, and the decay repeats across changeovers regardless of who did them.
- Material flush. The feed path still holds the previous variant's adhesive, powder or parts. Errors stop after roughly the dead volume has passed: a fixed part count.
- Detection bias. First-article inspection catches what sampling misses. The defect rate looks higher because the detection rate is higher.
Telling them apart
| Cause | Signature | Discriminating test |
|---|---|---|
| State mismatch | Constant rate from part 1, stops at a physical intervention | Compare detected tooling/fixture ID against the selected recipe |
| Parameter carryover | Error magnitude decays over the first N parts | Plot error vs part index; decay repeats across changeovers |
| Material flush | Stops after a fixed part count ≈ dead volume | Purge the path; count should drop to zero |
| Detection bias | "Stops" when inspection stops | Compare 100%-inspected vs sampled parts of the same variant |
Illustrative: a cell on an 8 s cycle runs 75 parts in ten minutes. If the first-article window is 3 parts, a carryover fault costs 3 parts per changeover; a state mismatch costs parts until detection, which is why that window runs long.
Trade-offs
A checklist is fine when the failure is caught downstream, the variant count is low, and the same person sets up and runs the changeover. It fails when the changeover is handed between shifts. A scanned tooling ID is cheap and catches identity, but not orientation or seating. An interlock is the only one that catches seating, and it costs an engineering pass per station plus a sensor that itself needs calibration. Pick by cost of escape, not by error rate.
What I'd do
Detect the fitted state rather than asking for confirmation, log every mismatch between selection and detection, and inspect the first parts at 100% until the mismatch rate is zero. Then re-measure, if the cluster survives, it was carryover or detection, not state.
Curated: · Written: · Reviewed: