RGB vs. depth cameras for robot skill training

The short version

• An RGB camera captures color and texture, cheap, high-resolution, and full of semantic detail, but with no direct sense of how far away anything is.
• A depth camera captures per-pixel distance, giving the robot metric 3D geometry for grasping and collision, but it is noisier, lower resolution, and fails on some surfaces.
• Most robot skill training today is RGB-dominant. Depth earns its place on precise, contact-rich, or spatially ambiguous tasks, often fused with RGB as RGB-D.

Vision is where most manipulation policies get almost all of their information about the world, so the camera you train on quietly shapes what the robot can and can’t learn. The choice usually comes down to RGB versus depth, and they are not competing versions of the same thing, they capture fundamentally different information. Understanding what each one actually gives a policy explains why some setups train robust skills and others struggle with tasks that look easy.

What an RGB camera captures

An RGB camera records color and texture: the same signal your phone captures, three channels of brightness at high resolution. That makes it extraordinarily information-dense. Edges, materials, text, the difference between a ripe and unripe tomato, the seam on a connector, all of it lives in RGB. It is also cheap, ubiquitous, and the format every modern vision model was pre-trained on, which matters enormously for transfer. What RGB does not give you is geometry. A single RGB frame can’t tell you whether an object is small and close or large and far, scale is ambiguous, and appearance shifts with lighting, so the same scene under different light can look like a different scene to the policy.

What a depth camera captures

A depth camera reports, for each pixel, how far the corresponding surface is from the sensor. Instead of color it produces a distance map, which is exactly the metric 3D geometry an RGB frame lacks: where surfaces are, how objects are shaped, how much clearance the gripper has. That is directly useful for the geometric core of manipulation, planning a grasp, approaching without collision, judging whether the hand cleared the bin wall. The distance is measured in one of a few ways, stereo (triangulating between two lenses), structured light (projecting a known pattern), or time-of-flight (timing a light pulse), but from the policy’s point of view the output is the same: geometry rather than appearance.

Where each one fails

RGB fails on geometry and lighting. It has no metric depth, so tasks that hinge on precise 3D relationships are inferred rather than measured, and a change in lighting or background can shift the input enough to break a policy that never saw that condition.

Depth fails on hard surfaces and detail. Reflective, transparent, and very dark objects, a glass, a chrome fixture, a black connector, confuse most depth sensors and come back as holes or noise. Depth maps are typically lower resolution than RGB, blur at object edges, and have a limited working range; structured-light and many stereo sensors also degrade badly in bright sunlight. And depth carries no semantics, it tells you a surface is there, not that it’s the handle you wanted to grab.

What today’s policies actually train on

Given all that, the center of gravity in robot learning has moved toward RGB. The vision-language-action models behind the current wave of robot foundation models are overwhelmingly RGB-based, because they inherit representations from internet-scale image and video pre-training, and depth data at that scale simply doesn’t exist. For a huge range of manipulation, well-lit RGB with a good policy is enough: the network learns the geometric cues it needs from texture, shading, and motion, the same way people judge depth with one eye closed.

When depth earns its place

Depth pays off when geometry is the bottleneck: high-precision insertion, bin picking into clutter, tasks with heavy occlusion, or anything where a few millimeters of misjudged distance means failure. In those cases teams usually don’t drop RGB, they fuse the two as RGB-D, so the policy gets semantics from color and metric structure from depth. The practical rule of thumb: start with RGB because it is cheap, transferable, and often sufficient, and add depth deliberately when a specific task proves it needs measured geometry, not more pixels.

Getting it into training

Whichever you choose, the camera stream is only useful if it lands in training aligned. Depth and RGB frames have to share a clock and a calibration so that a distance map lines up with the color frame and the action at the same instant, a few milliseconds of drift turns clean data into subtly mislabeled data. This is the same alignment problem every other stream in a demonstration has, and it’s why how you collect and validate the data matters as much as which sensor you picked. On Neuracore, RGB and depth feeds from any supported robot are ingested, time-aligned, and versioned together, so you can train on RGB, depth, or both without stitching the streams by hand.