This article is about the pair of plugins we built to do that, which the customer calls “ROS2 plugin for MAK one”, and what we learned on the way.

Where it started

The customer is an autonomy group developing an unmanned ground vehicle. Their testing problem might sound familiar: field trials are expensive, weather-dependent and hard to repeat, and the interesting cases are exactly the ones you cannot stage safely. They wanted two things from simulation:

• Camera imagery they could navigate on. Not a truth feed of positions, but rendered frames, in the visible and thermal bands, that their perception code would process exactly as it processes the real cameras.
• Scenarios with other traffic, at real places. Other vehicles moving through the scene, on the terrain of the areas they operate in, so a run could be set up at a named location and repeated with variations.

VR-Forces already does the second part well. VR-Vantage already renders the first. What was missing was the bridge to ROS 2, and the discipline to make the two products look, from the robot's side, like one vehicle with sensors on it.

How the pieces fit together

ROS2 plugin for MAK one is two plugins, one per product, each running a ROS 2 node inside the host process. VR-Forces and VR-Vantage talk to each other the way they always do, over DIS; the robot side sees only ROS 2 topics under a namespace named after the entity.

In VR-Forces, a component attached to the entity publishes what the vehicle's own sensors would report: IMU and compass, GPS as a NavSatFix plus the UTM position, odometry, and a separate wheel-odometry topic with a configurable random-walk drift, so a navigation stack cannot quietly cheat by trusting it. In the other direction it accepts drive commands, a standard Twist for a skid-steered vehicle or an AckermannDrive for a steered one, and feeds them into VR-Forces' own vehicle dynamics. The entity drives the way the simulation says it should, not the way a script teleports it.

In VR-Vantage, the plugin finds the imaged entity, creates an observer and a rendering channel for each camera defined in a settings file, and publishes every frame: colour imagery, or thermal when the camera is switched to a MWIR or LWIR SensorFX device; a depth image, always; CameraInfo with the channel's real intrinsics; and the static transforms from the vehicle body to each camera's optical frame. Each camera is controllable over ROS at runtime as well: pan and tilt, field of view, and the sensor band, so a test can switch a camera from visible to LWIR mid-run without touching the VR-Vantage GUI.

What turned out to be hard

We expected the thermal imagery to be the centre of the conversation. Every customer wants convincing thermal, and SensorFX delivers it. But in every meeting the customer came back to one word: latency. A navigation stack closes a control loop through the camera. If the frame it acts on is stale, the vehicle is steering by where things were, and the stack's tuning, done against real cameras, no longer holds. Rendering a good image was the easy promise. Delivering it fast enough, every frame, was the one we had to earn.

The first version did the obvious thing, reading back, compressing and publishing each camera in turn on the rendering thread; it worked, and every camera we added made it slower.

What fixed it was separating the work from the rendering. Getting the pixels off the graphics card became one shared step for all cameras instead of one per camera. Compressing and publishing moved into the background, so the renderer never waits for them. A camera keeps only its newest frames and lets old ones go, so a subscriber that falls behind receives what is current rather than a backlog. And nothing is captured at all for a topic nobody is listening to. The frame rate is now set by the graphics card and the scene rather than by the plugin, and tuning that pipeline took more of the project than any feature did.

The other lesson was quieter. Between the simulation and the robot sit several coordinate frames, and every one of them has to be right. VR-Forces reports positions in geocentric coordinates; the robot wants a local east-north-up world. The vehicle body is described three ways, by DIS, by VR-Vantage's model frame and by ROS. Each camera sits on the body at an offset, and the image has its own optical frame, where Z points down the lens, which is what the perception code actually reads. Seven layers, each a rotation that is easy to get almost right. When one is wrong, an autonomy stack does not crash. It drives slightly wrong, which is worse. Getting every one of them right once, and writing them down in an interface control document, mattered as much as any feature.

What's next

The aim is to close the gap between the two worlds for good: a robotics team should be able to plug in their hardware, point it at MAK ONE, and have their robot's world swapped for a simulated one with as little setup as possible. That means making the plugin more automatic, so that entities, cameras and topics come from the scenario rather than from configuration. The same approach is under construction for MAVLink, so that a flight controller can fly a VR-Forces aircraft the way a ROS stack now drives a VR-Forces vehicle.

Omer Salomon is a Software Engineer at Synergy Integration Ltd., MAK Technologies' distributor in Israel.