Background
Varjo XR-4 contains two chips, which we call vision processors; these connect to the bulk of the headset sensors, such as video see-through (VST) cameras, tracking cameras, and the IMU. Vision processors handle certain preprocessing steps on the sensor feeds and relay them over USB to the host PC. Because XR composition and the tracking algorithms run on the PC, that USB streaming is bandwidth-intensive and latency-critical.
The firmware on the vision processors is complex; it must push multiple
high-resolution streams through various IP blocks and then onward over USB. Each
vision processor has a single low-power ARM core; even memcpy() is so slow
that hardware offload throughout is the only option. The firmware we’ve been
shipping works, but it has inefficiencies that result in, for example, high USB
transfer latency variance, delays in camera control command processing, slow
start-up, and other fragility.
Caveat: The firmware is a key performance differentiator in Varjo XR-4, so I’m not sharing the numbers comparing it to the old one or most of the implementation details. The excitement in this post rests on measurements that aren’t publicly verifiable, so you will have to take my word for it – at least until Varjo Base 4.17 ships and XR-4 users can see the difference for themselves.
Challenge for Fable
This summer when Anthropic released Fable, the new model tier above Opus, I started wondering what would be a suitably hard task to truly benchmark its capabilities. That firmware came to mind quickly. I pointed Fable at the firmware, told it the product-level shortcomings of the current version, and asked if it could be improved. Fable proposed a rewrite plan, starting with explorations it called spikes, to de-risk the architecture. After agreeing on the goals and high-level approach through some back and forth, I left it to work.
I was on vacation, and Claude Code was running on my workstation at the office with the XR-4 connected to it. Programmatic control of the headset power supply and headset-internal peripherals was readily available through our software stack. So I assured Claude that it could drive it hard and explore how the vision processor chips work, without risk of permanent damage.
Progress was stunning:
- Day 5: Claude reported that it had seen a “typical office scene” through the VST cameras.
- Day 10: gaze tracking cameras were streaming at 200 Hz.
- Day 17: the feeds needed by SLAM were up.
- About a month in: the full XR-4 headset experience was running on the rewritten firmware.
And I did not need to touch the headset at all until I was ready to test it. All this with firmware that was a clean-slate rewrite; it had to write everything from scratch, from boot to MIPI drivers to various accelerator blocks and USB.
Paying for progress
During the project Claude had to tackle some truly gnarly issues that required reverse engineering undocumented hardware behavior. These were the tasks where paying for the very best model mattered. Opus would only make very slow incremental progress, while Fable competently deployed various expert-level strategies to triage the issue from multiple angles, and often in parallel. When the task got truly hard, the best model delivered staggeringly better progress per dollar.
Opus still wrote the bulk of the code; deploying Fable for everything would not have been cost-effective. Over time I also settled on a pattern where even the hardest tasks were “prepped” by Opus; once Opus had gathered evidence and done what it could, the problem was then handed over to Fable to crack.
Result
The new firmware has a completely different architecture from the old one and solves most of the hard problems differently. The resulting firmware significantly exceeds the old one in all measured indicators. It will be released as part of Varjo Base 4.17 and delivers a substantial upgrade to the XR-4 experience. All this in approximately two orders of magnitude less development time and cost than the original firmware.
AI-accelerated engineering delivered a win so decisive and dramatic that it is still hard to believe.
What actually made this work
So, why did this project turn out to be a success, and not turn into an unmaintainable, vibecoded mess? Given the project’s complexity, all ingredients for the latter existed.
Throughout the project I focused on guiding Claude on the requirements, architecture, and strategy. I used an evidence-first strategy that relied heavily on automated hardware-in-the-loop (HIL) testing. Claude was not allowed to call anything implemented unless proven by HIL tests. This resulted in an extensive and strict HIL test suite that made it possible to safely keep making changes even as complexity grew.
I also pushed Claude to root-cause and fix every flake, frame drop, and other abnormality. Once fixed, these became tests in the HIL suite that prevented regressions.
The USB stack was rewritten three times until we converged on software-QoS-over-bulk, which delivered on every system tested. Tens of patches to the USB IP vendor driver were required to weed out rare stalls or misbehavior in edge cases. Fault injection tests and observability surfaces made rare corner cases part of regular regression runs.
Manual review could not keep up with the volume of code from multiple parallel agents working on bringing up the firmware and polishing it. Instead, I prioritized reviewing plans, any changes to ratified requirements or architecture, and testing evidence. Exactly as you would if managing a larger engineering team.