Those two jobs need different physics’: Rebellions CEO says training and inference need different chips

The AI race started off with a pretty clear direction – bigger and better. The first waves were characterized by building bigger models, but it’s all change in the world of artificial intelligence and with enterprises, SMBs and consumers all finding use cases for the technology, the focus has shifted.

Now, AI firms and model developers are looking to realize a much tougher goal. Efficiency. Cost per token, performance per watt, output per input, it’s all about driving maximum efficiency.

One clear divide is between training and inference. While training models still requires huge amounts of resources, inference efficiency is starting to improve, and one company (Rebellions) now believes an opening for inference-first hardware could create a new market.

The company’s racks are said to consume around 16-20kW, compared with around 120kW for leading GPU-based inference systems that, for many use cases, are sheer overkill.

Rebellions’ rack costs are also said to be around one-third of the price, making AI inference more accessible and helping enterprises to deploy AI more widely.

This hardware shift could be the start of truly efficient AI

Memory is also another battleground, whereby huge trillion-parameter models are testing the limits of today’s hardware and the intertwined reliance on memory and compute. Something Rebellions says it’s looking to fix by working with the likes of SK Hynix and Samsung to align multiple roadmaps, instead of having to respond to shifts in architecture.

Ultimately, today’s black-and-white chip manufacturing landscape is now evolving, and Rebellions sees two key changes happening simultaneously. Firstly, training and inference hardware is starting to differ more drastically. Secondly, aligning multiple hardware roadmaps across memory, compute and more will drive more efficiency not just across deployments, but in terms of bringing new products to market.

I spoke to Rebellions CEO Sunghyun Park allows me to understand how and why inference and training hardware are starting to separate, as well as the importance of open standards and collaboration in the drive for all-round efficiency.

The AI chip market seems to be splitting between training-first and inference-first architectures. Why is that happening, and why now?

This split exists because training and inference are fundamentally different problems.

Training is how you build a model. It happens once, involves a small number of organizations, and rewards raw computational flexibility because the workload keeps shifting as research moves forward.

Inference is how you actually use a model: every query answered, every transaction processed, every decision an AI system makes in production. That happens billions of times a day across nearly every industry, and it’s where AI moves from R&D into revenue.

Those two jobs need different physics. Training requires maximum FLOPS. Inference requires efficiency, reliability, and economics that hold up when you’re serving users at scale.

The industry forced a training chip into that second job because that’s what existed. Now that inference has become the larger, more urgent market, that compromise no longer holds. Enterprises and governments are asking how fast they can deploy. That’s why the conversation is splitting now.

Where does Rebellions fit in that split, and what makes your approach to AI inference fundamentally different from your competitors?

We built for inference, from day one. Most first-generation AI chip companies emerged from the 2016-2017 training boom and adapted their architectures for inference afterward.

We started in 2020 – after that wave – with inference as the only target, which meant designing around what production AI actually needs instead of retrofitting a training chip.

The numbers reflect that choice. Our racks draw 16-20kW versus roughly 120kW for leading GPU-based inference systems, about a sixth of the power, in a market where power is the binding constraint for most operators.

Acquisition cost runs around $10 million per rack versus roughly $30 million, about a third of the cost. Our chiplet-based architecture also scales out rather than betting that a single device can handle a model’s full size, which matters now that production workloads are trillion-parameter mixture-of-experts models instead of the few-hundred-million-parameter models the first generation was built around.

It’s also why our architecture is memory-centric rather than compute-centric. The chiplet approach exists to keep memory close to logic as models scale, not just to add cores.

And we have three years of production deployments behind that architecture, not pilots. That’s the hardest part to replicate: real workloads, running at scale, today.

A year ago, everyone in AI was talking about chiplets. Now the conversation has shifted to memory. What changed?

The chiplet conversation was about architecture: breaking a chip into modular pieces that scale independently, rather than betting everything on a single monolithic die.

That mattered because it let the industry move past an assumption the first generation of accelerators made in 2016 and 2017, that a single device would always be big enough to run any model.

That assumption broke once mixture-of-experts and trillion-parameter models arrived.

The memory conversation is the layer underneath that. Once the architecture problem is solved, the constraint becomes physical: can you actually get enough high-bandwidth memory (HBM) to build what you’ve designed?

HBM is 3D-stacked memory, and how closely you can physically stack it to compute is as much of a bottleneck as raw supply.

Every AI accelerator company is competing for the same limited supply right now, and demand has outpaced what memory makers can produce. That’s the memory-logic co-design problem: architecture and memory supply are no longer separable decisions.

We’re in a different position because our investor relationships were bui...

Indexed and credited by AIPROPX. Originating outlet: TechRadar. Open at source →