When it comes to artificial intelligence, training models is a headache: it’s slow, expensive and requires a lot of hardware. To solve this problem, Apple developed an ingenious technology called ReDrafter, which it shared as open source earlier this year. The objective? Accelerate the process of generating “tokens” — these small blocks that make up the responses of AIs like ChatGPT.
A smart way to think faster
Traditionally, these tokens are produced one by one in a sequential process, much like writing a sentence letter by letter. This is where ReDrafter changes the game: this method uses an approach called “speculative decoding”. Rather than producing each token following a single path, ReDrafter generates multiple options in parallel, then validates the best one.
To achieve this, the technology relies on a recurrent neural network (RNN) and a tree structure. This may sound technical, but imagine an engine that tries multiple sentences at once, keeps the most relevant, and then continues. Result: up to 3.5 times more tokens generated per step, which drastically reduces training time.
For such technology to be usable on a large scale, it must work with GPUs, these super-processors often used for complex AI tasks. Apple therefore collaborated with Nvidia to integrate ReDrafter into the TensorRT-LLM framework, a tool designed to optimize calculations on Nvidia GPUs.
And it works! By testing a model of several tens of billions of parameters on Nvidia H100 GPUs (the stars of the moment), Apple observed a speed multiplied by 2.7 for the generation of tokens. In short, we go much faster with less material. Businesses benefit by reducing costs, and users benefit from faster query responses in the cloud.
For the general public, this means faster and perhaps more accessible AI services. Imagine asking a virtual assistant a question and getting a near-instant response, even during peak hours.
For developers and businesses, ReDrafter is a promise of efficiency. By integrating validation directly into the calculation engine, Nvidia and Apple were able to reduce unnecessary operations, while leaving room to design even more sophisticated models in the future.
This collaboration is part of a broader dynamic: Apple is also exploring other technologies, such as Amazon’s Trainium2 chips, to continue to push the performance of its artificial intelligence models. With ReDrafter, the foundations are laid for further progress, without exploding the energy bill.
???? To not miss any news on the Journal du Geek, subscribe on Google News. And if you love us, we have a newsletter every morning.
