The growing battle for AI inference is opening up on a new front: the millions of computers, gaming PCs and small servers with unused GPUs already sitting in homes and small businesses, with several new companies launching using a rather familiar strategy.
A startup from Abu Dhabi named Far Labs is preparing to launch a new platform called Far AI, while Evolving Edge of Austin, TX, is already operating its service in public beta. Other companies in the space include Bless Network, Salad, and Gradient, each with their own platforms that connect distributed computing resources with AI workloads.
This should sound familiar to you. It is known as distributed computing and has been around for almost three decades. The idea is to put a program on a PC and when the PC is idle the program executes a processing task. Because CPU utilization can shoot to 100%, most of these programs only run when the PC is idle and stop when the PC is in use.
These computing projects have a similar method of operation. They take a very large processing job and divide it up into bits that can be distributed to volunteer users. Because the PCs don’t communicate with each other, only the command server, they can only be used to process a job with no dependency on other results.
The most famous case was SETI@Home, which sifted through radio telescope recordings to look for signals from alien worlds. It launched in 1999 with a nifty-looking screensaver while it processed signal recordings from a radio telescope. It shut down in 2020 after 20 years and no results.
A more successful example is Folding@Home, which simulated how proteins fold and misfold. A misfold can lead to diseases like cancer and Alzheimer’s disease. Originally started at Stanford University, it has published more than 200 research papers for its findings in nearly 20 years of operation.
What’s going on here is no different, just the computational process. Rather than processing data like radio signals, these startups are using otherwise idle hardware to run inference workloads. The model could provide an alternative to relying exclusively on massive data centers while giving computer owners a way to monetize unused processing capacity.
AI inference—the process of running a trained AI model to generate an answer to a user request—has become an increasingly significant source of computing demand. Virtually every AI query on a chatbot like ChatGPT or Google Gemini is in fact an inference process, and they add up two very high workloads very quickly.
The good news is that individual inferencing tasks don’t require a great deal of horsepower. You don’t need a top of the line NVIDIA GPU to do inference, you can get by with the mid level GPU or even CPU.
That is creating an opportunity for distributed computing providers. Instead of sending every AI workload to a hyperscale data center, software identifies available computing resources across a network of independently operated machines.
Owners can decide when their hardware is available and potentially receive compensation for allowing AI workloads to run on it. That’s a big difference over the older distributed computing programs. No one was paid for lending their system to SETI@Home or Folding@Home but here you have a chance to make money. How much is still to be determined.
Another thing that needs to be worked out is the amount of performance needed for the model. Every AI large language model is different and will require different levels of computation for inference. So the software will have to be extremely flexible to handle changing computational demands from one model to the next.
Still, it has the potential to take the workload off of data centers and spread it around to individual PCs. There are just many variables that need to be worked out.
