Decentralised AI Training and Inference with BlockTrain

The article introduces Spheroid BlockTrain, a decentralized AI training protocol that partitions models into independently trainable blocks, optimizing local objectives derived from a global target. BlockTrain achieves a cross-entropy of 1.359 on byte-level WikiText, closely matching a full Transformer reference, while allowing each worker to focus on a single block, thus minimizing resource requirements. This approach enhances scalability and efficiency for AI practitioners by enabling decentralized training and inference across multiple hosts, significantly improving performance metrics like perplexity and reducing the need for centralized infrastructure.