Revolutionary Optimizer Boosts AI Performance

Researchers have developed an optimizer that significantly reduces the memory required to train large language models, allowing for much larger models to be trained on modern computers. The new optimizer, called TACO, works by selecting only the most important updates from each column of a weight matrix, making it much more efficient with its "memory" - the amount of data it needs to store and process during training. This allows for full-parameter fine-tuning of massive models that were previously too big to fit on even the largest computers, opening up new possibilities for AI research and applications.