AI Learns Better by Selectively Forgetting Less Important Information

Flexible AI Language Models: The Power of Selective Forgetting

A New Approach to Machine Learning

Researchers have developed a more adaptable and efficient type of machine learning model that periodically erases some of its knowledge. While this innovative approach may not replace the massive models powering the most popular applications, it could provide insights into how these programs comprehend language.

“We live in the same world. We conceptualize the same things with different words” in different languages, said Yihong Chen, the lead author of the recent paper. “That’s why you have this same high-level reasoning in the model. An apple is something sweet and juicy, instead of just a word.”

Overcoming Limitations in Traditional AI Training

Traditional AI language models rely on artificial neural networks, which require extensive computing power and are difficult to modify once trained. Mikel Artetxe and his colleagues have developed a method to circumvent these limitations by periodically resetting the embedding layer, which stores language-specific information, during the initial training phase. This allows the model to become accustomed to resetting and makes it easier to extend to new languages later on.

Improved Performance and Flexibility

The researchers compared their periodic-forgetting technique to the standard, non-forgetting approach using a common language model called Roberta. The forgetting model performed only slightly worse than the conventional one during initial training but showed significantly better results when retrained on other languages using smaller data sets and computational limits. This suggests that periodic forgetting enables the model to learn languages more effectively on a deeper level.

Mimicking Human Memory and Understanding

The periodic-forgetting approach resembles how the human brain functions, as explained by neuroscientist Benjamin Levy:

“Human memory in general is not very good at accurately storing large amounts of detailed information. Instead, humans tend to remember the gist of our experiences, abstracting and extrapolating. Enabling AI with more humanlike processes, like adaptive forgetting, is one way to get them to more flexible performance.”

Bringing AI Breakthroughs to More Languages

Artetxe and Chen hope that more flexible forgetting language models could help bring the latest AI advancements to a wider range of languages, particularly those with limited training materials. By adapting existing models to less common languages, such as Basque, this technology could enable the creation of multiple specialized language models rather than relying on a single, monolithic model.

The development of nimbler and more adaptable machine learning models through selective forgetting marks a significant step forward in the field of AI. By shedding light on how these programs understand language and mimicking human memory processes, this approach could pave the way for more diverse and accessible AI applications in the future.