Despite being significantly less efficient than frontier AI models, mechanistic interpretability models such as weight-sparse transformers can yield enormous dividends in boosting industry adoption and establishing societal trust in AI
Image Source: Getty Images
While the phenomenal rise of Generative Artificial Intelligence (AI) and Large Language Models (LLMs) has spawned a plethora of novel applications spanning a multitude of fields, the inner workings of these models remain largely unknown, rendering them vulnerable to various threats, including biases, hallucinations, and backdoors. In this regard, OpenAI’s novel work on the weight-sparse transformer model is encouraging and may offer a viable solution to the so-called “AI black box” problem. Though by no means comparable to the capabilities of advanced LLMs like GPT-5, weight-sparse transformers may offer new insights into the mystery surrounding the operational underpinnings of current AI models, thereby paving the way for a more secure and trustworthy future for AI adoption.
Neural networks consist of nodes or neurons arranged in multiple layers, which make predictions using pattern recognition. The majority of these nodes are connected to all others in adjacent layers, leading to “dense networks.” The strength of the connection between these nodes is determined by numerical values called weights and biases, which are iteratively adjusted to reduce error. Each node assigns a weight to its incoming connections, which is multiplied with the data if it exceeds a certain threshold value; otherwise, it is set to zero. Subsequently, the weighted inputs are added with a bias and passed through an activation function to give the final prediction.
Figure 1: Basic Structure of a Neural Network
Source: Medium
Generative AI largely works on a specific neural network architecture known as a “transformer,” which was developed in 2017 and constituted an evolutionary step forward for neural networks and machine learning, eventually leading to LLMs like ChatGPT.
A major issue with generative AI and LLMs is that they essentially function as black boxes. This is because transformer models are constituted of dense networks, which consist of a vast collection of nodes. These interact amongst themselves in complex ways, with each node representing multiple different features, a phenomenon known as superposition. Consequently, it is incredibly challenging to understand how transformer models actually work and isolate the exact cause behind anomalies such as hallucinations and biases, which have proven incredibly detrimental to the technology in multiple cases. For instance, one of the most infamous examples of the former occurred in 2023 when Google’s Bard chatbot incorrectly stated that the James Webb Space Telescope was the first to take pictures of a planet outside Earth’s solar system, which subsequently led to its parent company, Alphabet, losing more than US$100 billion in market value.
The pursuit to better understand AI models led to the development of the field known as “mechanistic interpretability”, which aims to reverse engineer dense neural networks and convert their underlying algorithms into human-understandable concepts.
The pursuit to better understand AI models led to the development of the field known as “mechanistic interpretability”, which aims to reverse engineer dense neural networks and convert their underlying algorithms into human-understandable concepts. Though several attempts have been made at achieving mechanistic interpretability, none have been particularly successful so far.
In a recent paper, OpenAI introduced the idea of “weight-sparse transformers”, which makes it much easier to understand how transformer models work. It does so by training transformers for simple, hand-crafted tasks where most of the weights are zero. For instance, one of the tasks employed was adding closing quotation marks at the end of a sentence.
This leads to simpler, fine-grained or “sparse” circuits wherein the nodes have a substantially reduced number of straightforward and interpretable connections, thereby limiting the complexity brought about by superposition. Therefore, sparse circuits essentially lead to untangled neural networks consisting of nodes with a limited number of connections.
Figure 2: Dense vs. Sparse Circuits
Source: OpenAI
However, weight-sparse transformers come at a cost. They essentially trade off capability for interpretability, making them 100 to 1000 times less efficient than dense models of comparable capability.
Though it is unlikely that this new class of sparse models can be scaled beyond the capability level of OpenAI’s GPT-3, they nonetheless present some interesting possibilities concerning the future of AI models. The primary reason for this stems from the fact that the OpenAI research team has been able to couple the weight-sparse transformers to dense models via the use of bridges. This implies that they may be able to explain the inner workings of frontier models even though they are themselves not scalable to their capability level. Furthermore, a sparse bridged model can potentially be trained on a narrow but critical task distribution such as deception, refusal, and goal-seeking, which could be incredibly useful in addressing AI safety threats.
Even though they are not a viable alternative to existing LLMs, weight-sparse transformers can be extremely helpful from the perspective of attaining a more comprehensive understanding of existing AI models.
So, even though they are not a viable alternative to existing LLMs, weight-sparse transformers can be extremely helpful from the perspective of attaining a more comprehensive understanding of existing AI models. Furthermore, while they may be less relevant from a commercial point of view, mechanistic interpretability models such as weight-sparse transformers are essential for opening the AI black box and uncovering how they function at an operational level. This is critical to understand and potentially do away with issues like hallucinations, which are severely impeding the effectiveness and applicability of generative AI, thereby limiting the utility of future advancements.
One of the fundamental problems with AI adoption at the moment is that AI models suffer from persistent security threats like hallucinations, which ultimately stem from a lack of understanding regarding their internal operation. This has led to several instances where AI chatbots have acted in an unpredictable and even malicious manner, including beckoning users to commit crimes. One such case involved a 17-year-old in Texas who received suggestive remarks from the Character.ai chatbot, insinuating that he should murder his parents for limiting his screen time. Beyond the societal threats posed by these models, this has simultaneously been quite detrimental to the pursuit of cultivating industry trust in the technology, thereby severely hampering wide-scale industry adoption.
In this context, although emerging approaches to AI development — such as mechanistic interpretability and glass-box models — may appear less commercially lucrative than frontier models, they can play a critical role in building human trust and boosting AI adoption. This is critical for maintaining investor confidence and can help tremendously in dictating a safe and predictable future for AI. More importantly, they can go a long way in addressing the existential threats posed by AI since malicious and errant outputs can, in principle, be effectively addressed and eliminated by employing these models.
Despite being less commercially lucrative at the moment, investing in mechanistic interpretability models like weight-sparse transformers must constitute a fundamental tenet of national AI goals in addition to pursuing sheer AI dominance through increasingly advanced LLMs and generative AI models.
Although AI has the historic potential to be deployed as a significant tool to augment human capabilities, it is severely handicapped by a fundamental lack of understanding regarding its actual functionality. While current AI development models being pursued by most nations lean heavily on outpacing peers and acquiring a competitive advantage — particularly given the dual-use nature of the technology — the fact is that no technology can truly progress without establishing human trust. Consequently, despite being less commercially lucrative at the moment, investing in mechanistic interpretability models like weight-sparse transformers must constitute a fundamental tenet of national AI goals in addition to pursuing sheer AI dominance through increasingly advanced LLMs and generative AI models. The future of AI utility and adoption hangs in the balance.
Prateek Tripathi is an Associate Fellow with the Centre for Security, Strategy and Technology (CSST) at the Observer Research Foundation.
The views expressed above belong to the author(s). ORF research and analyses now available on Telegram! Click here to access our curated content — blogs, longforms and interviews.
Prateek Tripathi is an Associate Fellow at the Centre for Security, Strategy and Technology. His work focuses on an emerging technologies and deep tech including quantum ...
Read More +
PREV
