Beyond Control: Risks and Opportunities to Govern Open-Weight AI Models

International institutions and researchers have begun discussing the benefits and risks associated with releasing publicly accessible weights of foundation models as part of the broader conversation on AI openness. Open-weight models are susceptible to unintended uses and malicious activities by bad actors. There is no ability to recall the models when released nor is it feasible to remove all copies. This article aims to explore the levers available to govern open-weight models beyond government and developer controls and proposes some possible approaches to responsible AI policy for consideration.

What Is an Open-Weight Model?

An open-weight model is a foundation AI model whose trained weights are publicly accessible, can be downloaded, modified and used by anyone. These models accounted for more than half of the market in early 2023^[1]. Their capabilities are growing and the frontier open-weight models are expected to catch up to proprietary AI models in 6 to 12 months^[2]. Top model developers are from the United States, China and France, with the Netherlands and Singapore serving as key provider hubs.

It is important to note that while these features of open-weight models seem to share similar concepts with open-source AI or open-source software, the nuances need to be clarified. The key distinction is that open-weight models do not release underlying training code and datasets^[3]. Additionally, contractual frameworks for open-source software licensing do not directly apply to AI model weights^[4]. This creates a barrier for those who wish to audit, replicate and understand the training process of open-weight models.

What Are the Levers Available to Govern Beyond Control?

In the recent G7 Digital Ministers Meeting, members recognized the similar role to open-source software that AI plays for innovation, research and economic growth but also highlighted the future work needed to address the potential risks around security vulnerabilities, malicious use concerns and supply-chain risks^[5].

This is a nascent space that needs to be monitored. Introducing early interventions to promote safe and responsible adoption of these AI models is as important as the promising benefits that AI openness provides. Here are some levers that could be considered to manage the potential risks:

Incentivize the Producers in the Supply Chain to Encourage Safety Over Speed

As one of the largest technology purchasers in the market, government can set the conditions to procure technology without changing legislation. Through its procurement power, government can encourage the communities of developers, startups and researchers to develop models that are fit for use by government and their regulated parties with considerations of safety and security in model development.

Intervene Upstream Before Model Release to Prevent Proliferation of Risks

Open-weight models spread quickly and it might not be operationally feasible to evaluate all deployed open-weight models and prevent downstream impacts. As model capabilities improve, it is evermore important to require evaluation before release to ensure safeguards are built-in to prevent possible downstream tampering.

Verify Safety and Security Claims by Third-Party Organizations and Independent Experts

For models that are developed with safety and security considerations, self-assessment can be subjective and might not reflect their actual performance. As AI becomes ubiquitous and is used on a daily basis, it is important that these claims are evaluated by third party organizations or independent experts to verify the true performance. This helps to eliminate possible conflicts of interest and ensure integrity of the evaluation.

Assess Models Against Tamper Resistance and Hazardous Capability Resistance

Many of the existing open-weight models lack information on tamper resistance training^[6]. While there are legitimate reasons for keeping the information confidential, it is important that developers have assessed their models against tamper resistance and hazardous capability resistance before deployment. Government can encourage these assessments by providing support through its research and standard-setting capabilities.

What Does It Mean for Canada?

Canada’s National Artificial Intelligence Strategy: AI for All demonstrates the country’s strategic direction in open-source AI and infrastructure to maintain flexibility, autonomy and cost efficiency. This direction is unclear as open-source AI is not defined in the strategy and policy measures designed for open-source ecosystems may not adequately address the risks and governance challenges associated with open-weight models. Furthermore, model developers might not have sufficient resources compared to proprietary AI model developers to fine-tune before releasing the model into the open-source community, making the marketplace crowded with noise and creating a possible higher chance of misuse.

Choosing the right models can be difficult for businesses and organizations without domain expertise and they need reliable ways and trustworthy sources to verify technical safeguards exist and to be informed when they do not. Canada can take lessons from its internationally-known, rigorous food safety regulations to solve this problem and encourage safe adoption of AI for All.

Labelling open-weight models for misuse-resistant evaluation by a third-party organization to signal safe use across distribution platforms is a possible remedy.

Open-weight models do not need to fly under the radar nor inherently create cyber risks. Planning around user adoption to ensure trust is built on credible scrutiny and promote the responsible use of AI will be the way forward in this dynamic AI ecosystem.