How CISOs can apply threat modelling to AI products in four steps
AI is fast becoming a cornerstone of how smart businesses operate. According to recent research by IBM, more than one in three (37%) UK businesses are actively using AI in their operations, and 41% are experimenting with this technology.
From identifying financial fraud, to predicting environment changes in supply chains, purpose-built AI models are becoming increasingly embedded into ways of working to improve efficiency and tackle industry challenges.
Ensuring these solutions are secure and resilient against threats from the offset, therefore, is vital to avoid massive disruption and potential data leaks. The NIST Artificial Intelligence Risk Management Framework (AI RMF) offers a valuable blueprint for responsible AI management, but this is just one starting line, not the finish line. To truly protect organizations, CISOs need to think like an attacker and expose those AI blindspots before someone else does.
This requires thinking beyond accidental errors and focusing on stopping deliberate attacks, biased outputs, and the potential for manipulation. By leveraging threat modelling – the process of identifying types of threats in the software development lifecycle (SDLC) – CISOs and developers can proactively address AI risks and build resilience into products.
The practical integration of threat modelling can be achieved by following these four key steps:
Senior Solutions Architect & Threat Modelling SME at IriusRisk.
Step 1: Lay the groundwork
When building a new AI system, start by developing a robust AI threat modelling policy to establish clear guidelines for its development, use and maintenance. The purpose of this is to map out the potential risks associated with your solution, and the potential impact of manipulated AI decisions within your organization.
Begin by thinking about your most critical AI systems. What are your organization’s highest priority risks associated with these systems? Consider the impact of data breaches, biased outcomes, and algorithm manipulation on different aspects of the organization such as its finances, reputation and safety.
When establishing risks, also consider where threats are coming from, both internally and externally. Once outlined, you can begin to mitigate against these risks by addressing bias testing, accountability mechanisms, and AI decision transparency – these are key areas that should be covered in your AI threat modelling policy.
Step 2: Threat model
Once you’ve defined your risk appetite and the threat model context, you can start to build out a more comprehensive threat modelling framework.
This can be as simple as creating a flowchart to visualize the way data moves within your AI systems and where it interacts with external systems. Here, you should pay special attention to the way sensitive data is handled and the potential points of exposure, prioritizing data protection and compliance in every step of the development process.
The more detailed your threat model is, the easier it will be to identify risks – even ones you haven’t thought of yet. When building out these connections, be specific. Don’t just identify that your monitoring AI integrates with a customer database, for example, but detail the exact fields pulled (PII, transaction history etc.), the transfer protocol (internal API call, nightly batch process), and any other software that also has access.
To bolster product resilience, consider the different scenarios where data integrity could be compromised, algorithms manipulated, or biases introduced, both through accidental errors and malicious intent. Also think about the likelihood of these events, as well as the impact. For example, a sophisticated adversarial input attack might be technically difficult, but a simple compromise of a weakly-secured third party data supplier could be more achievable and therefore more common. These are the vulnerabilities that should be addressed first.
Step 3: Stress test your system
Now that you’ve outlined the parameters of potential threats and have mitigated them, you should be stress testing your AI system against datasets. Using datasets from other countries with vastly different patterns of activity will help to reveal potential blind spots. For example, if you are a US-based financial organization, try testing financial datasets from another country.
If after stress testing there appears to be no issues, try subtly tweaking the format of legitimate-looking applications to expose unexpected weaknesses in how your AI handles input. The key is to be consistently monitoring and testing your AI systems, to stay one step ahead of threat actors and fix vulnerabilities before they cause bigger problems.
Step 4: Act & adapt
The cybersecurity landscape is constantly changing, and technology is rapidly advancing. To ensure your threat model keeps pace with these changes, regularly review them after incidents and look to incorporate new threat intelligence relevant to your industry.
In every cyber attack or breach, there’s a lesson to be learned. Even if the breach has happened to another organization, you can test and strengthen your own systems by mimicking the type of attack in a controlled and safe environment. Staying agile as a business means adapting to the new landscape, by understanding new and emerging threats and building resilience into product development in a timely way.
Conclusion
AI’s potential for both innovation and disruption is undeniable. As it becomes more intrinsic to the ways businesses operate, organisations need to ensure that these models are trialled and tested to build better resilience against emerging threats. Threat modelling isn’t just about mitigating risk, however, it’s a core element of responsible AI leadership that should be adopted by every organisation using this technology. CISOs who embed principles of threat modelling into AI development position themselves and their organizations at the forefront of trustworthy AI adoption, gaining a competitive advantage in an increasingly data-driven world.
We’ve featured the best encryption software.
This article was produced as part of TechRadarPro’s Expert Insights channel where we feature the best and brightest minds in the technology industry today. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro
AI is fast becoming a cornerstone of how smart businesses operate. According to recent research by IBM, more than one in three (37%) UK businesses are actively using AI in their operations, and 41% are experimenting with this technology. From identifying financial fraud, to predicting environment changes in supply chains,…
