Security and Privacy Risks in AI Systems | CSA AI Summit Q1 2025

Summary

In this talk, the speaker explores the security and privacy risks associated with modern AI models, particularly large language models (LLMs). They highlight vulnerabilities like backdoor attacks, prompt injection, and training data extraction. Additionally, they discuss how AI systems can inadvertently memorize sensitive data, exacerbating privacy concerns. The speaker also emphasizes the need for further research to secure the AI lifecycle, understand trade-offs between robustness, fairness, and explainability, and improve techniques like differential privacy to handle larger models. The presentation concludes by calling for stronger mitigation strategies and acknowledging funding agencies supporting this research.

Takeaways

• 😀 AI has made significant advancements in various fields, such as language technologies, healthcare, and cybersecurity, but it also comes with inherent security and privacy risks.
• 😀 There are two primary categories of AI risks: inherent risks (unintentional errors, biases, etc.) and adversarial risks (deliberate attacks aimed at disrupting AI systems).
• 😀 Adversarial machine learning (AML) studies attacks on AI systems, and research in this area has grown exponentially, with more than 11,000 papers published from 2021 to 2024.
• 😀 The lack of information-theoretic security guarantees in AI makes it challenging to create systems with proven security, unlike cryptography.
• 😀 Trustworthiness in AI involves seven key attributes, including safety, security, privacy, fairness, and transparency, but trade-offs exist between these attributes.
• 😀 Attacks on machine learning models can occur at different stages of learning, from training to deployment, with various attacker goals like availability breakdown, integrity violations, and privacy compromise.
• 😀 Evasion attacks target AI systems by causing incorrect predictions during testing with small perturbations to input data, leading to adversarial examples.
• 😀 Poisoning attacks involve manipulating training data or model weights to make AI systems produce incorrect predictions while maintaining overall accuracy.
• 😀 Privacy attacks, such as membership inference and training data extraction, pose significant threats, especially in models that memorize sensitive information like personally identifiable data.
• 😀 Generative AI faces unique security challenges, including prompt injection attacks, jailbreaking, and backdoor poisoning during fine-tuning, with a larger attack surface compared to predictive AI.
• 😀 There are ongoing open problems in AI security, such as understanding vulnerabilities in newer AI models, securing the AI lifecycle, and scaling privacy mitigations like differential privacy.

Q & A

What is the primary focus of the speaker's research in AI security and privacy?

The speaker's research focuses on identifying and mitigating security vulnerabilities and privacy risks in AI systems, especially large language models (LLMs), generative AI, and multimodal models. They also explore privacy attacks like training data extraction and backdoor vulnerabilities in these systems.

What is a backdoor attack, and how does it affect AI models?

A backdoor attack occurs when a malicious actor injects harmful inputs or triggers into a model during training, allowing them to exploit specific weaknesses in the model's behavior. The speaker notes that backdoor attacks are persistent even after applying safety alignment techniques like fine-tuning or adversarial training.

How do AI models such as LLMs suffer from privacy risks?

AI models, particularly LLMs, can inadvertently memorize sensitive information from their training data, such as personally identifiable information (PII), emails, phone numbers, and even code snippets. This memorization makes the models vulnerable to privacy attacks like training data extraction.

What was the significant finding of the 2021 collaboration on training data extraction attacks on GPT-2?

In the 2021 study, the team was able to extract about 600 memorized samples from GPT-2, including personal information like phone numbers and emails, as well as copyright notices and code snippets. The attack highlighted the increasing risk of data extraction as models grow larger.

What relationship between model size and memorization did the speaker mention?

The speaker noted that memorization increases with model size, meaning larger models tend to retain more sensitive information from their training data, making them more susceptible to privacy risks.

What are the main categories of attacks discussed in the transcript?

The speaker mentions several categories of attacks, including backdoor attacks, privacy attacks such as training data extraction, and other vulnerabilities affecting predictive and generative AI systems. They emphasize the need to secure the entire AI life cycle to prevent such attacks.

What is the role of differential privacy in mitigating AI privacy risks?

Differential privacy is a technique used to protect individual data in training datasets by adding noise to the data. While it is important for mitigating privacy risks, the speaker points out that more research is needed to scale it effectively for larger AI models.

Why are current AI security mitigations considered insufficient?

Current security mitigations are often heuristic-based, which means they rely on rules or assumptions that can be bypassed by motivated attackers. The speaker advocates for stronger, more robust guarantees to secure AI systems throughout their life cycle.

What open problems remain in AI security and privacy, according to the speaker?

The speaker identifies several open problems, including understanding the security vulnerabilities of newer AI models, better balancing trade-offs between robustness, fairness, accuracy, and explainability, and developing scalable privacy-preserving techniques for larger models.

How does the speaker suggest AI life cycle security could be improved?

The speaker suggests that to improve AI life cycle security, researchers need to focus on developing stronger security measures that provide better guarantees across all stages of the AI development and deployment process, beyond the current heuristic-based mitigations.