AI Agent Security — Attacks, Jailbreaking, and Defense · Red Teaming, Monitoring, and Secure Design of Agentic Systems
Red Teaming Automation: garak, PyRIT, PAIR — Tools Overview
Red Teaming, Monitoring, and Secure Design of Agentic Systems
Introduction
LLM red teaming automation enables systematic, scalable security testing without manually crafting every attacking prompt. This lesson covers three key tools: garak (open-source LLM vulnerability scanner), PyRIT (Python Risk Identification Toolkit from Microsoft), and PAIR (Prompt Automatic Iterative Refinement — algorithm for automated jailbreak search), as well as principles for selecting the right tool for a given scenario.