garak

garak (Generative AI Red-teaming and Assessment Kit) is an open-source large-language-model vulnerability scanner, described as 'the LLM vulnerability scanner'. The project was started by Leon Derczynski and is now maintained by NVIDIA under the Apache 2.0 license. Written in Python, it is used to automatically and systematically test the security of AI models and applications.

garak probes a model with a set of probes generating various attacks and weaknesses - including jailbreaks, prompt injection, training-data leakage, toxic content generation, hallucination, susceptibility to encoded instructions and more. The model's responses are scored by detectors, and the output is a report of vulnerabilities and robustness metrics. The tool supports many backends (e.g. Hugging Face, OpenAI, local models).

garak is used for AI red teaming, security assessment of models before deployment and tracking progress in attack robustness. It has about 8.7k GitHub stars and is one of the more popular open-source LLM security tools. It is available as a CLI tool and a Python library (pip).