Time Table

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by integrating retrieval mechanisms with their reasoning capabilities, enabling responses beyond their training data. However, the robustness of RAG systems remains an open question: Is our RAG system robust enough to avoid giving harmful or useless responses under various attacks?

This work explores the extensive attack surfaces of RAG systems, focusing on how attackers can manipulate either the retrieval phase or the LLM generation phase. For instance, attackers might obfuscate knowledge to mislead the retriever, causing LLMs to generate incorrect answers, or exploit LLMs’ preferences to reference poisoned information. Beyond incorrect answers, we demonstrate how attackers can deliver malicious instructions, such as disguising phishing links as reference links. In scenarios involving function calling, these techniques could potentially lead to remote code execution (RCE).

To address these threats, we introduce “BullyRAG,” the first open-source comprehensive framework for assessing RAG robustness. BullyRAG targets three main attack objectives: provide misinformation, lure into executing malicious instructions, and RCE. It includes over 10 attack techniques (e.g., invisible control character obfuscation and preference specialization), supports two RAG usage scenarios (question answering and function calling), and integrates with three inference engines (Hugging Face, Llama Cpp, and OpenAPI).

For an accurate evaluation, we also present a novel, auto-updating dataset sourced from ArXiv and news articles, ensuring it remains current and relevant while being excluded from any language model’s training data.

At the end, We will use BullyRAG to showcase the evaluation results of many powerful LLMs, aiming to provide an additional measurement perspective beyond accuracy when selecting models.

In conclusion, our research addresses critical aspects of RAG systems by uncovering vulnerabilities, providing a flexible evaluation framework, and offering an up-to-date dataset for comprehensive evaluation purposes, thereby enhancing the robustness of RAG systems.