BullyRAG:A Multi-Perspective RAG Robustness Evaluation Framework
DAY 1
11:00-
11:40
Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by integrating retrieval mechanisms with their reasoning capabilities, enabling responses beyond their training data. However, the robustness of RAG systems remains an open question: Is our RAG system robust enough to avoid giving harmful or useless responses under various attacks?
This work explores the extensive attack surfaces of RAG systems, focusing on how attackers can manipulate either the retrieval phase or the LLM generation phase. For instance, attackers might obfuscate knowledge to mislead the retriever, causing LLMs to generate incorrect answers, or exploit LLMs’ preferences to reference poisoned information. Beyond incorrect answers, we demonstrate how attackers can deliver malicious instructions, such as disguising phishing links as reference links. In scenarios involving function calling, these techniques could potentially lead to remote code execution (RCE).
To address these threats, we introduce “BullyRAG,” the first open-source comprehensive framework for assessing RAG robustness. BullyRAG targets three main attack objectives: provide misinformation, lure into executing malicious instructions, and RCE. It includes over 10 attack techniques (e.g., invisible control character obfuscation and preference specialization), supports two RAG usage scenarios (question answering and function calling), and integrates with three inference engines (Hugging Face, Llama Cpp, and OpenAPI).
For an accurate evaluation, we also present a novel, auto-updating dataset sourced from ArXiv and news articles, ensuring it remains current and relevant while being excluded from any language model’s training data.
At the end, We will use BullyRAG to showcase the evaluation results of many powerful LLMs, aiming to provide an additional measurement perspective beyond accuracy when selecting models.
In conclusion, our research addresses critical aspects of RAG systems by uncovering vulnerabilities, providing a flexible evaluation framework, and offering an up-to-date dataset for comprehensive evaluation purposes, thereby enhancing the robustness of RAG systems.
-
Location :
-
Track 3(Room 2)
-
-
Category :
-
Bluebox
-
-
Share :
Speakers
-
Sian-Yao Eric Huang
シアンヤオ・エリック・ホアン
Sian-Yao Huang is a data scientist technical lead at CyCraft Technology, where he is primarily responsible for adopting and creating sophisticated deep learning models to solve challenging cybersecurity issues like large-scale multifactorial anomaly detection, automatic AD security analysis, and massive user behavior retrieval. Huang is passionate about investigating opportunities to use top-notch ML approaches in the field of cybersecurity. His work has been published on IJCNN and CVPR, two of the world's leading machine learning conferences. In addition, he has also given technical presentations at cybersecurity conferences such as Black Hat USA, SINCON and SECCON.
-
Cheng-Lin Yang
チェンリン・ヤン
Dr. Cheng-Lin Yang, currently a data science director at CyCraft Technology, where he is responsible for organizing and leading the machine learning team. He received his PhD in Artificial Intelligence from the University of Edinburgh and his research focuses on constructing efficient and effective machine learning workflows and utilizing machine learning techniques to automate detection and response along each phase of the cyberattack kill chain. He was a speaker at Black Hat USA 2023, Troopers, FIRST CTI, SINCON, CYBERSEC, SECCON, PyCon Taiwan, and AWS Summit Taiwan.
-
Yen-Shan Chen
エンシャン・チェン
Yen-Shan (Lily) Chen serves as a data scientist intern at CyCraft Technology, specializing in leveraging advanced machine learning techniques. Her focus encompasses exploring diverse model architectures for contrastive learning of sentence embeddings and implementing in-context learning methods to enhance the generalization of NLP tasks. Alongside her internship, Chen pursues a double major in computer science and economics at National Taiwan University, where she actively contributes to the departmental student council's initiatives, engaging in web design projects, organizing academic workshops, and coordinating competitions. She also presented her poster in the recent Students' Information Technology Conference in Taiwan, where she shared her findings about potential biases in LLMs.