SA-RAG: Structuring Noisy Conversations for Retrieval-Augmented Generation

Large Language Models (LLMs) have been widely adopted in conversational applications. However, their reliance on parametric knowledge limits reliability in real-world scenarios that require dynamic or domain-specific information. Retrieval-Augmented Generation (RAG) addresses this limitation by incorporating external knowledge during generation, but existing RAG methods often struggle with noisy or irrelevant contexts, as they typically assume that high-quality structures are already available. In this work, we propose Structure-aware Retrieval-Augmented Generation (SA-RAG), a principled structure induction and generation framework that constructs reliable intermediate structures directly from noisy conversational data. SA-RAG uses tables as compact and controllable structured representations. We introduce a quality-aware structure induction framework that jointly models semantic and structural consistency. Furthermore, we explore both training-free and training-based structure generation strategies to further improve structure quality while maintaining fidelity. Experiments on two noisy real-world datasets show that SA-RAG significantly outperforms existing RAG baselines. Further analysis highlights that high-quality structural representations are crucial for robust RAG applications.