PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements

Abstract

Contract review is a complex and time-intensive task that typically demands specialized legal expertise, rendering it largely inaccessible to non-experts. Moreover, legal interpretation is rarely straightforward—ambiguity is pervasive, and judgments often hinge on subjective assessments. Compounding these challenges, contracts are usually confidential, restricting their use with proprietary models and necessitating reliance on open-source alternatives. To address these challenges, this thesis expands upon PAKTON: a fully open-source, end-to-end, multi-agent framework with plug-and-play capabilities, designed to handle the complexities of contract analysis through collaborative agent workflows and a novel retrieval-augmented generation (RAG) component, enabling automated legal document review that is more accessible and privacy-preserving. The proposed system is composed of three agents: (1) the Archivist, which interacts with the user and manages structured document input; (2) the Researcher, which retrieves relevant internal and external information using hybrid and graph-aware retrieval techniques; and (3) the Interrogator, which engages in multi-step reasoning to iteratively refine the final report. Each agent is tailored to its specific role, and through collaboration, they collectively produce a comprehensive outcome for the end user. PAKTON’s architecture departs from black-box LLM paradigms by explicitly exposing its reasoning process and highlighting the evidence spans—both within the user-provided contract and from external legal knowledge—that inform its conclusions. It leverages advanced retrieval techniques, including hybrid dense–sparse retrieval, graph-aware reranking, and context-sensitive chunking, to surface the most relevant evidence in response to user queries. The Interrogator agent operates through a multi-step refinement loop, systematically identifying reasoning gaps and enhancing the response with each iteration, resulting in a comprehensive report that reflects multiple perspectives. Evaluations across multiple benchmarks, including ContractNLI and LegalBench-RAG, demonstrate that PAKTON consistently outperforms both general-purpose and legal domain–fine-tuned models in quantitative metrics (e.g., accuracy, recall, precision) and qualitative dimensions (e.g., explainability, reasoning, completeness). Human and LLM-based evaluations further confirm PAKTON’s superiority in critical legal AI aspects such as transparency, ambiguity resolution, and evidentiary grounding, factors essential for supporting end-user decision-making. The framework is model-agnostic and suitable for deployment under privacy-preserving conditions, enhancing its practicality for real-world legal applications. With its modular architecture and rigorous benchmarking, PAKTON represents a significant step forward in legal AI, illustrating how multi-agent coordination and strategic retrieval can substantially elevate reasoning in high-stakes domains.