Skip to main content

Is Enterprise RAG Worth It? From Choosing Use Cases to Ongoing Governance

··5395 words·26 mins
Enterprise documents and data pass through retrieval to produce a verifiable answer, while access controls and governance mechanisms restrict information access.

Imagine an employee based in Sydney who is preparing to attend an industry conference in Melbourne. Their company’s travel policy sets the standard accommodation limit for Melbourne at AUD 250 per night. If a large conference or major event causes hotel prices to rise significantly, the limit can be raised to AUD 320 with prior approval from the employee’s line manager.

The rule may seem straightforward, but applying it involves several details: Does this event qualify for the exception? Must the employee obtain approval before booking? Is an approval email acceptable evidence for reimbursement? Relevant guidance might be scattered across internal websites, travel policies, expense pages, and earlier notices. If the employee finds an outdated version or misses a condition, the excess cost might not be reimbursed.

What the employee needs is not another list of search results, but a direct answer to a specific question:

I’m going to this conference in Melbourne next week. Can I claim AUD 295 per night for a hotel? If so, how do I obtain prior approval, and what records do I need to keep?

An AI-powered internal knowledge assistant designed for policy questions could find the current policy, explain the applicable conditions based on the employee’s travel dates and approval status, and cite the source material. In this article, Retrieval-Augmented Generation (RAG) primarily supports this kind of enterprise knowledge question answering: the system retrieves relevant material from an enterprise knowledge base, and a large language model uses that evidence to draft a response.

But technical feasibility alone does not mean an organization should build a RAG system. It must also assess how often such questions arise, whether existing search tools, human support, or established software products already meet the need, and whether the benefits can cover development and maintenance costs. If the organization adopts RAG, it must also address document versioning, access controls, and accountability for incorrect answers.

Organizations therefore face two linked questions: Which use cases justify RAG? Once a use case has been selected, how should the system be built and governed over time?

1. How RAG Gives Large Language Models Access to Enterprise Knowledge #

RAG retrieves relevant content from dispersed enterprise sources and provides it to a large language model to generate an answer.

Assessing whether such an internal assistant can answer reliably starts with understanding how large language models access knowledge and what RAG changes in that process.

Large language models (LLMs) learn patterns from vast amounts of text and predict what is most likely to appear next in a given context. Introduced in 2017, the Transformer architecture uses self-attention to model relationships across a sequence and has become a key foundation for modern large language models[1].

This training method gives models strong language understanding and generation capabilities but cannot guarantee factual accuracy. Models absorb some knowledge during training, but that knowledge is distributed across their parameters. Organizations cannot easily trace a particular answer to a specific source, nor can they immediately update the model by changing a policy document.

For general knowledge that repeatedly appears in public corpora, this limitation is not always obvious. Large language models can usually understand, summarize, and express this knowledge well. However, this does not mean that a model can search for current information on its own. An application can retrieve external information only when it is connected to search engines, databases, or other tools.

Enterprise-specific knowledge, however, is different. Such material is usually absent from public training corpora and is highly time-sensitive and context-dependent. The same business rule might vary by region, role, customer type, or effective date. Even if the model understands the user’s question, it may not know which material applies.

The basic idea behind Retrieval-Augmented Generation (RAG) is to find relevant material in an external knowledge base before generating an answer. Lewis et al. proposed the classic RAG architecture in 2020, enabling the model to use both knowledge encoded in its parameters and external retrieval results for knowledge-intensive tasks[2].

A typical enterprise RAG system comprises five stages:

  1. Parse documents and chunk text, tables, and other content.
  2. Build an index and record metadata such as version, source, and permissions.
  3. Retrieve candidate passages based on the user’s query.
  4. Filter and re-rank evidence.
  5. Generate an answer grounded in the evidence and include citations.

This pipeline helps find and organize citable knowledge for the model, but it cannot by itself determine how a policy applies to an individual’s circumstances. The accommodation question also involves employee identity, travel dates, the type of event, and approval status—information typically stored in identity, travel, expense, and approval systems.

With appropriate permissions, an internal assistant can query these systems through controlled interfaces and combine the results with policy evidence retrieved through RAG. RAG provides policy evidence, business systems provide the current state, and a rules engine applies explicit conditions. Approvals and exception decisions that require organizational authority remain with formal processes or authorized personnel. Connecting documents to a vector database does not automatically provide these capabilities.

Research on Dense Passage Retrieval (DPR) shows that systems can perform semantic retrieval by learning dense representations of questions and passages[3]. Such methods can identify content that uses different words but has a similar meaning. For example, an employee might ask, “After my parental leave ends, can I apply to work four days a week?” even though the company policy uses the term “flexible work arrangements.” The retrieval system needs to recognize the relationship between the two expressions, but whether the employee meets the eligibility criteria still depends on their circumstances, role requirements, and formal approval.

Organizations can adjust the material available to the system by updating the knowledge base, without retraining the model after every policy change. The system can also cite or link to original sources so users can verify the basis of an answer.

RAG changes how the model accesses knowledge; it does not resolve the underlying problems in an organization’s knowledge base. The organization must still address version conflicts, permission boundaries, and responsibility for interpreting rules.

2. The Core Challenge of Enterprise RAG: Knowledge Availability and Access Control #

RAG connects dispersed enterprise knowledge while authorization checks restrict what each user can retrieve and view.

Enterprise knowledge is rarely a neatly organized repository ready for a model to read. Policies, project records, and operational know-how may be scattered across document platforms, emails, and ticketing systems, or held only in the experience of individual employees. Different departments use different terminology, outdated versions may not be retired promptly, and ownership or scope may be unclear.

The more dispersed the knowledge, the stronger the incentive to bring it together with RAG. But enterprise knowledge cannot be open to everyone like public web pages. What an employee can view depends on their identity, role, department, region, and specific business context. Product descriptions in a sales report might be open to most employees, but access to customer lists, contract prices, and next quarter’s revenue forecasts should be restricted to specific roles.

Once an enterprise connects its document and data sources to a RAG system, information flows along a new path:

User Identity & Query → Authorization Check → Enterprise Knowledge Base → Retrieval & Re-ranking → Model Service → Answer, Citations & Logs

Improper handling at any stage can expose information more broadly than intended. The NIST Privacy Framework emphasizes that organizations need to identify and manage privacy risks throughout the entire data processing lifecycle[4]. For RAG, organizations must protect not only original documents but also retrieval results, model context, generated answers, caches, and usage logs.

Enterprise RAG is therefore not simply about giving more documents to a model. It creates a dynamic link between making knowledge available and enforcing access controls. The system can make more knowledge discoverable, but it must not expand a user’s existing permissions. To achieve this, organizations must address three boundaries:

BoundaryQuestion to Answer
Knowledge BoundaryWhich materials are still valid, and which can serve as the basis for an answer
Access BoundaryWhich materials the current user is authorized to retrieve and view
Accountability BoundaryWho confirms knowledge, interprets rules, and corrects errors

These three boundaries determine whether an answer can be relied upon in real work. Finding relevant text is only the starting point; the organization must also confirm that the material applies, that the user is authorized to view it, and that someone is responsible for handling errors. Whether building the system is justified depends on the scale of demand, total cost, and expected benefits.

3. Limitations of Common RAG Solutions #

Document parsing, retrieval, and generation can each introduce errors, producing fluent answers that rest on unreliable evidence.

Generating a final answer in a RAG system involves at least document processing, retrieval, filtering, and generation. Each stage can introduce errors, and different failures may produce the same result: a fluent but incorrect answer.

RAG Inherits the Knowledge Base’s Problems #

Enterprise documents are not always well-formatted plain text. Critical information may be embedded in PDF tables, scanned documents, flowcharts, attachments, footnotes, or passages that span multiple pages. Document parsers might separate table headers from values or ignore text within images. If chunks are too short, conditions and conclusions might fall into different passages; if they are too long, they might contain too much irrelevant information.

The travel expense problem at the beginning illustrates the potential impact of document processing. If the system places “the standard accommodation limit is AUD 250 per night” and “the limit can be increased to AUD 320 with prior approval” in different chunks, the model might see only the standard rate or only the exception amount.

Even if the system finds the complete policy clause, it still needs to combine that clause with travel dates, the type of event, and approval records to determine whether the exception applies. Structure-aware chunking, table parsing, and optical character recognition can reduce information loss, but cannot make business decisions on behalf of the organization.

Organizations also need to control knowledge sources, ingestion and update permissions, and relationships between current and superseded versions. They must clarify who has the authority to approve and interpret the material. RAG can retrieve knowledge, but it cannot independently determine which sources are authoritative. Connecting a disorganized knowledge base to a large language model does not make it orderly; the model may instead combine conflicting material into a seemingly coherent answer.

Similar Content Is Not Necessarily Applicable Evidence #

User queries may be vague, colloquial, or missing crucial details. “Can I still claim this?” does not specify the expense type, region, date, or employee identity. If the system relies only on semantic similarity, it may retrieve policies that sound relevant but do not apply.

Retrieval in enterprise settings must consider keywords and vector similarity alongside source authority, validity period, applicable region, and user permissions. When necessary details are missing, the system should ask clarifying questions rather than infer them on its own.

Giving more documents to the model cannot make this judgment automatically. Research by Liu et al. found that the position of relevant information in a long context can affect model performance; when key information is located in the middle of the input, some models use it less effectively, a phenomenon often referred to as “Lost in the Middle”[5]. This finding shows that expanding the context window alone does not guarantee that a model will use all the information within it reliably.

Some newer models have become markedly better at using long contexts[6]. For a small knowledge base that changes infrequently and has simple access controls, an organization can test supplying all relevant material directly to the model and use prompt caching to reduce repeated processing costs, rather than immediately building a complete retrieval pipeline[7]. Whether this approach is appropriate still depends on model capabilities, the size of the knowledge base, the update process, latency, cost, and access requirements; no fixed token count should be treated as a universal cutoff.

When the knowledge base grows, access rights vary by user, or the system must reliably select currently valid evidence from a large body of material, the organization still needs retrieval, filtering, and evidence organization. Long context and RAG are therefore not simple substitutes; organizations choose between them—or combine them—depending on the scale of the knowledge base, cost, and governance requirements.

Fluent Answers Do Not Make a Reliable System #

The model might incorrectly synthesize multiple sources, and the presence of citations does not mean they genuinely support the conclusion. Checking only the final answer still does not reveal whether the error originated in retrieval, evidence filtering, or generation.

Frameworks like RAGAS attempt to measure context relevance, answer relevance, and faithfulness separately[8]. Such automated evaluations can help teams compare system versions but cannot replace the judgment of business experts.

Organizations still need evaluation sets built from actual business questions, including policy conflicts, permission differences, ambiguous queries, changes over time, and missing material. They should assess retrieval, evidence, answers, citations, and refusal behavior separately. High-risk questions also require human review or clear escalation channels.

These risks show that organizations cannot optimize only the model and retrieval components; they must also govern the knowledge base, permissions, outputs, and operations. Even if the technical risks can be controlled, the organization must still decide whether the use case justifies RAG’s long-term costs.

Some Knowledge Problems Do Not Require RAG #

Not all enterprise knowledge problems require generative answers. If the number of documents is limited, their structure is clear, and updates are infrequent, improving directories, keyword search, and page navigation might be sufficient. If questions occur infrequently but rely heavily on expert judgment and particular circumstances, human support might be more economical and reliable.

The question is not “Can RAG replace an employee?” but how the total cost of ownership compares across solutions that meet the same service-quality and risk requirements. If existing search tools, workflows, human support, or established software products can solve the problem at a lower cost, the organization should choose the simpler option. Needing RAG capabilities and needing to build a RAG system in-house are two different questions.

4. How to Build Enterprise RAG: From Business Problems to Ongoing Governance #

An organization compares RAG with other options based on business value, knowledge quality, access risks, and cost.

Step One: Find Use Cases That Justify RAG #

The first step is not to choose a model or vector database, but to identify a recurring, costly problem for which the necessary domain knowledge can be maintained and verified. The organization can then compare RAG with traditional search, process improvement, human support, and established software products.

The travel query at the beginning illustrates a knowledge-retrieval problem, but one difficult query is not enough to justify a RAG project. The organization must measure how often similar questions arise, how much time employees spend finding and confirming the rules, and the amount of rework and the number of disputes caused by misinterpretation. It must then determine whether a travel platform, internal search, or human support could solve the problem at a lower cost.

For a small organization with few such inquiries, updating the travel page or designating an administrative contact might be more reasonable. For an organization with many geographically dispersed employees and frequently changing internal rules, this kind of knowledge service may be worth evaluating as a RAG use case.

Before a pilot, the organization should record a baseline for the existing process: time spent finding information, the number of repeated inquiries, the proportion escalated to experts, and rework or disputes caused by incorrect information. It should define in advance the conditions for continuing, changing, or ending the pilot, then measure not only answer accuracy but also adoption and end-to-end processing costs. A system will not deliver measurable value if it is rarely used or if employees spend more time verifying its answers than they previously spent searching.

Software development is a typical use case. Developers often know that documentation exists but not where a particular decision was recorded. Relevant knowledge might be scattered across code repositories, READMEs, API documentation, architectural decision records, wikis, tickets, and incident retrospectives. They might ask:

Why doesn’t the user account service use the company’s unified authentication component? Who made this design decision?

If the system can find the design records, tickets, and discussions from that time, RAG can organize the dispersed evidence. If the organization never recorded the decision, the system should report insufficient evidence rather than infer historical reasons from the current code. For precise searches of functions, call relationships, and dependencies, code search or static analysis is usually more appropriate than RAG.

The advantage of this use case is that the development team both uses and produces the relevant domain knowledge and can verify the evidence. The organization can measure the time required to find a verifiable answer, the number of repeated technical inquiries, onboarding time, and troubleshooting time.

Customer service and contact centers offer another use case beyond software development. Agents need to look up product descriptions, return and exchange policies, promotional terms, service outages, and escalation procedures every day. The information may be scattered across operating manuals, internal notices, and product knowledge bases, while customers may describe their problems differently from the terminology used in company policies.

RAG can help customer service agents find the relevant policy, its source, and the conditions under which it applies. Product, operations, and knowledge management teams maintain the material, while frontline employees identify errors and knowledge gaps through daily use.

Organizations can track answer accuracy, average handling time, first-contact resolution, escalation rates, and the volume of incorrect answers, so that shorter handling times do not come at the expense of service quality. The system can help employees find and interpret internal knowledge, but without formal authorization it should not approve refunds, modify accounts, or make other decisions that affect customers’ rights.

Although the two use cases come from different departments, they share similar selection criteria:

Selection CriterionSoftware DevelopmentCustomer Service or Contact Center
Frequent NeedRepeatedly looking up system and architectural knowledgeRepeatedly querying product and business rules
Dispersed KnowledgeCode repositories, wikis, tickets, design recordsManuals, announcements, product knowledge bases
Terminology MismatchUser terminology differs from code or project namesCustomer terminology differs from internal policy language
Clear OwnershipDevelopment team and system ownersProduct, operations, and knowledge management teams
Verifiable EvidenceCode, documentation, design recordsPolicies, product information, operational procedures
Measurable ValueLookup, onboarding, and troubleshooting time; number of repeated inquiriesAccuracy, handling time, first-contact resolution, and escalation rate

After confirming that a use case is suitable for RAG, the organization must compare buying an established software product with building an in-house system. If most knowledge already resides on one platform, that platform’s search and AI capabilities may be sufficient. Custom development is more likely to offer long-term value when the organization must connect multiple systems, synchronize complex permissions, build domain-specific evaluations, or embed knowledge queries in its own business workflows.

Here, “building in-house” usually does not mean training a large language model from scratch. It means integrating model services, data connectors, retrieval, access controls, evaluation, and operational processes into a complete system. The comparison must also include integration costs, vendor lock-in, data migration, and exit costs.

Bringing Top-Down and Bottom-Up Approaches Together #

Enterprise RAG can begin from two directions. Management, platform teams, and governance functions set strategic objectives, investment boundaries, shared platforms, and data, security, and audit requirements from the top down. Frontline employees and domain teams identify frequent problems, provide and verify domain knowledge, and document the costs and weaknesses of current practices from the bottom up. The two paths must meet in a clearly bounded use case with measurable value.

A purely top-down approach may produce goals that diverge from real work and a system that sees little sustained use. A purely bottom-up approach may produce a useful prototype but lack formal data-access authorization, a security review, a long-term budget, or clear responsibility for knowledge maintenance. Management should not manufacture demand for RAG through administrative mandates, and frontline teams should not treat a local prototype as a production system.

The two sides can then select a narrowly scoped pilot and decide whether to expand, adjust, or stop based on adoption, answer quality, total cost of ownership, and new risks—not simply because the organization has announced an AI strategy.

Step Two: Govern Knowledge First, Then Build the Index #

Each item indexed by the RAG system should carry metadata covering the source, its authority level, the owner, effective and expiration dates, applicable regions, business scope, access level, and relationship to superseded versions.

Knowledge owners are responsible for confirming whether materials are valid and how to interpret them; the technical team designs how the system parses, indexes, and retrieves materials. Neither can replace the other. Without this foundational information, even if the system finds semantically relevant text, it cannot reliably determine whether that text is a valid basis for the current query.

Organizations also need to control which sources and users can add content to the knowledge base, identify unverified material, prevent instructions embedded in documents from manipulating retrieval or generation, and protect sensitive information throughout the downstream pipeline[9].

In a software development use case, the system owner needs to confirm when architectural decision records are updated or become invalid. In a customer service use case, the product or operations team needs to clarify who published promotion rules and when those rules should be retired after a campaign ends.

Step Three: Enforce Access Controls During Retrieval #

The system cannot retrieve all material first and remove sensitive content only when generating the answer. Access controls must take effect before or during retrieval and remain enforced throughout context assembly, caching, citation generation, and logging. The system must find material that both applies to the question and falls within the current user’s permissions.

This requires the system to consider user identity, role, department, region, and business context, as well as the access levels of documents and text chunks. When permissions on the original file change, the index must be updated promptly.

Developers may not be authorized to retrieve security architecture documents, production incident records, or customer data. Customer service employees may query product rules, but that should not give them access to customer contract prices or internal risk flags.

Step Four: Constrain Generation with Evidence and Refusal Rules #

The system can use hybrid retrieval and re-ranking, then filter candidate materials based on metadata. When a query lacks necessary conditions, the system should ask the user for clarification; when materials conflict, it should flag the conflict; when evidence is insufficient, it should refuse to answer.

Questions involving formal approvals, policy exceptions, or individual rights should be referred to a clearly identified human decision-maker. If an organization uses RAG to answer the accommodation question, the system can explain when the AUD 320 limit applies and, if permitted, query the relevant business systems to check for an approval record. It cannot decide that an exception applies without approval, let alone approve the request on the line manager’s behalf. The system should identify the missing conditions and direct the employee to the appropriate approval process.

Evaluation should cover retrieval, evidence, answers, citations, and refusal behavior separately rather than assigning only an overall score to the final answer. This allows the team to locate errors and determine which metrics improved after a change to the model, index, or prompt—and whether other capabilities deteriorated.

When the system cannot find a design decision, it should state that it could not find a relevant record. When a policy does not cover a customer’s situation, the system should refer the case to a supervisor.

Step Five: Treat RAG as an Ongoing Organizational System #

Scale Deliberately After the Pilot #

Even after a successful pilot, the organization should not immediately expand to every knowledge domain. Each expansion requires a fresh review of knowledge quality, permission complexity, the consequences of errors, and maintenance responsibilities; success in one use case does not prove that another is suitable for RAG. An interview study by Jöhnk et al. also indicates that organizational readiness for AI depends not only on technical resources but also on strategy, data, knowledge, culture, and organizational commitment[10].

Involve Employees in System Development #

Project teams can work with the employees who perform these tasks to build question sets and record the time needed to find information, error-prone steps, and content that must be checked. After deployment, the team should examine whether employees find the correct sources faster, whether repeated inquiries decline, which questions are often escalated, and whether the workflow actually becomes simpler. Employees will quickly stop using the system if verifying its answers takes longer than their previous search process.

When employees do not use the system, the reason is not necessarily resistance to new technology. Frequent errors, concern that queries may be monitored, or extra steps that take more time than the system saves can all discourage use. Raisch and Krakowski argue that organizations must continually balance automation and augmentation: AI can replace some tasks while reshaping how people perform others[11]. The organization needs to explain what the system can handle, what it records, which answers must be checked, and how employees can report errors.

Clarify Who Can Interpret Rules and Who Must Correct Errors #

Once RAG begins interpreting policies, authoritative sources, retrieval ranking, prompts, and refusal rules will all influence the answers employees see. Research on algorithmic management indicates that algorithmic systems change information, evaluation, and control relationships within organizations[12].

If an organization leaves RAG entirely to the technical team, technical configuration can end up encoding policy judgments that belong to business, HR, legal, or management. When responsibilities, authority, and resources are not aligned, existing organizational ambiguity becomes embedded in retrieval ranking, prompts, and permission settings.

Project sponsors can help a pilot secure initial resources and coordinate across departments, but personal support cannot replace long-term authorization. After the pilot, the organization must turn temporary consensus into formal mechanisms: who approves content for the system, who interprets conflicting rules, who decides whether high-risk questions may be answered automatically, and who handles errors and exceptions. Sponsors can get collaboration started, but formal records, assigned responsibilities, and handover mechanisms keep it going. I discuss this transition in “Between Relationships and Rules: How to Get Things Done in Imperfect Organizations.”

A clearer division of responsibilities might include:

RolePrimary Responsibility
Business and Product OwnersDefine use cases, value objectives, and risk boundaries
Knowledge OwnersConfirm material authority, versions, scope of application, and interpretation rules
Technical TeamResponsible for retrieval, generation, evaluation, permission synchronization, and monitoring
Privacy, Risk and Compliance TeamReview data flows, high-risk uses, and accountability mechanisms
Frontline Employees and SpecialistsParticipate in question set development, pilot feedback, and error detection

Make Evaluation and Maintenance Ongoing Work #

The NIST AI Risk Management Framework uses four functions—Govern, Map, Measure, and Manage—to organize AI risk management[13]. NIST’s subsequent Generative AI Profile adds recommendations specific to generative AI[14]. While these frameworks do not dictate organizational structure, they emphasize that governance, context mapping, risk measurement, and continuous management need to work together.

Policies, permissions, models, and vendors will continue to change. Machine Learning Operations (MLOps) research emphasizes the ongoing work of managing data, models, deployment, monitoring, and cross-team collaboration[15]. RAG differs from traditional machine learning systems, but it still requires version management, regression testing, quality monitoring, incident response, and migration plans. Organizations must also measure usage, maintenance, and knowledge-update costs to determine whether the system’s value can justify long-term investment.

When code, products, or promotion rules change, the relevant documents, permissions, and indexes must be updated together; otherwise, a system that once passed validation will quickly become obsolete.

5. Future Outlook: From Focused Knowledge Tools to Organizational Infrastructure #

RAG serves as a controlled knowledge-access capability within enterprise systems, while agents use tools and take actions under human oversight and authorization.

For an enterprise RAG system that has demonstrated value and put the necessary governance in place, the next step might go beyond answering questions. The system could compare policy versions, detect knowledge conflicts, remind owners to update documents, and, with explicit authorization, help execute business workflows.

RAG primarily addresses how a system obtains the knowledge required for the current task. In some agentic architectures, it might not appear as a standalone application, but as a controlled knowledge-access capability that a model can invoke when needed. The model can also query databases, call business APIs, or use other software tools, and organize the returned information as context for the current task.

The risks of these capabilities depend on what the system can access and change. In the use cases discussed here, RAG primarily provides read-only access to knowledge. When a system goes on to initiate workflows or change business state, the organization is building more than a question-answering system: it is building an agentic system that can select tools and take action[16].

However, the more proactive the system becomes, the greater the accountability requirements. When the system only provides information, users can still detect errors by checking the source material. If it starts initiating approval requests for accommodation exceptions, updating records, or performing other operations, the same errors can directly affect the business. Organizations need stricter authorization, auditing, and human approval mechanisms; they cannot assume a system has the authority to act simply because it can take action.

The sales report example in the second chapter also illustrates that access control might need to evolve from the document level to the content and scenario levels. If the system can only determine whether a user may open an entire document, organizations face an all-or-nothing choice. Finer-grained control can reduce this dilemma but will also increase the complexity of identity, policy, index synchronization, and auditing systems.

RAG’s long-term value might also extend from knowledge retrieval to knowledge governance. By recording which questions cannot be answered, which files frequently conflict, and which critical knowledge is held by only a few employees, the system can reveal the organization’s knowledge gaps. RAG not only helps employees find knowledge; it may also show the organization where knowledge is outdated or missing, who holds critical expertise, and which processes still lack clear accountability.

These capabilities still depend on reliable permission systems, knowledge metadata, and organizational governance. For organizations with limited usage or governance capacity, standardized RAG capabilities in established software products may be more appropriate than an in-house system.

6. Conclusion #

An organization combines retrieval, access controls, evidence, accountability, and ongoing maintenance into a governable RAG knowledge system.

Organizations should not decide to deploy RAG and then look for documents to connect. They should start with a specific problem and compare search, process improvement, human support, established software products, and an in-house system. Many knowledge problems do not require RAG; even when an organization needs RAG capabilities, it may not need to build a system for the entire enterprise.

An organization should proceed only when demand reaches sufficient scale, the relevant knowledge can be governed, access can be controlled, and the expected long-term benefits are likely to outweigh the total cost. For these use cases, connecting a vector database is only the starting point. What the organization ultimately builds is not a model that is “better at answering questions,” but a knowledge system with explicit governance, traceable accountability, and sustainable operations.

RAG governance asks what the system may know; agent governance must also determine what it may decide and do, and whether its actions can be stopped or reversed.

References #

[1] Vaswani, A., et al. Attention Is All You Need. Advances in Neural Information Processing Systems, 2017. https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

[2] Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 2020. https://papers.nips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html

[3] Karpukhin, V., et al. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP, 2020. https://aclanthology.org/2020.emnlp-main.550/

[4] Boeckl, K., and Lefkovitz, N. NIST Privacy Framework: A Tool for Improving Privacy Through Enterprise Risk Management, Version 1.0. NIST, 2020. https://doi.org/10.6028/NIST.CSWP.01162020

[5] Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 2024. https://aclanthology.org/2024.tacl-1.9/

[6] Anthropic. Claude Opus 4.6. 2026. https://www.anthropic.com/news/claude-opus-4-6

[7] Anthropic. Contextual Retrieval. 2024. https://www.anthropic.com/engineering/contextual-retrieval

[8] Es, S., et al. RAGAs: Automated Evaluation of Retrieval Augmented Generation. EACL System Demonstrations, 2024. https://aclanthology.org/2024.eacl-demo.16/

[9] OWASP. Top 10 for LLM Applications 2025. 2024. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/

[10] Jöhnk, J., Weißert, M., and Wyrtki, K. Ready or Not, AI Comes—An Interview Study of Organizational AI Readiness Factors. Business & Information Systems Engineering, 2021. https://aisel.aisnet.org/bise/vol63/iss1/2/

[11] Raisch, S., and Krakowski, S. Artificial Intelligence and Management: The Automation–Augmentation Paradox. Academy of Management Review, 2021. https://doi.org/10.5465/amr.2018.0072

[12] Kellogg, K. C., Valentine, M. A., and Christin, A. Algorithms at Work: The New Contested Terrain of Control. Academy of Management Annals, 2020. https://doi.org/10.5465/annals.2018.0174

[13] NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023. https://doi.org/10.6028/NIST.AI.100-1

[14] Autio, C., et al. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1, 2024. https://doi.org/10.6028/NIST.AI.600-1

[15] Kreuzberger, D., Kühl, N., and Hirschl, S. Machine Learning Operations (MLOps): Overview, Definition, and Architecture. IEEE Access, 2023. https://doi.org/10.1109/ACCESS.2023.3262138

[16] NIST. Lessons Learned from the Consortium: Tool Use in Agent Systems. 2025. https://www.nist.gov/news-events/news/2025/08/lessons-learned-consortium-tool-use-agent-systems