Is Enterprise RAG Worth Implementing: From Scenario Selection to Ongoing Governance

Table of Contents
Imagine an employee working in Sydney preparing to attend an industry conference in Melbourne. Their company’s travel policy stipulates that for travel to Melbourne, the standard accommodation limit is AUD 250 per night; when large conferences or major events lead to a significant increase in hotel prices, this limit can be raised to AUD 320 with prior approval from their direct manager.
The rules may seem straightforward, but their actual execution involves numerous details: Does this event qualify for the exception? Must the employee obtain approval before booking? Can email approval serve as a basis for reimbursement? Relevant instructions might be scattered across internal websites, travel policies, reimbursement pages, and historical announcements. If the employee finds an outdated version or misses specific conditions, excess expenses might not be reimbursed.
What an employee truly needs is not another search result, but to directly ask:
I’m going to this conference in Melbourne next week. Can a hotel costing AUD 295 per night be reimbursed? If so, how do I obtain prior approval, and what materials do I need to keep?
If the enterprise had an AI assistant that could find the currently effective policy, explain the applicable conditions based on travel dates and approval status, and display the original basis, this question might be resolved faster. Retrieval-Augmented Generation (RAG) is one of the technical solutions to achieve such capabilities: the system first retrieves relevant materials from an enterprise knowledge base, and then a large language model organizes the answer based on these materials.
But just because it’s technically feasible doesn’t mean an enterprise should build a RAG system. The enterprise also needs to assess the frequency of such issues, whether existing search, human support, or mature software can already solve them, and whether the benefits of the system can cover its development and maintenance costs. If RAG is chosen, the enterprise also needs to address issues such as knowledge versions, access permissions, and answer accountability.
Therefore, enterprises face two consecutive questions: which business scenarios are worth using RAG for; and once a scenario is determined, how should this system be built and continuously governed?
How RAG Enables Large Language Models to Use Enterprise Knowledge #

To determine if such an internal assistant can reliably solve problems, one first needs to understand how large language models acquire knowledge and which part RAG alters in this process.
Large Language Models (LLMs) predict what is most likely to appear next in the current context by learning language patterns from vast amounts of text. The Transformer architecture, introduced in 2017, processes relationships between words through self-attention mechanisms, becoming a crucial technological foundation for modern large language models[1].
This training method endows models with strong language understanding and generation capabilities but cannot guarantee factual accuracy. While models absorb some knowledge during training, this knowledge is encoded dispersedly within its parameters. It is difficult for enterprises to accurately determine which material a particular answer originated from, and they cannot instantly update the model by modifying a policy document.
For general knowledge that repeatedly appears in public corpora, this limitation is not always obvious. Large language models can usually understand, summarize, and express this knowledge well. However, this does not mean the model itself can search for the latest information; only after being connected to search engines, databases, or other tools does it gain external retrieval capabilities.
Enterprise-specific knowledge, however, is different. This content usually does not enter public training corpora and is highly time-sensitive and contextual. The same business rule might vary depending on region, role, customer type, or effective date. Even if the model understands the user’s question, it may not know which material should be used.
The basic idea behind Retrieval-Augmented Generation (RAG) is to find relevant materials from an external knowledge base before generating an answer. Lewis et al. proposed the classic RAG architecture in 2020, enabling the model to use both its existing knowledge (encoded in parameters) and external retrieval results when handling knowledge-intensive tasks[2].
A typical enterprise RAG system usually comprises five stages:
- Parse documents and segment text, tables, and other content.
- Create an index for the text, recording metadata such as version, source, and permissions.
- Retrieve candidate materials based on the user’s query.
- Filter and re-rank evidence.
- Have the LLM generate an answer with sources based on the evidence.
This process primarily addresses how to find and organize citable knowledge for the model but cannot independently perform all business judgments related to personal status. The accommodation issue mentioned earlier also involves employee identity, travel dates, conference nature, and approval status, information typically stored in identity, travel, expense, and approval systems.
An internal enterprise assistant, with appropriate permissions, can query these systems via controlled interfaces and combine the query results with the policy basis found by RAG. RAG provides policy evidence, business systems provide current status, and a rules engine executes clear conditional judgments; approvals or exception decisions requiring organizational authorization are completed through formal processes or by authorized personnel. Connecting documents to a vector database does not automatically grant these capabilities.
Research such as Dense Passage Retrieval (DPR) shows that systems can perform semantic retrieval by learning dense representations of questions and passages[3]. Such methods can identify content that uses different words but has similar meanings. For example, an employee might ask: “After my parental leave ends, can I apply to work four days a week?” even though the company policy uses “flexible work arrangements.” The retrieval system needs to identify the relationship between the two expressions, but whether an employee meets the application conditions still depends on individual circumstances, job requirements, and formal approval.
Enterprises can adjust the materials used by the system by updating the knowledge base, without needing to retrain the model after every policy change. The system can also display citation sources for users to verify the answer’s basis.
However, RAG changes the entry point for the model to acquire knowledge; it does not automatically fix enterprise knowledge itself. Version conflicts, permission boundaries, and interpretative responsibilities within the knowledge base still need to be addressed by the enterprise.
The Core Contradiction of Enterprise RAG: Knowledge Connection and Access Control #

Enterprise knowledge is often not a neatly organized repository ready for a model to read. Policies, project records, and operational experience might be scattered across document platforms, emails, ticketing systems, and employees’ personal expertise; different departments use different terminology, outdated versions might not be promptly retired, and accountability and scope of application may not be clear.
The more dispersed the knowledge, the more enterprises hope to connect it through RAG. But enterprise knowledge cannot be open to everyone like public web pages. What materials an employee can view depends on their identity, role, department, region, and specific business scenario. Product descriptions in a sales report might be open to most employees, but customer lists, contract prices, and next quarter’s revenue forecasts can only be provided to specific roles.
Once an enterprise integrates its data into RAG, information flows along a new path:
User Identity & Query → Permission Judgment → Enterprise Knowledge Base → Retrieval & Re-ranking → Model Service → Answer, Citations & Logs
Improper handling at any stage could expand the scope of information exposure. The NIST Privacy Framework emphasizes that organizations need to identify and manage privacy risks throughout the entire data processing lifecycle[4]. For RAG, enterprises must protect not only original documents but also retrieval results, model context, generated answers, caches, and usage logs.
Therefore, enterprise RAG is not simply about giving more documents to a model, but about establishing a dynamic connection between knowledge availability and access control. The system can expand the discoverability of knowledge, but it cannot expand the user’s existing scope of permissions. To achieve this, enterprises must simultaneously address three boundaries:
| Boundary | Question to Answer |
|---|---|
| Knowledge Boundary | Which materials are still valid, and which can serve as the basis for an answer |
| Access Boundary | Which materials the current user is authorized to retrieve and view |
| Accountability Boundary | Who confirms knowledge, interprets rules, and corrects errors |
These three boundaries collectively determine whether an answer can enter a real workflow. The model finding relevant text is merely a starting point; the enterprise also needs to confirm if the material is applicable, if the user has permission to view it, and who handles errors if they occur. Whether an enterprise has reason to build such a system depends on the scale of use, total cost, and expected benefits.
Limitations of Common RAG Solutions #

The final answer from a RAG system goes through at least document processing, retrieval, filtering, and generation stages. Each stage can introduce errors, and different errors might ultimately manifest as the same result: the system provides a fluent but incorrect answer.
RAG Inherits Existing Issues from the Knowledge Base #
Enterprise data is not always well-formatted plain text. Much critical information exists in PDF tables, scanned documents, flowcharts, attachments, footnotes, and cross-page paragraphs. Document parsing programs might separate table headers from values, or ignore text within images. If text segments are too short, conditions and conclusions might fall into different snippets; if too long, they might include a lot of irrelevant information.
The travel expense problem at the beginning illustrates the potential impact of document processing. If the system segments “standard accommodation limit is AUD 250 per night” and “can be increased to AUD 320 per night with prior approval” into different snippets, the model might only see the standard rate, or only the exception amount.
Even if the system finds the complete terms, it still needs to combine them with travel dates, activity type, and approval records to determine if the exception applies. Structured segmentation, table parsing, and scanned document recognition can reduce information loss, but cannot make business judgments on behalf of the enterprise.
Enterprises also need to control knowledge sources, write permissions, and version replacement relationships, and clarify who has the authority to approve and interpret these materials. RAG can retrieve knowledge, but it cannot independently determine knowledge authority within an organization; disorder in the knowledge base will not disappear by connecting to a large language model. Instead, the model might organize conflicting materials into a seemingly coherent answer.
Retrieving Similar Content Does Not Mean Finding Applicable Evidence #
User queries might be vague, colloquial, or even lack crucial conditions. “Can I still claim this?” doesn’t specify expense type, region, date, or employee identity. If the system only retrieves based on semantic similarity, it might find regulations that are similar in content but not applicable.
Enterprise retrieval needs to simultaneously consider keyword and vector similarity, document authority level, validity period, applicable region, and user permissions. When conditions are insufficient, the system should ask clarifying questions, rather than attempting to complete business conditions on its own.
Giving more documents to the model cannot replace this judgment either. Research by Liu et al. found that the position of relevant information in a long context can affect model performance; when key information is located in the middle of the input, some models’ utilization effectiveness decreases, a phenomenon often referred to as “Lost in the Middle”[5]. More context also increases latency and inference costs.
Therefore, enterprise retrieval cannot merely judge which piece of text is most similar; it must also determine which material is applicable to the current user, time, and scenario. Long contexts expand the model’s capacity to process information, but cannot replace document filtering, re-ranking, and evidence organization.
Fluent Answers Do Not Equal Reliable Systems #
The model might incorrectly synthesize multiple materials, and the presence of citations does not mean they genuinely support the answer. Checking only the final answer still doesn’t reveal whether the error originated from retrieval, evidence filtering, or the generation process.
Frameworks like RAGAS attempt to measure context relevance, answer relevance, and faithfulness separately[6]. Such automated evaluations can help teams compare system versions but cannot replace the judgment of business experts.
Enterprises still need to establish a set of real-world questions covering policy conflicts, permission differences, ambiguous queries, temporal changes, and missing materials, and evaluate retrieval, evidence, answers, citations, and refusal to answer separately. High-risk questions also require manual review or clear escalation channels.
These failures indicate that enterprises cannot merely optimize the model and retrieval components; they must also govern knowledge, permissions, answers, and operational processes simultaneously. Even if an enterprise can control these technical risks, it still needs to determine: is this scenario worth incurring the long-term costs for RAG?
Some Knowledge Problems Do Not Require RAG #
Not all enterprise knowledge problems require generative answers. If the number of documents is limited, their structure is clear, and updates are infrequent, improving directories, keyword search, and page navigation might be sufficient. If questions occur infrequently but rely heavily on expert judgment and specific contexts, human support might be more economical and reliable.
What an enterprise needs to compare is not “Can RAG replace an employee?” but rather the total cost of ownership for different solutions under the same service quality and risk requirements. If existing search, workflows, employee support, or mature software can solve the problem at a lower cost, enterprises should opt for simpler solutions. Whether RAG capabilities are needed, and whether a RAG system needs to be built in-house, are two different questions.
Enterprise RAG Development Method: From Business Problems to Ongoing Governance #

Step One: Identify Business Cases Truly Worthy of RAG #
The first step for enterprise RAG is not choosing a model or a vector database, but identifying a problem that occurs frequently, has high existing processing costs, and involves professional knowledge that can be maintained and verified. Enterprises then compare RAG, traditional search, process improvement, human support, and mature software.
The travel expense query at the beginning indeed reflects a knowledge lookup problem, but a single difficult query is not enough to justify a RAG project. Enterprises also need to quantify the frequency of similar issues, the time employees spend searching for and confirming regulations, and rework or disputes caused by misinterpretations, and determine if travel platforms, internal search, or human support can solve them at a lower cost.
For small enterprises with limited inquiry volume, updating the travel page or designating an administrative contact might be more reasonable; for enterprises with a large number of geographically dispersed employees and frequently changing internal rules, such knowledge services might become a candidate scenario for RAG.
Before piloting, enterprises also need to record a baseline of existing processes, including the time employees spend finding information, the number of repetitive inquiries, the proportion of escalations to experts, and rework or disputes caused by incorrect information. Pilots should not only compare answer accuracy but also observe actual usage rates and end-to-end processing costs, and pre-define conditions for continuing, adjusting, or discontinuing investment. Even if one answer saves time, if the system is rarely used, or if employees need to spend more time verifying results, it’s difficult to generate actual returns.
Software development is a typical scenario. Developers often don’t fail to find any documentation; rather, they don’t know where a certain decision is recorded. Relevant knowledge might be scattered across code repositories, READMEs, API documentation, architectural decision records, Wikis, tickets, and incident retrospectives. They might ask:
Why doesn’t the user account service use the company’s unified authentication component? Who made this design decision?
If the system can find the design records, tickets, and discussions from that time, RAG can organize the dispersed evidence; if the organization never recorded this decision, the system should state insufficient evidence rather than inferring historical reasons from the current code. For precise searching of functions, call relationships, and dependencies, code search or static analysis is usually more appropriate than RAG.
The advantage of this scenario is that the development team is both the primary user of professional knowledge and the producer and verifier of that knowledge. Enterprises can measure the time required to find verifiable answers, the number of repetitive technical inquiries, the time for new hires to become familiar with systems, and fault diagnosis time.
Customer service or contact centers are another example outside of engineering teams. Customer service agents daily need to query product descriptions, return/exchange policies, promotion conditions, service outages, and issue escalation processes. Information might be scattered across operation manuals, internal announcements, and product knowledge bases, and customers’ natural language expressions might not align with the standard terminology used by the enterprise.
RAG can help customer service agents locate policies, displaying sources and applicable conditions. Product, operations, and knowledge management teams are responsible for maintaining materials, while frontline employees discover errors and knowledge gaps through daily use.
Enterprises can simultaneously observe answer accuracy, average handling time, first-call resolution rate, human escalation ratio, and the number of incorrect pieces of information, to avoid sacrificing service quality for shorter call times. The system can help employees find and interpret internal knowledge, but it should not automatically approve refunds, modify accounts, or make other decisions affecting customer rights without formal authorization.
Although the two scenarios come from different departments, they share similar judgment criteria:
| Judgment Criteria | Software Development | Customer Service or Contact Center |
|---|---|---|
| High-Frequency Need | Repeatedly looking up system and architectural knowledge | Repeatedly querying product and business rules |
| Dispersed Knowledge | Code repositories, Wikis, tickets, design records | Manuals, announcements, product knowledge bases |
| Semantic Differences | Query expressions differ from code or project names | Customer language differs from policy terminology |
| Clear Maintainers | Development team and system owners | Product, operations, and knowledge management teams |
| Verifiable Evidence | Code, documentation, design records | Policies, product information, operational procedures |
| Measurable Value | Lookup, onboarding, and diagnosis time; number of repetitive inquiries | Accuracy, handling time, first-call resolution rate, and escalation ratio |
After confirming a scenario is suitable for RAG, enterprises also need to compare purchasing mature products with building in-house. If knowledge is primarily concentrated within an existing platform, the search and AI capabilities provided by that platform might already be sufficient; if an enterprise must connect multiple systems, synchronize complex permissions, establish domain-specific evaluations, or embed knowledge queries into its own business processes, then custom development might offer long-term value.
Here, “building in-house” usually doesn’t mean training a large language model from scratch, but rather the enterprise independently combining model services, data connections, retrieval, permissions, evaluation, and operational mechanisms. Enterprises need to include integration costs, vendor lock-in, data migration, and exit costs in their comparison.
Converging Top-Down and Bottom-Up Approaches in Specific Scenarios. #
Enterprise RAG can be initiated from two directions simultaneously. Management, platform teams, and governance departments determine strategic objectives, investment boundaries, common platforms, and data, security, and audit requirements in a top-down manner. Frontline employees and specialized teams identify high-frequency problems, provide and verify professional knowledge, and record the costs and shortcomings of existing work methods in a bottom-up manner. Both paths ultimately need to converge in a business scenario with clear boundaries and measurable real value.
A purely top-down approach risks project goals diverging from real work, and the system might lack sustained users after deployment. A purely bottom-up approach, while potentially yielding useful prototypes, might lack formal data permissions, security reviews, long-term budgets, and knowledge maintenance responsibilities. Management should not create demand for RAG through administrative mandates, nor should frontline teams treat local prototypes directly as production systems.
Based on this, both parties select small-scale scenarios for piloting, and then decide to expand, adjust, or stop based on actual usage, answer quality, total cost of ownership, and new risks, rather than continuing investment simply because an AI strategy has been proposed.
Step Two: Govern Knowledge First, Then Build the Index #
Knowledge entering RAG must record source, owner, effective and expiration dates, applicable regions, business scope, authority level, access level, and replacement relationships with older versions.
Knowledge owners are responsible for confirming whether materials are valid and how to interpret them; the technical team is responsible for how the system parses, indexes, and retrieves materials. Neither can replace the other. Without this foundational information, even if the system finds semantically relevant text, it cannot reliably determine if it’s a valid basis for the current query.
Enterprises also need to control knowledge input sources, identify unverified content, guard against prompt injection instructions in documents affecting retrieval and generation, and prevent sensitive information leakage in subsequent stages[7].
In a software development scenario, the system owner needs to confirm when architectural decision records are updated or become invalid; in a customer service scenario, the product or operations team needs to clarify who published promotion rules and when they expire after an event ends.
Step Three: Integrate Permissions into the Retrieval Process #
The system cannot first retrieve all materials and then delete sensitive content during the answer generation stage. Access control must be enforced before or during retrieval, and extend throughout context assembly, caching, citation, and logging. The system must not only find the correct material but also the correct material that the current user is authorized to access.
This requires the system to simultaneously consider user identity, role, department, region, and business scenario, as well as the access levels of documents and text snippets. When original file permissions change, permissions in the index must also be synchronized promptly.
Regular developers may not have permission to retrieve security architecture, production incident, or customer data records; customer service personnel can query product rules, but should not thereby gain access to customer contract prices or internal risk flags.
Step Four: Constrain Generation with Evidence and Refusals to Answer #
The system can use hybrid retrieval and re-ranking, then filter candidate materials based on metadata. When a query lacks necessary conditions, the system should ask the user for clarification; when materials conflict, the conflict should be displayed; when evidence is insufficient, it should refuse to answer.
When involving formal approvals, policy exceptions, or individual rights, the system should escalate to a clear human accountable party. If an enterprise uses RAG in the future to answer the initial accommodation question, the system can explain under what conditions the AUD 320 limit applies, and, if permissions allow, query relevant business systems to confirm if an approval record exists; but it cannot independently determine an exception is valid without approval, much less act as a direct manager to grant approval. The system should explain which conditions are missing and guide the employee to the appropriate approval process.
Evaluation also needs to be broken down into retrieval, evidence, answers, citations, and refusals to answer, rather than just assigning an overall score to the final answer. Only then can the team pinpoint errors and determine which metrics a model, index, or prompt adjustment improved, and whether it harmed other capabilities.
When the system cannot find a design decision, it should state that records are missing; when policies do not cover specific customer situations, it should be escalated to a supervisor.
Step Five: Treat RAG as a Long-Term Organizational System #
Control the Pace of Expansion After Piloting. #
Even after a successful pilot, enterprises should not immediately expand to all knowledge domains. Each expansion requires re-evaluating data quality, permission complexity, consequences of errors, and maintenance responsibilities; the success of one scenario does not automatically prove another scenario is also suitable for RAG. Interview studies by Jöhnk et al. also indicate that an organization’s readiness for AI adoption depends not only on technical resources but also on strategy, data, knowledge, culture, and organizational commitment[8].
Involve Employees in System Development. #
Project teams can collaborate with actual employees to build question sets, recording the time currently needed to find information, error-prone steps, and content that requires verification. After deployment, observe whether employees find correct sources faster, if repetitive inquiries decrease, which questions are frequently escalated to human agents, and if workflows truly simplify after using the system. If employees find verifying answers more time-consuming than their original search methods, they will quickly stop using the system.
Employees not using the system isn’t necessarily just “not accepting new technology.” Frequent system errors, potential monitoring of query records, or new operations outweighing time saved can all reduce employee willingness to use the system. Raisch and Krakowski view automation and augmentation in enterprise AI applications as a relationship that requires continuous balance: AI might both replace some tasks and change how humans complete tasks[9]. Enterprises need to explain what the system can handle, what it records, which answers must be verified, and how employees can report errors.
Clarify Who Has the Authority to Interpret and Who Is Responsible for Error Correction. #
Once RAG begins interpreting policies, authoritative sources, retrieval ranking, prompts, and refusal-to-answer rules will all influence the answers employees see. Research on algorithmic management indicates that algorithmic systems change information, evaluation, and control relationships within organizations[10].
If an enterprise only entrusts RAG to the technical team, engineering configurations might shoulder policy judgments that originally belonged to business, HR, legal, or management. When responsibilities, permissions, and resources are not aligned, RAG will transform existing ambiguous organizational boundaries into retrieval ranking, prompts, and permission configurations.
Project sponsors can help pilots acquire initial resources and cross-departmental coordination, but individual support cannot replace long-term authorization. After the pilot, enterprises need to convert temporary consensus into formal mechanisms: who approves knowledge entering the system, who interprets conflicting rules, who decides whether high-risk questions can be answered automatically, and who is responsible for handling errors and exceptions. Sponsors can initiate collaboration, but documentation, responsibility allocation, and transferable mechanisms are what enable it to continue running. I discussed this transformation in “Between Relationships and Rules: How to Get Things Done in Imperfect Organizations.”
A clearer division of responsibilities might include:
| Role | Primary Responsibility |
|---|---|
| Business & Product Owners | Define scenarios, value objectives, and risk boundaries |
| Knowledge Owners | Confirm material authority, versions, scope of application, and interpretation rules |
| Technical Team | Responsible for retrieval, generation, evaluation, permission synchronization, and monitoring |
| Privacy, Risk & Compliance Team | Review data flows, high-risk uses, and accountability mechanisms |
| Frontline Employees & Specialists | Participate in question set development, pilot feedback, and error detection |
Make Evaluation and Maintenance Ongoing Work. #
The NIST AI Risk Management Framework uses four functions—Govern, Map, Measure, and Manage—to organize AI risk management[11]. NIST’s subsequent Generative AI Profile further expands on recommendations specific to generative AI[12]. While these frameworks do not dictate organizational structure, they emphasize that governance, scenario identification, risk measurement, and continuous management need to be interconnected.
Policies, permissions, models, and vendors will continue to change. Machine Learning Operations (MLOps) research emphasizes the continuous processes of data, models, deployment, monitoring, and cross-team collaboration[13]. RAG is not entirely the same as traditional machine learning systems, but it similarly requires version management, regression testing, quality monitoring, fault handling, and migration plans. Enterprises also need to measure invocation, maintenance, and knowledge update costs to confirm if the system’s value can cover long-term investment.
When code, product, and promotion rules change, relevant documents, permissions, and indexes all need to be synchronized and updated; otherwise, a system that once passed validation will quickly become obsolete.
Future Outlook: From Local Knowledge Tools to Organizational Infrastructure #

For enterprise RAG that has proven its value and established a governance foundation, the next step might go beyond merely answering questions. The system could further compare policy versions, detect knowledge conflicts, remind owners to update documents, and, with explicit authorization, assist in executing business processes.
When the system moves beyond providing answers to invoking tools, querying external systems, and altering business states, it begins to approach what are commonly referred to as agentic systems[14].
However, the more proactive the system, the higher the demands for accountability. When the system only provides information, users can still detect errors by checking the materials; if the system starts initiating accommodation exception approvals, updating records, or performing operations, the same errors could directly impact business. Enterprises need stricter authorization, auditing, and human approval mechanisms; they cannot assume a system has the authority to act simply because it can take action.
The sales report example in the second chapter also illustrates that permission control might need to evolve from document-level to content and scenario-level. If the system can only determine whether a user has permission to open an entire document, enterprises are forced to choose between “all open” and “all closed.” Finer-grained control can reduce this dilemma but will also increase the complexity of identity, policy, index synchronization, and auditing systems.
RAG’s long-term value might also extend from knowledge retrieval to knowledge governance. When the system continuously records which questions cannot be answered, which files frequently conflict, and which knowledge relies on a few employees, enterprises can, in turn, discover their own knowledge gaps. RAG not only helps employees find knowledge but might also enable organizations to see where knowledge breaks down, who holds it, and which processes still lack clear accountability.
The above capabilities still depend on reliable permission systems, knowledge metadata, and organizational governance. For enterprises with limited scale of use and governance capabilities, adopting standardized RAG capabilities offered by mature software might be more reasonable than building in-house systems.
Conclusion #

Enterprises should not decide to deploy RAG first and then search for documents to connect. Instead, they need to start from specific problems and compare search, process improvement, human support, mature software, and in-house systems. Many knowledge problems do not require RAG; even if an enterprise needs RAG capabilities, it might not need to build an in-house system covering the entire company.
Only when the problem has sufficient scale, knowledge can be governed, permissions can be controlled, and long-term benefits are likely to cover total costs, does an enterprise have a reason to proceed with development. For these scenarios, connecting to a vector database is merely a starting point. What an enterprise ultimately builds is not a model that is “better at answering questions,” but a knowledge system that is governed, accountable, and sustainably operational.
The same judgment applies to agentic systems: enterprises should first prove business value, then clarify what information the system can read, what tools it can invoke, what operations it can perform, and who is accountable for the results.
References #
[1] Vaswani, A., et al. Attention Is All You Need. Advances in Neural Information Processing Systems, 2017. https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
[2] Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 2020. https://papers.nips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
[3] Karpukhin, V., et al. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP, 2020. https://aclanthology.org/2020.emnlp-main.550/
[4] Boeckl, K., and Lefkovitz, N. NIST Privacy Framework: A Tool for Improving Privacy Through Enterprise Risk Management, Version 1.0. NIST, 2020. https://doi.org/10.6028/NIST.CSWP.01162020
[5] Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 2024. https://aclanthology.org/2024.tacl-1.9/
[6] Es, S., et al. RAGAs: Automated Evaluation of Retrieval Augmented Generation. EACL System Demonstrations, 2024. https://aclanthology.org/2024.eacl-demo.16/
[7] OWASP. Top 10 for LLM Applications 2025. 2024. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/
[8] Jöhnk, J., Weißert, M., and Wyrtki, K. Ready or Not, AI Comes—An Interview Study of Organizational AI Readiness Factors. Business & Information Systems Engineering, 2021. https://aisel.aisnet.org/bise/vol63/iss1/2/
[9] Raisch, S., and Krakowski, S. Artificial Intelligence and Management: The Automation–Augmentation Paradox. Academy of Management Review, 2021. https://doi.org/10.5465/amr.2018.0072
[10] Kellogg, K. C., Valentine, M. A., and Christin, A. Algorithms at Work: The New Contested Terrain of Control. Academy of Management Annals, 2020. https://doi.org/10.5465/annals.2018.0174
[11] NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023. https://doi.org/10.6028/NIST.AI.100-1
[12] Autio, C., et al. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1, 2024. https://doi.org/10.6028/NIST.AI.600-1
[13] Kreuzberger, D., Kühl, N., and Hirschl, S. Machine Learning Operations (MLOps): Overview, Definition, and Architecture. IEEE Access, 2023. https://doi.org/10.1109/ACCESS.2023.3262138
[14] NIST. Lessons Learned from the Consortium: Tool Use in Agent Systems. 2025. https://www.nist.gov/news-events/news/2025/08/lessons-learned-consortium-tool-use-agent-systems