Okay, let’s roll with the hype. Let’s assume that data quality isn’t an issue, that governance is in the bag, that data stewards are fully data-literate, and that data lineage is well-documented and catalogued. AI can seamlessly retrieve, process, and generate insights with precision and accountability in this ideal world. And yet, even under these perfect conditions, one major challenge remains: transparency.
The rapid advancement of Large Language Models (LLMs) and their integration with Retrieval-Augmented Generation (RAG) and Data Mesh architectures have created significant opportunities for AI applications in government and enterprises. These technologies offer the potential for increased efficiency, improved decision-making, and enhanced automation. However, they also pose challenges related to explainability, transparency, and governance. This is particularly important in the public sector, where AI systems must be accountable to policymakers and citizens.
For Agentic AI to become a reality—where AI operates with minimal human intervention—there must be complete confidence in every layer of the AI infrastructure. This requires transparent and open models that can be scrutinised and understood, verifiable and unbiased data, software that allows for independent oversight, and platforms that ensure interoperability and security. Without these foundations, AI remains a black-box technology, limiting its government adoption and raising legitimate concerns about security, fairness, and accountability.
LLMs, RAG, and Data Mesh: A Converging Ecosystem
LLMs have fundamentally changed how we interact with AI, making natural language processing more sophisticated. However, a core limitation remains: LLMs are trained on a fixed dataset and do not inherently possess real-time knowledge. While they may generate coherent responses, they can produce outdated, incorrect, or even entirely fabricated information.
Retrieval-augmented generation (RAG) addresses this problem by allowing AI to pull in external data during a query, ensuring that up-to-date, relevant information informs responses. RAG enhances the reliability of AI-generated outputs, making them more useful for enterprise and government applications. Yet, even RAG is not a silver bullet. The success of any AI system ultimately depends on the quality, governance, and accessibility of the data feeding into it. This is where Data Mesh becomes essential.
A Data Mesh approach decentralises data ownership, allowing different departments or organisations to manage their data sources while maintaining overarching governance standards. By integrating RAG with Data Mesh, organisations can ensure that AI systems operate with high-quality, well-managed data, making them more reliable and fit for purpose. These three elements—LLMs, RAG, and Data Mesh—form a robust AI ecosystem, but only if they are designed with transparency and accountability at their core.
The Probabilistic Nature of LLMs and the Need for Explainability
LLMs generate text probabilistically, meaning that responses are not deterministic but based on the statistical likelihood of specific word patterns appearing together. This fundamental characteristic leads to common issues such as hallucinations, where an AI generates incorrect but highly plausible-sounding responses, and bias, where the outputs reflect distortions in the training data.
These problems are particularly concerning for government applications. AI-driven decision-making in welfare benefits, law enforcement, and public health cannot afford to operate in a black box. If an LLM makes a recommendation that affects someone’s legal status, healthcare entitlement, or access to public funds, there must be a transparent and auditable rationale behind that decision.
Retrieval-augmented generation helps mitigate some of these risks by ensuring that AI responses are grounded in real-world data rather than being purely predictive. However, the system remains opaque unless the underlying data sources and retrieval mechanisms are transparent. The ability to audit and explain AI outputs is essential if governments are to build public trust in AI-driven decision-making.
Building a Reliable RAG Pipeline
A RAG system relies on multiple components to deliver accurate and relevant responses. The process begins with data sources, which can be structured (databases, APIs) or unstructured (documents, reports, web content). These data sources are transformed into embeddings—mathematical representations that allow AI to search for relevant content efficiently. These embeddings are stored in a vector database, which enables the LLM to retrieve the most appropriate data when generating responses.
The challenge, however, is ensuring that the data feeding into the RAG pipeline is continuously updated, high-quality, and compliant with regulations. The system may retrieve outdated or inaccurate information if the underlying data changes but the embeddings are not refreshed. Similarly, if data sources contain biases or inconsistencies, these will be reflected in AI outputs.
Robust data governance mechanisms must be in place to ensure RAG remains reliable. This includes version control for embeddings, data lineage tracking, and clear policies for handling sensitive information. Without these safeguards, RAG risks becoming another black box system, failing to provide the transparency needed for AI in the public sector.
Data Mesh: The Key to Scalable and Transparent AI
One of the biggest challenges in scaling AI across government departments is the fragmented nature of public sector data. Different agencies collect, store, and manage their data, often in siloed systems that are difficult to integrate. A centralised approach to data management has been attempted, but it has repeatedly failed due to complexity, governance challenges, and scalability issues.
Data Mesh offers a more sustainable approach by decentralising data ownership while maintaining centralised governance. Under this model, each department remains responsible for its data products, ensuring that domain experts control data quality and accessibility. At the same time, common standards for security, compliance, and interoperability are enforced at a higher level, allowing AI applications to access data seamlessly across the organisation.
For RAG-based AI systems, data retrieval can be decentralised and governed, ensuring that AI models have access to high-quality, up-to-date information while maintaining strict security and compliance controls. In a government context, this could allow different departments to access data (not data sharing) in a controlled and auditable manner, ensuring that AI-driven insights are based on a comprehensive and reliable information ecosystem. Data access is another deep topic which I’ve discussed before. If this isn’t done correctly, large-scale AI initiatives are moot. But as we are living the dream here, let’s assume we’ve got this right.
Agentic AI: Removing the Human-in-the-Loop
The long-term goal of AI development is to create systems that can operate autonomously with minimal human intervention. However, achieving this in a government setting requires high confidence in AI decision-making processes.
Human oversight is necessary to review AI-generated decisions, validate outputs, and intervene in cases of uncertainty or ethical concern. While this remains an essential safeguard, excessive reliance on human intervention reduces AI’s efficiency benefits. The only way to progressively remove humans from the loop is to establish AI systems that are transparent, auditable, and aligned with regulatory standards.
For this to happen, AI systems must be designed with explainability at every level. The models must be interpretable, the data must be traceable, and the infrastructure must be open and inspectable. This approach is desirable and essential in the public sector, where accountability is paramount.
Building AI for the Public Good
AI has the potential to transform government services, but only if it is implemented responsibly. LLMs, RAG, and Data Mesh represent a powerful combination of technologies, but their success depends on how they are designed, governed, and deployed.
A closed, black-box approach to AI risks undermining trust, creating inefficiencies, and introducing bias into public sector decision-making. By contrast, an open, transparent, and explainable AI ecosystem ensures that AI-driven services are accountable, reliable, and fair.
Governments should prioritise investments in open AI models, enforce transparent data governance, and create interoperable platforms that enable AI to benefit society effectively. The future of AI in the public sector should focus on enhancing human decision-making rather than replacing human oversight with unclear algorithms. Building systems that promote clarity, fairness, and trust is essential.
