Data locality, customer-controlled keys, tenant isolation, local repositories, identity and controlled update procedures inside the approved security boundary.
XaasIO AI Lake · Open Data Platform for AI
Enterprise Hadoop Data Platform, Extended for AI
XaasIO AI Lake is an enterprise distribution of the Apache Hadoop ecosystem, the same open stack that open data platform distributions ship: HDFS and Ozone storage, YARN, Spark, Flink and Tez, Hive, HBase, Phoenix and Trino, Iceberg tables, Kafka, NiFi and Oozie, Kerberos, Ranger, Knox and Atlas, managed with Ambari. On that stack XaasIO adds the layers AI needs, as extensions of the same components rather than a second platform: search and a vector database, a knowledge graph, the Context Lake and the Agent Lake, governed by the same Kerberos, Ranger and Atlas controls, in your data center, sovereign cloud or provider environment, with your data, keys, identity and operations under your control.
Open source by default: Apache Hadoop, Ozone, YARN, Hive, HBase, Phoenix, Spark, Flink, Tez, Kafka, NiFi, Oozie, Airflow, Iceberg, Ranger, Knox, Atlas and Ambari, with OpenSearch, Milvus and JanusGraph for the AI extensions. Upstream-aligned, customer-operated, co-managed or fully managed by XaasIO.
Architecture
One Open Data Platform. Four AI Extensions.
Storage, processing, resource management, data access, security and orchestration, named component by component, with the management plane beside them and the ecosystem the platform integrates with. The four AI extensions sit on top and reuse those components: the lake stores their data, YARN and Kubernetes run their jobs, Ranger and Atlas govern their output. Components are pinned per release and listed in the release SBOM.
AI extension
Search and Vector Database
Keyword, semantic and hybrid retrieval over documents, tables and events, with citations.
Extends: Spark on YARN for embedding jobs · Ozone or Ceph S3 for index segments · Ranger tags for filtering · OpenSearch and Milvus as the engines
AI extension
Knowledge Graph
Entities, relationships, dependencies and rules for multi-hop questions, impact analysis and graph-enhanced retrieval.
Extends: HBase as the graph store through JanusGraph · Atlas lineage and classifications as graph input · Spark and Flink for entity extraction
AI extension
Context Lake
The minimum necessary evidence for a user, task, model and policy, assembled from SQL, search, vectors and the graph, ranked, compressed and cited.
Extends: Hive and Trino SQL · Kafka and Flink for freshness · Knox single sign-on for identity · Ranger policies for entitlement
AI extension
Agent Lake
Agents, tools, prompts and memory as governed services, with the model gateway that routes to private or approved models.
Extends: HBase and Iceberg for registries and memory · Hive Metastore and catalog metadata as agent tools · Ranger audit for every agent action · XaasIO Kubernetes for the runtime
Storage
- Apache HDFS with erasure coding and tiered storage
- Apache Ozone, S3-compatible object store for lake zones, index segments and model artifacts
- Ceph S3 tier from XaasIO Block and Object Storage for cold data and backups
- HDFS Transparent Data Encryption with Ranger KMS
Processing
- Apache Spark for batch, SQL, streaming, MLlib and the embedding and entity-extraction jobs
- Apache Flink for stream processing
- Apache Tez for Hive, Apache MapReduce
- Apache Livy for REST job submission
Resource management
- Apache YARN with Capacity Scheduler queues per tenant and team
- Dynamic allocation and preemption
- Node labels for GPU and memory-heavy nodes
- XaasIO Kubernetes Platform for the cloud-native services beside the cluster
Data access
- Apache Hive with ACID and Iceberg tables
- Apache HBase and Apache Phoenix
- Trino for interactive and federated SQL
- Apache Zeppelin and Jupyter notebooks, Apache Superset dashboards
- JDBC and ODBC through Knox
Security
- Kerberos, with FreeIPA or Active Directory
- Apache Ranger: RBAC, ABAC, tag-based policies, masking, row filters, audit
- Apache Knox: perimeter and single sign-on (SAML, OIDC)
- Apache Atlas: metadata, lineage, classifications
- TLS everywhere; Ranger audit to XaasIO MLT or your SIEM
Orchestration
- Apache Oozie and Apache Airflow for scheduled pipelines
- Apache NiFi and NiFi Registry for data flows
- Apache Kafka for event streams and change data capture
- Temporal for durable agent workflows
Management
- Apache Ambari: provisioning, configuration, Kerberos wizard, alerts, and Helm-based lifecycle of the cloud-native services
- XaasIO MLT: platform, job, queue, index, retrieval and agent observability
- ZooKeeper, with the XaasIO release cycle for every component
Benefits
- Enterprise security
- Elastic scaling
- Cost efficient
- No lock-in
Features
- Full observability
- Data governance
- Multi-tenancy
- Audit and compliance
Component names are the upstream projects as packaged in each XaasIO AI Lake release. No deployment needs every component; the validated selection is documented in its design.
Same stack, extended
What Each AI Extension Builds On, and What XaasIO Adds
An open data platform is a data platform, not an AI engine. XaasIO AI Lake keeps it that way: the AI extensions reuse the storage, compute, catalog and security of the stack, and models remain replaceable consumers behind a gateway. The table shows the reuse line by line.
| AI extension | Builds on | What XaasIO adds | Governed by |
|---|---|---|---|
| Search and vector database | Spark jobs on YARN chunk and embed documents and tables; Ozone or Ceph S3 holds the index segments; Hive and Iceberg tables hold the sources | OpenSearch for keyword and hybrid search, Milvus for vector indexes at scale, an embedding lifecycle that records source snapshot, model and version, and a Retrieval API with re-ranking and citations | Ranger tags carried from source tables to every chunk and vector; Knox single sign-on; Atlas lineage from table to index |
| Knowledge graph | HBase as the storage backend, with the same Atlas lineage, classifications and glossary the platform already maintains; Spark and Flink for entity extraction | JanusGraph on HBase as the property graph, OpenSearch as its index, entity and relationship pipelines, ontology support, and graph expansion inside the Retrieval API | Ranger policies on entity types and sources; Atlas classifications as graph labels; every edge carries its source and timestamp |
| Context Lake | Hive and Trino for facts, Kafka and Flink for live state, Hive Metastore and Atlas for what exists and who owns it, Knox for the caller’s identity | A Context API that assembles identity-aware, task-relevant, fresh and policy-filtered evidence from SQL, search, vectors and the graph, ranks and compresses it, and returns provenance, policy decision and a token budget | Ranger entitlements evaluated per request; Atlas classifications drive masking and exclusion; context expiry rules; audit to XaasIO MLT |
| Agent Lake | HBase and Phoenix for registries, Iceberg tables for memory and traces, Kafka for events, YARN or XaasIO Kubernetes for the runtime, Oozie and Airflow for scheduled runs | Agent, tool and prompt registries, a harness with sandboxed tools, approvals, rollback and a kill switch, governed memory with classification, ageing and deletion, durable workflows on Temporal, and the model gateway | Ranger policies per agent identity; Ranger audit of every tool call and data access; Atlas lineage from memory to source |
| Catalog and metadata for agents | Hive Metastore, Iceberg catalog and Atlas metadata: schemas, partitions, owners, classifications and update times | Catalog, SQL, search and graph exposed as agent tools through the Model Context Protocol, so assistants discover and query data in natural language without a path around the policies | Ranger decides which metadata and rows a tool may return; Knox authenticates the agent; calls are logged |
| Model access | The same Kerberos and Knox identity, the same Ranger audit, GPUs from node-labeled YARN nodes or the XaasIO AI Factory | A model gateway that routes to private models on the XaasIO AI Token Factory first and to approved public providers only by policy, with metering per tenant and application | Per-tenant routing policies; prompt and response retention per application; usage events to the XaasIO Hyperscaler Platform |
| Observability for AI | XaasIO MLT collecting Ambari alerts, YARN and Spark metrics, Ranger audit and Kafka lag | Traces for retrieval, context assembly and agent runs; evaluation records per agent version; dashboards per tenant and application | The same tenants, roles and retention as the platform telemetry |
Every derived asset, from a chunk or vector to a graph edge, a context record or an agent memory, stays linked to its source table, Iceberg snapshot, Ranger policy and retention state, so corrections and deletions propagate from the source outward.
Data and AI, simplified
Everything Needed to Deliver Data and Knowledge as a Service
Teams register the data they own. XaasIO AI Lake coordinates storage, cataloging, processing, indexing, retrieval, context and access behind the scenes, on one stack.
Main components
Open Data Platform | Store raw and curated data in open formats on storage you own: HDFS and S3-compatible Ozone, with Hive and Iceberg tables, the Hive Metastore and Atlas for catalog and lineage, versioning and time travel. Query it with SQL and Spark without copying it into a proprietary warehouse. Technical reference: Apache Hadoop HDFS · Apache Ozone · Ceph S3 tier · Apache Hive · Apache Iceberg · Parquet and ORC · Hive Metastore · Apache Atlas |
Ingestion, Streaming and Processing | Load batch files, change data capture and event streams with NiFi and Kafka; enrich them in flight with Flink; transform at scale with Spark on YARN; and query the tables and external systems through Hive and Trino. Technical reference: Apache NiFi · Apache Kafka · Apache Flink · Apache Spark · Apache Tez · Trino · Apache Oozie and Airflow · Apache Livy |
Search, Vector Database and Knowledge Graph | Turn documents, tables and events into keyword indexes, vector embeddings and a graph of entities and relationships, computed by Spark on the cluster and stored on the same platform, so applications retrieve what is similar and understand how it is connected. Technical reference: OpenSearch · Milvus · JanusGraph on Apache HBase · Spark embedding and extraction jobs · Retrieval API |
Context Lake and Agent Lake | Assemble identity-aware, fresh, ranked and cited context for models and agents, and manage agents, tools, prompts and memory as governed services with approvals, rollback and a model gateway. Technical reference: Context API · Agent Registry · Temporal Workflows · Model Context Protocol tools · Model Gateway |
Identity and Governance | Separate tenants, queues, projects and workspaces with roles, attributes, classification, retention and policy. Integrate enterprise single sign-on through Knox and keep one Ranger audit trail across data, retrieval and agent actions. Technical reference: Kerberos with FreeIPA or Active Directory · Apache Ranger and Ranger KMS · Apache Knox · Apache Atlas · XaasIO IAM · Audit Archive |
Use the same platform through the:
Six layers
An Open Data Platform, Plus the Layers AI Needs
The data platform and the streaming and SQL layers are the enterprise Hadoop distribution. The vector database, knowledge graph, Context Lake and Agent Lake layers extend it and turn governed data into governed knowledge for models and agents.
Open Data Platform
Governed facts and history in open formats, on storage you own.
Raw and curated data
Parquet and ORC files
Documents and media
Open Data PlatformHive and Iceberg tables
Governed tables
Time travel, snapshots
Catalog and lineage
HDFS and S3-compatible Ozone are the data plane. Hive and Iceberg tables give it schemas, transactions, hidden partitioning and time travel.
Role
The sovereign data plane: raw, curated and gold zones on HDFS and S3-compatible Ozone, as Hive ACID tables, HBase tables or Iceberg tables with schema evolution, hidden partitioning and time travel; a Ceph S3 tier for cold data and backups.
Covers
Structured datasets, documents, media, features, embeddings, prompts, model artifacts and audit logs, each in its own zone with retention and legal-hold rules.
Catalog and lineage
The Hive Metastore and the Iceberg catalog for tables; Apache Atlas for metadata, lineage captured from Hive, Spark and Iceberg operations, classifications and a business glossary; Ranger tag-based policies on top of them.
Technical reference
Apache HDFS · Apache Ozone · Ceph S3 · Apache Hive · Apache HBase · Apache Iceberg · Parquet and ORC · Hive Metastore · Apache Atlas
Streams and Processing
Fresh data in, federated SQL out, without a proprietary engine in between.
NiFi flows and CDC
Kafka event streams
Batch and API loads
Stream ProcessingKafka, Flink, Spark, Trino
Fresh governed tables
Context events
Federated SQL
NiFi and Kafka move data, Flink enriches it, Spark transforms it on YARN, and Hive and Trino query the tables and external systems without copying.
Role
Batch, streaming and change-data-capture ingestion; stream enrichment; large-scale transformation on YARN; and interactive SQL over the tables and external systems.
Covers
Operational databases through CDC, event streams, files and API loads, NiFi data flows, scheduled pipelines and notebooks.
Performance
Spark for ETL, embeddings and feature generation; Flink for low-latency context processing; Tez for Hive; Trino for federated queries that avoid copying data; YARN queues and node labels for isolation and GPU placement.
Technical reference
Apache Kafka · Apache NiFi · Apache Flink · Apache Spark · Apache Tez · Trino · Apache Oozie and Airflow · Apache Livy
Vector Database
Semantic and hybrid retrieval with tenant scope, provenance and re-ranking built in, computed and stored on the platform.
Chunked documents
Spark embedding jobs
Ranger tags, tenant
Vector DatabaseMilvus and OpenSearch
Semantic similarity
Hybrid retrieval
Re-ranked citations
Every vector carries its source table, Iceberg snapshot, Ranger tags and embedding model, so results can be filtered, cited, re-embedded and deleted.
Role
Spark jobs on YARN chunk and embed documents, tables and events; Milvus indexes the vectors with its segments on Ozone or Ceph S3; OpenSearch indexes keywords; the Retrieval API combines both with Ranger tag filters, re-ranks and cites.
Covers
Retrieval-augmented generation, enterprise search, assistants, recommendations and agent context, with tenant, project and workspace isolation on every index.
Embedding lifecycle
Each vector records its source table and Iceberg snapshot, Ranger tags, embedding model, dimension and re-embedding triggers, so deletion and corrections propagate from source data to every index.
Technical reference
Milvus · OpenSearch hybrid search · Spark embedding jobs · Qdrant and pgvector options for small deployments · Feast feature store
Knowledge Graph
Entities, relationships, dependencies and rules, stored in the platform’s own HBase, for reasoning that similarity search cannot do.
Spark entity extraction
Atlas lineage, assets
Ontologies and rules
Knowledge GraphJanusGraph on HBase
Multi-hop reasoning
Impact and root cause
Graph-enhanced RAG
Vectors answer what is similar. The graph answers how things are related, which dependencies, rules, time or location connect them.
Role
Represent assets, services, customers, documents, agents and decisions as entities with typed, sourced relationships on JanusGraph over HBase, with OpenSearch as the index, and traverse them for multi-hop questions.
Covers
Application-to-infrastructure dependencies, impact and root-cause analysis, supply chains, identity relationships, maintenance history, fraud patterns and agent planning; Atlas lineage and classifications flow in as graph input.
Graph-enhanced retrieval
Graph expansion adds relationship paths to vector and keyword results, so answers explain how things connect and cite the path.
Technical reference
JanusGraph on Apache HBase · Gremlin · OpenSearch index backend · Apache Atlas lineage · RDF and ontology tooling where required
Context Lake
The minimum necessary evidence for a user, task, model and policy, delivered with provenance.
Identity and role
Live operational state
History and cases
Context Lakegoverned context
Minimum evidence
Provenance and age
Token budget
The delivery API returns evidence, the policy decision, provenance and a size budget. Temporary context expires; it does not become memory.
Role
Assemble identity-aware, task-relevant, fresh and policy-filtered context from Hive and Trino SQL, OpenSearch, Milvus and the graph; rank, compress and cite it; expire what no longer applies.
Context families
Identity and entitlement, task and process, operational and real-time state from Kafka and Flink, history and conversations, time and location, security and regulatory obligations.
Delivery
One API returns the evidence, the Ranger policy decision, provenance timestamps and a token or size budget, so nothing outside the requester’s entitlement reaches a model.
Technical reference
Context API · Context namespaces and events on Kafka and Iceberg · Policy-filtered retrieval through Ranger · Re-ranking and compression · Freshness and expiry
Agent Lake
Agents, tools, prompts and memory managed as governed services, with the model gateway they use.
Agent and tool registry
Prompts, workflows
Governed memory
Agent Lakeharness, model gateway
Traced runs, rollback
Human approvals
Private, hybrid models
Agents are managed as services with an owner, identity, permissions, version and evaluation record. Models remain replaceable consumers.
Role
A registry and runtime for agents with an owner, Kerberos identity, Ranger permissions, version and evaluation record; a harness with sandboxed tools, approvals, rollback and a kill switch; catalog, SQL, search and graph exposed as tools through the Model Context Protocol.
Memory
Working, episodic, semantic and procedural memory in HBase and Iceberg tables with classification, scope, ageing, expiry, correction and deletion. Promotion to organizational memory is explicit and auditable.
Model access
A gateway routes to private models on the XaasIO AI Token Factory, sovereign endpoints and approved public providers by policy, and meters usage. Models remain replaceable consumers of the lake.
Technical reference
Agent, tool and prompt registries · Temporal durable workflows · LangGraph · Model Context Protocol · Ranger audit · Model gateway
A data platform first
Five Questions an AI-Ready Data Platform Must Answer
Before a model is trained or an agent is given a tool, the platform has to answer for its data. The enterprise Hadoop stack answers these questions for tables; the XaasIO extensions make the same answers hold for chunks, vectors, graph edges, context records and agent memory.
Where did the data come from?
Atlas captures lineage from NiFi ingestion through Hive, Spark and Iceberg operations. XaasIO extends the chain to the embedding job, the index, the graph edge and the context record, so an answer can be traced back to a source table and snapshot.
Who can access it?
Kerberos authenticates, Knox signs on, Ranger decides: dataset-level grants, column masking, row filters and tag-based policies. The same tags travel with every chunk and vector, and the Context API evaluates the same policies per request.
What did it look like at training time?
Iceberg snapshots give every training run, embedding batch and evaluation a snapshot identifier, so a result can be reproduced or a record’s presence in a training set can be established.
Did sensitive data enter the pipeline?
Atlas classifications and Ranger masking keep classified columns out of embeddings and prompts by policy; Iceberg deletes and Ranger audit support data subject erasure, and deletions propagate to indexes, graph and memory.
Where does the data reside?
On premises, in a sovereign private cloud or in an air-gapped site, with no runtime dependency on an external service. Keys stay in Ranger KMS, and public model providers are reached only when a policy allows it.
Where does the model fit?
Outside the lake, as a replaceable consumer. Spark MLlib runs classical machine learning on the cluster; private language models run on the XaasIO AI Factory; the model gateway routes to them, and the lake owns the data, the context and the audit trail.
The platform supports these controls; it does not make a system compliant by itself. The assessment maps regulatory obligations, such as the EU AI Act’s data governance and record-keeping articles, to the controls each workload needs.
How it works
One Lifecycle. From Source Data to Governed Context.
01
Govern
Inventory and classify sources in Atlas, assign owners, Ranger policies, retention and authorized purpose before anything is embedded.
02
Ingest
Load batch, stream and CDC data with NiFi and Kafka; validate, catalog and store it in open zones on HDFS and Ozone; clean and transform it with Spark.
03
Index
Extract entities, create embeddings, build OpenSearch and Milvus indexes and update the JanusGraph knowledge graph, as jobs on the cluster.
04
Contextualize
Filter by identity and Ranger policy, rank, compress and cite the evidence; enforce freshness and expiry.
05
Serve and observe
Deliver context to applications, models and agents; trace, evaluate, correct and delete with propagation, observed in XaasIO MLT.
Every derived asset, from a chunk or vector to a graph edge or memory, stays linked to its source, version, policy and retention state.
Advanced Capabilities
Enterprise Data and Knowledge, Presented as Simple Choices
Open Table Formats and Time Travel
Hive ACID and Iceberg tables on HDFS and Ozone with schema evolution, snapshots, hidden partitioning and time travel, readable by Hive, Spark, Flink, Trino and any Iceberg-compatible engine.
Technical reference: Apache Hive · Apache Iceberg · Parquet and ORC · Hive Metastore and Iceberg catalog
Federated SQL Without Copying
Query platform tables, operational databases and external systems in one SQL layer, so analysts and agents reach data where it lives, with Ranger policies applied on the way.
Technical reference: Trino · Apache Hive · Connectors · Ranger Query Governance
Streaming and Change Data Capture
Capture changes from operational databases and events from applications and devices with NiFi and Kafka, enrich them in flight with Flink and keep tables, indexes and context current.
Technical reference: Apache NiFi · Apache Kafka · Apache Flink · XaasIO RDS as a CDC source
Hybrid Retrieval with Citations
Keyword, vector and graph results scored together, filtered by tenant, purpose and Ranger tags, re-ranked and returned with provenance, so every answer can be traced to its evidence.
Technical reference: OpenSearch · Milvus · Re-ranking · Graph Expansion
Graph-Enhanced RAG and Impact Analysis
Multi-hop reasoning over dependencies, timelines and rules for root-cause analysis, impact assessment, planning and explainable answers, on a graph that lives in the platform’s own HBase.
Technical reference: JanusGraph on HBase · Gremlin · Atlas lineage · Ontologies
Governed Memory and Agent Harness
Agent memory that is classified, scoped, aged and deletable, and an execution harness with Ranger-governed tool permissions, human approval, rollback and a kill switch.
Technical reference: Memory Lifecycle · Temporal · Approval Workflow · Kill Switch
Advanced capabilities depend on the validated storage, processing, retrieval and architecture selected for each deployment. No deployment needs every listed component.
Production data platform
Open Data Without the Integration Burden
XaasIO designs, deploys, validates and supports XaasIO AI Lake as one production platform rather than an unsupported collection of Apache projects, connectors and scripts.
Validated Architecture
Storage, table formats, catalog, processing engines, search, vector and graph stores, context services, identity and observability are tested as a complete solution, with the bug fixes needed to keep the components interoperable in production.
Controlled Lifecycle
Every pinned component follow a documented release, patching, and upgrade policy based on validated upstream releases, with Ambari-driven rolling upgrades.
Sovereign and Air-Gapped Ready
Flexible Operating Model
Choose customer-operated, co-managed, fully managed or dedicated SRE Pod operations.
Built for enterprise private cloud, sovereign and regulated cloud, and cloud service provider or MSP environments.
Where it runs
Cluster Nodes on XaasIO Infrastructure, Cloud-Native Services on XaasIO Kubernetes
The Hadoop cluster wants servers with local disks; the newer services want containers. XaasIO AI Lake runs both on the same XaasIO infrastructure, with one identity and one security configuration shared between them.
Bare Metal and XaasIO Virtualization
HDFS, Ozone, YARN and HBase nodes on bare metal provisioned by XaasIO, or as KVM virtual machines on XaasIO Virtualization and HCI for smaller clusters, with node labels for GPU and memory-heavy nodes.
XaasIO Compute
Clusters as OpenStack projects with quotas and self-service, Nova instances for the cluster roles, Ironic for bare-metal data nodes, and the Ceph S3 tier from XaasIO Block and Object Storage for cold data and backups.
XaasIO Kubernetes
Trino, Superset, OpenSearch, Milvus, JanusGraph services, the Context and Agent services and the model gateway as Helm-deployed services on the XaasIO Kubernetes Platform, with the Kerberos realm, Ranger policies and directory connection applied from the cluster configuration.
Air-gapped sites run every component inside the boundary with local package, chart, image and model repositories; updates arrive as signed offline bundles on the XaasIO release cycle.
Console included, platforms optional
Start with the Lake. Add Private Models and More Cloud Services When You Need Them.
XaasIO AI Lake includes the XaasIO Hyperscaler console for data services, with Ambari as the cluster management tool underneath. Hosting private models on your own GPUs is the job of the XaasIO AI Factory Platform. Delivering Compute, RDS and other XaasIO modules through the same console needs the XaasIO Hyperscaler Platform.
Included
Included with XaasIO AI Lake
- XaasIO Hyperscaler console for data services, Ambari for the cluster
- The open data platform: storage, streaming, processing, SQL and governance
- Search, vector database and knowledge graph extensions
- Context Lake and Agent Lake with the model gateway
- Tenants, queues, projects, workspaces, roles and single sign-on
Optional
Add XaasIO AI Factory Platform
- Private model hosting on your GPUs
- Token-metered model APIs through the AI Token Factory
- Training and fine-tuning on lakehouse data
- Hybrid routing to approved public providers by policy
Optional
Requires XaasIO Hyperscaler Platform
- Additional XaasIO modules in the same console
- Multi-service catalogs and tenant management
- FinOps, metering, billing and payments
Modules delivered through XaasIO Hyperscaler Platform
XaasIO Compute
OpenStack private cloud for the cluster’s virtual machines, bare-metal nodes, networks and persistent storage.
XaasIO Block and Object Storage
Ceph block volumes and the S3-compatible object tier for cold data, backups and index segments.
XaasIO RDS
Managed PostgreSQL, MySQL, MariaDB and MongoDB, and a CDC source for the lake.
XaasIO AI Factory + HPC
Model training, fine-tuning, LLM and SLM development and HPC computing.
XaasIO AI Token Factory
Governed inference and token-metered model APIs, the model gateway’s private endpoint.
XaasIO MLT
Monitoring, logging and telemetry for the lake, its pipelines, retrieval and agents.
These modules are optional and are not required to deploy or operate XaasIO AI Lake. Adding them to the console requires the XaasIO Hyperscaler Platform.
Migration Paths and Support Packs
Separately scoped support for assessment, migration, cutover and managed operations, with a path for every starting point:
Cloudera CDP, CDH and HDP, TDP and legacy Hadoop: the same Apache APIs, so Hive tables, HBase applications, Spark jobs, Ranger policies and Atlas metadata carry over
Azure HDInsight and Amazon EMR: the same open stack on your own hardware, with ADLS and S3 data landed on Ozone or Ceph S3
Databricks, Snowflake and cloud data warehouses: Iceberg tables on the platform with Spark and Trino, one workload group at a time
Proprietary search, SaaS vector databases and graph services: re-indexed from the governed tables into the AI extensions, with policies attached from the start
Software supply chain assurance
Every supported XaasIO AI Lake release includes an upstream SBOM and XaasIO release SBOM, covering components, container images, dependencies, licenses, vulnerabilities, provenance, and signed artifacts. SPDX or CycloneDX outputs support SOC 2-aligned controls and audit evidence.
Upstream SBOM
XaasIO release SBOM
Components and container images
Licenses and vulnerabilities
Provenance and signed artifacts
SPDX or CycloneDX output
SOC 2-aligned audit evidence
Existing tables, jobs, pipelines, policies, indexes and prompts are reviewed for reuse before cutover
SPDX and CycloneDX outputs are available for every supported release
Frequently asked questions
What is XaasIO AI Lake?
An open-source data platform for analytics and AI, delivered and supported as one product. Its base is an enterprise distribution of the Apache Hadoop ecosystem: HDFS and Ozone storage, YARN, Spark, Flink, Tez and MapReduce, Hive, HBase, Phoenix and Trino, Iceberg tables, Kafka, NiFi, Oozie and Airflow, Kerberos, Ranger, Knox and Atlas, managed with Ambari. On that base XaasIO adds four AI extensions that reuse the same components: search with a vector database, a knowledge graph, the Context Lake, which assembles governed context for models and agents, and the Agent Lake, which manages agents, tools, prompts and memory. It runs in your data center, sovereign cloud or provider environment, and XaasIO designs, validates, supports and can operate it.
Is XaasIO AI Lake a Hadoop distribution? How does it relate to ODP and Cloudera?
Yes. The base of XaasIO AI Lake is an enterprise Apache Hadoop distribution built from current upstream releases of the Apache projects, with the bug fixes needed to keep them interoperable in production, packaged, hardened and supported by XaasIO under its release cycle. It is the same open stack that open data platform distributions such as ODP ship, and the same project set that Cloudera CDP, CDH and HDP estates run, so Hive tables, HBase applications, Spark jobs, Ranger policies, Atlas metadata and Kerberos integration carry over with version-upgrade testing rather than a rewrite. What XaasIO adds on top is the AI extensions, the XaasIO Hyperscaler console and the validation for sovereign, air-gapped and service-provider deployments.
How are the AI features extensions of the stack rather than a second platform?
Each extension reuses what the platform already has. Spark jobs on YARN compute embeddings and extract entities; Ozone or Ceph S3 holds vector index segments; HBase is the storage backend of the knowledge graph through JanusGraph; Hive, Trino, Kafka and Flink feed the Context Lake; HBase and Iceberg tables hold agent registries and memory; Kerberos and Knox authenticate every call; Ranger tags and policies travel from the source tables to every chunk, vector, edge, context record and agent action; Atlas lineage extends from table to index to answer. There is no separate data copy, identity or audit trail for AI.
Which open-source components does the platform use?
The open data platform: Apache Hadoop (HDFS, YARN, MapReduce), Apache Ozone, Apache Spark, Apache Flink, Apache Tez, Apache Hive, Apache HBase, Apache Phoenix, Trino, Apache Iceberg, Apache Kafka, Apache NiFi and NiFi Registry, Apache Oozie, Apache Airflow, Apache Livy, Apache Zeppelin, Apache Superset, Apache Ranger and Ranger KMS, Apache Knox, Apache Atlas, Kerberos with FreeIPA, ZooKeeper and Apache Ambari, with a Ceph S3 tier from XaasIO Block and Object Storage. The AI extensions: OpenSearch for keyword and hybrid search, Milvus for vector indexes, JanusGraph on HBase for the knowledge graph, Temporal for durable agent workflows, and the XaasIO Context, Agent and gateway services. Components are pinned per release and listed in the release SBOM.
How is an AI Lake different from a data lake or a lakehouse?
A data lake stores raw data cheaply. A lakehouse adds open table formats, transactions, schema evolution and SQL over that storage, which the platform provides with Hive and Iceberg tables. XaasIO AI Lake keeps both and adds what models and agents need: semantic search through a vector database, relationship knowledge through a knowledge graph, identity-aware context assembly through the Context Lake, and governed agents and memory through the Agent Lake. Every derived asset, from a chunk or vector to a graph edge or memory, stays linked to its source, version, policy and retention state.
Is the platform an AI engine? Where do the models run?
No. The platform is a governed, auditable, versioned and sovereign data foundation, and the models stay outside it as replaceable consumers. Spark MLlib runs classical machine learning on the cluster. Private language and embedding models run on your GPUs through the XaasIO AI Factory Platform, whose AI Token Factory is the private endpoint behind the model gateway; where policy allows, the gateway can route to approved public providers, with every request logged and metered. The lake owns the data, the context and the audit trail, whichever model is used.
Why does the platform include both a vector database and a knowledge graph?
They answer different questions. A vector database finds what is similar in meaning, which suits document search, retrieval-augmented generation and recommendations. A knowledge graph records how entities are connected and what depends on what, which suits impact analysis, root cause, compliance and multi-step planning. XaasIO AI Lake scores keyword, vector and graph results together, filters them by tenant, purpose and Ranger policy, and returns them with citations, so applications get both similarity and relationships from one retrieval call.
Which search, vector and graph engines are used, and why these?
OpenSearch provides keyword and hybrid search and is also the index backend of the knowledge graph; XaasIO MLT already runs it, so one engine serves both. Milvus is the vector database: it stores its index segments on S3-compatible object storage, which means Ozone or the Ceph S3 tier, and scales to very large vector estates; Qdrant and pgvector are options for small deployments. JanusGraph is the graph database because it runs on the platform’s own HBase, the same storage backend Apache Atlas uses for its metadata graph. The validated selection for each deployment is documented in its design.
What is the Context Lake?
A governed service that assembles the minimum necessary evidence for a specific user, task, model and policy. It draws on Hive and Trino SQL, OpenSearch, Milvus and the graph, and on live state from Kafka and Flink, filters by identity and Ranger entitlement, ranks and compresses the evidence, attaches provenance and freshness timestamps, applies a token or size budget and expires context that no longer applies. Applications and agents call one Context API rather than integrating each store separately.
What is the Agent Lake?
A registry and runtime that treats agents, tools, prompts and memory as governed services. Each agent has an owner, a Kerberos identity, explicit Ranger permissions, a version and an evaluation record. Runs execute in a harness with sandboxed tools, human approval for sensitive actions, tracing, rollback and a kill switch. The catalog, SQL, search and graph are exposed to agents as tools through the Model Context Protocol, so an assistant can discover and query data in natural language without a path around the policies. Memory is classified, scoped, aged, correctable and deletable, and promotion to shared organizational memory is explicit and auditable.
What does Apache Iceberg add to a Hadoop platform?
Open tables with ACID transactions, schema evolution, hidden partitioning, snapshots and time travel on HDFS and Ozone, readable and writable from Hive, Spark, Flink and Trino. For AI it gives every training run, embedding batch and evaluation a snapshot identifier, so results can be reproduced and a record’s presence in a training set can be established, and it supports row-level deletes for data subject erasure with the deletion recorded in Ranger audit.
Is there S3-compatible object storage in the platform?
Yes, twice. Apache Ozone is the S3-compatible object store inside the Hadoop stack, used for lake zones, vector index segments and model artifacts beside HDFS. The Ceph S3 tier from XaasIO Block and Object Storage adds versioned, replicated object storage for cold data, backups and multisite copies. Spark, Trino, NiFi and the AI extensions address both through the S3 API.
What runs on Kubernetes?
The cluster roles, HDFS, Ozone, YARN, HBase and the services managed by Ambari, run on bare metal or virtual machines from XaasIO Compute or Virtualization. The cloud-native services run as Helm-deployed services on the XaasIO Kubernetes Platform: Trino, Superset, OpenSearch, Milvus, the JanusGraph services, the Context and Agent services and the model gateway, with the Kerberos realm, keytabs, Ranger policies and directory connection applied from the cluster configuration so both halves share one security model.
How is security and governance handled?
Kerberos authenticates users and services, Knox provides the perimeter and single sign-on with SAML or OIDC, and Ranger decides access with dataset-level grants, column masking, row filters and tag-based policies, with every decision audited. Atlas records metadata, lineage and classifications. HDFS transparent encryption with Ranger KMS and TLS protect data at rest and in transit. The AI extensions inherit all of it: the same identities, the same tags on every derived asset, the same audit trail for retrieval, context and agent actions, forwarded to XaasIO MLT or your SIEM.
Does our data leave the environment? Are public AI services used?
No, by default. Data, indexes, graphs, context and memory stay inside the deployment boundary, encrypted with keys you control. The model gateway routes to private models first and uses approved public providers only where an explicit policy allows it, with the request logged and metered. Prompts, context and responses are retained according to the policy you set per application.
Can XaasIO AI Lake run in an air-gapped environment?
Yes. The fully air-gapped pattern runs every layer inside the boundary: local package, image, chart and model repositories, the cluster, Ozone and Ceph S3, catalogs, processing, search, vector and graph stores, context and agent services, identity, policy and observability. Updates arrive only as signed, scanned and approved offline bundles on the XaasIO release cycle.
Can a cloud service provider offer it as a multi-tenant managed service?
Yes. Tenants, projects and workspaces are isolated at the storage, YARN queue, namespace, index, graph, context and agent levels, with per-tenant keys, quotas and audit trails. Usage events are produced for every service. With the XaasIO Hyperscaler Platform, providers publish the data, retrieval, knowledge and agent services in a self-service catalog and rate, invoice and settle them alongside Compute, storage and databases.
Is the XaasIO Hyperscaler console included?
Yes. XaasIO AI Lake includes the XaasIO Hyperscaler console for the lake services: tenants, projects, workspaces, data sources, tables, indexes, graphs, context applications, agents, access and usage, with Ambari as the cluster management tool underneath. Extending the same console to Compute, Block and Object Storage, RDS, AI Factory, MLT and other modules, multi-service catalogs, billing and payments requires the XaasIO Hyperscaler Platform.
Are other XaasIO modules required?
No. The platform runs on bare metal or virtual machines from XaasIO Compute or Virtualization, or on your own servers, with the XaasIO Kubernetes Platform for the cloud-native services. XaasIO Block and Object Storage, RDS, AI Factory and MLT are optional and integrate directly: Ceph provides the S3 tier, RDS databases become change-data-capture sources, AI Factory hosts private models and MLT observes the cluster, pipelines, retrieval and agents.
Who operates XaasIO AI Lake?
You choose: customer-operated with XaasIO support, co-managed, fully managed by XaasIO, or a dedicated XaasIO SRE Pod. Every deployment includes validated architecture, documented release and upgrade policy, SBOMs for every release and 24×7 support options.
Apache, Hadoop, HDFS, Ozone, YARN, Hive, HBase, Phoenix, Tez, Spark, Flink, Kafka, NiFi, Oozie, Airflow, Livy, Zeppelin, Superset, Ranger, Knox, Atlas, Ambari, Iceberg, ZooKeeper, Trino, Ceph, OpenSearch, Milvus, JanusGraph, Qdrant, Gremlin, Temporal, Kerberos, FreeIPA, Microsoft, Active Directory, Azure, HDInsight, Amazon, EMR, Cloudera, ODP, Clemlab, Acceldata, Databricks, Snowflake and other product names are trademarks of their respective owners and are named for identification and comparison only. XaasIO is not affiliated with, endorsed by or sponsored by these organizations.
Build what the world depends on.
Planning a cloud for government, financial services, pharmaceuticals, healthcare, energy, transport, industry, service-provider operations, or private AI? Work with XaasIO to define the architecture, security boundary, and operating model your environment requires.