Platform architecture
Control plane and data plane architecture for multi-cloud data and AI
Vegalake separates the services that decide from the compute that touches your data. A global control plane manages identity and accounts. A control plane in each region coordinates every product. A data plane in the same cloud and region, on AWS, Azure, GCP or your own infrastructure, runs queries, connectors and pipelines inside an isolated boundary for each workspace, on storage encrypted with that workspace’s own key.
- The control plane decides. The data plane executes.
- Orchestration, permissions and metadata live in the control plane. Record data moves only through the data plane and workspace storage.
- Every workspace is its own boundary
- Each workspace gets its own encryption key, storage, access role and table catalog. Its compute runs in isolated microVMs that serve no other workspace.
- One permission model, checked everywhere
- Grants are defined once and replicated to every service, which checks them locally on each request. AI agents act with the permissions of the person they work for.
The system at a glance
Four tiers, one platform
The edge sends sign-in and account traffic to the global control plane and product traffic to the regional control plane that hosts your workspace. The regional control plane dispatches work to the regional data plane, which reads and writes workspace storage. Each workspace runs in the cloud and region you choose, on AWS, Azure or GCP, and regions are independent, so data and compute stay where you put them. The data plane can also be self-hosted in your own cloud account or data center while Vegalake operates the control plane.
- Vegalake service · metadata and control
- Touches customer data
- Outside system
- Instruction · carries no record data
- Data or request
Control plane
The services that decide
The control plane owns identity, permissions, orchestration and metadata. It is split in two. The global tier holds what is the same everywhere: who you are and which organizations and workspaces you belong to. The regional tier runs next to your data and coordinates every product for the workspaces in that region.
| Component | What it owns | How it works |
|---|---|---|
| EdgeGlobal | Every browser and API request enters here. | A global CDN with a web application firewall. A path router sends account and sign-in paths to the global tier and workspace paths to the regional host for that workspace. |
| IdentityGlobal | Sign-in, single sign-on, two-factor authentication, SCIM provisioning, API keys. | Acts as an OAuth and OIDC issuer and signs short-lived tokens with rotating keys. Only the Account service may call its internal API; browsers and apps never reach it directly. |
| AccountGlobal | Organizations, users, groups, roles, service principals, workspaces, clouds and regions. | The source of truth for every grant. It provisions and tears down workspaces, stores AI provider settings, and streams each permission change to every region. |
| Session gatewayGlobal and regional | Browser sessions. | Runs the OAuth flow server-side with PKCE. Tokens are stored encrypted on the server; the browser holds only a session cookie and a CSRF token. Each workspace gets its own regional session, created silently after the first sign-in. |
| VegaFlowRegional | QuickFlow ingestion, VegaPipelines, connections, triggers, approvals, monitors and alert rules. | Plans each run and dispatches the work to the data plane. Resolves connection secrets and mints storage credentials scoped to a single run. |
| VegaDurableFlowRegional | Durable execution for every workflow on the platform. | A multi-tenant workflow engine that records executions, attempts and schedules, so a run survives restarts and resumes where it stopped. |
| VegaDB serviceRegional | Sessions, worksheets, query history, loads, publications and the table catalog. | Checks permission, then admits each statement to an interactive or pipeline queue before it reaches a warehouse. Creates the catalog and default warehouse for every new workspace. Private catalog option: the table catalog can run entirely inside the data plane, for customers who want no table or column names held in the control plane. |
| Vega AIRegional | The assistant and agents that work across every product. | The agent engine calls your chosen model through a model gateway and reaches products through one set of platform tools, the same tools external AI clients use over MCP. |
| VegaGraphRegional | The organization graph: data catalog, software service catalog, metrics, lineage and classification. | Once graph mirroring is turned on, every product mirrors its objects and lineage into the graph. It stores metadata only, never data values or secrets. |
| VegaLensRegional | Datasets, charts, dashboards and packs. | Stores definitions only. Every chart runs as a VegaDB query with the permissions of the person viewing it. |
| SecretsRegional | Connection credentials, personal vaults and model provider keys. | Encrypts each workspace’s secrets with that workspace’s own key. Code running in a pipeline never sees a secret value. |
| OrchestratorRegional | Where compute runs and what it may reach. | Reconciles the desired state of warehouses, connector pods and sandboxes, and applies network policy and approved private destinations. |
Data plane
The compute that touches your data
The data plane runs in the same cloud and region as its workspace. Each organization’s compute runs in its own namespace, inside lightweight virtual machines. Trusted platform processes hold the credentials; the code they run does not.
| Component | What it does | Isolation |
|---|---|---|
| VegaDB engineWarehouses | Massively parallel SQL over managed tables, streaming tables and outside sources. Every workspace has a default warehouse that stops when idle; dedicated warehouses scale out for heavy or steady workloads. | Each warehouse serves one workspace and authenticates to the control plane with its own certificate. Each session is pinned to its own engine connection. |
| Connector runtimeVegaFlowConnect | Reads from and writes to databases, SaaS apps, files, streams and APIs. Micro-batches are staged in workspace storage and loaded from there. | Runs with credentials scoped to the current run’s storage prefix. Connections to private networks run on dedicated clusters with approved destinations. |
| Pipeline sandboxesVegaPipelines | Run Python transformations, assets and AI agents written with the VegaPipelines SDK. | Each sandbox is a microVM bound to a single workspace and never reused. Secrets reach it only through a narrow broker, for the IDs bound to that run. |
| Dedicated clustersVegaFlow | Long-lived nodes for workspaces with steady load or private network access. | Trusted node processes run alongside disposable sandboxes for user code. |
| Vega Tables engineCollaborative data | Serves live records, formulas and realtime collaboration for every base. | Record values live in workspace storage, rebuilt from each base’s own log. |
Where the data plane runs
- ManagedOn AWS, Azure or GCP, operated by Vegalake
- Pick a cloud and region for each workspace. Vegalake runs the data plane there, and workspaces in different clouds sit side by side in one organization.
- Self-hostedIn your own cloud account
- The data plane runs in your AWS, Azure or GCP account, so compute, storage and keys stay under your control. It connects out to the Vegalake control plane, which needs no inbound access to your network.
- Self-hostedOn your own infrastructure
- Run the data plane on Kubernetes or bare metal in your data center. The same products, permissions and control plane work across every deployment.
Tenancy and isolation
What every workspace gets
An organization holds workspaces, and each workspace lives in one cloud and region. When you create a workspace, the platform builds its boundary in this order. If a step fails it retries, and if it cannot finish it rolls back everything it created. Deleting a workspace removes its grants at once and tears down its resources in the background.
- Encryption keyA key used only by this workspace. Its storage and its secrets are encrypted with it.
- StorageDedicated object storage in the workspace’s cloud for warehouse data and for pipeline and connector staging, with public access blocked.
- Access roleA cloud role that only this organization’s compute can assume, limited by a permissions boundary to the workspace’s own buckets and key.
- AddressA workspace hostname in its region, so every request lands next to its data.
- Catalog and warehouseA table catalog for the workspace and a default warehouse that starts on demand.
Data flow
How data moves
Two paths carry almost all customer data: ingestion into VegaDB, and queries out of it. In both, the control plane coordinates the work while the records stay in the data plane and workspace storage.
Ingestion: a QuickFlow run
Query: from a worksheet, dashboard, API or BI tool
- Graph syncMetadata flows into one graph
- Tables, pipelines, dashboards and bases mirror their current state and lineage into VegaGraph, and a nightly reconcile keeps it complete. Only metadata crosses: SQL text and data values never do. Mirroring is opt-in, so with the catalog kept in the data plane, table and column names can stay there too.
- AIAgents work through the same doors as people
- The agent loop runs in the regional control plane and calls the model provider you configure, using a key held in your workspace vault. Every tool call carries a grant signed by Account, so the agent can do only what its user can do, and writes wait for approval.
Permission flow
How permissions flow
Identity proves who is calling. Account decides what they may do and replicates that decision to every region. Each product service checks the grant locally, on every request, before any work reaches the data plane.
The permission model
A principal (a user, group, role or service principal) holds a permission on an object type at one of three scopes. Grants inherit downward, from organization to workspace to feature to a single resource. Whoever creates a resource receives full rights on it, and granting access to others requires the permission-management right at that level or above.
| Scope | Applies to | Example |
|---|---|---|
| Organization | Every workspace in the organization. | A platform admin role that manages users, groups and workspaces. |
| Workspace | Every resource of that type in one workspace. | Analysts may use the workspace and read every VegaLens dashboard in it. |
| Resource | One named object. | A finance group may edit one Vega Tables base and run one pipeline. |
- PeopleSessions without tokens in the browser
- Sign-in uses your identity provider over SAML or OIDC, with SCIM provisioning, two-factor policy per organization, device trust and breached-password checks. Signed tokens stay on the server.
- Apps and automationService principals and API keys
- Programs authenticate as service principals with API keys and receive grants like any other principal, so automation is governed by the same model as people.
- ServicesSigned calls between components
- Services identify each other with signed, purpose-bound credentials. Loads into VegaDB, graph updates and event publishing each carry their own signed capability, limited to one organization and workspace.
- ComputeCredentials that expire with the run
- Data-plane workloads prove their identity to the control plane before they receive anything. Their cloud credentials are short-lived and limited to the run’s own storage prefix.
VegaDB
Why VegaDB is faster
Most analytical databases were designed around hardware assumptions from a decade ago. VegaDB was designed around the hardware that exists now. At startup it detects what the machine offers, whether bare metal or a cloud VM, and adapts. If RDMA is available, it uses RDMA. It takes control of memory, threads and networking from the operating system so that every core, cache line and network link does useful work.
| The old assumption | What modern hardware does | What VegaDB does |
|---|---|---|
| Memory is the fast path. Disk is slow, so read sequentially and load everything into RAM. | NVMe drives read tens of gigabytes per second with microsecond latency, close to RAM bandwidth. | Treats compressed columns in memory as a cold cache and keeps the hottest 20% decompressed. Its format is tuned for decompression speed over compression ratio, because decompression, not I/O, is the real limit. |
| The network is slow. Avoid shuffling data between nodes. | 100 to 200 Gbps networks and RDMA are available in the cloud for a few dollars an hour. TCP alone can saturate a CPU at about 3 Gbps. | Moves data over RDMA on InfiniBand or RoCE where present, measured above 150 Gbps on cloud VMs with the CPU idle. Uses its own zero-copy RPC layer over TCP everywhere else. |
| Analytical work is bound by disk and network I/O. | An L1 cache read takes 1 to 2 ns; RAM takes over 100 ns. A core can scan about 4 billion values a second and hash about half a billion. SIMD multiplies that. | Breaks work into blocks that fit in CPU cache and runs vectorized SIMD operators over them, so data stays in cache from operator to operator. |
| The operating system’s scheduler and allocator are good enough. | Machines are networks of NUMA nodes. The OS moves threads across cores and sockets, flushing caches, which can cost 10 to 100 times in throughput. | Pins one thread to each core, keeps memory local to its NUMA node, allocates up front and uses its own allocator for the rest. |
- LanguageWritten in Rust and C++
- Managed runtimes trade memory for speed and become hard to tune above 64 GB. Servers now carry a terabyte of RAM. Rust and C++ give VegaDB full control over every allocation and thread.
- ThreadsOne pinned thread per core
- Threads never migrate between cores or NUMA nodes, so caches stay warm through memory and networking. Work is divided so that every core stays busy, even on a small query.
- MemoryAllocated before the query runs
- Each operator’s memory is reserved when the query starts, which keeps latency predictable. 300 MB per core is enough for full-speed processing; the rest of RAM serves as data cache. A custom allocator replaced mimalloc and cut fragmentation significantly.
- NetworkingZero-copy from memory to wire
- The in-memory format is also the wire format, so nodes operate on received data without deserializing it. Shuffles run at about 6 GB per second per core, which is why group-by and join workloads are so fast.
- ExecutionA pipelined, non-blocking engine
- A cost-based optimizer builds a DAG of operators. Data flows between them in cache-sized blocks with built-in backpressure. Join shuffles send each block as soon as it is ready, holding latency in single-digit milliseconds, and join strategies are chosen at runtime from the actual data.
- StorageIndexes and statistics built on load
- Columnar data is split into multi-level partitions with fine-grained indexes and statistics on every column, plus a sparse key index that fits in memory. Updates and deletes are written beside the data and merged on read, then compacted automatically. Time travel comes for free.
Built for real time and many users
- Streaming tables
- A hybrid row and column format commits each write in single-digit milliseconds and ingests a million rows per second per core. VegaDB compacts it to columnar automatically, so queries stay as fast as on batch tables, with no separate Flink or Beam pipeline to run.
- Real-time materialized views
- Views with complex aggregates update as their sources change, continuously, near real time or on a schedule, all in SQL. Their resources can be capped so they never slow other work.
- Workload management
- Built for hundreds of users running thousands of concurrent queries. Short queries take priority, long ones get their own allocation, admins set per-user limits, and a query hint can raise a business-critical job to the top.
- Warm restarts
- VegaDB saves its cache metadata and rebuilds the cache when a warehouse starts. The first queries of the day run within 10 to 20% of fully warm speed. Commercial serverless warehouses typically run 50 to 60% slower until their caches refill.
- Fixed, serverless or both
- Run warehouses on fixed machines with local NVMe for heavy steady load, or serverless on object storage for scheduled work, often at a tenth of the cost. Mix both in one workspace.
- ACID, with familiar SQL
- Full ACID transactions at read-committed isolation. Readers never block each other, and bulk appends never block each other. Speaks ANSI SQL and Spark SQL, and serves clients over Apache Arrow Flight SQL, so BI tools and applications receive columnar results end to end through standard Flight SQL drivers.
Benchmarks
The numbers
Results from the VegaDB white paper, using public benchmarks against widely used engines. The TPC-H comparison ran on the same instances as Databricks Photon, so the speed difference is also the price-performance difference.
- faster on TPC-H SF1000 total runtime than Databricks Photon, on the same instances
- 14.8×
- faster H2O group-by on 1 billion rows than Databricks; 9.5× faster than ClickHouse
- 21.0×
- faster H2O join on 100 million rows than Databricks; 16.7× faster than Snowflake
- 23.7×
- to load about 1 TB of TPC-H data with statistics built, against 2.24 min for Databricks
- 56 s
H2O group-by
1 billion rows, 9 columns. Seconds, lower is faster.
H2O join
100 million rows, 7 columns. Seconds, lower is faster.
TPC-H, scale factor 1000: VegaDB speed-up over Databricks Photon
Times faster per query on the same instances. The dashed line marks the total across all 22 queries.
Show TPC-H timings
| Query | Databricks Photon (ms) | VegaDB (ms) | Speed-up |
|---|---|---|---|
| Q1 | 3,818 | 816 | 4.7× |
| Q2 | 2,956 | 118 | 25.1× |
| Q3 | 3,988 | 143 | 27.9× |
| Q4 | 4,600 | 193 | 23.8× |
| Q5 | 6,210 | 252 | 24.6× |
| Q6 | 940 | 53 | 17.7× |
| Q7 | 4,530 | 310 | 14.6× |
| Q8 | 5,730 | 262 | 21.9× |
| Q9 | 8,668 | 479 | 18.1× |
| Q10 | 3,530 | 250 | 14.1× |
| Q11 | 2,295 | 94 | 24.4× |
| Q12 | 3,454 | 121 | 28.5× |
| Q13 | 3,247 | 446 | 7.3× |
| Q14 | 720 | 85 | 8.5× |
| Q15 | 1,450 | 110 | 13.2× |
| Q16 | 1,690 | 48 | 35.2× |
| Q17 | 7,030 | 395 | 17.8× |
| Q18 | 1,093 | 754 | 1.4× |
| Q19 | 4,079 | 168 | 24.3× |
| Q20 | 2,020 | 121 | 16.7× |
| Q21 | 13,270 | 617 | 21.5× |
| Q22 | 2,880 | 119 | 24.2× |
| Total | 88,198 | 5,954 | 14.8× |
Show H2O timings
| Product | Group-by, 1B rows (s) | Join, 100M rows (s) |
|---|---|---|
| VegaDB Distributed, in-memory | 27 | 14 |
| Polars Single node, in-memory | 143 | 43 |
| ClickHouse Distributed, in-memory | 256 | 159 |
| Snowflake Distributed, in-memory | 513 | 234 |
| Databricks Distributed, in-memory | 568 | 332 |
How these were run. VegaDB ran on its default settings, with no tuning for TPC-H and no separate statistics step, queried right after loading. The Databricks setup was tailored for the benchmark and ran ANALYZE TABLE before the queries. VegaDB loaded the roughly 1 TB dataset in 56 seconds, against 2.24 minutes for Databricks, and many VegaDB cores sat idle during the run while every Databricks executor ran at full load. Newer results publish with the Vegalake launch, together with a public benchmark repository so anyone can replicate and verify every posted result.
Why it matters
What this architecture gives you
- Your records stay on the data path
- Micro-batches move from source to workspace storage to engine. The control plane carries instructions, references and metadata, which keeps the surface that can see your data small.
- Isolation is the default
- A dedicated key, storage, role and catalog for every workspace, and microVM sandboxes that never serve two workspaces. Isolation comes from the architecture; no setting has to be remembered.
- One permission model for people, apps and agents
- Grants are defined once in Account and enforced in every product. An AI agent can never do more than the person it works for.
- Compute that scales on its own
- Warehouses start on demand, stop when idle and scale out without touching the control plane. Heavy pipeline work and interactive dashboards never compete in the same queue.
- A faster engine is a smaller bill
- 14.8 times faster on the same instances means the same work on a fraction of the hardware, or far more work on the same budget.
- Your cloud, your region
- Run each workspace on AWS, Azure or GCP, in the region your data and regulators require. Moving between clouds changes where the data plane runs, not how you work.
- Self-host the data plane
- Keep compute, storage and keys in your own cloud account or data center, while Vegalake operates the control plane for you.
- Products that already know each other
- Every product shares its metadata and lineage through VegaGraph, so lineage, triggers and AI context work across the platform with no integration to build.
Vegalake / platform architecture