QuickFlows

A QuickFlow is the quickest useful thing in VegaFlow. It connects one source to one destination, holds the discovered stream selections and sync behavior, runs on a managed cluster and can have a schedule.

Use a pipeline when the work needs a general DAG, code project or model/task graph. Do not turn a two-system replication job into a pipeline merely to look sophisticated.

Definition

A QuickFlow definition includes:

• name and status;
• cluster ID;
• source and destination connection references;
• selected source streams and their destination configuration;
• one inline schedule in the standard create/update flow;
• optional run_now and validate_only behavior.

Source and destination connection IDs must differ. The connector ID and type in each reference must match the stored connection row. The creator needs create permission plus use permission on the cluster and both connections.

For a first run, follow the VegaFlow quickstart to validate both connections and prepare the cluster.

Stream and column selection

Use the saved connection's discovered catalog to select valid stream and column identifiers.

Identifiers cannot be blank. Apply column rules per selected stream where the schema supports them.

Source modes

Snapshot reads the selected source as a complete point-in-time dataset. Omit cursor strategy, cursor fields, resume token and initial-snapshot controls in this mode.

Incremental cursor requires at least one cursor. A single cursor can use the default single strategy; multiple cursors require an explicit strategy. Use non-empty cursor fields, valid ordering and tie-breakers that make progress deterministic.

Use only modes and fields supported by the current connector schema and API contract. Resume-token support depends on the connector and runtime path.

Schema evolution

When a schema-evolution configuration is supplied, its enabled value is required. Enable it when the destination should accept source changes; otherwise keep destination expectations explicit. Review mapping and cursor compatibility when a source schema changes.

Create or update

POST /api/v1/vegaflow/orgs/{org_id}/workspaces/{workspace_id}/vegaflow/quickflows
Content-Type: application/json

{
  "name": "orders-to-lakehouse",
  "cluster_id": "clu_01...",
  "source": {
    "connection_id": "con_source...",
    "connector_id": "postgres",
    "connector_type": "postgres"
  },
  "destination": {
    "connection_id": "con_destination...",
    "connector_id": "postgres",
    "connector_type": "postgres"
  },
  "streams": [
    {
      "source_namespace": "public",
      "source_name": "orders",
      "destination_namespace": "raw",
      "destination_name": "orders",
      "sync_mode": "incremental",
      "cursor_field": ["updated_at"],
      "primary_key": [["id"]]
    }
  ],
  "schedule": {
    "schedule_kind": "cron",
    "cron_expression": "*/10 * * * *",
    "timezone": "UTC",
    "enabled": true
  },
  "run_now": false
}

The create route upserts by name in the workspace and writes an immutable version snapshot. If the inline schedule is null, it removes the existing inline schedules. run_now=true queues a run after the definition is committed.

List and detail

GET /quickflows is deliberately a summary endpoint. It returns IDs, name, status, connection and cluster IDs, latest run ID/status and the next enabled schedule time. It does not load secret-bearing connection config, every stream, attempts or the full schedule graph.

Use GET /quickflows/{quickflow_id} for the complete model. This separation keeps a workspace with hundreds of flows from making hundreds of nested reads just to draw a table.

Run lifecycle

queued → dispatching → running → succeeded
                         ├──────→ failed
                         └──────→ canceling → canceled

POST /quickflows/{quickflow_id}/runs first verifies the flow, permission and cluster readiness, then ensures the flow has no other active run. It creates the parent run in queued. Stream runs are created later during dispatch preparation, after the current source catalog is resolved.

VegaFlow moves the run to dispatching, verifies the selected compute environment and creates one independently visible stream run for each selected source stream. Retries create new attempts under that stream run; they do not erase the failed attempt or replay successful streams without an explicit request.

Cancel and retry

DELETE .../quickflows/{quickflow_id}/runs/{run_id}
POST   .../quickflows/{quickflow_id}/runs/{run_id}/retry-failed-streams

The current DELETE route means cancel, not historical deletion. Terminal runs are left unchanged. A queued run without a workflow binding can be cancelled directly; active work enters canceling while the durable workflow is asked to stop.

Retry-failed-streams creates new attempts only for failed stream work, which is useful when a large flow moved 99 tables and one changed schema. Confirm the source and destination are still compatible before retrying.

Schedules

QuickFlows support list, create, get, update and delete schedule endpoints under /quickflows/{quickflow_id}/schedules. Store time zones explicitly. A cron without a time zone eventually becomes somebody's daylight-saving incident.

Prove the configuration with a successful manual run before adding recurrence. Pause and resume preserve the schedule definition while controlling future launches. See schedule operations.

Update or delete a flow

Review active schedules before changing connections, cluster, selection or cursor settings; those changes affect future runs. Remove or retire dependent schedules before deleting the QuickFlow. Collection and instance permissions govern these operations.

Versions

List immutable definition versions with:

GET .../quickflows/{quickflow_id}/versions
GET .../quickflows/{quickflow_id}/versions/{version_id}

Run history points at the version that was executed. Editing a definition cannot rewrite what a previous run meant.