Icehouse Ingest

Every stream, every file,already in Iceberg.

Icehouse Ingest is the managed ingestion layer of Starburst's Icehouse architecture. Stream Kafka topics and land S3 files into query-ready Iceberg tables. No Flink jobs to manage. No custom pipelines. No stitching layers together.

Start Free →Read the Icehouse Overview →

How it works

From source to query-ready, automatically

One managed path for both streaming and file sources. Point Starburst at your data, and it handles the rest.

Kafka & S3

Point at a Kafka topic or an S3 bucket. No connectors to deploy.

Raw table

Every message or file lands in a live Iceberg table in your own storage.

Transform

Schema is inferred and mapped to relational columns automatically.

Query-ready Iceberg tables

Ready to query from Starburst, or any Iceberg-compatible engine.

The comprehensive place for the Iceberg lakehouse

One platform for ingestion, maintenance, and query — no code required

Most vendors solve one piece of the Iceberg lakehouse: a catalog, an engine, or a pipeline. Starburst ingests the data, maintains the tables, and queries the result, all from the same platform, so nothing about ingestion locks you into a catalog, a storage vendor, or a query engine.

Kafka · MSK · Confluent

Amazon S3 files

Icehouse Ingest

Streaming + File IngestSchema ManagementTable Maintenance

Apache Iceberg tables

Any catalog: Glue, Polaris, Unity, Starburst Metastore

Your object storage: S3, ADLS, GCS

Starburst runs ingestion, transformation, table maintenance and query, on tables any Iceberg-compatible engine can also read.

No code

Point and configure, don't pipeline

Connect a Kafka topic or an S3 bucket through Starburst. No Flink jobs, no connectors, no custom code to deploy or maintain.

Zero ops

Serverless from ingest to maintenance

Capacity scales automatically with throughput. No clusters to size, no maintenance jobs to schedule, nothing idle to pay for.

Fully managed

A live pipeline, not a one-time load

Tables update continuously as new events and files arrive, and stay compacted and query-ready without manual intervention.

StarburstApache IcebergMetastore Optionality

Together, that's the Icehouse: a high performance, open, vendor-neutral lakehouse, built on open standards at every layer, including the catalog.

Powerful where it counts, simple where it matters

Enterprise-scale ingestion, without the operational weight

Streaming and file ingest are built to handle serious volume. Running them should not feel that way.

Powerful

Streaming ingest

100GB/s

Benchmarked throughput, Kafka to Iceberg

<30s

Event to query-ready, no micro-batch delay

Exactly-once delivery, never duplicated or dropped

File ingest

20M

Files supported in a single source path

30 min

Continuous polling, no scheduler to manage

JSON / CSV / TSV

Compressed or uncompressed, auto-detected

Simple

Nothing to run, nothing to tune

Point at a topic or a bucket, no pipeline code to write
Schema inferred and evolved automatically as sources change
One control plane for both streaming and file sources
Table maintenance included, no separate compaction jobs
Nothing to do costs nothing — you pay for work, not idle capacity

Challenges we hear about building an Iceberg lakehouse

Getting data into Iceberg shouldn't be a project

Teams that adopt Iceberg run into the same wall: ingestion is the hardest part of standing up a lakehouse, and it is rarely where anyone wants to spend engineering time.

No Iceberg expertise at scale

Most teams don't have deep Iceberg specialists on staff.

Ingestion into Iceberg is too complex to DIY

Hand-rolled Kafka Connect or Flink jobs are fragile and costly to maintain.

Table maintenance is a major operational burden

Compaction, snapshot expiry, and file cleanup need constant tuning.

No visibility into whether it's working

Teams want observability to see the impact of moving to Iceberg.

Two paths, one Iceberg table

Streaming and file ingestion, natively managed

Icehouse Ingest covers both ways data reaches the lake. Point either one at an Iceberg table and Starburst runs the pipeline for you.

Streaming Ingest

Kafka topics into managed Iceberg tables, in real time. Any Kafka-compliant topic — Apache Kafka, MSK, Confluent — with JSON and Avro formats, Schema Registry integration, and AWS PrivateLink support.

Kafka topic → Raw table → Transform → Live table

JSON and Avro message formats

Schema Registry integration with automatic schema migration

Throughput per partition: 1, 2, 4, 8, or 16 MB/s, auto-scales with partition count

Start from latest or earliest offset

Benchmarked to 100GB/s throughput

File Ingest

S3 files into managed Iceberg tables, continuously. Continuous polling every 30 minutes across up to 20 million files in a single source path.

S3 bucket → Raw table → Transform → Live table

JSON, CSV, and TSV file formats

Compressed or uncompressed files

Automatic schema inference from JSON

Raw table to transform table pipeline out of the box

Iceberg-native by design

Native support for the Iceberg spec, not a workaround around it

Schema management, partitioning, sorting, snapshot expiration, and data retention are handled the way Iceberg intends, automatically, for every table Icehouse Ingest creates.

Schema management

Auto-inferred column mapping, Schema Registry integration for Avro topics, and backfill or apply-forward evolution as sources change.

Partitioning

Configure up to 5 partition columns to match how tables are queried. No manual re-partitioning jobs.

Sorting

Sort columns configured alongside partitioning, keeping files organized for fast scans.

Snapshot expiration

Old snapshots are expired automatically as part of continuous, serverless table maintenance.

Data retention

Keep data as long as you need, with configurable retention and rewind to a savepoint from the last 30 days.

Any Iceberg REST metastoreQueryable by any Iceberg-compatible engineOTEL metrics for real-time monitoringAWS PrivateLink supportAutomated maintenance via Icehouse LakeOps

Built for the moments that matter

What teams ingest into Iceberg with Starburst

Fraud detection

Join live event streams with historical behavioral patterns to flag risk before a payment clears.

Operational dashboards

Give ops teams a live view of what's happening right now, without adding refresh load to the warehouse.

Live personalization

Act on what a customer is doing right now by joining clickstream data with catalog and purchase history.

Compliance & audit trails

Land regulated event data in near real time so financial services and audit teams always query the current state.

IoT & equipment telemetry

Stream sensor data from industrial equipment or connected devices to catch failures before they cause downtime.

AI & ML feature pipelines

Keep Iceberg tables fresh and well maintained so feature pipelines and AI agents are always working off current data.

Why not build it yourself

Skip the custom pipeline

Kafka Connect and Flink jobs, or a stitched-together Firehose and Tableflow setup, can get data into Iceberg. They just come with a second system to run.

	DIY pipeline	Icehouse Ingest
Getting data into an Iceberg table	Custom Flink or Connect jobs to build and maintain	Point at a topic or bucket, Starburst runs it
Exactly-once guarantees	Deduping logic you write and own	Built in — no double-counted or dropped events
Table maintenance after ingest	Separate compaction jobs to schedule and tune	Handled automatically by Icehouse LakeOps
Querying the result	Locked to whatever engine wrote the pipeline	Open Iceberg tables, queryable by any engine

Go deeper on Icehouse

Icehouse Overview

The full Icehouse architecture: Starburst, Iceberg, and metastore optionality.

Explore the Icehouse Overview

Icehouse LakeOps

Serverless table maintenance and lake observability for every Iceberg table.

Explore Icehouse LakeOps

See Icehouse Ingest in action

Free on Starburst Galaxy. Stream a topic or point at a bucket, watch the Iceberg table fill in.

Start Free →