Apache SeaTunnel
Apache SeaTunnel is a distributed integration platform that runs batch, streaming, and change-data-capture pipelines between databases, message queues, data lakes, lakehouses, and SaaS applications without hand-rolled scripts. Pipelines are declared once in a configuration file and executed on your choice of the native Zeta engine, Apache Flink, or Apache Spark, so the same job definition travels across engines without connector rewrites. Real-time CDC captures row changes from major databases with a snapshot-first handoff that never locks the source, while checkpoint-backed recovery restarts jobs from the last completed checkpoint with exactly-once semantics. When an upstream table gains a column, schema evolution propagates the change downstream automatically instead of pausing the pipeline for manual surgery. Roughly two hundred connectors cover relational databases, Kafka, object storage, search indexes, vector stores, and business apps, and the multi-table routing keeps every record on the right destination path. Engineers can dry-run configurations against sampled data before a full launch, isolate failures per table, and route bad records to dead-letter queues. Release 3.0.0 adds AI-assisted configuration generation from natural language, Python sources and transforms, and document parsing with text chunking for RAG preparation. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.
PeerDB
Replicate PostgreSQL to data warehouses 10x faster than conventional CDC tools, proven across 400+ companies including AutoNation and LC Waikiki collectively moving 200 TB monthly. The architecture pairs a Rust-based Nexus query layer implementing the PGWire protocol with Go-based Flow workers orchestrated by Temporal. Because Nexus speaks native Postgres wire protocol, any client tool (pgAdmin, psql, Grafana, Tableau, Flyway) can manage replication through standard SQL commands like CREATE MIRROR. Three streaming modes serve different needs: log-based CDC via logical replication slots, cursor-based streaming through timestamp or integer columns, and XMIN-based capture for tables lacking logical replication. Parallel initial load achieves consistent snapshots through transaction snapshotting and CTID range scans, reducing 100+ GB migrations from days to minutes. Native TOAST column handling processes large JSONB payloads and IoT data efficiently without row expansion penalties. Destinations include ClickHouse, Snowflake, BigQuery, Kafka, Azure Event Hubs, Google PubSub, S3, and PostgreSQL with in-flight SQL transformations. Schema change propagation, partitioned table support, and slot growth alerts ensure production reliability. Docker Compose bundles Temporal, catalog Postgres, Flow API, workers, and the Next.js monitoring UI. Deployable on RepoCloud with dedicated VPS resources under AGPL-3.0.