Logo
Deploy Now

SaaS Alternative

Apache Airflow

Categories

Data Pipeline ETL Data Engineering MLOps Workflow Automation Data Orchestration Data Observability

Stars

16,045

Forks

2,254

Watchers

154

Developer links

Last Updated: 08-21-2026 12:38 PM (UTC)

Dagster

With nearly 16,000 GitHub stars, 5.7 million monthly PyPI downloads, and 400+ contributors, Dagster is the most widely adopted asset-centric data orchestration platform — replacing task-oriented schedulers like Apache Airflow with a declarative model where every pipeline is defined as Python functions producing data assets such as tables, datasets, machine learning models, and reports. The built-in asset graph provides automatic lineage tracking across your entire data platform, showing exactly how data flows from ingestion through transformation to downstream consumption in a single unified view. Declarative Automation goes beyond cron scheduling with event-driven conditions that intelligently trigger materializations based on upstream freshness, data quality signals, and dependency state. The integrated data catalog auto-generates documentation from asset metadata, ensuring it never drifts out of sync with production. Native first-class integrations connect dbt, Snowflake, BigQuery, Databricks, Fivetran, Airbyte, Spark, Great Expectations, Tableau, Power BI, AWS, GCP, and Azure without custom glue code. The web UI visualizes asset graphs, run history, schedules, sensors, and partitioned materializations with built-in alerting via Slack and PagerDuty. Dagster Pipes enables executing arbitrary code in external environments including Spark clusters, Kubernetes Jobs, and cloud functions. Deploy via Docker Compose on a single VM with separate containers for the webserver, daemon, and code locations, or use official Helm charts for production Kubernetes with K8sRunLauncher scaling each run as an independent Job. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Dagster
Dagster
Dagster
Dagster
Dagster

Benefits

  • Asset-Centric Pipeline Declaration
  • Define pipelines as Python functions producing typed data assets with automatic dependency resolution, integrated lineage, and built-in quality checks that eliminate manual DAG wiring.
  • Unified Observability and Catalog
  • Auto-generated data catalog with always-current documentation, asset health dashboards, freshness monitoring, and proactive failure alerting through Slack and PagerDuty integrations.
  • Native Data Stack Integrations
  • First-class connectors for dbt, Snowflake, BigQuery, Databricks, Fivetran, Airbyte, Spark, and Great Expectations map external tools directly into the unified asset graph.
  • Flexible Deployment Architecture
  • Separate webserver, daemon, and code location containers enable independent scaling and upgrades via Docker Compose for single VPS or Helm charts for production Kubernetes clusters.

Features

  • Declarative Automation
  • Event-driven scheduling beyond cron that triggers asset materializations based on upstream freshness, dependency state, and configurable quality conditions.
  • Asset Lineage Graph
  • Automatic dependency tracking across ingestion, transformation, and delivery with a visual web UI showing data flow, partition status, and materialization history.
  • Dagster Pipes
  • Execute arbitrary code in external environments including Spark clusters, Kubernetes Jobs, AWS Lambda, and cloud-managed services while reporting metadata back to Dagster.
  • Partitioned Processing
  • Time-based and custom partitioning enables incremental materialization, processing only changed data slices instead of full refreshes for cost-efficient pipeline execution.
  • Built-in Testing Framework
  • Unit test individual assets and full pipelines locally with in-process execution, resource mocking, and type-checked IO managers before deploying to production.