This document defines the Pipeline Engine (RAP)
sub-system, part of the Debrief Rewrite under the Shared Services category.
Feature / Criteria | Airflow | Prefect | Dagster | Custom Runner (JS + Optional Python) |
---|---|---|---|---|
In-browser support | ❌ None | ❌ None | ❌ None | ✅ JS-native steps run in-browser |
Local container execution | ⚠️ Heavyweight | ✅ Designed for this | ✅ Designed for this | ✅ With Node.js or Python backend |
Future server-hosted scaling | ✅ Battle-tested | ✅ Native via Prefect Orion | ✅ Native via Dagster Cloud | ⚠️ Requires work to scale server infra |
Step registry across JS + Python | ❌ Python-only | ⚠️ Python-only | ⚠️ Python-first | ✅ Explicit mapping in JS and Python |
Analyst-triggered runs | ⚠️ Hard to streamline | ✅ Simple in UI/API | ✅ Simple with GraphQL | ✅ Direct UI integration |
Lightweight local install/dev | ❌ Requires DB + scheduler | ✅ CLI or Docker Compose | ⚠️ Slightly heavier | ✅ As lightweight as needed |
Visual pipeline debugging | ✅ Gantt UI | ✅ Flow UI | ✅ Graph UI | ⚠️ Basic unless built separately |
Support for long-running steps | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Not in-browser, only via server |
Loop/branch/async support | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Requires explicit implementation |
Declarative JSON pipeline format | ❌ Python DAGs only | ✅ JSON available | ⚠️ YAML/Python focus | ✅ Designed to be JSON-native |
Extensibility (MCP / LLM support) | ⚠️ Limited | ✅ Python dynamic graphs | ✅ Strongly typed graph model | ✅ Easily adapted to LLM-generated steps |
Phase | Strategy |
---|---|
Now | Build a Custom Runner in JS with pluggable function registry |
Mid-term | Add an optional Python backend runner (Flask + Celery/RQ or similar) |
Future | Evaluate Prefect or Dagster for hosted orchestration |
Airflow? | Only consider if tight scheduling, SLA enforcement, or legacy tie-ins arise |
TBD
TBD
TBD