Awesome ClickHouse › ETL and Data Processing
MooseStack is a developer framework designed for building real-time analytical backends with a focus on high-performance analytical infrastructure such as ClickHouse, Redpanda, Temporal, and Redis. It offers a modern, code-first approach to managing analytical data pipelines and infrastructure, enabling developers to declare tables, streams, workflows, and APIs in TypeScript or Python. MooseStack aims to simplify the integration and management of complex analytical stacks by providing type-safe, modular components that can be adopted incrementally. The framework supports a quick setup process, allowing developers to bootstrap a backend in just five minutes using its CLI tool. MooseStack includes pre-integrated components for ClickHouse (a fast columnar database optimized for analytical queries), Redpanda (a streaming platform), and Temporal (for workflow orchestration), among others. It supports hot-reload development, enabling live schema migrations and local development environments that run all necessary services. MooseStack is modular, with distinct modules for OLAP (managing ClickHouse tables and materialized views), streaming (real-time pipelines with Kafka/Redpanda), workflows (ETL pipelines with Temporal), and APIs (type-safe ingestion and query endpoints with auto-generated OpenAPI documentation). Additional tooling includes deployment, migration, and observability features. The project provides comprehensive documentation, quickstart guides, and examples in both TypeScript and Python, demonstrating how to create ClickHouse tables, Redpanda streaming topics, and API endpoints for data ingestion and consumption. MooseStack can be deployed using Boreal, a zero-configuration deployment platform by 514 Labs, or self-hosted with detailed deployment instructions. Overall, MooseStack is designed to bring the best practices of modern web development to the construction of analytical backends, making it easier and faster for developers to build scalable, real-time analytics applications leveraging ClickHouse and other cutting-edge technologies.
https://github.com/514-labs/moosestack
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.