Awesome ClickHouse › ETL and Data Processing
Data Forge is a comprehensive modern data stack playground designed to help data engineers practice end-to-end workflows and best practices locally without the risks and costs associated with cloud environments. It integrates a variety of industry-standard tools and technologies, all orchestrated through Docker Compose, to simulate a real data platform environment. The stack includes storage solutions like MinIO for S3-compatible object storage and Hive Metastore for centralized metadata management. For compute, it features Trino for interactive SQL queries and Apache Spark for distributed batch and streaming processing. Streaming and change data capture are handled by Apache Kafka, Schema Registry, and Debezium. The platform also incorporates PostgreSQL as the primary OLTP database and ClickHouse as a columnar analytics database, enabling efficient data warehousing and analytics. Workflow orchestration is managed by Apache Airflow 3, while data visualization and exploration are supported through Apache Superset and JupyterLab. Additionally, Data Forge includes a data generator that produces realistic retail data for Kafka topics and Postgres tables, facilitating practical experimentation. The project emphasizes ease of setup with Docker and Docker Compose, recommending sufficient system resources for smooth operation. Users can start core services, add compute and orchestration layers, and access various services through provided URLs and default credentials. The architecture supports modular profiles for different components, and comprehensive documentation guides users through learning paths, development, and troubleshooting. Data Forge is open-source under the MIT license and acknowledges the contributions of multiple open-source communities. It serves as a hands-on gym for data engineers to build skills in managing modern data stacks, from data lakes and lakehouses to streaming and analytics, all within a controlled local environment.
https://github.com/fortiql/data-forge
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.