Awesome ClickHouse › ETL and Data Processing
The flow-pipeline project by Cloudflare is a comprehensive set of tools and examples designed to facilitate the collection, processing, and storage of network flow data such as sFlow, NetFlow, and IPFIX. It leverages GoFlow, a collector developed by Cloudflare, to capture flow data and supports integration with Kafka, Zookeeper, and databases like Postgres and Clickhouse for scalable data ingestion and storage. The project includes a demo pipeline setup that listens on standard ports for sFlow and NetFlow data, demonstrating how flow data can be collected, processed, and inserted into databases for analysis. A key feature of the flow-pipeline is its modular architecture, which allows users to add processors that enrich flow data by mapping IP addresses to countries, ASNs, or customer information. This enrichment is achieved by consuming data from Kafka, processing it, and reinjecting it back into Kafka or directly into the database. The project also provides a mock insertion tool that simulates flow data generation and sends protobuf messages to Kafka, enabling testing without actual network traffic. The project supports both Postgres and Clickhouse databases, with a particular emphasis on Clickhouse for high-performance flow data storage and querying. Clickhouse integration uses a Kafka engine and protobuf format for efficient data ingestion, with materialized views to store data persistently and perform aggregations. The project includes Docker Compose configurations for easy deployment of the entire pipeline, including Grafana for visualization with a Clickhouse data source plugin. Overall, flow-pipeline is a powerful and flexible solution for network flow data collection and analysis, suitable for development, prototyping, and production environments where high throughput and data enrichment are required. It provides a practical example of how to build scalable flow processing pipelines using modern data infrastructure components.
https://github.com/cloudflare/flow-pipeline
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.