Awesome ClickHouse › ETL and Data Processing
file.d is a high-performance tool designed for building data pipelines that efficiently read, process, and output events. It is primarily developed to read from files but supports a wide range of input, action, and output plugins, making it versatile for various data processing needs. The project emphasizes speed, reliability, and predictability, boasting performance metrics that significantly surpass similar tools like vector, filebeat, logstash, fluentd, and fluent-bit. For instance, on a MacBook Pro 2017 with two physical cores, file.d can achieve throughput rates of 1.7GB/s when reading files to devnull and 1.0GB/s when decoding JSON before outputting to devnull. This performance is attributed to its efficient use of pooling to limit memory consumption and a commitment mechanism that ensures no data loss. file.d is container, cloud, and Kubernetes native, making it suitable for modern infrastructure environments. It is configured simply using YAML files and integrates well with Prometheus for metrics transformation and Vault for secure storage of sensitive information. The tool is well-tested and used in production environments, including Kubernetes clusters with over 3000 CPU cores. The project includes a rich set of plugins categorized into inputs (e.g., dmesg, file, kafka, http), actions (e.g., json_decode, mask, modify, parse_re2), and outputs (e.g., clickhouse, elasticsearch, kafka, postgres). Notably, it supports ClickHouse as an output plugin, enabling efficient data storage and querying in ClickHouse databases. file.d is open-source and encourages community contributions. It is still under active development and not yet at version 1.0.0, so users are advised to test pipelines carefully in development or staging environments before production deployment. The project also provides extensive documentation, including quick start guides, installation instructions, examples, and architectural details, along with a community Telegram channel for support and discussion.
https://github.com/ozontech/file.d
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.