jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
mintance/nginx-clickhouse
The project "nginx-clickhouse" is a simple and efficient tool designed to parse NGINX logs and transport them into a ClickHouse database. It serves as a bridge between NGINX web server logs and the ClickHouse analytical database, enabling users to store, query, and analyze their web traffic data effectively. The tool supports parsing logs based on custom NGINX log formats defined by the user, making it adaptable to various logging configurations. The project provides a straightforward setup process, including building from source or using a pre-built Docker image for easy deployment. Users can configure the tool by specifying the log file path, the log format, and ClickHouse database credentials in a YAML configuration file. This configuration also maps NGINX log variables to the corresponding ClickHouse table columns, ensuring seamless data ingestion. The ClickHouse table schema example provided demonstrates how to structure the database to store detailed request information such as remote address, user, request time, status, bytes sent, HTTP referer, user agent, and various upstream timings. This rich dataset allows for comprehensive analysis and monitoring of web server performance and user behavior. The tool supports continuous log parsing with configurable intervals, making it suitable for real-time or near-real-time data ingestion. Additionally, it integrates well with visualization tools like Grafana, enabling users to create dashboards for monitoring NGINX traffic and performance metrics. Overall, nginx-clickhouse is a valuable utility for system administrators, DevOps engineers, and data analysts who want to leverage ClickHouse's powerful analytical capabilities to gain insights from NGINX logs. Its ease of use, configurability, and Docker support make it a practical choice for managing and analyzing web server logs at scale.
https://github.com/mintance/nginx-clickhouse
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.