Telegram Group & Telegram Channel
πŸ–₯ SQL Flow

SQL Flow позиционируСтся ΠΊΠ°ΠΊ Β«DuckDB для ΠΏΠΎΡ‚ΠΎΠΊΠΎΠ²Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ…Β» β€” лёгковСсный Π΄Π²ΠΈΠΆΠΎΠΊ stream-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚ΠΊΠΈ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡŽΡ‰ΠΈΠΉ ΠΎΠΏΠΈΡΡ‹Π²Π°Ρ‚ΡŒ вСсь pipeline СдинствСнным языком SQL ΠΈ слуТащий ΠΊΠΎΠΌΠΏΠ°ΠΊΡ‚Π½ΠΎΠΉ Π°Π»ΡŒΡ‚Π΅Ρ€Π½Π°Ρ‚ΠΈΠ²ΠΎΠΉ Apache Flink.

πŸ” ΠšΠ»ΡŽΡ‡Π΅Π²Ρ‹Π΅ возмоТности:

- Π˜ΡΡ‚ΠΎΡ‡Π½ΠΈΠΊΠΈ (Sources): Kafka, WebSocket-стримы, HTTP-webhooks ΠΈ Π΄Ρ€.
- ΠŸΡ€ΠΈΡ‘ΠΌΠ½ΠΈΠΊΠΈ (Sinks): Kafka, PostgreSQL, Π»ΠΎΠΊΠ°Π»ΡŒΠ½Ρ‹Π΅ ΠΈ S3-ΠΏΠΎΠ΄ΠΎΠ±Π½Ρ‹Π΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π°, Π»ΡŽΠ±Ρ‹Π΅ Ρ„ΠΎΡ€ΠΌΠ°Ρ‚Ρ‹, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹Π΅ ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ DuckDB (JSON, Parquet, Iceberg ΠΈ Ρ‚.Π΄.).
- SQL-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚Ρ‡ΠΈΠΊ (Handler): встраиваСт DuckDB + Apache Arrow; ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ Π°Π³Ρ€Π΅Π³Π°Ρ‚Ρ‹, ΠΎΠΊΠΎΠ½Π½Ρ‹Π΅ Ρ„ΡƒΠ½ΠΊΡ†ΠΈΠΈ, UDF ΠΈ динамичСский Π²Ρ‹Π²ΠΎΠ΄ схСмы.
- Π£ΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ ΠΎΠΊΠ½Π°ΠΌΠΈ: in-memory tumbling-windows, Π±ΡƒΡ„Π΅Ρ€Π½Ρ‹Π΅ Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹.
- ΠΠ°Π±Π»ΡŽΠ΄Π°Π΅ΠΌΠΎΡΡ‚ΡŒ: встроСнныС Prometheus-ΠΌΠ΅Ρ‚Ρ€ΠΈΠΊΠΈ (с Ρ€Π΅Π»ΠΈΠ·Π° v0.6.0).

πŸ”— АрхитСктура

ΠšΠΎΠ½Π²Π΅ΠΉΠ΅Ρ€ описываСтся YAML-Ρ„Π°ΠΉΠ»ΠΎΠΌ с Π±Π»ΠΎΠΊΠ°ΠΌΠΈ `source β†’ handler β†’ sink`.
Π’ΠΎ врСмя выполнСния:

1. Source считываСт ΠΏΠΎΡ‚ΠΎΠΊ (Kafka, WebSocket …).
2. Handler выполняСт SQL-Π»ΠΎΠ³ΠΈΠΊΡƒ Π² DuckDB.
3. Sink сохраняСт Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚Ρ‹ Π² Π²Ρ‹Π±Ρ€Π°Π½Π½ΠΎΠ΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π΅.

βœ… Быстрый старт (β‰ˆ 5 ΠΌΠΈΠ½ΡƒΡ‚)


# ΠΏΠΎΠ»ΡƒΡ‡ΠΈΡ‚ΡŒ ΠΎΠ±Ρ€Π°Π·
docker pull turbolytics/sql-flow:latest

# тСстовая ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΊΠ° ΠΊΠΎΠ½Ρ„ΠΈΠ³ΡƒΡ€Π°Ρ†ΠΈΠΈ
docker run -v $(pwd)/dev:/tmp/conf \
-v /tmp/sqlflow:/tmp/sqlflow \
turbolytics/sql-flow:latest \
dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

# запуск ΠΏΡ€ΠΎΡ‚ΠΈΠ² Kafka
docker-compose -f dev/kafka-single.yml up -d # ΠΏΠΎΠ΄Π½ΡΡ‚ΡŒ Kafka

docker run -v $(pwd)/dev:/tmp/conf \
-e SQLFLOW_KAFKA_BROKERS=host.docker.internal:29092 \
turbolytics/sql-flow:latest \
run /tmp/conf/config/examples/basic.agg.mem.yml --max-msgs-to-process=10000


β–ͺ Github

@sqlhub
Please open Telegram to view this post
VIEW IN TELEGRAM



tg-me.com/sqlhub/1874
Create:
Last Update:

πŸ–₯ SQL Flow

SQL Flow позиционируСтся ΠΊΠ°ΠΊ Β«DuckDB для ΠΏΠΎΡ‚ΠΎΠΊΠΎΠ²Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ…Β» β€” лёгковСсный Π΄Π²ΠΈΠΆΠΎΠΊ stream-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚ΠΊΠΈ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡŽΡ‰ΠΈΠΉ ΠΎΠΏΠΈΡΡ‹Π²Π°Ρ‚ΡŒ вСсь pipeline СдинствСнным языком SQL ΠΈ слуТащий ΠΊΠΎΠΌΠΏΠ°ΠΊΡ‚Π½ΠΎΠΉ Π°Π»ΡŒΡ‚Π΅Ρ€Π½Π°Ρ‚ΠΈΠ²ΠΎΠΉ Apache Flink.

πŸ” ΠšΠ»ΡŽΡ‡Π΅Π²Ρ‹Π΅ возмоТности:

- Π˜ΡΡ‚ΠΎΡ‡Π½ΠΈΠΊΠΈ (Sources): Kafka, WebSocket-стримы, HTTP-webhooks ΠΈ Π΄Ρ€.
- ΠŸΡ€ΠΈΡ‘ΠΌΠ½ΠΈΠΊΠΈ (Sinks): Kafka, PostgreSQL, Π»ΠΎΠΊΠ°Π»ΡŒΠ½Ρ‹Π΅ ΠΈ S3-ΠΏΠΎΠ΄ΠΎΠ±Π½Ρ‹Π΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π°, Π»ΡŽΠ±Ρ‹Π΅ Ρ„ΠΎΡ€ΠΌΠ°Ρ‚Ρ‹, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹Π΅ ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ DuckDB (JSON, Parquet, Iceberg ΠΈ Ρ‚.Π΄.).
- SQL-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚Ρ‡ΠΈΠΊ (Handler): встраиваСт DuckDB + Apache Arrow; ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ Π°Π³Ρ€Π΅Π³Π°Ρ‚Ρ‹, ΠΎΠΊΠΎΠ½Π½Ρ‹Π΅ Ρ„ΡƒΠ½ΠΊΡ†ΠΈΠΈ, UDF ΠΈ динамичСский Π²Ρ‹Π²ΠΎΠ΄ схСмы.
- Π£ΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ ΠΎΠΊΠ½Π°ΠΌΠΈ: in-memory tumbling-windows, Π±ΡƒΡ„Π΅Ρ€Π½Ρ‹Π΅ Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹.
- ΠΠ°Π±Π»ΡŽΠ΄Π°Π΅ΠΌΠΎΡΡ‚ΡŒ: встроСнныС Prometheus-ΠΌΠ΅Ρ‚Ρ€ΠΈΠΊΠΈ (с Ρ€Π΅Π»ΠΈΠ·Π° v0.6.0).

πŸ”— АрхитСктура

ΠšΠΎΠ½Π²Π΅ΠΉΠ΅Ρ€ описываСтся YAML-Ρ„Π°ΠΉΠ»ΠΎΠΌ с Π±Π»ΠΎΠΊΠ°ΠΌΠΈ `source β†’ handler β†’ sink`.
Π’ΠΎ врСмя выполнСния:

1. Source считываСт ΠΏΠΎΡ‚ΠΎΠΊ (Kafka, WebSocket …).
2. Handler выполняСт SQL-Π»ΠΎΠ³ΠΈΠΊΡƒ Π² DuckDB.
3. Sink сохраняСт Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚Ρ‹ Π² Π²Ρ‹Π±Ρ€Π°Π½Π½ΠΎΠ΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π΅.

βœ… Быстрый старт (β‰ˆ 5 ΠΌΠΈΠ½ΡƒΡ‚)


# ΠΏΠΎΠ»ΡƒΡ‡ΠΈΡ‚ΡŒ ΠΎΠ±Ρ€Π°Π·
docker pull turbolytics/sql-flow:latest

# тСстовая ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΊΠ° ΠΊΠΎΠ½Ρ„ΠΈΠ³ΡƒΡ€Π°Ρ†ΠΈΠΈ
docker run -v $(pwd)/dev:/tmp/conf \
-v /tmp/sqlflow:/tmp/sqlflow \
turbolytics/sql-flow:latest \
dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

# запуск ΠΏΡ€ΠΎΡ‚ΠΈΠ² Kafka
docker-compose -f dev/kafka-single.yml up -d # ΠΏΠΎΠ΄Π½ΡΡ‚ΡŒ Kafka

docker run -v $(pwd)/dev:/tmp/conf \
-e SQLFLOW_KAFKA_BROKERS=host.docker.internal:29092 \
turbolytics/sql-flow:latest \
run /tmp/conf/config/examples/basic.agg.mem.yml --max-msgs-to-process=10000


β–ͺ Github

@sqlhub

BY Data Science. SQL hub




Share with your friend now:
tg-me.com/sqlhub/1874

View MORE
Open in Telegram


Data Science SQL hub Telegram | DID YOU KNOW?

Date: |

Telegram and Signal Havens for Right-Wing Extremists

Since the violent storming of Capitol Hill and subsequent ban of former U.S. President Donald Trump from Facebook and Twitter, the removal of Parler from Amazon’s servers, and the de-platforming of incendiary right-wing content, messaging services Telegram and Signal have seen a deluge of new users. In January alone, Telegram reported 90 million new accounts. Its founder, Pavel Durov, described this as β€œthe largest digital migration in human history.” Signal reportedly doubled its user base to 40 million people and became the most downloaded app in 70 countries. The two services rely on encryption to protect the privacy of user communication, which has made them popular with protesters seeking to conceal their identities against repressive governments in places like Belarus, Hong Kong, and Iran. But the same encryption technology has also made them a favored communication tool for criminals and terrorist groups, including al Qaeda and the Islamic State.

A project of our size needs at least a few hundred million dollars per year to keep going,” Mr. Durov wrote in his public channel on Telegram late last year. β€œWhile doing that, we will remain independent and stay true to our values, redefining how a tech company should operate.

Data Science SQL hub from hk


Telegram Data Science. SQL hub
FROM USA