Telegram Group & Telegram Channel
πŸ–₯ SQL Flow

SQL Flow позиционируСтся ΠΊΠ°ΠΊ Β«DuckDB для ΠΏΠΎΡ‚ΠΎΠΊΠΎΠ²Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ…Β» β€” лёгковСсный Π΄Π²ΠΈΠΆΠΎΠΊ stream-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚ΠΊΠΈ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡŽΡ‰ΠΈΠΉ ΠΎΠΏΠΈΡΡ‹Π²Π°Ρ‚ΡŒ вСсь pipeline СдинствСнным языком SQL ΠΈ слуТащий ΠΊΠΎΠΌΠΏΠ°ΠΊΡ‚Π½ΠΎΠΉ Π°Π»ΡŒΡ‚Π΅Ρ€Π½Π°Ρ‚ΠΈΠ²ΠΎΠΉ Apache Flink.

πŸ” ΠšΠ»ΡŽΡ‡Π΅Π²Ρ‹Π΅ возмоТности:

- Π˜ΡΡ‚ΠΎΡ‡Π½ΠΈΠΊΠΈ (Sources): Kafka, WebSocket-стримы, HTTP-webhooks ΠΈ Π΄Ρ€.
- ΠŸΡ€ΠΈΡ‘ΠΌΠ½ΠΈΠΊΠΈ (Sinks): Kafka, PostgreSQL, Π»ΠΎΠΊΠ°Π»ΡŒΠ½Ρ‹Π΅ ΠΈ S3-ΠΏΠΎΠ΄ΠΎΠ±Π½Ρ‹Π΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π°, Π»ΡŽΠ±Ρ‹Π΅ Ρ„ΠΎΡ€ΠΌΠ°Ρ‚Ρ‹, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹Π΅ ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ DuckDB (JSON, Parquet, Iceberg ΠΈ Ρ‚.Π΄.).
- SQL-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚Ρ‡ΠΈΠΊ (Handler): встраиваСт DuckDB + Apache Arrow; ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ Π°Π³Ρ€Π΅Π³Π°Ρ‚Ρ‹, ΠΎΠΊΠΎΠ½Π½Ρ‹Π΅ Ρ„ΡƒΠ½ΠΊΡ†ΠΈΠΈ, UDF ΠΈ динамичСский Π²Ρ‹Π²ΠΎΠ΄ схСмы.
- Π£ΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ ΠΎΠΊΠ½Π°ΠΌΠΈ: in-memory tumbling-windows, Π±ΡƒΡ„Π΅Ρ€Π½Ρ‹Π΅ Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹.
- ΠΠ°Π±Π»ΡŽΠ΄Π°Π΅ΠΌΠΎΡΡ‚ΡŒ: встроСнныС Prometheus-ΠΌΠ΅Ρ‚Ρ€ΠΈΠΊΠΈ (с Ρ€Π΅Π»ΠΈΠ·Π° v0.6.0).

πŸ”— АрхитСктура

ΠšΠΎΠ½Π²Π΅ΠΉΠ΅Ρ€ описываСтся YAML-Ρ„Π°ΠΉΠ»ΠΎΠΌ с Π±Π»ΠΎΠΊΠ°ΠΌΠΈ `source β†’ handler β†’ sink`.
Π’ΠΎ врСмя выполнСния:

1. Source считываСт ΠΏΠΎΡ‚ΠΎΠΊ (Kafka, WebSocket …).
2. Handler выполняСт SQL-Π»ΠΎΠ³ΠΈΠΊΡƒ Π² DuckDB.
3. Sink сохраняСт Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚Ρ‹ Π² Π²Ρ‹Π±Ρ€Π°Π½Π½ΠΎΠ΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π΅.

βœ… Быстрый старт (β‰ˆ 5 ΠΌΠΈΠ½ΡƒΡ‚)


# ΠΏΠΎΠ»ΡƒΡ‡ΠΈΡ‚ΡŒ ΠΎΠ±Ρ€Π°Π·
docker pull turbolytics/sql-flow:latest

# тСстовая ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΊΠ° ΠΊΠΎΠ½Ρ„ΠΈΠ³ΡƒΡ€Π°Ρ†ΠΈΠΈ
docker run -v $(pwd)/dev:/tmp/conf \
-v /tmp/sqlflow:/tmp/sqlflow \
turbolytics/sql-flow:latest \
dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

# запуск ΠΏΡ€ΠΎΡ‚ΠΈΠ² Kafka
docker-compose -f dev/kafka-single.yml up -d # ΠΏΠΎΠ΄Π½ΡΡ‚ΡŒ Kafka

docker run -v $(pwd)/dev:/tmp/conf \
-e SQLFLOW_KAFKA_BROKERS=host.docker.internal:29092 \
turbolytics/sql-flow:latest \
run /tmp/conf/config/examples/basic.agg.mem.yml --max-msgs-to-process=10000


β–ͺ Github

@sqlhub
Please open Telegram to view this post
VIEW IN TELEGRAM



tg-me.com/sqlhub/1874
Create:
Last Update:

πŸ–₯ SQL Flow

SQL Flow позиционируСтся ΠΊΠ°ΠΊ Β«DuckDB для ΠΏΠΎΡ‚ΠΎΠΊΠΎΠ²Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ…Β» β€” лёгковСсный Π΄Π²ΠΈΠΆΠΎΠΊ stream-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚ΠΊΠΈ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡŽΡ‰ΠΈΠΉ ΠΎΠΏΠΈΡΡ‹Π²Π°Ρ‚ΡŒ вСсь pipeline СдинствСнным языком SQL ΠΈ слуТащий ΠΊΠΎΠΌΠΏΠ°ΠΊΡ‚Π½ΠΎΠΉ Π°Π»ΡŒΡ‚Π΅Ρ€Π½Π°Ρ‚ΠΈΠ²ΠΎΠΉ Apache Flink.

πŸ” ΠšΠ»ΡŽΡ‡Π΅Π²Ρ‹Π΅ возмоТности:

- Π˜ΡΡ‚ΠΎΡ‡Π½ΠΈΠΊΠΈ (Sources): Kafka, WebSocket-стримы, HTTP-webhooks ΠΈ Π΄Ρ€.
- ΠŸΡ€ΠΈΡ‘ΠΌΠ½ΠΈΠΊΠΈ (Sinks): Kafka, PostgreSQL, Π»ΠΎΠΊΠ°Π»ΡŒΠ½Ρ‹Π΅ ΠΈ S3-ΠΏΠΎΠ΄ΠΎΠ±Π½Ρ‹Π΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π°, Π»ΡŽΠ±Ρ‹Π΅ Ρ„ΠΎΡ€ΠΌΠ°Ρ‚Ρ‹, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹Π΅ ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ DuckDB (JSON, Parquet, Iceberg ΠΈ Ρ‚.Π΄.).
- SQL-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚Ρ‡ΠΈΠΊ (Handler): встраиваСт DuckDB + Apache Arrow; ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ Π°Π³Ρ€Π΅Π³Π°Ρ‚Ρ‹, ΠΎΠΊΠΎΠ½Π½Ρ‹Π΅ Ρ„ΡƒΠ½ΠΊΡ†ΠΈΠΈ, UDF ΠΈ динамичСский Π²Ρ‹Π²ΠΎΠ΄ схСмы.
- Π£ΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ ΠΎΠΊΠ½Π°ΠΌΠΈ: in-memory tumbling-windows, Π±ΡƒΡ„Π΅Ρ€Π½Ρ‹Π΅ Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹.
- ΠΠ°Π±Π»ΡŽΠ΄Π°Π΅ΠΌΠΎΡΡ‚ΡŒ: встроСнныС Prometheus-ΠΌΠ΅Ρ‚Ρ€ΠΈΠΊΠΈ (с Ρ€Π΅Π»ΠΈΠ·Π° v0.6.0).

πŸ”— АрхитСктура

ΠšΠΎΠ½Π²Π΅ΠΉΠ΅Ρ€ описываСтся YAML-Ρ„Π°ΠΉΠ»ΠΎΠΌ с Π±Π»ΠΎΠΊΠ°ΠΌΠΈ `source β†’ handler β†’ sink`.
Π’ΠΎ врСмя выполнСния:

1. Source считываСт ΠΏΠΎΡ‚ΠΎΠΊ (Kafka, WebSocket …).
2. Handler выполняСт SQL-Π»ΠΎΠ³ΠΈΠΊΡƒ Π² DuckDB.
3. Sink сохраняСт Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚Ρ‹ Π² Π²Ρ‹Π±Ρ€Π°Π½Π½ΠΎΠ΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π΅.

βœ… Быстрый старт (β‰ˆ 5 ΠΌΠΈΠ½ΡƒΡ‚)


# ΠΏΠΎΠ»ΡƒΡ‡ΠΈΡ‚ΡŒ ΠΎΠ±Ρ€Π°Π·
docker pull turbolytics/sql-flow:latest

# тСстовая ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΊΠ° ΠΊΠΎΠ½Ρ„ΠΈΠ³ΡƒΡ€Π°Ρ†ΠΈΠΈ
docker run -v $(pwd)/dev:/tmp/conf \
-v /tmp/sqlflow:/tmp/sqlflow \
turbolytics/sql-flow:latest \
dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

# запуск ΠΏΡ€ΠΎΡ‚ΠΈΠ² Kafka
docker-compose -f dev/kafka-single.yml up -d # ΠΏΠΎΠ΄Π½ΡΡ‚ΡŒ Kafka

docker run -v $(pwd)/dev:/tmp/conf \
-e SQLFLOW_KAFKA_BROKERS=host.docker.internal:29092 \
turbolytics/sql-flow:latest \
run /tmp/conf/config/examples/basic.agg.mem.yml --max-msgs-to-process=10000


β–ͺ Github

@sqlhub

BY Data Science. SQL hub




Share with your friend now:
tg-me.com/sqlhub/1874

View MORE
Open in Telegram


Data Science SQL hub Telegram | DID YOU KNOW?

Date: |

Importantly, that investor viewpoint is not new. It cycles in when conditions are right (and vice versa). It also brings the ineffective warnings of an overpriced market with it.Looking toward a good 2022 stock market, there is no apparent reason to expect these issues to change.

Look for Channels Online

You guessed it – the internet is your friend. A good place to start looking for Telegram channels is Reddit. This is one of the biggest sites on the internet, with millions of communities, including those from Telegram.Then, you can search one of the many dedicated websites for Telegram channel searching. One of them is telegram-group.com. This website has many categories and a really simple user interface. Another great site is telegram channels.me. It has even more channels than the previous one, and an even better user experience.These are just some of the many available websites. You can look them up online if you’re not satisfied with these two. All of these sites list only public channels. If you want to join a private channel, you’ll have to ask one of its members to invite you.

Data Science SQL hub from sa


Telegram Data Science. SQL hub
FROM USA