Telegram Group & Telegram Channel
πŸ–₯ SQL Flow

SQL Flow позиционируСтся ΠΊΠ°ΠΊ Β«DuckDB для ΠΏΠΎΡ‚ΠΎΠΊΠΎΠ²Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ…Β» β€” лёгковСсный Π΄Π²ΠΈΠΆΠΎΠΊ stream-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚ΠΊΠΈ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡŽΡ‰ΠΈΠΉ ΠΎΠΏΠΈΡΡ‹Π²Π°Ρ‚ΡŒ вСсь pipeline СдинствСнным языком SQL ΠΈ слуТащий ΠΊΠΎΠΌΠΏΠ°ΠΊΡ‚Π½ΠΎΠΉ Π°Π»ΡŒΡ‚Π΅Ρ€Π½Π°Ρ‚ΠΈΠ²ΠΎΠΉ Apache Flink.

πŸ” ΠšΠ»ΡŽΡ‡Π΅Π²Ρ‹Π΅ возмоТности:

- Π˜ΡΡ‚ΠΎΡ‡Π½ΠΈΠΊΠΈ (Sources): Kafka, WebSocket-стримы, HTTP-webhooks ΠΈ Π΄Ρ€.
- ΠŸΡ€ΠΈΡ‘ΠΌΠ½ΠΈΠΊΠΈ (Sinks): Kafka, PostgreSQL, Π»ΠΎΠΊΠ°Π»ΡŒΠ½Ρ‹Π΅ ΠΈ S3-ΠΏΠΎΠ΄ΠΎΠ±Π½Ρ‹Π΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π°, Π»ΡŽΠ±Ρ‹Π΅ Ρ„ΠΎΡ€ΠΌΠ°Ρ‚Ρ‹, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹Π΅ ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ DuckDB (JSON, Parquet, Iceberg ΠΈ Ρ‚.Π΄.).
- SQL-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚Ρ‡ΠΈΠΊ (Handler): встраиваСт DuckDB + Apache Arrow; ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ Π°Π³Ρ€Π΅Π³Π°Ρ‚Ρ‹, ΠΎΠΊΠΎΠ½Π½Ρ‹Π΅ Ρ„ΡƒΠ½ΠΊΡ†ΠΈΠΈ, UDF ΠΈ динамичСский Π²Ρ‹Π²ΠΎΠ΄ схСмы.
- Π£ΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ ΠΎΠΊΠ½Π°ΠΌΠΈ: in-memory tumbling-windows, Π±ΡƒΡ„Π΅Ρ€Π½Ρ‹Π΅ Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹.
- ΠΠ°Π±Π»ΡŽΠ΄Π°Π΅ΠΌΠΎΡΡ‚ΡŒ: встроСнныС Prometheus-ΠΌΠ΅Ρ‚Ρ€ΠΈΠΊΠΈ (с Ρ€Π΅Π»ΠΈΠ·Π° v0.6.0).

πŸ”— АрхитСктура

ΠšΠΎΠ½Π²Π΅ΠΉΠ΅Ρ€ описываСтся YAML-Ρ„Π°ΠΉΠ»ΠΎΠΌ с Π±Π»ΠΎΠΊΠ°ΠΌΠΈ `source β†’ handler β†’ sink`.
Π’ΠΎ врСмя выполнСния:

1. Source считываСт ΠΏΠΎΡ‚ΠΎΠΊ (Kafka, WebSocket …).
2. Handler выполняСт SQL-Π»ΠΎΠ³ΠΈΠΊΡƒ Π² DuckDB.
3. Sink сохраняСт Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚Ρ‹ Π² Π²Ρ‹Π±Ρ€Π°Π½Π½ΠΎΠ΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π΅.

βœ… Быстрый старт (β‰ˆ 5 ΠΌΠΈΠ½ΡƒΡ‚)


# ΠΏΠΎΠ»ΡƒΡ‡ΠΈΡ‚ΡŒ ΠΎΠ±Ρ€Π°Π·
docker pull turbolytics/sql-flow:latest

# тСстовая ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΊΠ° ΠΊΠΎΠ½Ρ„ΠΈΠ³ΡƒΡ€Π°Ρ†ΠΈΠΈ
docker run -v $(pwd)/dev:/tmp/conf \
-v /tmp/sqlflow:/tmp/sqlflow \
turbolytics/sql-flow:latest \
dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

# запуск ΠΏΡ€ΠΎΡ‚ΠΈΠ² Kafka
docker-compose -f dev/kafka-single.yml up -d # ΠΏΠΎΠ΄Π½ΡΡ‚ΡŒ Kafka

docker run -v $(pwd)/dev:/tmp/conf \
-e SQLFLOW_KAFKA_BROKERS=host.docker.internal:29092 \
turbolytics/sql-flow:latest \
run /tmp/conf/config/examples/basic.agg.mem.yml --max-msgs-to-process=10000


β–ͺ Github

@sqlhub
Please open Telegram to view this post
VIEW IN TELEGRAM



tg-me.com/sqlhub/1874
Create:
Last Update:

πŸ–₯ SQL Flow

SQL Flow позиционируСтся ΠΊΠ°ΠΊ Β«DuckDB для ΠΏΠΎΡ‚ΠΎΠΊΠΎΠ²Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ…Β» β€” лёгковСсный Π΄Π²ΠΈΠΆΠΎΠΊ stream-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚ΠΊΠΈ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡŽΡ‰ΠΈΠΉ ΠΎΠΏΠΈΡΡ‹Π²Π°Ρ‚ΡŒ вСсь pipeline СдинствСнным языком SQL ΠΈ слуТащий ΠΊΠΎΠΌΠΏΠ°ΠΊΡ‚Π½ΠΎΠΉ Π°Π»ΡŒΡ‚Π΅Ρ€Π½Π°Ρ‚ΠΈΠ²ΠΎΠΉ Apache Flink.

πŸ” ΠšΠ»ΡŽΡ‡Π΅Π²Ρ‹Π΅ возмоТности:

- Π˜ΡΡ‚ΠΎΡ‡Π½ΠΈΠΊΠΈ (Sources): Kafka, WebSocket-стримы, HTTP-webhooks ΠΈ Π΄Ρ€.
- ΠŸΡ€ΠΈΡ‘ΠΌΠ½ΠΈΠΊΠΈ (Sinks): Kafka, PostgreSQL, Π»ΠΎΠΊΠ°Π»ΡŒΠ½Ρ‹Π΅ ΠΈ S3-ΠΏΠΎΠ΄ΠΎΠ±Π½Ρ‹Π΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π°, Π»ΡŽΠ±Ρ‹Π΅ Ρ„ΠΎΡ€ΠΌΠ°Ρ‚Ρ‹, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹Π΅ ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ DuckDB (JSON, Parquet, Iceberg ΠΈ Ρ‚.Π΄.).
- SQL-ΠΎΠ±Ρ€Π°Π±ΠΎΡ‚Ρ‡ΠΈΠΊ (Handler): встраиваСт DuckDB + Apache Arrow; ΠΏΠΎΠ΄Π΄Π΅Ρ€ΠΆΠΈΠ²Π°Π΅Ρ‚ Π°Π³Ρ€Π΅Π³Π°Ρ‚Ρ‹, ΠΎΠΊΠΎΠ½Π½Ρ‹Π΅ Ρ„ΡƒΠ½ΠΊΡ†ΠΈΠΈ, UDF ΠΈ динамичСский Π²Ρ‹Π²ΠΎΠ΄ схСмы.
- Π£ΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ ΠΎΠΊΠ½Π°ΠΌΠΈ: in-memory tumbling-windows, Π±ΡƒΡ„Π΅Ρ€Π½Ρ‹Π΅ Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹.
- ΠΠ°Π±Π»ΡŽΠ΄Π°Π΅ΠΌΠΎΡΡ‚ΡŒ: встроСнныС Prometheus-ΠΌΠ΅Ρ‚Ρ€ΠΈΠΊΠΈ (с Ρ€Π΅Π»ΠΈΠ·Π° v0.6.0).

πŸ”— АрхитСктура

ΠšΠΎΠ½Π²Π΅ΠΉΠ΅Ρ€ описываСтся YAML-Ρ„Π°ΠΉΠ»ΠΎΠΌ с Π±Π»ΠΎΠΊΠ°ΠΌΠΈ `source β†’ handler β†’ sink`.
Π’ΠΎ врСмя выполнСния:

1. Source считываСт ΠΏΠΎΡ‚ΠΎΠΊ (Kafka, WebSocket …).
2. Handler выполняСт SQL-Π»ΠΎΠ³ΠΈΠΊΡƒ Π² DuckDB.
3. Sink сохраняСт Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚Ρ‹ Π² Π²Ρ‹Π±Ρ€Π°Π½Π½ΠΎΠ΅ Ρ…Ρ€Π°Π½ΠΈΠ»ΠΈΡ‰Π΅.

βœ… Быстрый старт (β‰ˆ 5 ΠΌΠΈΠ½ΡƒΡ‚)


# ΠΏΠΎΠ»ΡƒΡ‡ΠΈΡ‚ΡŒ ΠΎΠ±Ρ€Π°Π·
docker pull turbolytics/sql-flow:latest

# тСстовая ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΊΠ° ΠΊΠΎΠ½Ρ„ΠΈΠ³ΡƒΡ€Π°Ρ†ΠΈΠΈ
docker run -v $(pwd)/dev:/tmp/conf \
-v /tmp/sqlflow:/tmp/sqlflow \
turbolytics/sql-flow:latest \
dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

# запуск ΠΏΡ€ΠΎΡ‚ΠΈΠ² Kafka
docker-compose -f dev/kafka-single.yml up -d # ΠΏΠΎΠ΄Π½ΡΡ‚ΡŒ Kafka

docker run -v $(pwd)/dev:/tmp/conf \
-e SQLFLOW_KAFKA_BROKERS=host.docker.internal:29092 \
turbolytics/sql-flow:latest \
run /tmp/conf/config/examples/basic.agg.mem.yml --max-msgs-to-process=10000


β–ͺ Github

@sqlhub

BY Data Science. SQL hub




Share with your friend now:
tg-me.com/sqlhub/1874

View MORE
Open in Telegram


Data Science SQL hub Telegram | DID YOU KNOW?

Date: |

Telegram hopes to raise $1bn with a convertible bond private placement

The super secure UAE-based Telegram messenger service, developed by Russian-born software icon Pavel Durov, is looking to raise $1bn through a bond placement to a limited number of investors from Russia, Europe, Asia and the Middle East, the Kommersant daily reported citing unnamed sources on February 18, 2021.The issue reportedly comprises exchange bonds that could be converted into equity in the messaging service that is currently 100% owned by Durov and his brother Nikolai.Kommersant reports that the price of the conversion would be at a 10% discount to a potential IPO should it happen within five years.The minimum bond placement is said to be set at $50mn, but could be lowered to $10mn. Five-year bonds could carry an annual coupon of 7-8%.

Spiking bond yields driving sharp losses in tech stocks

A spike in interest rates since the start of the year has accelerated a rotation out of high-growth technology stocks and into value stocks poised to benefit from a reopening of the economy. The Nasdaq has fallen more than 10% over the past month as the Dow has soared to record highs, with a spike in the 10-year US Treasury yield acting as the main catalyst. It recently surged to a cycle high of more than 1.60% after starting the year below 1%. But according to Jim Paulsen, the Leuthold Group's chief investment strategist, rising interest rates do not represent a long-term threat to the stock market. Paulsen expects the 10-year yield to cross 2% by the end of the year. A spike in interest rates and its impact on the stock market depends on the economic backdrop, according to Paulsen. Rising interest rates amid a strengthening economy "may prove no challenge at all for stocks," Paulsen said.

Data Science SQL hub from in


Telegram Data Science. SQL hub
FROM USA