StackFollow on WhatAreYouBuilding.AI

Data Engineer

Databases

ClickHouse

Column-oriented database built for fast analytical queries at scale.

PostgreSQL

Open-source relational database with strong SQL compliance and extensibility.

Snowflake

Cloud-native data warehouse with separated storage and compute.

Orchestration

Apache Airflow

Platform to author, schedule, and monitor workflows as code.

Dagster

Data orchestrator focused on asset-based pipelines and observability.

dbt

SQL-first transformation tool for building and testing data models in the warehouse.

Streaming & Processing

Apache Flink

Stream-processing framework for stateful computations over unbounded data.

Apache Kafka

Distributed event streaming platform for high-throughput data pipelines.

Apache Spark

Unified analytics engine for large-scale batch and streaming data processing.