RAG data pipelines for data engineers — the four-stage ingest → chunk → embed → index → retrieve pipeline, chunking strategies (fixed, recursive, semantic, hierarchical), embedding model selection and vector store layout, hybrid dense + BM25 search, cross-encoder reranking, freshness SLOs, CDC-driven incremental reindex, tombstone deletes, blue/green embedding model upgrades, and per-tenant ACL pushdown. Each section ships a worked interview answer with code, a step-by-step trace

Jun 16, 202682 min read

Read article

De InterviewSql

Vector Databases for Data Engineers: Pinecone vs Weaviate vs Qdrant vs pgvector

Gowtham Potureddi

Vector databases for data engineers — Pinecone vs Weaviate vs Qdrant vs pgvector compared on hosting, index types, scale, and ops. Covers HNSW, IVFFlat, scalar / product quantization, and DiskANN; hybrid retrieval with metadata filter pushdown and rerankers; memory sizing per million vectors at dim=768 / 1536; multi-tenant namespaces, drift on embedding-model upgrades, and blue / green collection swaps. Each section ships a worked interview answer with code, a step-by-step trace.

Jun 16, 202681 min read

Read article

De InterviewSql

Semantic Layer Showdown: Cube vs dbt Semantic Layer vs Looker LookML

Gowtham Potureddi

Semantic layer showdown for analytics engineers — Cube.dev, dbt Semantic Layer (MetricFlow), and Looker LookML compared on data model, BI tool fan-out, caching, governance, and migration risk. Walks through the headless-BI premise, the three platforms' data models (cubes vs semantic_models vs LookML views/explores), defining the same Weekly Active Users metric in each engine, and a consumer-fan-out playbook for Tableau / Power BI / Hex / Mode / embedded apps / LLM agents. Each section ships

Jun 16, 202682 min read

Read article

De InterviewSql

MetricFlow & dbt Metrics: Single Source of Truth for KPIs

Gowtham Potureddi

MetricFlow and dbt metrics tutorial — the dbt metrics layer as the KPI single source of truth, MetricFlow architecture (semantic models, measures, dimensions, metrics, saved queries, the MetricFlow server), the anatomy of a metric definition with entity / measure / dimension / filter and ratio / cumulative / derived variants, the query flow from metric to BI / Python via the Semantic Layer API, and a migration playbook from BI calculated fields to dbt semantic models.

Jun 16, 202675 min read

Read article

De InterviewSql

dbt Model Contracts, Constraints & Versioning: Production Patterns

Gowtham Potureddi

Gowtham Potureddi

DuckDB for data engineering — the in-process OLAP engine, vectorized executor, Arrow zero-copy, MVCC snapshots, Parquet-first scans with partition pruning and httpfs / S3, local ETL with Python and dbt-duckdb, pytest-driven CI, and the four-quadrant deployment matrix that pins DuckDB to laptop, CI runner, edge, and notebook workloads while naming the anti-patterns (long-running OLAP clusters, concurrent writes, multi-user serving). Every section ships a worked interview answer with code,

Jun 14, 202679 min read

Read article

De InterviewSql

Iceberg REST Catalog, Nessie & Polaris: Open Lakehouse Catalogs Compared

Gowtham Potureddi

Iceberg REST catalog, Project Nessie, and Apache Polaris compared for senior data engineers picking an open lakehouse catalog — the REST OpenAPI surface (createNamespace, loadTable, commitTable, vendCredentials), Nessie's git-like branches / tags / atomic multi-table commits, and Polaris's principals / roles / grants for multi-tenant Iceberg with RBAC.

Jun 14, 202670 min read

Read article

De InterviewSql

Apache Hudi Merge-on-Read vs Copy-on-Write: Picking the Right Table Type

Gowtham Potureddi

Apache Hudi Merge-on-Read vs Copy-on-Write — the senior data engineer's playbook for picking the right Hudi table type. Walk the timeline (commit, deltacommit, compaction, clean, rollback, savepoint), the file-group anatomy (base parquet + delta log avro), the CoW rewrite path that keeps reads fast, the MoR base-plus-logs path that keeps writes fast, the compaction and cleaner contracts (inline vs async scheduler, KEEP_LATEST_COMMITS vs KEEP_LATEST_FILE_VERSIONS)

Jun 13, 202665 min read

Read article

De InterviewSql

Delta Lake Change Data Feed (CDF) & Z-Ordering: Performance Tuning

Gowtham Potureddi

Delta Lake performance tuning guide for senior data engineers — Change Data Feed (CDF) row-state mechanics with _change_type / _commit_version / _commit_timestamp, Z-Ordering as a space-filling curve for multi-dimensional file skipping, OPTIMIZE bin-packing into 1 GB target files, auto compaction and optimized writes, VACUUM retention with the 168-hour floor, deletion vectors for row-level deletes without file rewrites, and liquid clustering as the 2024+ successor to Z-Order. Each section ships

Jun 13, 202673 min read

Read article

De InterviewSql

Apache Iceberg Branching, Tagging & WAP: Production Patterns

Gowtham Potureddi

Apache Iceberg branching, tagging and write-audit-publish (WAP) production patterns for senior data engineers — snapshot atomicity and the apache iceberg vs delta lake decision, the createBranch / fastForward / cherryPick lifecycle, immutable tags for month-end audit and golden datasets, the branch-based WAP pattern that replaces spark.wap.id, and the production maintenance cadence for snapshot expiration, OPTIMIZE / rewrite_data_files, rewrite_manifests, remove_orphan_files, and schema

Jun 13, 202680 min read

Read article

De InterviewSql

Lambda vs Kappa Architecture: When Each Wins in 2026

AI agents in the data stack — a practitioner's reference for 2026 covering the agent loop, MCP tool servers, lineage agents that answer natural-language metadata questions, anomaly-detection agents that go from metric to root cause in one pass, and auto-repair agents that ship every change as a confidence-gated, shadow-tested PR. Each section ships a worked engineering answer with code, a step-by-step trace, an output table, and a concept-by-concept why-this-works. Practice data engineering on P

Jun 11, 202680 min read

Read article

Top 50 Data Engineering Interview Questions & Answers (2026 Guide)

New Tags

The Only 5 Skills You Need to Become a Data Engineer in 2026

Nick

A definitive guide to the 5 core skills required for modern Data Engineering. Learn how Extraction, ETL, Warehousing, Delivery, and Orchestration fit together to build your career.

Mar 5, 202612 min read

Read article