{"id":4491,"date":"2022-09-27T20:00:45","date_gmt":"2022-09-27T10:00:45","guid":{"rendered":"http:\/\/bicortex.com\/?p=4491"},"modified":"2022-09-30T09:54:35","modified_gmt":"2022-09-29T23:54:35","slug":"data-build-tool-dbt-the-emerging-standard-for-building-sql-first-data-transformation-pipelines-part-1","status":"publish","type":"post","link":"http:\/\/bicortex.com\/bicortex\/data-build-tool-dbt-the-emerging-standard-for-building-sql-first-data-transformation-pipelines-part-1\/","title":{"rendered":"Data Build Tool (DBT) – The Emerging Standard For Building SQL-First Data Transformation Pipelines – Part 1"},"content":{"rendered":"

Note: Part 2 of this post can be found HERE<\/a>.<\/p>\n

Introduction<\/h3>\n

It\u2019s never a dull moment when working in IT and although Data Warehousing domain was not subjected to the hamster wheel of relentless innovation in the first few decades when Oracle, IBM, Microsoft and SAP reigned supreme, with the advent of cloud computing, it too had to adapt and change. For me, the most prolific changes included the separation of storage and compute, in-database machine learning, on-demand elasticity and server-less database models. The resulting upending of the status quo also had a large impact on the good, old-fashion ETL (Extract Transform Load) paradigm which started to shift to the new, more cloud-aligned architecture and many businesses contemplating Data Warehouse modernization are jumping on the ELT bandwagon. This is also where a suite of new tools started to emerge, and one company with its flagship product started to make serious inroads in this market.<\/p>\n

dbt (data build tool) emerged as a development framework that combines modular SQL with software engineering best practices to make data transformation reliable, fast, and fun. It makes data engineering activities accessible to people with data analyst skills to transform the data in the warehouse using simple SELECT statements, effectively creating your entire transformation process with code. You can write custom business logic using SQL, automate data quality testing, deploy the code, and deliver trusted data with a comprehensive documentation side-by-side with the code. This is more important today than ever due to the shortage of data engineering professionals in the marketplace. Anyone who knows SQL can now build production-grade data pipelines, reducing the entry barriers that previously limited staffing capabilities for legacy technologies. In short, dbt turns your data analysts into engineers and allows them to own the entire analytics engineering workflow.<\/p>\n

\"\"<\/a><\/p>\n

dbt has two core workflows: building data models and testing data models. It fits nicely into the modern data stack and is cloud agnostic – meaning it works within each of the major cloud ecosystems: Azure, GCP, and AWS. However, the biggest advantage of dbt is its new approach to building pipelines which traditionally have been quite clunky and inefficient. Some of the most prolific issues with the standard ETL workflow are:<\/p>\n