DBT(Data Build Tool)
In the evolving world of data engineering, a new star has emerged, and it’s not a product from one
of the tech giants. Meet dbt (Data Build Tool), a game-changer that’s reshaping the landscape of
data transformation and management.
Originally developed by Fishtown Analytics in 2016, dbt started as an open-source project
designed to simplify data workflows. Since then, it has evolved into a crucial tool for modern data
teams, bridging the gap between traditional data warehousing solutions and contemporary software
practices. Here’s why dbt is making waves and why it’s poised to lead the charge in data
engineering.
1. Built-In Version Control
Version control is a cornerstone of modern software development, and dbt gets it right by
integrating seamlessly with git. This integration ensures that teams can collaborate effectively on
data projects, maintaining a clear history of changes and reducing the risk of breaking changes.
Unlike many traditional data tools, which treat version control as an afterthought, dbt makes it a
fundamental part of its workflow, enhancing both collaboration and code stability.
2. User-Friendly and Efficient
One of dbt’s standout features is its simplicity. It allows users to create complex data pipelines
quickly using SQL and Jinja templating. This means you don’t need to master a new language to
handle data transformations — dbt leverages SQL best practices to streamline the process. Its
intuitive approach makes data engineering accessible and efficient, setting it apart from more
cumbersome tools that require extensive learning curves.
3. Automated Documentation
Documentation is often neglected in traditional data warehousing solutions, leading to confusion
and errors. dbt addresses this issue by automatically generating documentation for data
transformations and visualizing data lineage. This built-in documentation is standardized and easy
to understand, providing a clear view of how data moves and transforms through your pipelines.
This feature significantly reduces the risk of miscommunication and enhances data transparency.
4. Broad Vendor Compatibility
dbt’s versatility is another reason for its growing popularity. It supports a wide range of data
warehousing platforms, including Amazon Redshift, Google BigQuery, Snowflake, Azure Synapse
Analytics, and Databricks. Additionally, it works with popular open-source databases like MySQL
and PostgreSQL. This broad compatibility makes dbt a flexible choice for various data
environments and uses cases.
5. Comprehensive Testing
Ensuring data quality is critical, and dbt excels in this area with its robust testing capabilities. Users
can run built-in data quality checks or create custom tests using SQL queries. This proactive
approach to testing helps maintain high data quality and reliability, a feature often missing from
other data warehousing solutions.
Challenges and Considerations
Despite its strengths, dbt isn’t without limitations. It requires an existing data warehouse for
storage and query processing, which means it can’t function as a stand-alone solution. Additionally,
while dbt is user-friendly, newcomers may face a learning curve and will benefit from the extensive
documentation and community support available.
Conclusion
dbt is redefining data engineering by integrating modern software engineering practices with data
management. Its focus on version control, ease of use, automated documentation, compatibility,
and testing positions it as a leading tool in the field. As the data engineering landscape continues
to evolve, dbt’s innovative approach promises to drive future advancements and set new standards
for data transformation and management.
Use case: Rocket Money modernizes financial reporting
with dbt Cloud
• More than 3,000 Tests implemented to ensure data quality
• No deficiencies in SOX audit after modernizing system
"Having this automated Quote-to-Cash system run in dbt with our test suite allows us to
confidently and quickly close our books each month." said Amber Oar, Staff Analytics Engineer
Data-driven digital finance
Founded in 2015 Rocket Money has gained rapid recognition for its personal finance app. Through
an array of features such as budgeting, spending tracking, bill negotiation, and subscription
cancellation, Rocket Money allows its users to precisely control their spending.
Like many other financial organizations, Rocket Money’s business model leans heavily on reliable
data and analytics, with processes such as revenue recognition, cash forecasting, and monthly close
depending on high-quality data.
The need for a quote-to-cash system
The Rocket Money team quickly found that as the business expanded, keeping track of high-
volume information wasn’t a simple task. The company has many different revenue streams and
works with several different payment providers, which led to the requirement of a single system
to serve as a source of truth to combine these disparate data sources.
Amber Oar, Staff Analytics Engineer, and her team quickly realized that if the business was to
marshal the data needed to keep decision-makers up-to-date with the latest information, it needed
a single source of truth for financial reporting - one that was GAAP (Generally Accepted
Accounting Principles) compliant and easy to audit.
“We didn’t want different metrics being calculated in different ways by different teams when they’re
all supposed to be the same thing,” explained Amber. “Whenever we are calculating revenue for
a given month for our company, we want to make sure everyone is pulling it the same way and
ending up with the same number.”
The team needed a quote-to-cash system to help them ensure their financial data was as accurate—
and accessible—as possible. To comply with GAAP, the new system needed to track users and
transactions through the entire accounting lifecycle, from purchase orders and invoices to revenue,
cash, and accounts receivable.
Mapping complex revenue streams
Given the business’s complex financial workflow, the Rocket Money team knew they needed
support from specialized data tools. They singled out dbt Cloud to support the new system.
“The previous process was very time consuming. We needed a system that would allow us to scale
and also quickly close the financials each month.” said Amber.
One of the first challenges was mapping Rockey Money’s complex web of revenue streams. This
was a time-consuming task that stretched across multiple teams, which included product,
engineering, and accounting all needing to coordinate efforts.
Fortunately, dbt Cloud significantly assisted the team in managing this complexity. The tool’s
modular architecture allowed them to reduce masses of complex financial logic into more
manageable, maintainable layers. By breaking down intricate processes and data flows into simpler
components, the team could more efficiently manage and update their financial systems.
Building trust with rigorous testing
One of the cornerstones of any financial system is reliability; when cash is on the line, you don’t
just need accurate numbers, you need to be able to prove they are accurate. The Rocket Money
team ensures data quality with a rigorous testing scheme.
Before introducing the quote-to-cash system, a lot of this testing was conducted by the finance and
accounting team. As the product scaled, the team realized that this would not be sustainable long
term. Now, it runs 3,000 tests (that test both quote-to-cash and other production data models) daily,
which include business logic and accounting logic checks to ensure that everything is operating as
expected.
These tests create accountability for the data produced by the new system and allow for easy
collaboration across teams. This comprehensive testing gives the Rocket Money team confidence
that upstream changes will not cause unintended regressions.
Guaranteeing compliance with auditability
In 2021, Rocket Money was acquired by a publicly traded company. Now that the business was
part of a publicly traded company, the company’s accounts needed to be consistently easy to audit.
This means having not just clear data but also solid lineage and documentation.
dbt Cloud set up the team for scaling success by supplying access to auto-generated documentation
and audit logging. In addition, data diffs, a tool that compares datasets, furthered auditability.
“We wanted to make sure that we had an audit-worthy system that could pass any audit even before
we needed it,” said Amber.
This careful planning paid off when Rocket Money underwent its first SOX audit—an annual
requirement under the US Sarbanes-Oxley Act.
“We passed with zero deficiencies,” smiled Amber. “That's great. This system is exactly the kind
of thing auditors look at during SOX audits.”
Streamlining financial processes
Introducing an automated system sped up the team’s month-end close tasks and provided a single
source of truth to power the business’ enterprise reporting.
“Introducing dbt and the Quote-to-Cash process has made close a lot less stressful because we
have a testing suite that runs every single day,” explained Amber. “If there is a bug that is going
to impact our accounting team, we know immediately, and we're working with the development
team to fix it before the month closes.”
The extra visibility has also improved communication between teams. Rocket Money’s
engineering team is now more aware of the accounting impacts of their decisions—such as
releasing products at the very end of the month. The product team can confidently assess the
impacts of experiments because the system supports production of key financial KPIs at the
granularity of a purchase.
Driving impact across Rocket Money
Introducing the dbt Cloud-powered quote-to-cash system has transformed Rocket Money’s
financial processes and unlocked cross-team alignment.
"Now the accounting team feels like they're shipping like the larger product and engineering
team,” said Amber. “They're aware of changes before they are made that may impact the revenue
streams."
Looking ahead, the team is building on the scalable foundation to implement advanced analytics
across the business as changes are made to existing revenue streams and new ones are rolled out.
