The Fallacy of the Developer Way
In the fast-evolving world of data, we are nearly half a decade into the widespread adoption of engineering best practices to drive more efficient workflows and higher-impact data products. The promise of the developers' way, to create a “multiplier effect on the output of an analytics team,” is compelling, and it leads most teams to embrace an engineering mindset.
“Developers are like power users...but we are all beginning to work and think like developers…as we adopt [the developer's way] it unleashes our ability to focus on the best use of time.” - David Ulevitch, General Partner at A16Z
The adoption of the engineering mindset has led to the explosion of new tools that enable data and analytics engineering workflows. Most notable is the widespread use of dbt, which enables teams to transform their data using the same practices as software engineers use to build applications.
Yet the dirty secret underlying these innovations is that data teams still find themselves frustrated, and often complaining that they need to navigate the chaos of their workflows, as Benn Stancil describes it. “You don’t hear engineers saying, ‘This is chaos!’ Why are data teams so bad at this?” Stancil says.
Data teams say that they need more time to focus on the deep work of engineering amidst all of their competing demands. Every day, they face the clash between the relatively objective and linear nature of development work and the iterative and messy reality of data work. This can simmer under the surface for teams until it bubbles up and explodes as interruptions undermine the productivity of data teams, and the business finds it does not have what it needs.
While applying a pure engineering mindset to agile project management, for example, is a helpful starting point for data teams, it is merely just a starting point. If data teams want to keep up with the competing demands put on them by the modern organization, simply embracing JIRA is not—and will never be—enough.
A Diagnosis: Why Agile Alone Falls Short for Data Teams
If we look to DevOps as a guide for how data teams should work and organize themselves, we should first recognize it includes several fundamental practices. These include Version Control, CI/CD, Shift Left for security and testing, and of course Agile.
But just because you have embraced DataOps, and use dbt for CI/CD & testing, and GitHub for version control, does not mean that Agile is the right fit for your data team.
Agile is where data and data products tend to go to die…data is an interactive process of asking questions, learning something, usually asking another question…so that folks can make decisions or that we can build products... whereas software engineering is that we know what we are doing, and we [just iterate on how to do it]. - Hilary Mason, CEO of Hidden Door; former GM of Machine Learning at Cloudera.
At Workstream.io, we also believe that data work is more interconnected, collaborative, and less modular than software engineering workflows. Data pros have different project management needs than engineers because data teams typically are:
Face daily needs to balance the competing demands of core engineering work with countervailing forces such as incident management, or data consumer support
Because engineers typically wear just the developer hat and are supported by additional roles like PMs, customer support, and DevOps teams, relatively linear agile frameworks work well.
While Agile is flexible enough to allow for changing priorities, it is still designed for engineering workflows that are linear, predictable, and measurable. As a successor to the assembly line, agile frameworks help engineering teams transform inputs like tickets into outputs like features, or customer support tickets into resolutions.
On the other hand, agile in isolation does a really bad job of helping data teams manage the multi-directional vector of their workflow. Data teams deal with inputs and outputs that can come from any direction (ex: a stakeholder request might drive current engineering work, or vice versa), and a single team does all the work to prioritize, support and execute. Most data teams don’t have the luxury of a PM to prioritize feature work, a customer support team to serve as a traffic cop for bugs and support tickets, or a DevOps team to handle outages.
Data teams that go no further than agile often end up:
Don't Go Chasing Waterfall, Simplify Your Kanban with Connected Data Workflows
All of this generally means that data teams want to keep their core agile simple and that the solution to their project management woes is not more complex.
A Kanban board meets the needs of most core data and analytics engineers, and so teams should avoid introducing other, more process-oriented frameworks like SCRUM (which folks often wrongly use interchangeably with agile), Feature Driven Development, or SAFe.
Instead, the root cause of a data team’s success rests in their ability to manage these priorities across their other competing vectors of work. Any core agile workflow must be sufficiently flexible to accommodate the needs of data incident management, day-to-day support workflows, or data consumer enablement, among others.
Agile tools such as JIRA, or even newer startups like Linear, do not do a good job of this. At Workstream.io, we support clients using both JIRA and Linear, and teams seek out our integrations because they don’t have a PM to manage priorities, and are also responsible for handling data incident management and data consumer support.
Fundamentally, the world these tools were designed for is linear (pun intended). A world that is simpler, and more modular than those which data teams inhabit. To do it better, data teams need to keep their agile simple and focus on managing the competing demands of their other interconnected workflows.
Conclusion
While Agile has proven invaluable in driving data work forward, its future is limited if data teams do not evolve it to their unique needs.
By recognizing the iterative and interconnected nature of data workflows, data teams can better design core agile processes that help them navigate all of the competing demands on their time. In the ever-evolving landscape of data management, adapting methodologies to suit the specific needs of data teams is key to unlocking their full potential. If you’re ready to streamline and integrate all of your data workflows, check out this 5-Part demo series on how your data teams can do better with Workstream.io for free in 2024—shit show not included.
And don't miss out on our April Promo, if you activate a free account and Slack alerts by 4/31, we'll send you $100 to sweeten your tax refund.
In this article, we discuss how data-driven teams can tame entropy and empower stakeholders with a single access plane for analytics.
Data asset entropy has been an issue for as long as data has been used in decision making. But in the modern organization, where analytics tools have exploded in number and type, entropy has become a huge stumbling block to effective data-driven operations.
The goal of your workflow is not just to uncover valuable wisdom from your data. It is crucial to communicate those actionable data observations to the organization which can use them to make decisions.
Receive regular updates about Workstream, and on our research into the past, present and future of how teams make decisions.
