Data by itself is often useless. However, by transforming the data into human understandable insights, organizations can harness the power of data to make meaningful decisions that drive outcomes.
But making this happen is easier said than done. And traditional BI approaches often don’t meet the realities of the fast-paced and ever-changing needs of today’s world.
Therefore, to increase the odds of project success, adopt an Agile Business Intelligence project management approach.
What is Agile Business Intelligence?
Simply put, Agile Business Intelligence is the application of Agile practices to deliver BI products. OK. I admit that’s not the most helpful definition, so let’s take a step back…
What is Agile?
Agile is a mindset that emphasizes:
- Open collaboration – both within the product team and with stakeholders
- Rapid iteration to deliver small incremental usable items of value
- Continuous inspection and improvement to improve the team’s velocity over time
- Empowering the product team to self-manage
- Flexibility to adjust plans as needed
Apply these principles to BI, and you get Agile Business Intelligence. There are numerous Agile approaches to delivering a product. We’ll assess three. But first, let’s consider why we should care about Agile BI.
Why Agile Business Intelligence?
Shortcomings of Traditional Business Intelligence Projects
Traditional business intelligence projects focus on pre-planning each phase in the business intelligence life cycle. The project builds each infrastructure layer one at a time – with a significant investment in complex backend systems for the IT environment and data warehousing. The user-facing systems come later in the project. “Testing” is often a final phase.
On the surface, this traditional approach makes some sense. But dive deeper and you’ll find that this careful pre-planned step-by-step approach leads to numerous issues:
- Unknown Needs: End-users often don’t know what they want. Asking them to define every filter and chart upfront can lead to a plan that doesn’t actually serve user needs.
- Difficult to Change: Even if the end-users understand their current needs, they might not know which metrics will be important next year.
- Testing Errors: The cost of defects increases as the project continues. Imagine multiplying the same query error over and over again! If you wait until the end of the project to test, you might wind up with errors that are tricky to de-tangle.
- Delayed Value: Think of the 80-20 rule. 80% of the value comes from 20% of the solution. So why defer value delivery until everything is “complete”?
- Delayed Feedback Loop: You don’t know how people will use your dashboard until – well, they actually use it. Traditional approaches rely heavily on representations of the product like data flow diagrams or wireframes. These are indeed useful for feedback — but are not as rich as an incremental version of a useable BI system.
Agile BI – A better Approach
Instead of traditional business intelligence project management approaches, organizations should use an Agile Business Intelligence approach. When using an agile approach, rather than planning everything upfront and then delivering the whole system at the end, Agile BI looks to deliver multiple small usable increments that progress toward the finish line. This has many benefits.
- Higher Engagement: A common failure for BI projects is the lack of adoption. By involving users upfront and throughout the business intelligence development life cycle, end-users will take more ownership, understand the system’s value, and be more likely to use it.
- Flexibility: Did you plan for COVID or the impact of sudden inflation or (insert whatever major shock to your business or world when you’re reading this)? Probably not. But by having, short flexible plans such as outlined in a roadmap, you can more quickly adjust accordingly.
- Quicker Value: By delivering thin vertical slices of end-to-end value, users start to get value – even before the project ends. Instead of having a complete backend data warehouse as an initial deliverable, focus on delivering an important table with a useful dashboard built on top of it.
- Higher Motivation: What’s great about BI projects, is that you can literally see your results. It’s much more motivating to see a dashboard live with partial functionality as opposed to a complete wireframe that sits somewhere in the depths of a shared drive.
Example Kanban Board for a typical BI Dashboard Project
Agile Business Intelligence with Kanban
What Is Kanban?
Originating out of manufacturing, Kanban is a simple process flow that makes work highly visible and aims to decrease time-to-delivery.
It is not a comprehensive project management framework, but rather, can serve as a base process from which to build a solid framework.
Indeed, many Scrum teams use Kanban to manage their Sprint Backlogs. It is also an explicit artifact for Data Driven Scrum teams.
Agile Business Intelligence with Kanban
What Is Kanban?
Originating out of manufacturing, Kanban is a simple process flow that makes work highly visible and aims to decrease time-to-delivery.
It is not a comprehensive project management framework, but rather, can serve as a base process from which to build a solid framework.
Indeed, many Scrum teams use Kanban to manage their Sprint Backlogs. It is also an explicit artifact for Data Driven Scrum teams.
How Can a BI Team Implement Kanban?
It’s easy to start! Keep the Kanban board in focus to sustain results.
- Set up Kanban Board: This board is divided into various columns, each one representing the various work item phases. Most simply, this can be “To Do”, “Doing”, and “Done” but a more practical board is shown below.
- Define Work In Progress Limits. To prevent too much work from piling up into a potential bottleneck, each column has a Work In Progress Limit that sets the maximum number of items that can be in that phase at any given point in time.
- Define Work Items: The team defines the work that needs to be done and splits the overall work requirements into small bite-sized work units. Each item becomes a “card” in the initial column of the Kanban Board.
- Prioritize Work Units: The collective team or a specific team member such as Product Owner will determine which work items to deliver first based on value, effort, and dependencies.
- Implement Work: Team members focus on a small number of work items at each time. When a team member has the capacity, they will “pull” a work item forward from a previous phase.
- Measure Progress: The team inspects the board at least daily. Over time, they analyze their flow to identify and adopt practices to increase the throughput and minimize cycle times.
When Does Kanban Work for BI?
Given its light, flexible structure, Kanban can work for (nearly) any Business Intelligence team. Specific use cases include:
- Teams that field an incoming flow of “work ticket” requests from stakeholders to deliver functionality like filters, visualizations, or data sets (for self-service BI).
- Teams that struggle to estimate how long specific work items will take to complete.
- Teams that want a minimalist approach toward project management.
- Teams that focus on minimizing the time to deliver a specific request.
- Organizations that might pull team members onto different initiatives (i.e. team continuity can be fluid).
However, a team needs more than just Kanban, since as noted previously, Kanban is not a comprehensive project management framework, but rather, can serve as a base process from which to build a solid framework. Below are two frameworks that are more comprehensive.
Agile Business Intelligence with Scrum
What is Scrum?
Scrum is the most popular Agile approach for software teams. Relative to Kanban, it is more comprehensive in that it defines:
- 5 Values: Commitment, focus, openness, respect, and courage
- 3 Team Roles: The scrum master, product owner, and developers
- 5 Events: The sprint, sprint planning, daily scrum, sprint review, and sprint retrospective
- 3 Artifacts: The product backlog, the sprint backlog, and the increment
The Scrum Guide is Scrum’s definitive guide. It emphasizes the team as a self-managing cohesive unit focused on delivering functional deliverables on a pre-defined schedule (such as every two weeks).
How Can a BI Team Use Scrum?
The team need is organized with each team member taking on one or more of the 3 defined roles:
- The Scrum Master facilitates the overall process as a servant leader for the team and the broader organization.
- The Product Owner defines the overall Product Vision. They break the Vision down into smaller units of work in the Product Backlog which serves as a wish list of deliverables.
- The Developers are the team members who directly work on the product. This includes data engineers, data analysts, BI developers, and quality engineers.
The Sprint (the pre-defined period of time to do a burst of work) beings with Spring Planning which is when the Developers peel off the top priority backlog items from the Product Backlog into the Sprint Backlog. This Sprint Backlog represents their commitment by the end of the Sprint. During the Sprint, the team meets at least daily (Daily Scrum) to develop short-term plans. They collaborate closely as a cohesive unit to deliver to the Sprint Goal. At the end of the Sprint, the team solicits feedback of its Increment (completed work) during Sprint Review. They then assess how they can work better during Sprint Retrospective.
When does Scrum Work for BI?
Scrum works great for many BI Teams. However, given its additional definition, Scrum may not be appropriate for all teams. Specifically:
- Scrum requires predictable continuity of the team members. Short projects (less than a couple of months) may not hit a critical mass of time to allow the team to perform. Moreover, the team members should be fully dedicated to the Product and should not be pulled around for other initiatives.
- Scrum upsets traditional top-down management philosophies. Therefore, Scrum only works in organizations that empower their Scrum teams to own their processes and deliverables.
- The work should be well-understood before the start of each Sprint. This should not be a constraint for too many BI teams (but is a major complication for Data Science Scrum teams).
Beyond those exceptions, Scrum can be an effective process for business intelligence teams. It offers numerous benefits over Kanban. Specifically:
- The team focuses on value with each Sprint. For example, an early Sprint might deliver a set of filters, visualizations, or a dashboard module. Then the follow-up Sprint might improve on that set of deliverables. A third Sprint could focus on a new set of functionality. The cohesiveness of each set of functionality shows progress and builds momentum.
- The overall structure is clearer. In Scrum, you know who serves which role, when your meetings are, and what the upcoming deadlines are.
Agile Business Intelligence with Data Driven Scrum
What is Data Driven Scrum?
Data Driven Scrum (DDS) is a new Agile framework specifically designed to help data driven teams improve their communication and collaboration.
At its core, DSS is a modified version of Scrum with similar meetings, roles, and artifacts.
However, DDS changes some of Scrum’s most constraining aspects to be more flexible for the research-oriented nature of data driven projects.
The biggest shift is the DDS concept of an Iteration (versus a Scrum sprint).
Specifically:
| Scrum | Data Driven Scrum | |
|---|---|---|
| Logical Unit | Sprint | Iteration |
| Time Constraint | Each Sprint is the same length on a pre-scheduled basis. | Iterations lengths and start/end based on the completion of work. |
| Overlapping Constraint | There is only one Sprint at a time. | Iterations can overlap based on the timing of work completion. |
| Meetings | Ceremonies coincide with the pre-defined cadence of Sprints. | Just like Scrum, the meetings occur on a regular calendar bases, but Iterations don’t necessarily start or end with those meetings. |
Key differences in the cadence of work between Scrum and DDS
How Can a BI Team use DDS?
A BI Team would use DDS with the same roles and (with some modifications) similar meetings to Scrum. However, the refined Iteration definition restructures the way the team lines up it work.
- A Scrum BI team will constantly face a conundrum in setting up each Sprint. Commit too much, and you’ll miss your Sprint Goal. Undercommit and you’ll have unproductive downtime toward the Sprint’s end. On the other hand, DDS teams don’t have to stress about this. They don’t have to develop granular estimates for each work item (or “story”) to figure out what “fits” into the upcoming Sprint. Rather DDS teams can focus on the logical cohesive set of data cleaning, filters, visualizations, and other deliverables that combine together into an Iteration – without concern as to whether that work can be done in a pre-defined time window.
- A Scrum BI team plans on a pre-defined cadence to line up each Sprint on a calendar basis. However, a DDS BI team starts an Iteration as soon as it has the capacity to do so. This flexibility opens up scenarios whereby – for example – a data engineer can start setting up the data structures for the following dashboard in “Iteration 2” while the BI developers are still wrapping up work on the current dashboard in “Iteration 1”.
When does DDS Work for BI Teams?
DDS has more structure than Kanban but is more flexible than Scrum. As such, the use cases of DDS fall somewhere in between. It works well for teams that:
- Enjoy the visual flow of work afforded by Kanban
- Want to focus on an incremental set of deliverables defined in an Iteration
- But don’t want to be constrained by timeboxed Sprints
Learn More
With the ever-changing world before us, Agile Business Intelligence processes can support how we can rapidly convert data into meaningful information that drive business decisions. Kanban, Scrum, and Data Driven Scrum are three Agile approaches that can help you deliver value from your BI initiatives.
To learn more:
- A comparison of traditional and self-service BI from softwareadvice.com
- Business intelligence projects feature many of the same elements as Data Analytics Project Management.
- What are the factors that impact the selection of a project management methodology? Many of the same factors outlined in this research paper from Jeff and me also apply to business intelligence.
