Project Management in a BI/DW/PM environment


Business Intelligence and data warehouse projects are often seen as costly and never ending on time or even worst as costly projects that never deliver any benefits. As with any ICT related project, this can be true in some cases but quite often the root cause is incorrect or lack of project management during those projects.

BI/DW projects are not different from any other type of projects; they need to have clear deliverables and benefits defined, the efforts needs to be estimated and planned in advance and their execution must be controlled in order to guarantee success.

But then why are so many Business Intelligence & Data Warehousing projects running out of control or failing ? Why are some very experienced project managers failing to manage this type of projects ? Maybe because Business Intelligence and datawarehouse projects have some specificities that are too often underestimated or ignored. In this insight we will review the benefits to be expected from good project management for a BI/DW project as well as the specificities of those projects. We will finally define some best practices to tailor the existing project management methodology.

Project Management is needed as for any project

As for any project, Business Intelligence or data warehouse projects need a project management methodology to plan, monitor and control their activities in order to achieve the project objectives within the expected performance targets for time, cost, quality, scope, benefits and risks.


Project Management in a BI/DW/PM environment

Project Management is an on-going activity which will last during the complete lifetime of the project and will constantly review those aspects to ensure that the project is still matching the targets defined during the initiate phase.

· Costs: The cost of the project must be defined and monitored to be able to compare it with the expected benefits and to assess if the project is/remains affordable. This statement is valid for a BI or DW project as for any other project as it has to deliver value to the organization where it is run.

· Timescales: Like any project, a Business Intelligence & Data Warehousing needs a plan and its execution must be monitored and controlled to avoid the "never ending on-time” feeling that the users can have. It will also help the project manager to report on the progress and to communicate with the other parties involved.

· Quality: The expected quality must be clearly defined at the beginning of the project in order to be able to assess it correctly and to spend the appropriate effort during the project to reach it. Quite often this part is neglected during the initiation of a BI/DW project which can lead to a final product which is not matching the quality expectation or to a project where significant efforts are spent to reach a quality which is not needed by the business. A correct definition of the quality expectation and needed accuracy of the data will avoid those issues.

· Scope: The scope of the project needs to be clearly defined during the ‘Initiate' phase in order to be able to assess the impact of possible changes, which might occur during execution. This is a challenging task for a BI/DW project as quite often the scope is unclear or vague at the beginning of the project and refined later, but the necessary efforts should be made to avoid this. An incorrect or incomplete scope definition generally leads to projects exceeding budget due to uncontrollable changes.

· Risk: BI/DW projects are mainly focusing on "data” so most of the risks encountered by the project will come from there. It is important to clearly define the limits of risk the project is ready to accept in order to be able to decide corrective action or to stop the project if those risks occur. It might seem obvious but quite often Business Intelligence & Data Warehousing projects are running over budget because unidentified risks, like data quality issues, occur during their execution and require a lot of effort to handle them.

· Benefits: Maybe the most difficult aspect to define for a BI/DW project but maybe also the most important one. Why is the project started ? What are the benefits expected from it by the organization ? The Business Intelligence & Data Warehousing projects don't have usually a direct effect on the business of the organization so it is difficult to define measurable benefits that will be reached thanks to the product delivered by the project. But other aspects should also be considered like the number of FTEs needed to produce the monthly reporting or the moment by when the weekly figures can be delivered thanks to the new solution. Those benefits can be measured and therefore will help to measure the benefits and the performance of the project.

The existing project management methodologies provide a good framework and tools to do all those tasks. They will help the project manager to plan, monitor and control the project in a structured way but in the context of a BI/DW project they will also require some adaptations to be used in the most efficient way possible.

Business Intelligence and Datawarehouse projects specificities

BI/DW projects have specificities which make that the usage of standard project management methodologies might be difficult without some tailoring. This tailoring should not denature the methodology but adapt it to the needs of this type of projects. The main specificities which influence the project methodology can be described as followed:

· Multiple competence and technology

In a Business Intelligence & Data Warehousing project a wide range of technologies and competences are involved. From the technical and business side the resources involved will have different backgrounds and experiences but they will need to work closely and intensively together to accomplish their tasks.

This will require intensive communication and coordination work in order to keep the team focused on the same objectives as well as accurate planning in order to ensure that everybody is delivering what is expected by other team members. Prior to the kick-off, specific attention must be given to defining the role and tasks of all the groups involved and to make sure that this is known by everybody within the project team. This will help to avoid communication problems or frustration during the project.

· External dependencies

The execution and success of a BI/DW project will strongly depend on external products or circumstances on which the project will have no or little influence. The source systems and the data they contain will be the main materials used by the project and by definition use of data is exploratory. Therefore less predictable data issues and opportunities will appear. Those issues and opportunities will occur with a higher frequency than for other project types and will need to be handled carefully to ensure that because of them the project will still deliver the expected products without exceeding the defined costs and time targets.

The project management methodology will then need to be adapted to have a light and efficient issue management processes to treat those issues with a minimum overhead work and without losing control over the project.

· Continuous Requirement Flow

In the context of a BI/DW project it is very difficult to have final requirements set prior its actual kick off. Business changes, regulatory changes or simply new analysis opportunities will lead to requirement changes during the project. They shouldn't be ignored as they might have a direct influence on the delivered value and business justification of the project. If -for example- the data of the reporting delivered does not comply with new regulation rules or with the new strategy of the organization, the project has reduced business value. But in the meantime some of the new or changing requirements may need to be rejected or postponed to avoid a continuous expansion of the project scope which will lead to a never ending project.

Therefore the project management methodology in place should have a strong change management process in order to be able to classify and integrate those changes within the project with a minimal disruptive impact. As for the issue management process it should be light enough to not create too much overhead work but also clearly defined to capture and treat all the changes without losing track of the project objectives.

BI/DW project management methodology

So then what it is the ideal project methodology for Business Intelligence & Data Warehousing projects? The answer is none. Depending on the context, structure and standards of the organization as well as on the size and complexity of the project the methodology used can and will be different but some general principles should always be followed.

· Hands-on project management

In the complex and changing environment of a BI/DW project it is important to have a hands-on, pragmatic approach to project management. All possible changes, risks and issues need to be captured as soon as possible and to be handled in a structured approach to track, treat and classify them without creating significant overhead.

So the concept of the project manager being in his ivory tower and overlooking the project from a distance is doomed to fail. He needs to have ‘the feet on the ground' and be accessible by having frequent contact with his team members. He also needs to have enough empowerment to threat most of the issues himself or -if this is not possible- have a direct access to the project sponsor(s). Without that, the risk of having significant delay and exceeding costs due to a long decision process or due to too many changes on the scope, will be very high.

· Iterative approach

All the BI/DW projects should follow an iterative approach rather than a big bang approach. The plan should be divided in short and manageable phases (maximum 2 to 3 months of development time) which will allow an easier control and assessment of the progress. Most probably the first phases will deliver none or limited products but the scope and benefits can be reviewed at the end of each phase and possible changes to the project due to new needs, discovered issues or external factors can be easily integrated in the next phase. It will also make the project testing and validation process easier to control and to execute.

But usually this kind of approach is harder to sell. So the project manager will need to push for such an approach as this will help him to avoid two typical failure causes; the lost of interest of the users for the project due to too long development time and a mismatch between the delivered products and the users expectations that might appear.

· Users and management involvement

Finally users and management should be involved at all stages of the project and not only at the beginning and at the end of it. The exploratory nature of a BI/DW project requires constant review and validation of the data as well as of the requirements by the users. Assumptions and decisions will need to be made by management all the time. This is also a very good way to ensure the acceptation of the new products when they are released.

The project manager will ensure that involvement by maintaining constant communication channels between the final users, management and the project team.

It becomes then clear that the role and personality of the project manager is central in this kind of projects. The person selected to fulfil the PM role will need to be experienced in project management techniques but also in business intelligence and data warehousing to be able to handle such a project successfully. He should have a structured and at the same time pragmatic project management style to be flexible enough to cope with changing environment of a BI or DW project without losing track of the project objectives.


To manage a BI/DW project is fundamentally not different from managing any other kind of project. It requires to plan, monitor and control the activities in order to ensure that the project delivers the expected value to the organization.

But the resources and technological diversity, the continuous information flow and their exploratory nature are Business Intelligence & Data Warehousing projects' specificities that make standard project management methodologies not directly applicable to them. They will require some tailoring in order to use them in the best way possible to support the project without creating too much overhead.

As such, the project manager will face the dilemma between applying a strict project management approach or a more pragmatic one. Each organization and project is different and in the context of a BI/DW project the project methodology to be used will need to be adapted. Therefore possibly the most important decision of the project is to appoint the right project manager, combining the right mix of project management and Business Intelligence & Data Warehousing skills & experience.