Management of Data Warehousing Projects: Challenges and Best Practices I., we were introduced to a DWH anomaly, examined the role of data quality in relation to data warehouses and partner systems, and discussed the difficulties of positioning completed developments before management. We also presented the possible impacts of data corrections and explained a DWH term that has taken root in English-speaking regions.  

In this article, we discuss the fine-tuning of data warehouse projects from various perspectives, and examine some typical project management situations from both the project manager's and the vendor's points of view.  

Deadline-Optimized Developments

Project managers who optimize their projects for meeting deadlines often – even unconsciously – resort to the option of project compression. This can easily lead to situations where tasks on the critical path, and consequently certain project phases, overlap. 

By project compression, we primarily mean the "Fast Tracking" technique according to PMI (Project Management Institute) terminology, which is particularly characteristic of IT projects in domestic insurance companies and banks. Fast Tracking refers to the parallelization of tasks originally planned to be executed sequentially. In contrast, a less frequently used compression technique is "Crashing", which involves bringing in additional resources. 

Let's look at an example: 

• At 90% completion of the source system data cleaning task, we already begin defining the building data mining logics.  

• As a result of the above, we can only complete the definition of data mining logics up to 80% completion.  

• Following the completion of the source system data cleaning task, it becomes necessary to update and finalize the specification of the data mining logics. Although the remaining tasks account for only 20% of the total work, executing them requires twice the planned resource budget, i.e., around 40%.   

It is easy to see that in the case of insufficiently planned, deadline-optimized developments, the frustration of the specialists participating in the project increases significantly. In a vendor role, we can particularly often find ourselves in a situation where – in the absence of proper communication – we cannot validate the time effort actually required for the remaining 20% of tasks, which may be up to twice as much. (Based on the example above, assuming a project-based contract structure.) 

If a project manager optimizes the project for meeting deadlines, it is advisable to communicate this clearly already in the initial phase.  

Tension typically arises when, in a deadline-optimized project, the project manager sets the expectation for participants to prepare estimates optimized for resource efficiency. 

Colleagues responsible for estimation often price in these circumstances, which in practice is called a development buffer. This can fund the extra work resulting from the overlapping of project tasks, which – especially in the absence of sufficient project management experience – is a common phenomenon in data warehousing projects.  

However, it is not advisable to list the development buffer separately in the estimates, as it is not linked to a predefined deliverable and can thus trigger resentment from a sponsor not well-versed in the specific topic. This development buffer is often dispersed across sub-tasks, which can be disadvantageous because the DWH project manager is less aware of the real financial consequences arising from the reorganization of project tasks.  

From a vendor's perspective, it is worth highlighting that as a result of reorganizing the work, X person-days of cost-free extra tasks have arisen. It is important to make them aware of this, because if this happens repeatedly, sooner or later we will have to request additional resources from the project. This is easier to do if we have previously performed certain extra tasks without an extra charge, even at the expense of the development buffer.  

In general, there are two cases where the use of a development buffer is truly justified: 

• There is no opportunity for thorough project preparation and/or 

• The project is optimized for deadlines, so it may be necessary to buffer less efficient resource utilization. 

If we work with an external vendor on a project basis as a client-side project manager, it is highly important to discuss in advance the pricing of individual developments, as well as to understand the vendor's ideas about individual sub-tasks, their dependencies, and the pricing policy applied during any reorganization of work. These are typically not included in the proposal. 

Developments Optimized for Resource Utilization, the Cost of Context Switching 

In addition to the above, we can also optimize our data warehouse developments for resource utilization; in this case, tasks on the critical path are not compressed. However, as project managers, we can easily find ourselves in a position where we take the path of least resistance and do not plan this way, disregarding the maintenance of the long-term motivation of project members. 

In this case, all essential prerequisites for performing the task are available, and there is no need to stop and then restart sub-tasks in progress. So, remaining with the example above, the data cleaning operations on the partner system data have finished, and then the specification and subsequent development of the data mining logics can be carried out without interruption on the files tested on the source side.  

While business experts typically think in business definitions and abstract business concepts, developers are more characterized by analytical, slow thinking, where the cost of context switching appears exponentially when tasks are interrupted and restarted. We can save this cost with project planning optimized for resource utilization, and we typically pay its price when the project starts running out of steam. The latter concept is used to describe human thinking, but its roots trace back to the memory usage of processors.   

Estimation of Developments 

At this point, we only examine how the pricing of our DWH development takes shape, taking into account international standards regarding the accuracy of estimates (which, by the way, also corresponds to domestic practice). Before moving forward, it is important to state: we can speak of indicative estimates, budget plan estimates, and detailed, precise estimates. What we optimize our estimation for realistically has an impact on the detailed estimate - even shifting it from -5% up to +30%.  

Conclusion: 

Reading the above, we can easily see that as professional leaders or team leads, we might be more interested in projects optimized for resource efficiency, whereas as project managers, we can easily shift towards work organization optimized for deadlines.  

If the stability and sustainability of our data warehouse, and keeping our colleagues motivated, are of value to us, we should try to optimize our projects for resource efficiency when working with the vendor and enforce this in the vendor's pricing.  

When working with the vendor, it is worthwhile to provide project conditions where the vendor can develop a sense of ownership (regardless of the fact that the code of the given system is typically not exclusively theirs), even by entering into strategic partnerships or applying shared ownership. 

To eliminate development and management difficulties occurring during the further development of data warehouses, we can use the CLARA, PWM, and Dexter applications.   

Let us understand and have high-level decision-makers accept that data warehouse projects often have characteristics that make the use of traditional project management tools only partially cost-effective. If we let this be forced upon us anyway, long-term sustainability, maximization of business value, and even our colleagues' professional commitment will surely suffer.