From our previous article, we could see that implementing a strict development policy (which combines permissive and restrictive rules with appropriate emphasis) provides an opportunity to work with Junior developers on the principle of "a more transparent architecture leads to a lower entry barrier".
However, IT domains with permissive standards can easily fall victim to the innovation drive of individual senior professionals or banking sectors. Driven by personal career goals, business pressure, or perhaps new technologies, the result can be an IT landscape that is motivating in the short term, but unsustainable and difficult to operate in the long term. Enterprise IT systems can easily fall into the trap of forced innovation. When should we start wondering if we have also fallen into it?!
Only senior colleagues can be employed in the given area;
Successor systems have already been introduced alongside several systems intended for decommissioning or currently being phased out;
Parallel use of various applications built on DWH with similar purposes;
Disproportionately many pilot projects;
Excessive communication of innovation values;
Recently introduced systems (in an enterprise environment, this can be as much as 3-5 years) are being decommissioned;
Many vendors are present;
Constant questioning of our own processes, continuous modification of our own IT processes;
Frequent application of Data Lake;
Frequent application of Data Vault.
While the first items on the list probably need no explanation, Data Lake or Data Vault do... To understand the application of the Data Lake concept in an "innovation squeeze", we first need to understand the main data warehousing trends and their relationship to it, which we write about in our upcoming article "The Data Lake Concept and the Used Battery".
Using Data Vault can be rewarding in special cases where it is worth significantly increasing the complexity of our loading processes and data model, which in turn noticeably increases the cost of our additional developments. Conversely, if we have applied Data Vault modeling techniques reasonably and not as an end in itself, faster-running processes, highly normalized data, lower storage requirements, and lower computing capacity demands can tip the scales in our favor.
In many areas of IT, we encounter business domains where high-level innovation is essential. However, it is important to recognize that the world of data warehousing is a fairly mature field.
An important aspect – among many others not included in this article – is that professional standards can also provide a sense of security, which in the long run reduces developer turnover. Of course, an interim refactoring or rethinking of policies can easily push the team into the anger phase of the change curve outlined by Kübler-Ross, an approach that well describes the psychological phases of team members' attitude toward innovation: denial, anger, bargaining, depression, acceptance. However, even then, we should strive to build a good, long-term professional environment rather than securing the already familiar but unsustainable working environment of developers.
As a data warehouse manager, one of our hardest tasks can be to avoid falling into the trap of forced innovation while not striving to maintain the status quo at all costs. Paradoxically, the professional fulfillment of today's managers can lie in introducing new technologies or managing less justified pilot projects. The success associated with these is credited to the manager in question, unlike the extra risks and technical debts that arise over a 5-10 year horizon, the responsibility for managing which will be inherited by the successor of the colleague who has since changed jobs.
A challenge can be the innovation demands coming from business areas (for example, introducing an MS BI in an Oracle environment), which in some cases can significantly affect the IT infrastructure surrounding our data warehouse and the competency needs of developers and business colleagues. These can even bring about political realignments in the IT-business relationship, as well as unplanned or long-delayed refactoring tasks.
Digital immunity
Although we have seen many SDLC (Software development life cycle) approaches in recent decades, the data warehouse projects of the 2000s were characterized by development in the waterfall model, where the strict sequence of project phases (specification, development, testing, deployment) and traditional DWH project management techniques (which we will write about in a later article) forced the given development to be produced in accordance with the quality expectations of that time.
As the business environment accelerated, these quality expectations changed during joint work with IT departments, including data warehouse departments. To use today's fashionable term: the need for VUCA (Volatile, Uncertain, Complex, Ambiguous) compliant demand fulfillment emerged, where business needs faithfully reflect the uncertainty, volatility, complexity, and ambiguity characteristic of the business environment.
At this point, we need to get to know the concept of digital immunity, where the concept was brought to life partly by the changes in business expectations detailed above, and partly by the conceptual shifts perceptible in the IT management triangle.
By digital immunity of software, we mean a combination of practices and technologies where quality expectations point towards rapid implementation, rapid bug fixing, and rapid adaptation. We increasingly encounter the term "Minimum Viable Product" (MVP) in data warehouse pilot projects. [5]
A methodology that promotes digital immunity can be the ErgonomX methodology, which also received recognition from the IT-Business Award. By rethinking user interfaces, it is capable of providing a competitive advantage, but this also includes the introduction of a CI/CD pipeline, which is less common in domestic data warehouses.
Somewhere between the hype and reality…
As a data warehouse manager, we encounter many concepts used by businesses that can lead us into an innovation trap, forcing us onto a path of accumulating technical debt. At the same time, it is important to see that every era has its fashionable scientific fields, be it BI, AI, ML, DevOps, Multi-cloud, or Data Governance. It is important to see beyond the terms, even during a business-driven system implementation, as using these terms can easily become an entry point to management. Whichever new method or tool we decide on, it is important to understand the underlying content - somewhere between the hype and reality.
