Project Assurance
19.March.2024
Data Analytics and Artificial Intelligence: Essential Tools for a Major Project Assurance Strategy.
Assurance Strategies
Although there are many ways of implementing an Assurance Strategy, perhaps the most structured and widely used is the Three Lines of Defence (3 LoD) Model, which will be used as an assurance strategy exemplar for this article.
Following the financial crisis of 2008/9, there was a realisation that many organisations did not have a structured approach to managing risk and did not provide assurance to Board level that risks were being appropriately identified and controlled. In January 2013, the Institute of Internal Auditors published their 3 LoD Model. Its aim was to provide a comprehensive framework to consider the overall arrangements for managing risk and exercising control within an organisation.
Project Surprises
Setting up a 3 LoD organisation is not easy or guaranteed to achieve its aim. Crossrail, for example, had a full 3 LoD system in place and yet, on 31 August 2018, when the project had been inaccurately reporting to be 97% complete, there was an announcement of a 9-month delay, which eventually became 4 years with the inevitable resulting cost overrun. The question is why, with all that governance in place, did Crossrail not know that their project schedule was out of control.
Data Analytics and AI
This article is not a Lessons Learned for Crossrail who are by no means alone in their experience. Rather, it looks at the 3 LoD Model, to see how it should be applied to major projects and the essentiality of using Data Analytics and AI (DA&AI) to enable meaningful insights and assurance. In so doing, it shows how modern DA&AI tools, that are available off the shelf, need to be used as part of the governance framework in order to assure against such surprises.
The 3 LoD Model
Let’s start by looking at a typical 3 LoD Model definition:
- First line: Management (process owners) has the primary responsibility to own and manage risks associated with day-to-day operational activities. Other accountabilities assumed by the first line include design, operation, and implementation of controls.
- Second line: The second-line function enables the identification of emerging risks in daily operation of the business. It does this by providing compliance and oversight in the form of frameworks, policies, tools, and techniques to support risk
- Third line: The third-line function provides objective and independent assurance, to assess whether the first- and second-line functions are operating effectively.
Project Risk
The financial sector roots are evident in this definition with emphasis on compliance, day-to-day operational activities, and risk management. Day-to-day operational activities are controlled by processes and regulations which can be audited against by the 3rd LoD, with the 2nd LoD ensuring the risk controls framework is in place. Indeed, this applies in a major project setting with designs to follow, regulations to comply with and managing external risks such as supply chain stability, labour shortages, cost increases etc. However, in addition to these risks, which should be part of any risk controls framework, major projects are held at gunpoint by time.
Schedule is King
Much effort is put into controlling cost often with complex governance regimes, but cost is an outcome, mostly of time slippage. Quality issues, for example, will cost in terms of rework but this can be overshadowed by the impact of the delay on other parts of the projects. To put this into perspective, a major international organisation undertook a study across thousands of its projects and found that time contingency is worth twice that of budget. In other words, if we were to put as much effort into schedule control and forward-looking predictive risk assessment as we do cost control, we could see twice the effect. The reality is that we rarely do, and it is particularly difficult for the 2nd or 3rd LoDs to provide genuine insightful assurance of schedules that are owned and controlled by the 1st LoD. The result is, according to research from Oxford University, that fewer than one in ten major projects deliver on time. Schedule assurance is often a monthly meeting with a thick project report and a deck of slides with the project lead, or his/her scheduler, assuring everyone that everything is on track.
Other Tools
Earned Value Management (EVM) is a useful monitoring tool but, it deals with what ‘should’ have been completed and what ‘should’ have been spent. EVM is therefore a lag indicator rather than a risk management tool as required by an assurance framework. Furthermore, in major projects, tasks that can be seemingly innocuous and cost very little, can have a disproportionate impact to the overall schedule and budget, because of their interdependencies and criticality. So ‘value’ in a major project should also be a measure of schedule criticality and the potential overall impact on the project should the task not be completed on time. In major projects, it is essential that DA&AI tools are used that can assess value in terms of project criticality as well as cost/budget.
Major Project Complexity
Major projects are very complex with countless interdependencies hidden in the depths of the schedule, having knock-on effects with each other. Indeed, these interdependencies mean the idea of a ‘single critical’ path in the constantly changing environment of a major project, is a misnomer. Expecting humans to be able to manage all that complexity and identify the level of criticality of each task in their heads is unrealistic and can only be achieved through DA&AI. In terms of assurance, periodic auditing alone is unlikely to be an effective control because major projects can slip months overnight due to their constantly changing nature. It is also a difficult and time-consuming exercise to effectively manually audit the contents of a complex schedule. A Quantitative Schedule Risk Analysis (QSRA) exercise for example will take weeks to complete on a major project and it is based on judgement and therefore subject to optimism bias or, at worst, strategic misrepresentation (cover up).
Surprise Avoidance
The previously referred to surprises, happened when the problems had likely been buried deep in the data for months if not years, unidentified by traditional governance regimes and project management tools. Using DA&AI, risks can be controlled by assessing on a continuous basis the criticality of every task in the schedule, recommending where to apply strict controls, and identifying what impacts seemingly minor slippages can have. Additionally, task intensity can be measured in any given period enabling resource smoothing, and task prioritisation can be done automatically. By analysing the data in this way, surprise slippages can be avoided, and accurate forecasting achieved.
The role of the 2nd LoD
DA&AI should be a fundamental part of the risk management process and should be the basis of a major project risk framework. The risk controls framework implemented or overseen by a 2nd LoD or any assurance strategy, needs to be based on a deep assessment of the data and close tracking of all critical tasks as lead indicators. The role of the 2nd LoD is then to compliance check that the 1st LoD is using the framework correctly to identify the strategic risks to the project and is applying adequate mitigations and/or reporting accurately.
In Sum
Time is the biggest enemy for major projects and the threats are buried in the data. There are DA&AI tools that have been developed specifically to look deep into schedules to identify risks that would otherwise not be visible and produce prioritised task lists to keep the project on track. These modern-day DA&AI tools hold the project line and provide assurance on a near continuous basis, in a way that simply cannot be done with traditional tools and techniques and periodic auditing. Total visibility of the data and the ability to analyse and test it using algorithmic techniques provides a transparency which is not only a great enabler for the frontline but is also critical to any governance framework in order to provide meaningful insights and assurance of the schedule. Furthermore, using data that is readily available in existing tools, such as Primavera of Microsoft Project, it provides assurance with a much-reduced need for audit and thereby reducing the impact on productivity. Technical reviews and compliance checks will still be needed, but the crucial schedule assurance can be done remotely through the data.
Thinking Shift
It is time to move our thinking into the data world to simplify the complexity of major projects and start to deliver on time and budget in a consistent manner. For this to happen, we need to embrace DA&AI and place it at the heart of our project assurance strategy.
Dr Peter Ewen
peter.ewen@hanoverfox.com