As projects become increasingly larger and more complex, can Artificial Intelligence help us manage requirements more efficiently and accurately? Can it be the key to delivering projects on time and budget?
Technological innovation has the potential to make our lives easier and better. However, the complexity of the technology can be exponential.
Figure 1 demonstrates this increase in complexity [1]. Can you believe there are more line of codes in your mobile phone than in the Mars Curiosity Rover? And what if we were to tell you there are ten times more lines of code in a car than in a Boeing 787 developed in 2009?
This increasing complexity comes with its challenges in specifying, designing and building these systems. In this article we explore how human intelligence could use some artificial help to manage increasingly complex set of requirements while meeting the project budget and schedule.
Systems Engineering & Requirements Management
Efficient systems engineering and requirements management processes need to be part of the solution to this problem. Inaccurate requirements gathering is identified as the second major cause of project failure – 39% according to the Project Management Institute’s Pulse of the Profession survey 2017 [2]: see figure 2.
CIO Magazine published a report that stated as many as “71% of software projects that fail do so because of poor requirements management, making it the single biggest reason for project failure – bigger than bad technology, missed deadlines or change management fiascos.” [3]
Therefore, investing a lot of effort in activities, such as requirements gathering and elicitation, in the early stages of projects is key to success. But there are constraints (time, budget) on how much effort can be invested so early in projects. We need to invest wisely and efficiently. And, most importantly, we need to strive for continuous improvement.
AI is a maturing technology revolutionising many industries [4]. AI has its benefits, but can it be relied on to support requirements management?Are those benefits worthwhile at this stage of its maturity?
Artificial Intelligence & Requirements Management
Before we start, we need to define key terms of reference, including AI, Natural Language Process, Machine Learning, and AI assistant.
AI is the theory and development of computer systems able to perform tasks normally requiring human intelligence (source: Oxford Languages). AI is composed of many subfields, but we will focus on two in particular:
Natural Language Processing (NLP): This refers to the application of computational techniques to the analysis and synthesis of natural language and speech (Oxford Languages).
Machine Learning (ML): The use and development of computer systems that are able to learn and adapt without following explicit instructions, by using algorithms and statistical models to analyse and draw inferences from patterns in data (Oxford Languages). Deep Learning, while different from a technical point of view, can be considered as a subset of Machine Learning in our case.
From a requirement management point of view, the use of NLP means a program could “understand” requirements. There are many tools already available for requirement analysis using pattern/word recognition.
With NLP, the concept can be extended for semantic analysis based on a requirement context, for example analysis against the INCOSE Guide to Writing Requirements [5], or analysis in the context of a large rail infrastructure project and so on.
Then ML will qualify the analysis to determine if the requirement is a quality requirement in the context of analysis (clear requirement, in conflict with another requirements, etc.). Feedback on the analysis can then be used to reinforce the knowledge base to make it more accurate over time.
For the purposes of this article, we will call programs using AI to support Requirements Management “AI Assistants”.
Possessing the capability to automatically assess the quality of requirements in a matter of seconds, AI Assistants look really promising. On paper, at least. But how do they perform when applied on to the complexities of a real-life rail project?
Acmena’s Experience Using AI Assistants on Rail Projects
To provide a bit of context, when engaged to support the systems engineering approach for the integration of a complex railway digital transformation project, Acmena had to develop more than four thousand requirements with very little time and resources.
We mostly relied on Subject Matter Experts (SME) to define the requirements with no specific training, only ad-hoc support from the systems engineering team. In the end, the requirements were “good enough” to progress to the next phase of the project and kick-off procurement activities; however, in their current form, the requirements were far from perfect and represented a risk for future project phases.
Requirement quality and correctness is a risk for large scale projects [6]. The potential impact of low-quality requirements increases as the project progresses through the design, construction, testing and commissioning phases. If this risk materialises, it could lead to longer than expected verification and validation campaigns or require redesigning part of the solution.
To quantify and control this risk, we decided to use one of the most advanced AI Assistants available in the market, IBM Engineering Requirements Quality Assistant (RQA) [7].
To deploy RQA on the project, Acmena followed the approach documented in Table 1 below. We significantly increased our productivity by providing the initial requirement quality analysis for approximately 4,000 requirements in just minutes instead of the estimated four weeks it would take if it were to be done manually.
The quality of requirements, or the lack of, was analysed correctly about 70% of the time. It was easy to identify major categories for false positives/false negatives and correct them in a few days. In less than five days, we were able to provide an analysis for 4,000 requirements that was correct more than 90% of the time.
However, it was a very difficult process to get requirements updated to a minimum level of quality (see Table 1 steps 1 and 2). The risks posed by low quality requirements were not concrete enough (realisation would happen in a mid-term future and not in the next project phase) and contractual issues linked to the update were very real and seemed too big to manage.
So, what have we learned?
Lessons Learned
The first lessons we’ve learned is how AI Assistants should be considered. AI assistants are not programs with bugs to be fixed. They are resources you can train to improve their performance over time. And these resources will not quit because the competition offers free breakfast or some extra cash. This is an investment you control 100%.
The perfect AI assistant does not exist. There is still input required from humans to support the AI Assistant, but the perfect requirements engineer who can infallibly evaluate requirements eight hours a day does not exist either. We need to accept reality and a solution that is good, but not perfect and manage the risks accordingly.
AI Assistants bring a level of speed and consistency to requirements analysis that we humans cannot compete with. The larger the number of requirements, the more useful the AI Assistant will be, which will be of significant benefit as projects continue to grow in size or complexity.
To make the requirement writing process more efficient, the AI Assistant needs to be deployed to support the initial writing of requirements. A minimum acceptable score (see Table 1 steps 1 and 2) must be defined and made a criterion for a requirement to enter the review process.
Doing so will bring many benefits, including:
• Less time spent in the review process with more focus on the “meaning” of the requirement.
• Upskilling the Subject Matter Expert writing the requirements and improving the quality of requirements for future projects.
• In a market with limited supply of (good) systems engineers, allow them to focus on other high-value activities that cannot yet be automated.
Despite the obvious benefits it may bring, it is important not to underestimate the pushback on new technology.
We have seen that it takes time for people to trust AI Assistants. It is a natural first reaction to push back on a tool that will judge the quality of your work. As a result, it will take time to train people to work with AI Assistants and consider them as a support function rather than a judgement tool. It will also take time to train the AI Assistant and bring it to an accuracy closer to 100%.
Where is AI Going Next?
Based on our experience, AI technologies are clearly still in their infancy. That being said, we also believe that now is the time for project stakeholders to seriously consider investing in an AI Assistant, not just for the benefits it can deliver now, but to develop its future potential.
With the help of ML, AI technologies are progressing at a terrific pace. Most of the current limitations, such as lack of domain knowledge or lack of accuracy, could be lifted as adoption increases. As AI assistants reach maturity, they might directly develop the requirements with limited supervision.
The scope of an AI assistant could also be expanded to analyse a set of requirements or validate/create requirement traceability to other engineering artefacts, such as Project Hazard Logs.
And this could be even more true if AI Assistants are embraced by the whole supply chain (government, asset owners, suppliers, designers, consultancies, software developers, etc.).
The benefits to an industry could be enormous and include:
• Reduced time spent in the definition space – faster project delivery.
• Reduced cost of issues due to low quality requirements – do more with less money.
• Better utilisation of technical resources to solve engineering problems rather than requirement management issues.
• Be an immediate part of the solution to the current skills and competency shortage without waiting to train systems engineers for five years and then let them accrue experience.
Over the last few years Acmena, along with a handful of other pioneering service providers, has taken the first few steps in realising the enormous benefit AI Assistants can bring to the rail industry. There is no doubt that right now we have the privilege to be working in some very exciting times.
Where we go from here will be up to you.
Nicolas Teulier | Principal Consultant
References
[1] David Notkin – University of Washington, Paul G. Allen School of Computer Science and Engineering – https://courses.cs.washington.edu /courses/cse503/11sp/lect-3-complexity-proving-adts.pdf
[2] https://www.pmi.org/-/media/pmi/documents/
public/pdf/learning/thought-ladership/pulse /pulse-of-the- profession-2017.pdf
[3] https://web.accompa.com/requirements-management/
[4] Artificial Intelligence Roadmap – https://www.csiro.au/en/research/technology-space/ai/artificial-intelligence-roadmap
[5] INCOSE Guide to Writing Requirements v3.1 2022 (https://connect.incose.org/Pages/Product-Details.aspx?ProductCode=TechGuideWR2022Soft)
[6] Gruhl (NASA) 1992: projects which spend less than 5% of total project costs on the requirements engineering process experienced an 80% to 200% cost overrun, whereas those that invested 8% to 14% were able to meet their costs or incur less than 60% overrun (https://www.incose.org/ docs/default-source/texas-gulf-coast/incose_ meeting_ 04_15_2021.pdf?sfvrsn=254567c7_0)
[7] https://www.ibm.com/products/requirements-quality-assistant
