Methodological Aspects and Models for Assessing the Effectiveness of Artificial Intelligence in Project Management
Keywords:
model, machine learning, benchmark, performance assessment, methodology, cognitive models, systems analysis, project management, artificial intelligence, decision support systemAbstract
Rapid integration of artificial intelligence (AI) into project management offers significant potential to improve productivity through data automation, performance monitoring and schedule optimization. However, challenges such as “effective inefficiency” and the variability of AI model output complicate the assessment of its effectiveness. This article analyses the methodological aspects of evaluating AI effectiveness in project management, classifies existing methods (benchmarks, explainable AI, mutual information, psychometrics), identifies key challenges (biases, lack of standards, ethical constraints), and proposes novel metrics- indicator of new competency activation (INCA), novelty coefficient in AI-Driven Project Management (NCAPM) and dynamic assessment of transition to new efficiency enabled by AI (DATNE) to measure innovation. The potential of these approaches for transport infrastructure projects is indicated, where AI allows for the creation of fundamentally new opportunities in planning, service forecasting, and resource optimisation. Future directions include hybrid metrics and integration with decision support systems. The study underscores the need for interdisciplinary approaches to adapt AI evaluation to resource constrained project management environments.
References
Müller, R., Locatelli, G., Holzmann, V., Nilsson, M., & Sagay, T. (2024). Artificial intelligence and project management: Empirical overview, state of the art, and guidelines for future research. Project Management Journal, 55(1), 9-15. https://doi.org/10.1177/87569728231225198.
Mills, S., & Spencer, D. A. (2025). Efficient Inefficiency: Organisational challenges of realising economic gains from AI. Journal of Business Research, 189, 115128. https://doi.org/10.1016/j.jbusres.2024.115128.
Edwards, J. (2025). How To Measure AI Efficiency and Productivity Gains. In: InformationWeek, ed. AI and Machine Learning Insights. Available from:. https://www.informationweek.com/machine-learning-ai/how-to-measure-ai-efficiency-and-productivity-gains.
Burden, J. (2024). Evaluating ai evaluation: Perils and prospects. arXiv preprint arXiv:2407.09221. https://doi.org/10.48550/arXiv.2407.09221.
Challapally, A., & Pease, C. (2025). AI trends and innovations in 2025. In: Artificial Intelligence News, ed. Annual AI Report. Available from: https://www.artificialintelligence-news.com/wp-content/uploads/2025/08/aireport2025.pdf.
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., ... & Bowman, S. (2019). Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32б 1-15. https://doi.org/10.48550/arXiv.1905.00537.
Zhang, M., Wang, H., Li, J., & Gao, H. (2020). Learned sketches for frequency estimation. Information Sciences, 507, 365-385. https://doi.org/10.1016/j.ins.2019.08.045.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144). https://doi.org/10.1145/2939672.2939778.
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., ... & Herrera, F. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion, 58, 82-115. https://doi.org/10.1016/j.inffus.2019.12.012.
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. https://doi.org/10.48550/arXiv.1702.08608.
Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30. https://doi.org/10.48550/arXiv.1705.07874.
Olah, C., Mordvintsev, A., & Schubert, L. (2017). Feature visualization. Distill, 2(11), e7. https://distill.pub/2017/feature-visualization.
Tishby, N., Pereira, F. C., & Bialek, W. (2000). The information bottleneck method. arXiv preprint physics/0004057. https://doi.org/10.48550/arXiv.physics/0004057.
Hernandez-Orallo, J., 2017. Evaluating intelligence across species and machines. The Measure of All Minds, ed. Cognitive Science Series. Cambridge, UK: Cambridge University Press, 50–100. https://doi.org/10.1017/9781316596654.
McKinsey & Company. (2023). Generative AI’s impact on business in 2023. In: The State of AI Report, ed. Digital Transformation Insights. New York, NY: McKinsey & Company, 10–25. Available from: https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/the-state-of-ai-in-2023.
WIPO (2023). Trends in AI technology development. Technology Trends 2023: Artificial Intelligence, ed. IP Research Series. Geneva, Switzerland: World Intellectual Property Organization.
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6), 1-35. https://doi.org/10.1145/3457607.
MIT (2025). Business applications of AI in 2025. The State of AI in Business 2025, ed. Technology Insights. Cambridge, MA: MIT Press, pp.5–20. Available from: https://www.mit.edu/ai-report-2025.
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. https://doi.org/10.48550/arXiv.1702.08608.
Nesterenko O., & Kulbovskyi I. (2024). Mathematical framework of transformer-based artificial intelligence architectures in large language models for the development of intelligent agents. Science and technology today, 5(46). http://dx.doi.org/10.52058/2786-6025-2025-5(46)-1860-1872.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright: This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
