Managing AI and Data Science Projects: Unveiling Secrets from Industry Experts for Real-world Success

Authors

  • Samuel Habtemariam School of Science and Engineering, Atlantic International University, Pioneer Plaza, 900 Fort Street Mall 905, Honolulu, Hawaii 96813, USA
  • Daniel Yihdego Microsoft, Azure Data Science & ML, Bellevue, Washington, USA
  • Edward Lambert School of Science and Engineering, Atlantic International University, Pioneer Plaza, 900 Fort Street Mall 905, Honolulu, Hawaii 96813, USA

DOI:

https://doi.org/10.14738/tmlai.1302.18536

Keywords:

Data science, artificial intelligence (AI), project management

Abstract

Data science projects are becoming increasingly critical in today’s data-driven landscape, yet effectively managing their complexities remains challenging, and this often results in the project's underperformance and or failure. This study provides comprehensive guidelines for critical aspects of data science project management - from initiation and implementation to scaling. Both primary and secondary data were gathered using mixed research methodology. Twenty-six (26) experts from international companies were engaged with a well-structured questionnaire, and selective companies successful in previous data science projects were approached to gather primary data. For secondary data, papers published in reputable journals were researched systematically to support claims from the primary data. However, based on an extensive literature review and critical sampling of opinions, significant challenges for data science projects identified include a lack of feedback-constrained timeframes, project complexity, the inability to meet high stakeholder expectations, the inability to enable informed decisions, and the iterative nature of data science is underscored, along with nuanced expositions on navigating the potential challenges of science data projects. Finally, other best practices identified through literature searches and case studies include regular audits, effective metadata management, data privacy and security, optimal cloud storage for cost-effectiveness, and computation acceleration. Further, recommendations were provided, which contributed to positioning this paper as a valuable guide for practitioners, researchers, organizations, and individuals in the field.

References

Al-Saqqa, S., Sawalha, S., & Abdel-Nabi, H. (2020). Agile software development methodologies and trends. Computer Science and IT Research Journal, 14(11), 246-270.

Beckman, M., Cetinkaya, M., Horton, N. J., Rundel, C. W., Sullivan, A. J., & Tackett, M. (2020). Implementing version control with Git and GitHub as a learning objective in statistics and data science courses. Journal of Statistics and Data Science Education, 29(suppl. 1), 1–35. https://doi.org/10.1080/10691898.2020.1790417

Beltramin, D., Lamas, E. V., & Bousquet, C. (2022). Ethical issues in the utilization of black boxes for artificial intelligence in medicine. Advances in Informatics, Management, and Technologies in Healthcare, 249-252.

Bozic, V. (2024). The impact of artificial intelligence on business intelligence. International Journal of Business Intelligence, 18(2), 55-68.

Cazacu, M., & Titan, E. (2020). Adapting CRISP-DM for social sciences. Broad Research in Artificial Intelligence and Neuroscience, 11(2), 99-106.

Chinthamu, N., & Karukuri, M. (2023). Data science and applications. Journal of Data Science and Intelligence Systems, 1(2), 81–91.

Daraojimba, C., Nwasike, C. N., Adegbite, A. O., Ezeigweneme, C. A., & Gidiagba, J. O. (2024). A comprehensive review of Agile methodologies in project management. Computer Science and IT Research Journal, 5(1), 190-218.

Foote, K. D. (2021). A brief history of data science. Data Governance. Retrieved from https://www.dataversity.net

Garczarek, U., & Steuer, D. (2019). Approaching ethical guidelines for data scientists. Journal of Data Ethics, 4(1), 55-68.

Hawkins, J. (2022). Getting data science done. Business Expert Press.

Heising, W. (2012). The integration of ideation and project portfolio management – A key factor of sustainable success. International Journal of Project Management, 30(5), 582–595. https://doi.org/10.1016/j.ijproman.2011.11.004

Hotz, N. (2024). Agile data science. Data Science Process Alliance. Retrieved from https://www.datasciencepm.com/agile-data-science/

Ibraheem, I. F. (2018). The effects of stakeholder engagement and communication management on project success. MATEC Web of Conferences, 25(3), 1-12.

Igwenagu, C. (2016). Fundamentals of research methodology and data collection. Research Gate, 1–47.

Joel, A. (2022). Data science life cycle: Detailed explanation. Data Science. Addis Ababa University.

Kang, E., & Hwang, H. (2023). The importance of anonymity and confidentiality for conducting survey research. Journal of Research and Publications Ethics, 4(1), 33-45.

Kuphanga, D. (2024). Questionnaires in research: Their role, advantages, and main aspects. Journal of Research Methodology, 12(4), 1–8.

Lurie, Y., & Mark, S. (2015). Professional ethics of software engineers: An ethical framework. Science and Engineering Ethics, 22, 1245-1263. https://doi.org/10.1007/s11948-015-9665-x

Luz, A. (2023). Machine learning operations. University of Melbourne Press.

Maimon, O., & Rokach, L. (2005). Chapter 1: Introduction to knowledge discovery in databases. In Data Mining and Knowledge Discovery Handbook (pp. 1-17). Springer.

Majeed, A., & Lee, S. (2020). Anonymization techniques for privacy-preserving data publishing: A comprehensive survey. IEEE Access, 4, 1-25.

Makinen, S., Skogstrom, H., Laaksonen, E., & Mikkonen, T. (2021). Who needs MLOps: What data scientists seek to accomplish, and how can MLOps help? 1st Workshop on AI Engineering-Software Engineering for AI (WAIN), 55-62.

Mao, Y., Wang, D., Muller, M., & Varshney, K. R. (2019). How do data scientists work together with domain experts in scientific collaborations: To find the right answer or to ask the right questions? Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1-25.

Martinez, I., Viles, E., & Olaizola, I. G. (2021). Data science methodologies: Current challenges and future approaches. Big Data Research, 24(3), 102-115.

Martínez-Plumed, F., Contreras-Ochando, L., Ferri, C., Hernandez-Orallo, J., Kull, M., Lachiche, N. J. A. H., Ramírez-Quintana, M. J., & Flach, P. A. (2019). CRISP-DM twenty years later: From data mining processes to data science trajectories. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2019.2962680

Martin-Santana, S., Perez-Gonzalez, C. J., Colebrook, M., Roda-Gracia, J. L., & Gonzalez-yanes, P. (2018). Deploying a scalable data science environment using Docker. Journal of Cloud Computing, 15(4), 45-57.

Medeiros, M., Hoppen, N., & Macada, C. G. (2024). Data science or business: Benefits, challenges, and opportunities. Data Science for Business. Emerald Publishing Limited.

Mensah, C. B. (2023). Artificial intelligence and ethics: A comprehensive review of bias mitigation, transparency, and accountability in AI systems. AI Ethics, 5(1), 33-49.

Nguyen, G., Dlugolinsky, S., Bobak, M., Tran, V., Garcia, A. L., Heredia, I., Malik, P., & Hluchy, L. (2019). Machine learning and deep learning frameworks and libraries for large-scale data mining: A survey. Artificial Intelligence Review, 52(1), 77–124.

Omonije, A. (2024). Agile methodology: A comprehensive impact on modern business operations. International Journal of Science and Research, 13(2), 55-70.

Ruf, P., Madan, M., Reich, C., & Abdeslam, D. O. (2021). Demystifying MLOps and presenting a recipe for the selection of open-source tools. Applied Science, 11(19), 39.

Schulz, M., Neuhaus, U., Kaufmann, J., & Kuhnel, S. (2022). DASC-PM v1.1: A process model for data science projects. NORDKADEMIE gAG Hochschule der Wirtschaft.

Downloads

Published

2025-04-25

How to Cite

Habtemariam, S., Yihdego, D., & Lambert, E. (2025). Managing AI and Data Science Projects: Unveiling Secrets from Industry Experts for Real-world Success. Transactions on Engineering and Computing Sciences, 13(02), 116–133. https://doi.org/10.14738/tmlai.1302.18536