Understanding the Capabilities of Data Quality Measurement and Monitoring
Article Sidebar
Abstract:
Background of Study: Ensuring high-quality data is essential for organizations that depend on analytics, automation, and regulatory compliance.
Aims and Scope of Paper: This paper explores the foundational concepts and evolving practices of two interrelated capabilities: data quality measurement and data quality monitoring. While measurement focuses on quantifying attributes such as accuracy, completeness, consistency, and timeliness, monitoring emphasizes the continuous detection and alerting of anomalies during data operations.
Methods: This paper examines the application of frameworks like Total Data Quality Management (TDQM), ISO 8000, and Data Management Association Data Management Body of Knowledge (DAMA DMBOK), alongside emerging tools such as rule-based engines, metadata-driven platforms, and AI-driven anomaly detection systems.
Results: Findings reveal a persistent gap in systems that integrate both measurement and monitoring effectively, hindering long-term data governance. This paper discusses a case study of the Data Quality framework implementation in the Healthcare sector. It was found that the healthcare organization implemented the Total Data Quality Management (TDQM) framework and Apache Griffin to ensure the accuracy, completeness, consistency, timeliness, and validity of clinical and IoT data through continuous monitoring, automated validation, and anomaly detection. Governance mechanisms aligned with ISO 8000 and HIPAA standards ensured full compliance, traceability, and accountability across all data quality and auditing processes. This study contributes to a deeper understanding of how integrated data quality practices can support digital transformation and operational resilience across industries.
Conclusion: The paper concludes by recommending the adoption of continuous quality measurement practices aligned with governance policies and supported by both human expertise and automation, arguing that data quality must be embedded as a dynamic and strategic function within the digital enterprise. While measurement emphasizes the quantification of data attributes such as accuracy, completeness, consistency, and timeliness, monitoring focuses on the continuous detection and alerting of data anomalies during data operations.
Keywords: Anomaly Detection, Data Governance, Data Quality, Data Quality Measurement, Data Quality Monitoring
Copyright (c) 2025 Normi Sham Awang Abu Bakar, Alea Syaffa Mohd Sabri, Aisya Sufiah Hazlan
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
References
Batini, C., Scannapieco, M., & Rula, A. (2024). From data quality to data value: A new paradigm for data-driven organizations. Information Systems Frontiers, 26(2), 445–462. https://doi.org/10.1007/s10796-023-10426-5
Bertossi, L., & Geerts, F. (2020). Data Quality and Explainable AI. J. Data and Information Quality, 12(2), 1–9. https://doi.org/10.1145/3386687
Buelvas, J. H., Múnera, D., & Gaviria, N. (2023). DQ-MAN: A tool for multi-dimensional data quality analysis in IoT-based air quality monitoring systems. Internet of Things, 22(100769), 1–18. https://doi.org/10.1016/j.iot.2023.100769
Cai, L., & Zhu, Y. (2015). The challenges of data quality and data quality assessment in the big data era. Data Science Journal, 14, 2. https://doi.org/10.5334/dsj-2015-002
Cortes, C., Sanz, C., Etcheverry, L., & Marotta, A. (2024). Data Quality Management for Responsible AI in Data Lakes. VLDB Workshops. https://api.semanticscholar.org/CorpusID:273878300
Ehrlinger, L., & Wöß, W. (2022). A Survey of Data Quality Measurement and Monitoring Tools. Frontiers in Big Data, 5, 1–30. https://doi.org/10.3389/fdata.2022.850611
Fu, Q., Nicholson, G. L., & Easton, J. M. (2024). Understanding data quality in a data-driven industry context: Insights from the fundamentals. Journal of Industrial Information Integration, 42, 100729. https://doi.org/10.1016/j.jii.2024.100729
Gudivada, V. N., Apon, A., & Ding, J. (2017). Data Quality Considerations for Big Data and Machine Learning : Going Beyond Data Quality Considerations for Big Data and Machine Learning : Going Beyond Data Cleaning and Transformations. International Journal on Advances in Software, 10(1), 1–20. https://www.researchgate.net/publication/318432363_Data_Quality_Considerations_for_Big_Data_and_Machine_Learning_Going_Beyond_Data_Cleaning_and_Transformations
Kahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F. (2012). A Pragmatic Framework for Single-site and Multisite Data Quality Assessment in Electronic Health Record-based Clinical Research. Medical Care, 50(1), 28–39. https://doi.org/10.1097/MLR.0b013e318257dd67
Martín, L., Sánchez, L., Lanza, J., & Sotres, P. (2023). Development and evaluation of Artificial Intelligence techniques for IoT data quality assessment and curation. Internet of Things, 22(April), 100779. https://doi.org/10.1016/j.iot.2023.100779
Miller, R., Hin, S., Chan, M., & Whelan, H. (2025). A Comparison of Data Quality Frameworks : A Review. Big Data Cogn. Comput., 9(93), 4–13. https://doi.org/10.3390/bdcc9040093
Mohammed, S., Budach, L., Feuerpfeil, M., Ihde, N., Nathansen, A., Noack, N., Patzlaff, H., Naumann, F., & Harmouch, H. (2025). The effects of data quality on machine learning performance on tabular data. Information Systems, 132, 102549. https://doi.org/10.1016/j.is.2025.102549
Nguyen, T., Nguyen, H.-T., & Nguyen-Hoang, T.-A. (2025). Data quality management in big data: Strategies, tools, and educational implications. Journal of Parallel and Distributed Computing, 200, 105067. https://doi.org/10.1016/j.jpdc.2025.105067
PLC, E. (2022). Global data management research: Data quality and business performance. Experian PLC.
Poon, L., Farshidi, S., Li, N., & Zhao, Z. (2021). Unsupervised Anomaly Detection in Data Quality Control. In Proceedings of the 2021 IEEE International Conference on Big Data, 2326–2336. https://doi.org/10.1109/BigData52589.2021.9671672
Precisely, & University, D. (2024). Data quality and governance survey: The data quality confidence gap. Precisely.
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024). The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review. NPJ Digital Medicine, 7(1), 203. https://doi.org/10.1038/s41746-024-01196-4
Segreto, T., & Teti, R. (2023). Data quality evaluation for smart multi-sensor process monitoring using data fusion and machine learning algorithms. Production Engineering, 17(2), 197–210. https://doi.org/10.1007/s11740-022-01155-6
Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33. https://web.mit.edu/tdqm/www/tdqmpub/beyondaccuracy_files/beyondaccuracy.html
Zhu, X., Xu, W., & Wang, Q. (2017). Apache Griffin: An open-source framework for data quality solutions BT - Proceedings of the IEEE International Conference on Big Data. 655–664. https://doi.org/10.1109/BigData.2017.8258015
