This title appears in the Scientific Report :
2019
Please use the identifier:
http://hdl.handle.net/2128/24952 in citations.
A New Tool for Automated Quality Control of Environmental Data in Open Web Services
A New Tool for Automated Quality Control of Environmental Data in Open Web Services
We report on the development of a new software tool (auto-qc) for automated quality control (QC) of environmental timeseries data. Novel features of this tool include a flexible Python software architecture, which makes it easy for users to configure the sequence of tests as well as their statistica...
Saved in:
Personal Name(s): | Kaffashzadeh, Najmeh (Corresponding author) |
---|---|
Kleinert, Felix / Schultz, Martin | |
Contributing Institute: |
Jülich Supercomputing Center; JSC |
Imprint: |
2019
|
Document Type: |
Preprint |
Research Program: |
Earth System Data Exploration Artificial Intelligence for Air Quality Data-Intensive Science and Federated Computing |
Link: |
Get full text OpenAccess OpenAccess |
Publikationsportal JuSER |
We report on the development of a new software tool (auto-qc) for automated quality control (QC) of environmental timeseries data. Novel features of this tool include a flexible Python software architecture, which makes it easy for users to configure the sequence of tests as well as their statistical parameters, and a statistical concept to assign each value a probability of being a correct value. There are many occasions when it is necessary to inspect the quality of environmental datasets, from first quality checks during real-time sampling and data transmission to assessing the quality of long-term monitoring data from measurement stations. Erroneous data can have a substantial impact on the statistical data analysis and, for example, lead to wrong estimates of trends. Existing QC workflows largely rely on individual investigator knowledge and have often been constructed from practical considerations alone. Our tool aims to complement traditional data quality analyses and adds some insights into the nature of the individual tests that are being applied. |