Managing Data Acquisition, Cleansing and Transformation in an Agriculture Data Warehouse

Abstract

Pakistan is the world’s fourth largest cotton producer. The country relies heavily on cotton yield to sustain and enhance its export and economic growth. Several state run organizations have been monitoring the cotton crop for decades through pest-scouting, agriculture and meteorological data-gathering processes. This non-digitized and non-standardized dirty data is of little use for strategic analysis and decision support. This paper is based on the data collection and cleansing issues of that cotton pest-scouting data consisting of approximately 15,000 sheets from 20 cotton-growing districts of Punjab province. Various real-life agriculture data management and data quality problems are discussed and explained in this paper using several real examples.