README.md 5.79 KB
Newer Older
oparisot's avatar
oparisot committed
1 2
# gapIt

Olivier Parisot's avatar
Olivier Parisot committed
3
Copyright 2014 - 2019 Luxembourg Institute of Science and Technology (LIST - [http://www.list.lu/](http://www.list.lu/)). Any use of this software constitutes full acceptance of all terms of the software's license.
oparisot's avatar
oparisot committed
4 5


Olivier Parisot's avatar
Olivier Parisot committed
6 7 8 9
## Synopsis

gapIT is an interactive software for gap filling in hydrological time series.

oparisot's avatar
oparisot committed
10
Given as input hydrological discharge data series measured at gauging stations, it performs an automated calculation of missing data using different data-infilling techniques. Donor station(s) are automatically selected based on Dynamic Time Warping, geographical proximity and upstream/downstream relationships among stations. For each gap, the tool computes several flow estimates through various data-infilling techniques, including interpolation, multiple regression, regression trees and neural networks. The visual application provides the possibility for the user to select different donor station(s) w.r.t. those automatically selected.
Olivier Parisot's avatar
Olivier Parisot committed
11 12 13 14 15 16 17 18 19 20 21

The results are validated by randomly creating artificial gaps of different lengths and positions along the entire records. Using the Root Mean Squared Error and the Nash-Sutcliffe coefficient as performance measures, the method is evaluated based on a comparison with the actual measured discharge values.

The interactive but automated approach of gapIT, coupled with a visual inspection system for user-defined refinement, allows for a standardized objective infilling, where subjective decisions are allowed but are at the same time traceable.


## Motivation

Missing records in hydrological databases represent a loss of information and a serious drawback in water management (measures of river flow for example). An incomplete time series prevents the computation of hydrological statistics and indicators. Also, records with data gaps are not suitable as input or validation data for hydrological or hydrodynamic modeling.


Olivier Parisot's avatar
Olivier Parisot committed
22
## Installation / execution
Olivier Parisot's avatar
Olivier Parisot committed
23

oparisot's avatar
oparisot committed
24
### Requirements
oparisot's avatar
oparisot committed
25

oparisot's avatar
oparisot committed
26
Hardware: RAM>=4Gb.
oparisot's avatar
oparisot committed
27

oparisot's avatar
oparisot committed
28
Software: 
Olivier Parisot's avatar
Olivier Parisot committed
29 30
* JDK (>=1.7 - [http://www.oracle.com/technetwork/java/javase/downloads/](http://www.oracle.com/technetwork/java/javase/downloads/)).
* Maven (>=3.0.2 - [https://maven.apache.org/](https://maven.apache.org/))
Olivier Parisot's avatar
Olivier Parisot committed
31

Olivier Parisot's avatar
Olivier Parisot committed
32

oparisot's avatar
oparisot committed
33
### Compilation
oparisot's avatar
oparisot committed
34

Olivier Parisot's avatar
Olivier Parisot committed
35 36
    mvn clean package -Dmaven.test.skip=true
    
Olivier Parisot's avatar
Olivier Parisot committed
37
(*Some tests are currently not ok, and the build fails without the '-Dmaven.test.skip=true' option.*)
Olivier Parisot's avatar
Olivier Parisot committed
38

Olivier Parisot's avatar
Olivier Parisot committed
39

oparisot's avatar
oparisot committed
40
### Data preparation
Olivier Parisot's avatar
Olivier Parisot committed
41

Olivier Parisot's avatar
Olivier Parisot committed
42
unzip [data_fake2.zip](./data_fake2.zip) in the same directory
oparisot's avatar
oparisot committed
43

Olivier Parisot's avatar
Olivier Parisot committed
44

oparisot's avatar
oparisot committed
45 46
### Execution for Windows OS

Olivier Parisot's avatar
Olivier Parisot committed
47
    run.bat
oparisot's avatar
oparisot committed
48

oparisot's avatar
oparisot committed
49
### Execution for other OS
Olivier Parisot's avatar
Olivier Parisot committed
50

Olivier Parisot's avatar
Olivier Parisot committed
51
    java -jar -Xmx4G target/gapIt.jar
Olivier Parisot's avatar
Olivier Parisot committed
52 53 54 55


## Data

oparisot's avatar
oparisot committed
56
gapIt was used to fill gaps in water discharge time series at Luxembourg (see the 'References' section).
oparisot's avatar
oparisot committed
57
During tests phases, it was used too for the processing of water level time series.
oparisot's avatar
oparisot committed
58

oparisot's avatar
oparisot committed
59 60
Here, fake data are provided within the distribution [data_fake2.zip] (./data_fake2.zip). These data were derived from real data using a random method. To get the real data or to integrate your own data, please contact Olivier Parisot (olivier.parisot@list.lu) or Ivonne Trebs (ivonne.trebs@list.lu).

Olivier Parisot's avatar
Olivier Parisot committed
61
In this archive ([data_fake2.zip](./data_fake2.zip)), the required data files are:
oparisot's avatar
oparisot committed
62 63 64 65 66 67
* *all_valid_q_series_complete2.arff*: the time series representing the measured discharge for each station, under the [ARFF format] (https://weka.wikispaces.com/ARFF).
* *knowledgeDB20-discharge.arff*: the knowledge database containing the Case-Based Reasoning data, under the ARFF format too.
* *stations_coordinates.txt*: the coordinates of each station.
* *stations_relationships_1.xml*: the upstream/downstreams relationships among the stations (file 1/2).
* *stations_relationships_2.xml*: the upstream/downstreams relationships among the stations (file 2/2).
* *shapeCountry.jpg*: a picture to show the shape of the studied country.
oparisot's avatar
oparisot committed
68

Olivier Parisot's avatar
Olivier Parisot committed
69

oparisot's avatar
oparisot committed
70 71
## Screenshots

oparisot's avatar
oparisot committed
72
![pic1](/pictures/picture0.png)
oparisot's avatar
oparisot committed
73

oparisot's avatar
oparisot committed
74
![pic2](/pictures/picture2.png)
oparisot's avatar
oparisot committed
75

oparisot's avatar
oparisot committed
76
![pic3](/pictures/picture3.png)
oparisot's avatar
oparisot committed
77

oparisot's avatar
oparisot committed
78

oparisot's avatar
oparisot committed
79 80
## Video

Olivier Parisot's avatar
Olivier Parisot committed
81
[gapIt short demo](/video/GapIT_Video2.wmv)
oparisot's avatar
oparisot committed
82

oparisot's avatar
oparisot committed
83

Olivier Parisot's avatar
Olivier Parisot committed
84 85
## References

oparisot's avatar
oparisot committed
86
This section lists scientific publications in which gapIt is used. 
oparisot's avatar
oparisot committed
87

Olivier Parisot's avatar
Olivier Parisot committed
88
 * Laura Giustarini, Olivier Parisot, Mohammad Ghoniem, Renaud Hostache, Ivonne Trebs, Benoît Otjacques: **«A user-driven case-based reasoning tool for infilling missing values in daily mean river flow records»**, «Environmental Modelling and Software», Elsevier, 5/2016 [(link)](http://www.sciencedirect.com/science/article/pii/S1364815216301050)
oparisot's avatar
oparisot committed
89

Olivier Parisot's avatar
Olivier Parisot committed
90
 * Laura Giustarini, Olivier Parisot, Mohammad Ghoniem, Renaud Hostache, Ivonne Trebs, Benoît Otjacques:**«gapIT: a user-driven case-based reasoning tool for infilling missing values in daily mean river flow records»**, «Tag der Hydrologie 2016», Koblenz, Germany, 17/3/2016 [(link)](https://scholar.google.lu/citations?view_op=view_citation&hl=fr&user=OeqhaZ4AAAAJ&sortby=pubdate&citation_for_view=OeqhaZ4AAAAJ:4TOpqqG69KYC)
oparisot's avatar
oparisot committed
91

Olivier Parisot's avatar
Olivier Parisot committed
92
 * Laura Giustarini, Olivier Parisot, Mohammad Ghoniem, Ivonne Trebs, Nicolas Médoc, Olivier Faber, Renaud Hostache, Patrick Matgen, Benoît Otjacques: **«Data-infilling in daily mean river flow records: first results using a visual analytics tool (gapIT)»**, «European Geosciences Union General Assembly 2015 (EGU 2015), Geophysical Research Abstracts Volume 17», Vienna, Austria, 4/2015 [(link)](http://adsabs.harvard.edu/abs/2015EGUGA..1710462G)
oparisot's avatar
oparisot committed
93

Olivier Parisot's avatar
Olivier Parisot committed
94
 * Olivier Parisot, Laura Giustarini, Olivier Faber, Renaud Hostache, Ivonne Trebs, Mohammad Ghoniem: **«gapIT: Un outil visuel pour l'imputation de valeurs manquantes en hydrologie»**, «15ème conférence internationale sur l'extraction et la gestion des connaissances (EGC 2015)», Luxembourg, 1/2015 [(link)](http://editions-rnti.fr/?inprocid=1002107&PHPSESSID=ks64gh4ktvuilvujuu9lcva5i4&lg=en&PHPSESSID=ks64gh4ktvuilvujuu9lcva5i4)
Olivier Parisot's avatar
Olivier Parisot committed
95

oparisot's avatar
oparisot committed
96
If you also have employed gapIt, send an email to olivier.parisot@list.lu to have your publication listed here.
Olivier Parisot's avatar
Olivier Parisot committed
97

oparisot's avatar
oparisot committed
98

Olivier Parisot's avatar
Olivier Parisot committed
99 100
## License

Olivier Parisot's avatar
Olivier Parisot committed
101
Licensed under GNU General Public License version 3, see [LICENSE.txt](./LICENSE.txt).