README.md 5.96 KB
Newer Older
weevil-bot's avatar
weevil-bot committed
1
2
# gapIt

Olivier Parisot's avatar
Olivier Parisot committed
3
Copyright 2014 - 2019 Luxembourg Institute of Science and Technology (LIST - [http://www.list.lu/](http://www.list.lu/)). Any use of this software constitutes full acceptance of all terms of the software's license.
weevil-bot's avatar
weevil-bot committed
4
5


Olivier Parisot's avatar
Olivier Parisot committed
6
7
8
9
## Synopsis

gapIT is an interactive software for gap filling in hydrological time series.

weevil-bot's avatar
weevil-bot committed
10
Given as input hydrological discharge data series measured at gauging stations, it performs an automated calculation of missing data using different data-infilling techniques. Donor station(s) are automatically selected based on Dynamic Time Warping, geographical proximity and upstream/downstream relationships among stations. For each gap, the tool computes several flow estimates through various data-infilling techniques, including interpolation, multiple regression, regression trees and neural networks. The visual application provides the possibility for the user to select different donor station(s) w.r.t. those automatically selected.
Olivier Parisot's avatar
Olivier Parisot committed
11
12
13
14
15
16
17
18
19
20
21

The results are validated by randomly creating artificial gaps of different lengths and positions along the entire records. Using the Root Mean Squared Error and the Nash-Sutcliffe coefficient as performance measures, the method is evaluated based on a comparison with the actual measured discharge values.

The interactive but automated approach of gapIT, coupled with a visual inspection system for user-defined refinement, allows for a standardized objective infilling, where subjective decisions are allowed but are at the same time traceable.


## Motivation

Missing records in hydrological databases represent a loss of information and a serious drawback in water management (measures of river flow for example). An incomplete time series prevents the computation of hydrological statistics and indicators. Also, records with data gaps are not suitable as input or validation data for hydrological or hydrodynamic modeling.


Olivier Parisot's avatar
Olivier Parisot committed
22
## Installation / execution
Olivier Parisot's avatar
Olivier Parisot committed
23

weevil-bot's avatar
weevil-bot committed
24
### Requirements
weevil-bot's avatar
weevil-bot committed
25

weevil-bot's avatar
weevil-bot committed
26
Hardware: RAM>=4Gb.
weevil-bot's avatar
weevil-bot committed
27

weevil-bot's avatar
weevil-bot committed
28
Software: 
Olivier Parisot's avatar
Olivier Parisot committed
29
30
* JDK (>=1.7 - [http://www.oracle.com/technetwork/java/javase/downloads/](http://www.oracle.com/technetwork/java/javase/downloads/)).
* Maven (>=3.0.2 - [https://maven.apache.org/](https://maven.apache.org/))
Olivier Parisot's avatar
Olivier Parisot committed
31

Olivier Parisot's avatar
Olivier Parisot committed
32

weevil-bot's avatar
weevil-bot committed
33
### Compilation
weevil-bot's avatar
weevil-bot committed
34

Olivier Parisot's avatar
Olivier Parisot committed
35
36
    mvn clean package -Dmaven.test.skip=true
    
Olivier Parisot's avatar
Olivier Parisot committed
37
(*Some tests are currently not ok, and the build fails without the '-Dmaven.test.skip=true' option.*)
Olivier Parisot's avatar
Olivier Parisot committed
38

Olivier Parisot's avatar
Olivier Parisot committed
39

weevil-bot's avatar
weevil-bot committed
40
### Data preparation
Olivier Parisot's avatar
Olivier Parisot committed
41

Olivier Parisot's avatar
Olivier Parisot committed
42
unzip [data_fake2.zip](./data_fake2.zip) in the same directory
weevil-bot's avatar
weevil-bot committed
43

Olivier Parisot's avatar
Olivier Parisot committed
44

weevil-bot's avatar
weevil-bot committed
45
46
### Execution for Windows OS

Olivier Parisot's avatar
Olivier Parisot committed
47
    run.bat
weevil-bot's avatar
weevil-bot committed
48

weevil-bot's avatar
weevil-bot committed
49
### Execution for other OS
Olivier Parisot's avatar
Olivier Parisot committed
50

Olivier Parisot's avatar
Olivier Parisot committed
51
    java -jar -Xmx4G target/gapIt.jar
Olivier Parisot's avatar
Olivier Parisot committed
52
53
54
55


## Data

weevil-bot's avatar
weevil-bot committed
56
gapIt was used to fill gaps in water discharge time series at Luxembourg (see the 'References' section).
weevil-bot's avatar
weevil-bot committed
57
During tests phases, it was used too for the processing of water level time series.
weevil-bot's avatar
weevil-bot committed
58

weevil-bot's avatar
weevil-bot committed
59
60
Here, fake data are provided within the distribution [data_fake2.zip] (./data_fake2.zip). These data were derived from real data using a random method. To get the real data or to integrate your own data, please contact Olivier Parisot (olivier.parisot@list.lu) or Ivonne Trebs (ivonne.trebs@list.lu).

Olivier Parisot's avatar
Olivier Parisot committed
61
In this archive ([data_fake2.zip](./data_fake2.zip)), the required data files are:
weevil-bot's avatar
weevil-bot committed
62
63
64
65
66
67
* *all_valid_q_series_complete2.arff*: the time series representing the measured discharge for each station, under the [ARFF format] (https://weka.wikispaces.com/ARFF).
* *knowledgeDB20-discharge.arff*: the knowledge database containing the Case-Based Reasoning data, under the ARFF format too.
* *stations_coordinates.txt*: the coordinates of each station.
* *stations_relationships_1.xml*: the upstream/downstreams relationships among the stations (file 1/2).
* *stations_relationships_2.xml*: the upstream/downstreams relationships among the stations (file 2/2).
* *shapeCountry.jpg*: a picture to show the shape of the studied country.
weevil-bot's avatar
weevil-bot committed
68

Olivier Parisot's avatar
Olivier Parisot committed
69

weevil-bot's avatar
weevil-bot committed
70
71
## Screenshots

weevil-bot's avatar
weevil-bot committed
72
![pic1](/pictures/picture0.png)
weevil-bot's avatar
weevil-bot committed
73

weevil-bot's avatar
weevil-bot committed
74
![pic2](/pictures/picture2.png)
weevil-bot's avatar
weevil-bot committed
75

weevil-bot's avatar
weevil-bot committed
76
![pic3](/pictures/picture3.png)
weevil-bot's avatar
weevil-bot committed
77

weevil-bot's avatar
weevil-bot committed
78

weevil-bot's avatar
weevil-bot committed
79
80
## Video

Olivier Parisot's avatar
Olivier Parisot committed
81
[gapIt short demo](/video/GapIT_Video2.wmv)
weevil-bot's avatar
weevil-bot committed
82

weevil-bot's avatar
weevil-bot committed
83

Olivier Parisot's avatar
Olivier Parisot committed
84
85
## References

weevil-bot's avatar
weevil-bot committed
86
This section lists scientific publications in which gapIt is used. 
weevil-bot's avatar
weevil-bot committed
87

Olivier Parisot's avatar
Olivier Parisot committed
88
 * Laura Giustarini, Olivier Parisot, Mohammad Ghoniem, Renaud Hostache, Ivonne Trebs, Benoît Otjacques: **«A user-driven case-based reasoning tool for infilling missing values in daily mean river flow records»**, «Environmental Modelling and Software», Elsevier, 5/2016 [(link)](http://www.sciencedirect.com/science/article/pii/S1364815216301050)
weevil-bot's avatar
weevil-bot committed
89

Olivier Parisot's avatar
Olivier Parisot committed
90
 * Laura Giustarini, Olivier Parisot, Mohammad Ghoniem, Renaud Hostache, Ivonne Trebs, Benoît Otjacques:**«gapIT: a user-driven case-based reasoning tool for infilling missing values in daily mean river flow records»**, «Tag der Hydrologie 2016», Koblenz, Germany, 17/3/2016 [(link)](https://scholar.google.lu/citations?view_op=view_citation&hl=fr&user=OeqhaZ4AAAAJ&sortby=pubdate&citation_for_view=OeqhaZ4AAAAJ:4TOpqqG69KYC)
weevil-bot's avatar
weevil-bot committed
91

Olivier Parisot's avatar
Olivier Parisot committed
92
 * Laura Giustarini, Olivier Parisot, Mohammad Ghoniem, Ivonne Trebs, Nicolas Médoc, Olivier Faber, Renaud Hostache, Patrick Matgen, Benoît Otjacques: **«Data-infilling in daily mean river flow records: first results using a visual analytics tool (gapIT)»**, «European Geosciences Union General Assembly 2015 (EGU 2015), Geophysical Research Abstracts Volume 17», Vienna, Austria, 4/2015 [(link)](http://adsabs.harvard.edu/abs/2015EGUGA..1710462G)
weevil-bot's avatar
weevil-bot committed
93

Olivier Parisot's avatar
Olivier Parisot committed
94
 * Olivier Parisot, Laura Giustarini, Olivier Faber, Renaud Hostache, Ivonne Trebs, Mohammad Ghoniem: **«gapIT: Un outil visuel pour l'imputation de valeurs manquantes en hydrologie»**, «15ème conférence internationale sur l'extraction et la gestion des connaissances (EGC 2015)», Luxembourg, 1/2015 [(link)](http://editions-rnti.fr/?inprocid=1002107&PHPSESSID=ks64gh4ktvuilvujuu9lcva5i4&lg=en&PHPSESSID=ks64gh4ktvuilvujuu9lcva5i4)
Olivier Parisot's avatar
Olivier Parisot committed
95

Olivier Parisot's avatar
Olivier Parisot committed
96
97
98
99
100
101
102
If you also have employed gapIt, send an email to [Olivier Parisot](olivier.parisot@list.lu) to have your publication listed here.


## Contact

Any question? Please contact [Olivier Parisot](olivier.parisot@list.lu) or visit the [LIST website](https://www.list.lu/en/contact/).

Olivier Parisot's avatar
Olivier Parisot committed
103

weevil-bot's avatar
weevil-bot committed
104

Olivier Parisot's avatar
Olivier Parisot committed
105
106
## License

Olivier Parisot's avatar
Olivier Parisot committed
107
Licensed under GNU General Public License version 3, see [LICENSE.txt](./LICENSE.txt).