README.md 36 KB
Newer Older
1
# OTBTF
remi cresson's avatar
remi cresson committed
2

Cresson Remi's avatar
Cresson Remi committed
3
This remote module of the [Orfeo ToolBox](https://www.orfeo-toolbox.org) provides a generic, multi purpose deep learning framework, targeting remote sensing images processing.
remi cresson's avatar
remi cresson committed
4
It contains a set of new process objects that internally invoke [Tensorflow](https://www.tensorflow.org/), and a bunch of user-oriented applications to perform deep learning with real-world remote sensing images.
Cresson Remi's avatar
Cresson Remi committed
5
Applications can be used to build OTB pipelines from Python or C++ APIs. 
remi cresson's avatar
remi cresson committed
6
7
8
9

*Main highlights*
 - Sampling,
 - Training, supporting save/restore/import operations (a model can be trained from scratch or fine-tuned),
Cresson Remi's avatar
Cresson Remi committed
10
 - Serving models with support of OTB streaming mechanism. Meaning (1) not limited by images sizes, (2) can be used as a "lego" in any OTB pipeline and preserve streaming, (3) MPI support available (use multiple processing unit to generate one single output image)
remi cresson's avatar
remi cresson committed
11

Cresson Remi's avatar
Cresson Remi committed
12
*Portfolio*
Cresson Remi's avatar
Cresson Remi committed
13

Cresson Remi's avatar
Cresson Remi committed
14
Below are some screen captures of deep learning applications performed at large scale with OTBTF.
Cresson Remi's avatar
Cresson Remi committed
15
 - Image to image translation (Spot-7 image --> Wikimedia Map using CGAN)
Cresson Remi's avatar
Cresson Remi committed
16
<img src ="doc/images/pix2pix.png" />
Cresson Remi's avatar
Cresson Remi committed
17

Cresson Remi's avatar
Cresson Remi committed
18
 - Landcover mapping (Spot-7 images --> Building map using semantic segmentation)
Cresson Remi's avatar
Cresson Remi committed
19
<img src ="doc/images/landcover.png" />
Cresson Remi's avatar
Cresson Remi committed
20

Cresson Remi's avatar
Cresson Remi committed
21
 - Image enhancement (Enhancement of Sentinel-2 images at 1.5m  using SRGAN)
Cresson Remi's avatar
Cresson Remi committed
22
<img src ="doc/images/supresol.png" />
Cresson Remi's avatar
Cresson Remi committed
23

Cresson Remi's avatar
Cresson Remi committed
24
You can read more details about these applications on [this blog](https://mdl4eo.irstea.fr/2019/)
remi cresson's avatar
remi cresson committed
25

26
# How to install
27

Cresson Remi's avatar
Cresson Remi committed
28
For now you have two options: either use the existing **docker image**, or build everything yourself **from source**.
29

Cresson Remi's avatar
Cresson Remi committed
30
## Docker image
Cresson Remi's avatar
Cresson Remi committed
31

32
Use the latest image from dockerhub:
33
```
34
docker pull mdl4eo/otbtf1.6
35
docker run -u otbuser -v $(pwd):/home/otbuser mdl4eo/otbtf1.6 otbcli_PatchesExtraction -help
36
```
Cresson Remi's avatar
Cresson Remi committed
37
Please note that for now, TensorFlow and OTB are built with the minimal optimization flags, no CUDA/OpenCL enabled, no AVX and such for CPU. **Contributions are welcome: add more Dockerfiles, e.g. with OpenCL or CUDA support, CPU optimisations, etc.**
38
The dockerfiles corresponding to the images available on dockerhub are provided in the `tools/dockerfiles/` path of this repository.
39

Cresson Remi's avatar
Cresson Remi committed
40
## Build from sources
41

Cresson Remi's avatar
Cresson Remi committed
42
See [here](doc/HOWTOBUILD.md) to see how to build the remote module from sources.
remi cresson's avatar
remi cresson committed
43
44

# New applications
Cresson Remi's avatar
Cresson Remi committed
45

remi cresson's avatar
remi cresson committed
46
Let's describe quickly the new applications provided.
Cresson Remi's avatar
Cresson Remi committed
47

remi cresson's avatar
remi cresson committed
48
## PatchesExtraction
Cresson Remi's avatar
Cresson Remi committed
49

50
51
52
53
This application performs the extraction of patches in images from a vector data containing points. 
The OTB sampling framework can be used to generate the set of selected points. 
After that, you can use the **PatchesExtraction** application to perform the sampling of your images.
We denote _input source_ an input image, or a stack of input images (of the same size !). 
54
The user can set the `OTB_TF_NSOURCES` environment variable to select the number of _input sources_ that he wants.
55
56
57
58
59
60
61
For example, for sampling a Time Series (TS) together with a single Very High Resolution image (VHR), a number of 2 sources is required: 1 input images list for time series and 1 input image for the VHR.
The sampled patches will be extracted at each positions designed by the points, only if they are entirely lying inside all _input sources_ extents.
For each _input source_, patches sizes must be provided.
For each _input source_, the application export all sampled patches as a single multiband raster, stacked in rows.
For instance, for *n* samples of size *16 x 16* from a *4* channels _input source_, the output image will be a raster of size *16 x 16n* with *4* channels. 
An optional output is an image of size *1 x n* containing the value of one specific field of the input vector data. 
Typically, the *class* field can be used to generate a dataset suitable for a model that performs pixel wise classification. 
remi cresson's avatar
remi cresson committed
62

Cresson Remi's avatar
Cresson Remi committed
63
![Schema](doc/images/patches_extraction.png)
remi cresson's avatar
remi cresson committed
64

remi cresson's avatar
remi cresson committed
65
66
67
68
69
```
This application extracts patches in multiple input images. Change the OTB_TF_NSOURCES environment variable to set the number of sources.
Parameters: 
        -source1            <group>          Parameters for source 1 
MISSING -source1.il         <string list>    Input image(s) 1  (mandatory)
Cresson Remi's avatar
Cresson Remi committed
70
MISSING -source1.out        <string> [pixel] Output patches for image 1  [pixel=uint8/uint16/int16/uint32/int32/float/double/cint16/cint32/cfloat/cdouble] (default value is float) (mandatory)
remi cresson's avatar
remi cresson committed
71
72
73
MISSING -source1.patchsizex <int32>          X patch size for image 1  (mandatory)
MISSING -source1.patchsizey <int32>          Y patch size for image 1  (mandatory)
MISSING -vec                <string>         Positions of the samples (must be in the same projection as input image)  (mandatory)
Cresson Remi's avatar
Cresson Remi committed
74
75
        -outlabels          <string> [pixel] output labels  [pixel=uint8/uint16/int16/uint32/int32/float/double/cint16/cint32/cfloat/cdouble] (default value is uint8) (optional, off by default)
MISSING -field              <string>         field of class in the vector data  (mandatory)
remi cresson's avatar
remi cresson committed
76
77
78
79
80
81
82
83
84
        -inxml              <string>         Load otb application from xml file  (optional, off by default)
        -progress           <boolean>        Report progress 
        -help               <string list>    Display long help (empty list), or help for given parameters keys

Use -help param1 [... paramN] to see detailed documentation of those parameters.

Examples: 
otbcli_PatchesExtraction -vec points.sqlite -source1.il $s2_list -source1.patchsizex 16 -source1.patchsizey 16 -field class -source1.out outpatches_16x16.tif -outlabels outlabels.tif
```
Cresson Remi's avatar
Cresson Remi committed
85

86
## Build your Tensorflow model <a name="buildmodel"></a>
Cresson Remi's avatar
Cresson Remi committed
87

88
89
90
91
You can build models using the TensorFlow Python API as shown in the `./python/` directory.
Models must be exported in **SavedModel** format.
When using a model in OTBTF, the important thing is to know the following parameters related to the _placeholders_ (the inputs of your model) and _output tensors_ (the outputs of your model).
 - For each _input placeholder_:
Cresson Remi's avatar
Cresson Remi committed
92
   - Name
93
94
   - **Receptive field**
 - For each _output tensor_:
Cresson Remi's avatar
Cresson Remi committed
95
   - Name 
96
97
   - **Expression field**
   - **Scale factor**
remi cresson's avatar
remi cresson committed
98

Cresson Remi's avatar
Cresson Remi committed
99
![Schema](doc/images/schema.png)
remi cresson's avatar
remi cresson committed
100

101
102
103
104
105
106
The **scale factor** descibes the physical change of spacing of the outputs, typically introduced in the model by non unitary strides in pooling or convolution operators.
For each output, it is expressed relatively to one single input of the model called the _reference input source_.
Additionally, the names of the _target nodes_ must be known (e.g. "optimizer").
Also, the names of _user placeholders_, typically scalars placeholders that are used to control some parameters of the model, must be know (e.g. "dropout_rate").
The **receptive field** corresponds to the input volume that "sees" the deep net.
The **expression field** corresponds to the output volume that the deep net will create.
remi cresson's avatar
remi cresson committed
107
108

## Train your Tensorflow model
109
110
111

Here we assume that you have produced patches using the **PatchesExtraction** application, and that you have a **SavedModel** stored in a directory somewhere on your filesystem.
The **TensorflowModelTrain** application performs the training, validation (against test dataset, and against validation dataset) providing the usual metrics that machine learning frameworks provide (confusion matrix, recall, precision, f-score, ...).
112
113
114
You must provide the path of the **SavedModel** to the `model.dir` parameter.
The `model.restorefrom` and `model.saveto` corresponds to the variables of the **SavedModel** used respectively for restoring and saving them.
Set you _input sources_ for training (`training` parameter group) and for validation (`validation` parameter group): the evaluation is performed against training data, and optionally also against the validation data (only if you set `validation.mode` to "class").
115
For each _input sources_, the patch size and the placeholder name must be provided.
116
117
118
119
Regarding validation, if a different name is found in a particular _input source_ of the `validation` parameter group, the application knows that the _input source_ is not fed to the model at inference, but is used as reference to compute evaluation metrics of the validation dataset.
Batch size (`training.batchsize`) and number of epochs (`training.epochs`) can be set.
_User placeholders_ can be set separately for training (`training.userplaceholders`) and validation (`validation.userplaceholders`).
The `validation.userplaceholders` can be useful if you have a model that behaves differently depending the given placeholder. 
120
121
122
Let's take the example of dropout: it's nice for training, but you have to disable it to use the model at inference time. 
Hence you will pass a placeholder with "dropout\_rate=0.3" for training and "dropout\_rate=0.0" for validation. 
Of course, one can train models from handmade python code: to import the patches images, a convenient method consist in reading patches images as numpy arrays using OTB applications (e.g. **ExtractROI**) or GDAL, then do a np.reshape to the dimensions wanted.
remi cresson's avatar
remi cresson committed
123

Cresson Remi's avatar
Cresson Remi committed
124
![Schema](doc/images/model_training.png)
remi cresson's avatar
remi cresson committed
125

remi cresson's avatar
remi cresson committed
126
127
128
```
Train a multisource deep learning net using Tensorflow. Change the OTB_TF_NSOURCES environment variable to set the number of sources.
Parameters: 
129
130
131
132
133
134
        -model                        <group>          Model parameters 
MISSING -model.dir                    <string>         Tensorflow model_save directory  (mandatory)
        -model.restorefrom            <string>         Restore model from path  (optional, off by default)
        -model.saveto                 <string>         Save model to path  (optional, off by default)
        -training                     <group>          Training parameters 
        -training.batchsize           <int32>          Batch size  (mandatory, default value is 100)
Cresson Remi's avatar
Cresson Remi committed
135
        -training.epochs              <int32>          Number of epochs  (mandatory, default value is 100)
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
        -training.userplaceholders    <string list>    Additional single-valued placeholders for training. Supported types: int, float, bool.  (optional, off by default)
MISSING -training.targetnodes         <string list>    Names of the target nodes  (mandatory)
        -training.outputtensors       <string list>    Names of the output tensors to display  (optional, off by default)
        -training.usestreaming        <boolean>        Use the streaming through patches (slower but can process big dataset)  (optional, off by default, default value is false)
        -training.source1             <group>          Parameters for source #1 (training) 
MISSING -training.source1.il          <string list>    Input image (or list to stack) for source #1 (training)  (mandatory)
MISSING -training.source1.patchsizex  <int32>          Patch size (x) for source #1  (mandatory)
MISSING -training.source1.patchsizey  <int32>          Patch size (y) for source #1  (mandatory)
MISSING -training.source1.placeholder <string>         Name of the input placeholder for source #1 (training)  (mandatory)
        -training.source2             <group>          Parameters for source #2 (training) 
MISSING -training.source2.il          <string list>    Input image (or list to stack) for source #2 (training)  (mandatory)
MISSING -training.source2.patchsizex  <int32>          Patch size (x) for source #2  (mandatory)
MISSING -training.source2.patchsizey  <int32>          Patch size (y) for source #2  (mandatory)
MISSING -training.source2.placeholder <string>         Name of the input placeholder for source #2 (training)  (mandatory)
        -validation                   <group>          Validation parameters 
Cresson Remi's avatar
Cresson Remi committed
151
        -validation.step              <int32>          Perform the validation every Nth epochs  (mandatory, default value is 10)
152
153
154
155
156
157
158
159
160
161
162
163
        -validation.mode              <string>         Metrics to compute [none/class/rmse] (mandatory, default value is none)
        -validation.userplaceholders  <string list>    Additional single-valued placeholders for validation. Supported types: int, float, bool.  (optional, off by default)
        -validation.usestreaming      <boolean>        Use the streaming through patches (slower but can process big dataset)  (optional, off by default, default value is false)
        -validation.source1           <group>          Parameters for source #1 (validation) 
        -validation.source1.il        <string list>    Input image (or list to stack) for source #1 (validation)  (mandatory)
        -validation.source1.name      <string>         Name of the input placeholder or output tensor for source #1 (validation)  (mandatory)
        -validation.source2           <group>          Parameters for source #2 (validation) 
        -validation.source2.il        <string list>    Input image (or list to stack) for source #2 (validation)  (mandatory)
        -validation.source2.name      <string>         Name of the input placeholder or output tensor for source #2 (validation)  (mandatory)
        -inxml                        <string>         Load otb application from xml file  (optional, off by default)
        -progress                     <boolean>        Report progress 
        -help                         <string list>    Display long help (empty list), or help for given parameters keys
remi cresson's avatar
remi cresson committed
164
165
166
167

Use -help param1 [... paramN] to see detailed documentation of those parameters.

Examples: 
168
otbcli_TensorflowModelTrain -source1.il spot6pms.tif -source1.placeholder x1 -source1.patchsizex 16 -source1.patchsizey 16 -source2.il labels.tif -source2.placeholder y1 -source2.patchsizex 1 -source2.patchsizex 1 -model.dir /tmp/my_saved_model/ -training.userplaceholders is_training=true dropout=0.2 -training.targetnodes optimizer -model.saveto /tmp/my_saved_model/variables/variables
remi cresson's avatar
remi cresson committed
169
```
Cresson Remi's avatar
Cresson Remi committed
170

171
172
As you can note, there is `$OTB_TF_NSOURCES` + 1 sources because we often need at least one more source for the reference data (e.g. terrain truth for land cover mapping).

remi cresson's avatar
remi cresson committed
173
## Serve the model
174
175
176
177
178
179
180
181
182
183
184

The **TensorflowModelServe** application perform model serving, it can be used to produce output raster with the desired tensors. 
Thanks to the streaming mechanism, very large images can be produced. 
The application uses the `TensorflowModelFilter` and a `StreamingFilter` to force the streaming of output. 
This last can be optionally disabled by the user, if he prefers using the extended filenames to deal with chunk sizes. 
However, it's still very useful when the application is used in other composites applications, or just without extended filename magic. 
Some models can consume a lot of memory. 
In addition, the native tiling strategy of OTB consists in strips but this might not always the best. 
For Convolutional Neural Networks for instance, square tiles are more interesting because the padding required to perform the computation of one single strip of pixels induces to input a lot more pixels that to process the computation of one single tile of pixels.
So, this application takes in input one or multiple _input sources_ (the number of _input sources_ can be changed by setting the `OTB_TF_NSOURCES` to the desired number) and produce one output of the specified tensors.
The user is responsible of giving the **receptive field** and **name** of _input placeholders_, as well as the **expression field**, **scale factor** and **name** of _output tensors_.
185
The first _input source_ (`source1.il`) corresponds to the _reference input source_.
186
187
188
As explained [previously](#buildmodel), the **scale factor** provided for the _output tensors_ is related to this _reference input source_.
The user can ask for multiple _output tensors_, that will be stack along the channel dimension of the output raster.
However, if the sizes of those _output tensors_ are not consistent (e.g. a different number of (x,y) elements), an exception will be thrown.
remi cresson's avatar
remi cresson committed
189

Cresson Remi's avatar
Cresson Remi committed
190
![Schema](doc/images/classif_map.png)
remi cresson's avatar
remi cresson committed
191
192


remi cresson's avatar
remi cresson committed
193
```
Cresson Remi's avatar
Cresson Remi committed
194
Multisource deep learning classifier using TensorFlow. Change the OTB_TF_NSOURCES environment variable to set the number of sources.
remi cresson's avatar
remi cresson committed
195
Parameters: 
196
197
198
199
200
201
        -source1                <group>          Parameters for source #1 
MISSING -source1.il             <string list>    Input image (or list to stack) for source #1  (mandatory)
MISSING -source1.rfieldx        <int32>          Input receptive field (width) for source #1  (mandatory)
MISSING -source1.rfieldy        <int32>          Input receptive field (height) for source #1  (mandatory)
MISSING -source1.placeholder    <string>         Name of the input placeholder for source #1  (mandatory)
        -model                  <group>          model parameters 
Cresson Remi's avatar
Cresson Remi committed
202
MISSING -model.dir              <string>         TensorFlow model_save directory  (mandatory)
203
204
205
        -model.userplaceholders <string list>    Additional single-valued placeholders. Supported types: int, float, bool.  (optional, off by default)
        -model.fullyconv        <boolean>        Fully convolutional  (optional, off by default, default value is false)
        -output                 <group>          Output tensors parameters 
Cresson Remi's avatar
Cresson Remi committed
206
        -output.spcscale        <float>          The output spacing scale, related to the first input  (mandatory, default value is 1)
207
208
209
210
211
MISSING -output.names           <string list>    Names of the output tensors  (mandatory)
        -output.efieldx         <int32>          The output expression field (width)  (mandatory, default value is 1)
        -output.efieldy         <int32>          The output expression field (height)  (mandatory, default value is 1)
        -optim                  <group>          This group of parameters allows optimization of processing time 
        -optim.disabletiling    <boolean>        Disable tiling  (optional, off by default, default value is false)
Cresson Remi's avatar
Cresson Remi committed
212
213
        -optim.tilesizex        <int32>          Tile width used to stream the filter output  (mandatory, default value is 16)
        -optim.tilesizey        <int32>          Tile height used to stream the filter output  (mandatory, default value is 16)
214
215
216
217
MISSING -out                    <string> [pixel] output image  [pixel=uint8/uint16/int16/uint32/int32/float/double/cint16/cint32/cfloat/cdouble] (default value is float) (mandatory)
        -inxml                  <string>         Load otb application from xml file  (optional, off by default)
        -progress               <boolean>        Report progress 
        -help                   <string list>    Display long help (empty list), or help for given parameters keys
remi cresson's avatar
remi cresson committed
218
219
220
221

Use -help param1 [... paramN] to see detailed documentation of those parameters.

Examples: 
222
otbcli_TensorflowModelServe -source1.il spot6pms.tif -source1.placeholder x1 -source1.rfieldx 16 -source1.rfieldy 16 -model.dir /tmp/my_saved_model/ -model.userplaceholders is_training=false dropout=0.0 -output.names out_predict1 out_proba1 -out "classif128tgt.tif?&streaming:type=tiled&streaming:sizemode=height&streaming:sizevalue=256"
remi cresson's avatar
remi cresson committed
223
```
Cresson Remi's avatar
Cresson Remi committed
224

remi cresson's avatar
remi cresson committed
225
## Composite applications for classification
226

remi cresson's avatar
remi cresson committed
227
228
229
Who has never dreamed to use classic classifiers performing on deep learning features?
This is possible thank to two new applications that uses the existing training/classification applications of OTB:

remi cresson's avatar
remi cresson committed
230
**TrainClassifierFromDeepFeatures**: is a composite application that wire the **TensorflowModelServe** application output into the existing official **TrainImagesClassifier** application. 
remi cresson's avatar
remi cresson committed
231
232
233
234
235
```
Train a classifier from deep net based features of an image and training vector data.
Parameters: 
        -source1                     <group>          Parameters for source 1 
MISSING -source1.il                  <string list>    Input image (or list to stack) for source #1  (mandatory)
236
237
MISSING -source1.rfieldx             <int32>          Input receptive field (width) for source #1  (mandatory)
MISSING -source1.rfieldy             <int32>          Input receptive field (height) for source #1  (mandatory)
remi cresson's avatar
remi cresson committed
238
MISSING -source1.placeholder         <string>         Name of the input placeholder for source #1  (mandatory)
Cresson Remi's avatar
Cresson Remi committed
239
240
        -model                       <group>          Deep net inputs parameters 
MISSING -model.dir                   <string>         TensorFlow model_save directory  (mandatory)
remi cresson's avatar
remi cresson committed
241
242
243
        -model.userplaceholders      <string list>    Additional single-valued placeholders. Supported types: int, float, bool.  (optional, off by default)
        -model.fullyconv             <boolean>        Fully convolutional  (optional, off by default, default value is false)
        -output                      <group>          Deep net outputs parameters 
Cresson Remi's avatar
Cresson Remi committed
244
        -output.spcscale             <float>          The output spacing scale, related to the first input  (mandatory, default value is 1)
remi cresson's avatar
remi cresson committed
245
MISSING -output.names                <string list>    Names of the output tensors  (mandatory)
246
247
        -output.efieldx              <int32>          The output expression field (width)  (mandatory, default value is 1)
        -output.efieldy              <int32>          The output expression field (height)  (mandatory, default value is 1)
Cresson Remi's avatar
Cresson Remi committed
248
        -optim                       <group>          Processing time optimization 
249
        -optim.disabletiling         <boolean>        Disable tiling  (optional, off by default, default value is false)
Cresson Remi's avatar
Cresson Remi committed
250
251
252
253
254
255
256
257
        -optim.tilesizex             <int32>          Tile width used to stream the filter output  (mandatory, default value is 16)
        -optim.tilesizey             <int32>          Tile height used to stream the filter output  (mandatory, default value is 16)
        -ram                         <int32>          Available RAM (Mb)  (optional, off by default, default value is 128)
MISSING -vd                          <string list>    Vector data for training  (mandatory)
        -valid                       <string list>    Vector data for validation  (optional, off by default)
MISSING -out                         <string>         Output classification model  (mandatory)
        -confmatout                  <string>         Output confusion matrix  (optional, off by default)
        -sample                      <group>          Sampling parameters 
remi cresson's avatar
remi cresson committed
258
259
260
261
        -sample.mt                   <int32>          Maximum training sample size per class  (mandatory, default value is 1000)
        -sample.mv                   <int32>          Maximum validation sample size per class  (mandatory, default value is 1000)
        -sample.bm                   <int32>          Bound sample number by minimum  (mandatory, default value is 1)
        -sample.vtr                  <float>          Training and validation sample ratio  (mandatory, default value is 0.5)
262
        -sample.vfn                  <string>         Field containing the class integer label for supervision  (mandatory, no default value)
Cresson Remi's avatar
Cresson Remi committed
263
        -elev                        <group>          Elevation parameters 
remi cresson's avatar
remi cresson committed
264
265
266
        -elev.dem                    <string>         DEM directory  (optional, off by default)
        -elev.geoid                  <string>         Geoid File  (optional, off by default)
        -elev.default                <float>          Default elevation  (mandatory, default value is 0)
Cresson Remi's avatar
Cresson Remi committed
267
        -classifier                  <string>         Classifier parameters [libsvm/boost/dt/gbt/ann/bayes/rf/knn/sharkrf/sharkkm] (mandatory, default value is libsvm)
remi cresson's avatar
remi cresson committed
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
        -classifier.libsvm.k         <string>         SVM Kernel Type [linear/rbf/poly/sigmoid] (mandatory, default value is linear)
        -classifier.libsvm.m         <string>         SVM Model Type [csvc/nusvc/oneclass] (mandatory, default value is csvc)
        -classifier.libsvm.c         <float>          Cost parameter C  (mandatory, default value is 1)
        -classifier.libsvm.nu        <float>          Cost parameter Nu  (mandatory, default value is 0.5)
        -classifier.libsvm.opt       <boolean>        Parameters optimization  (mandatory, default value is false)
        -classifier.libsvm.prob      <boolean>        Probability estimation  (mandatory, default value is false)
        -classifier.boost.t          <string>         Boost Type [discrete/real/logit/gentle] (mandatory, default value is real)
        -classifier.boost.w          <int32>          Weak count  (mandatory, default value is 100)
        -classifier.boost.r          <float>          Weight Trim Rate  (mandatory, default value is 0.95)
        -classifier.boost.m          <int32>          Maximum depth of the tree  (mandatory, default value is 1)
        -classifier.dt.max           <int32>          Maximum depth of the tree  (mandatory, default value is 65535)
        -classifier.dt.min           <int32>          Minimum number of samples in each node  (mandatory, default value is 10)
        -classifier.dt.ra            <float>          Termination criteria for regression tree  (mandatory, default value is 0.01)
        -classifier.dt.cat           <int32>          Cluster possible values of a categorical variable into K <= cat clusters to find a suboptimal split  (mandatory, default value is 10)
        -classifier.dt.f             <int32>          K-fold cross-validations  (mandatory, default value is 10)
        -classifier.dt.r             <boolean>        Set Use1seRule flag to false  (mandatory, default value is false)
        -classifier.dt.t             <boolean>        Set TruncatePrunedTree flag to false  (mandatory, default value is false)
        -classifier.gbt.w            <int32>          Number of boosting algorithm iterations  (mandatory, default value is 200)
        -classifier.gbt.s            <float>          Regularization parameter  (mandatory, default value is 0.01)
        -classifier.gbt.p            <float>          Portion of the whole training set used for each algorithm iteration  (mandatory, default value is 0.8)
        -classifier.gbt.max          <int32>          Maximum depth of the tree  (mandatory, default value is 3)
        -classifier.ann.t            <string>         Train Method Type [back/reg] (mandatory, default value is reg)
        -classifier.ann.sizes        <string list>    Number of neurons in each intermediate layer  (mandatory)
        -classifier.ann.f            <string>         Neuron activation function type [ident/sig/gau] (mandatory, default value is sig)
        -classifier.ann.a            <float>          Alpha parameter of the activation function  (mandatory, default value is 1)
        -classifier.ann.b            <float>          Beta parameter of the activation function  (mandatory, default value is 1)
        -classifier.ann.bpdw         <float>          Strength of the weight gradient term in the BACKPROP method  (mandatory, default value is 0.1)
        -classifier.ann.bpms         <float>          Strength of the momentum term (the difference between weights on the 2 previous iterations)  (mandatory, default value is 0.1)
        -classifier.ann.rdw          <float>          Initial value Delta_0 of update-values Delta_{ij} in RPROP method  (mandatory, default value is 0.1)
        -classifier.ann.rdwm         <float>          Update-values lower limit Delta_{min} in RPROP method  (mandatory, default value is 1e-07)
        -classifier.ann.term         <string>         Termination criteria [iter/eps/all] (mandatory, default value is all)
        -classifier.ann.eps          <float>          Epsilon value used in the Termination criteria  (mandatory, default value is 0.01)
        -classifier.ann.iter         <int32>          Maximum number of iterations used in the Termination criteria  (mandatory, default value is 1000)
        -classifier.rf.max           <int32>          Maximum depth of the tree  (mandatory, default value is 5)
        -classifier.rf.min           <int32>          Minimum number of samples in each node  (mandatory, default value is 10)
        -classifier.rf.ra            <float>          Termination Criteria for regression tree  (mandatory, default value is 0)
        -classifier.rf.cat           <int32>          Cluster possible values of a categorical variable into K <= cat clusters to find a suboptimal split  (mandatory, default value is 10)
        -classifier.rf.var           <int32>          Size of the randomly selected subset of features at each tree node  (mandatory, default value is 0)
        -classifier.rf.nbtrees       <int32>          Maximum number of trees in the forest  (mandatory, default value is 100)
        -classifier.rf.acc           <float>          Sufficient accuracy (OOB error)  (mandatory, default value is 0.01)
        -classifier.knn.k            <int32>          Number of Neighbors  (mandatory, default value is 32)
        -classifier.sharkrf.nbtrees  <int32>          Maximum number of trees in the forest  (mandatory, default value is 100)
        -classifier.sharkrf.nodesize <int32>          Min size of the node for a split  (mandatory, default value is 25)
        -classifier.sharkrf.mtry     <int32>          Number of features tested at each node  (mandatory, default value is 0)
        -classifier.sharkrf.oobr     <float>          Out of bound ratio  (mandatory, default value is 0.66)
        -classifier.sharkkm.maxiter  <int32>          Maximum number of iteration for the kmeans algorithm.  (mandatory, default value is 10)
        -classifier.sharkkm.k        <int32>          The number of class used for the kmeans algorithm.  (mandatory, default value is 2)
Cresson Remi's avatar
Cresson Remi committed
315
        -rand                        <int32>          User defined random seed  (optional, off by default)
remi cresson's avatar
remi cresson committed
316
317
318
319
320
        -inxml                       <string>         Load otb application from xml file  (optional, off by default)
        -progress                    <boolean>        Report progress 
        -help                        <string list>    Display long help (empty list), or help for given parameters keys

Use -help param1 [... paramN] to see detailed documentation of those parameters.
Cresson Remi's avatar
Cresson Remi committed
321
322
323

Examples: 
None
remi cresson's avatar
remi cresson committed
324
```
remi cresson's avatar
remi cresson committed
325
326
327
328

**ImageClassifierFromDeepFeatures** same approach with the official **ImageClassifier**.

```
remi cresson's avatar
remi cresson committed
329
330
331
332
Classify image using features from a deep net and an OTB machine learning classification model
Parameters: 
        -source1                    <group>          Parameters for source 1 
MISSING -source1.il                 <string list>    Input image (or list to stack) for source #1  (mandatory)
333
334
MISSING -source1.rfieldx            <int32>          Input receptive field (width) for source #1  (mandatory)
MISSING -source1.rfieldy            <int32>          Input receptive field (height) for source #1  (mandatory)
remi cresson's avatar
remi cresson committed
335
336
MISSING -source1.placeholder        <string>         Name of the input placeholder for source #1  (mandatory)
        -deepmodel                  <group>          Deep net model parameters 
Cresson Remi's avatar
Cresson Remi committed
337
MISSING -deepmodel.dir              <string>         TensorFlow model_save directory  (mandatory)
remi cresson's avatar
remi cresson committed
338
339
340
        -deepmodel.userplaceholders <string list>    Additional single-valued placeholders. Supported types: int, float, bool.  (optional, off by default)
        -deepmodel.fullyconv        <boolean>        Fully convolutional  (optional, off by default, default value is false)
        -output                     <group>          Deep net outputs parameters 
Cresson Remi's avatar
Cresson Remi committed
341
        -output.spcscale            <float>          The output spacing scale, related to the first input  (mandatory, default value is 1)
remi cresson's avatar
remi cresson committed
342
MISSING -output.names               <string list>    Names of the output tensors  (mandatory)
343
344
345
346
        -output.efieldx             <int32>          The output expression field (width)  (mandatory, default value is 1)
        -output.efieldy             <int32>          The output expression field (height)  (mandatory, default value is 1)
        -optim                      <group>          This group of parameters allows optimization of processing time 
        -optim.disabletiling        <boolean>        Disable tiling  (optional, off by default, default value is false)
Cresson Remi's avatar
Cresson Remi committed
347
348
        -optim.tilesizex            <int32>          Tile width used to stream the filter output  (mandatory, default value is 16)
        -optim.tilesizey            <int32>          Tile height used to stream the filter output  (mandatory, default value is 16)
remi cresson's avatar
remi cresson committed
349
350
351
MISSING -model                      <string>         Model file  (mandatory)
        -imstat                     <string>         Statistics file  (optional, off by default)
        -nodatalabel                <int32>          Label mask value  (optional, off by default, default value is 0)
352
353
MISSING -out                        <string> [pixel] Output image  [pixel=uint8/uint16/int16/uint32/int32/float/double/cint16/cint32/cfloat/cdouble] (default value is uint8) (mandatory)
        -confmap                    <string> [pixel] Confidence map image  [pixel=uint8/uint16/int16/uint32/int32/float/double/cint16/cint32/cfloat/cdouble] (default value is double) (optional, off by default)
remi cresson's avatar
remi cresson committed
354
355
356
357
358
359
        -ram                        <int32>          Ram  (optional, off by default, default value is 128)
        -inxml                      <string>         Load otb application from xml file  (optional, off by default)
        -progress                   <boolean>        Report progress 
        -help                       <string list>    Display long help (empty list), or help for given parameters keys

Use -help param1 [... paramN] to see detailed documentation of those parameters.
Cresson Remi's avatar
Cresson Remi committed
360
361
362

Examples: 
None
remi cresson's avatar
remi cresson committed
363
```
Cresson Remi's avatar
Cresson Remi committed
364

remi cresson's avatar
remi cresson committed
365
Note that you can still set the `OTB_TF_NSOURCES` environment variable.
Cresson Remi's avatar
Cresson Remi committed
366
367
368
369
370

# How to use

## The basics

Cresson Remi's avatar
Cresson Remi committed
371
Here we will try to provide a simple example of doing a classification using a deep net that performs on one single VHR image.
372
373
374
Our data set consists in one Spot-7 image, *spot7.tif*, and a training vector data, *terrain_truth.shp* that describes sparsely forest / non-forest polygons.
First, we compute statistics of the vector data : how many points can we sample inside objects, and how many objects in each class.
We use the **PolygonClassStatistics** application of OTB.
remi cresson's avatar
remi cresson committed
375
```
376
otbcli_PolygonClassStatistics -vec terrain_truth.shp -field class -in spot7.tif -out vec_stats.xml
remi cresson's avatar
remi cresson committed
377
378
```
Then, we will select some samples with the **SampleSelection** application of the existing machine learning framework of OTB.
379
Since the terrain truth is sparse, we want to sample randomly points in polygons with the default strategy of the **SampleSelection** OTB application.
remi cresson's avatar
remi cresson committed
380
```
381
otbcli_SampleSelection -in spot7.tif -vec terrain_truth.shp -instats vec_stats.xml -field class -out points.shp
remi cresson's avatar
remi cresson committed
382
```
383
384
Now we extract the patches with the **PatchesExtraction** application. 
We want to produce one image of 16x16 patches, and one image for the corresponding labels.
remi cresson's avatar
remi cresson committed
385
```
386
otbcli_PatchesExtraction -source1.il spot7.tif -source1.patchsizex 16 -source1.patchsizey 16 -vec points.shp -field class -source1.out samp_labels.tif -outpatches samp_patches.tif
remi cresson's avatar
remi cresson committed
387
```
388
389
390
391
392
393
Now we have two images for patches and labels. 
We can split them to distinguish test/validation groups (with the **ExtractROI** application for instance).
But here, we will just perform some fine tuning of our model.
The **SavedModel** is located in the `outmodel` directory.
Our model is quite basic: it has two input placeholders, **x1** and **y1** respectively for input patches (with size 16x16) and input reference labels (with size 1x1).
We named **prediction** the tensor that predict the labels and the optimizer that perform the stochastic gradient descent is an operator named **optimizer**.
394
We perform the fine tuning and we export the new model variables directly in the _outmodel/variables_ folder, overwritting the existing variables of the model.
395
We use the **TensorflowModelTrain** application to perform the training of this existing model.
remi cresson's avatar
remi cresson committed
396
```
397
otbcli_TensorflowModelTrain -model.dir /path/to/oursavedmodel -training.targetnodesnames optimizer -training.source1.il samp_patches.tif -training.source1.patchsizex 16 -training.source1.patchsizey 16 -training.source1.placeholder x1 -training.source2.il samp_labels.tif -training.source2.patchsizex 1 -training.source2.patchsizey 1 -training.source2.placeholder y1 -model.saveto /path/to/oursavedmodel/variables/variables
remi cresson's avatar
remi cresson committed
398
399
400
```
Note that we could also have performed validation in this step. In this case, the `validation.source2.placeholder` would be different than the `training.source2.placeholder`, and would be **prediction**. This way, the program know what is the target tensor to evaluate. 

401
402
After this step, we use the trained model to produce the entire map of forest over the whole Spot-7 image.
For this, we use the **TensorflowModelServe** application to produce the **prediction** tensor output for the entire image.
remi cresson's avatar
remi cresson committed
403
```
404
otbcli_TensorflowModelServe -source1.il spot7.tif -source1.placeholder x1 -source1.rfieldx 16 -source1.rfieldy 16 -model.dir /path/to/oursavedmodel -output.names prediction -out map.tif uint8
remi cresson's avatar
remi cresson committed
405
```
406

Cresson Remi's avatar
Cresson Remi committed
407
## Begin with provided models
Cresson Remi's avatar
Cresson Remi committed
408

Cresson Remi's avatar
Cresson Remi committed
409
410
In the `python` folder are provided some [ready-to-use deep networks, with documentation and scientific references](doc/EXAMPLES.md).
**Feel free to contribute with your own architecture!**
Cresson Remi's avatar
Cresson Remi committed
411
412

## Tutorial
Cresson Remi's avatar
Cresson Remi committed
413

Cresson Remi's avatar
Cresson Remi committed
414
A tutorial is available at [MDL4EO's blog](https://mdl4eo.irstea.fr/2019/01/04/an-introduction-to-deep-learning-on-remote-sensing-images-tutorial/)