An error occurred while loading the file. Please try again.
-
Remi Cresson authoredcc666abd
<!doctype html><html><head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> <meta name="apple-mobile-web-app-capable" content="yes"> <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> <title>OTBTF, The Orfeo ToolBox extension for deep learning</title> <link rel="stylesheet" href="revealjs/dist/reset.css"> <link rel="stylesheet" href="revealjs/dist/reveal.css"> <link rel="stylesheet" href="a11y-light.css"> <link rel="stylesheet" href="otb.css" id="theme"> <style type="text/css"> .reveal pre { width: 512px } .slide-number { opacity:0 } </style></head><body> <div class="reveal"> <div class="slides"> <!------------------------------------------------------------------------ PREMIER SLIDE -------------------------------------------------------------------------> <section data-background="illustrations/blank.png" background-size="contain"> <h1> Status of OTBTF </h1> <h2> The Orfeo ToolBox extension for deep learning </h2> </br> <p> Rémi Cresson<sup>1</sup>, Nicolas Narçon<sup>1</sup>, Vincent Delbar<sup>2</sup></p> <small>(1) French National Research Institute for Agriculture, Food and the Environment (INRAE), <br> (2) LaTeleScop</small> <br> <br> <br> <br> <img width="30%" data-src="illustrations/foss4g_logo.png"> </section> <!------------------------------------------------------------------------ WHAT IS OTBTF -------------------------------------------------------------------------> <section> <section> <h1>What is OTBTF?</h1> </section> <section> <h2>In short</h2> <ul> <li><h>Generic</h> framework for deep learning on rasters</li> <li>Developped at INRAE for <h>research</h>, <h>education</h> and <h>production</h> </li> <li>Use <h>deep learning</h> techniques on geospatial images</li> <ul> <li>Create datasets (samples selection, patches extraction)</li> <li>Train models (CLI, Python)</li> <li>Apply models in OTB applications</li> </ul> </ul> <br>7172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140
<br>
<img width="15%" data-src="illustrations/logo.png">
</section>
<section>
<h2>Repository and docker images</h2>
<img width="5%" data-src="illustrations/repos_git.jpg" style="float:left;padding-left:20%">
<pre style="width:800px"><code data-trim class="bash">
git clone https://github.com/remicres/otbtf.git
</pre></code>
<img width="4%" data-src="illustrations/repos_docker.jpg" style="float:left;padding-left:20%">
<pre style="width:800px"><code data-trim class="bash" >
docker pull mdl4eo/otbtf:3.3.0-cpu
docker pull mdl4eo/otbtf:3.3.0-gpu # GPU enabled
</pre></code>
</section>
<section>
<img width="40%" data-src="illustrations/ensembles.png">
</section>
<section>
<h4>TensorFlow graphs</h4>
<img width="17.5%" data-src="illustrations/computational_graph.gif">
<p><small>Source: the TensorFlow website</small></p>
</section>
<section>
<h4>Example: scalar product</h4>
<br>
<img width="20%" data-src="illustrations/graph_1.png">
<br>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:15%;margin:30px">
<pre style="width:1050px"><code data-trim class="python">
import tensorflow as tf
x1 = tf.keras.Input(shape=[None, None, None], name="x1")
x2 = tf.keras.Input(shape=[None, None, None], name="x2")
# Scalar product
y = tf.reduce_sum(tf.multiply(x1, x2), axis=-1)
# Create model
model = tf.keras.Model(inputs={"x1": x1, "x2": x2}, outputs={"y": y})
model.save("/tmp/my_savedmodel")
</code></pre>
<img width="48px" data-src="illustrations/cli.png" style="float:left;padding-left:15%;margin:30px">
<pre style="width:1050px"><code data-trim class="bash">
export OTB_TF_NSOURCES=2
otbcli_TensorflowModelServe \
-source1.il "input_img_1.tif" -source2.il "input_img_2.tif" \
-model.dir "/tmp/my_savedmodel" -model.fullyconv on \
-out "output.tif"
</code></pre>
</section>
</section>
<!------------------------------------------------------------------------
FEATURES
------------------------------------------------------------------------->
<section>
<section>
<h1>Features</h1>
141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210
</section>
<section>
<h2>Out of the box</h2>
<ul>
<li>
Additional <h>applications</h> for the Orfeo ToolBox
<ul>
<li>For users (GIS, teaching)</li>
<li>For production</li>
</ul>
</li>
<li>
<h>Python API</h> (<y>NEW!</y>)
<ul>
<li>Dedicated to model training</li>
<li>For data scientists</li>
<li>To go large scale with distributed training</li>
</ul>
</li>
</ul>
<br><br>
</section>
<section>
<h2>OTB Applications</h2>
<ul>
<li>
<h>TensorflowModelServe</h>: Inference on real world remote sensing products
</li>
<li>
<g>PatchesExtraction</g>: extract patches in images
</li>
<li>
<h>PatchesSelection</h>: for patches selection from rasters
</li>
<li>
<g>TrainClassifierFromDeepFeatures</g>: train traditionnal classifiers that use features from deep nets
</li>
<li>
<g>ImageClassifierFromDeepFeatures</g>: use traditionnal classifiers with features from deep nets
</li>
<li>
<g>LabelImageSampleSelection</g>: select patches from a label image
</li>
<li>
<g>DensePolygonClassStatistics</g>: fast terrain truth polygons statistics
</li>
<li>
<g>TensorflowModelTrain</g>: training/validation (educational purpose)
</li>
</ul>
<br><br>
</section>
<section>
<h3>TensorflowModelServe</h3>
<h4>Streamable inference: a key feature to go large scale</h4>
<img width="60%" data-src="illustrations/pipeline.png">
<p><small>Typical pipeline for inference in production</small></p>
</section>
<section>
<h3>Patches extraction</h3>
<h4>CLI example</h4>
<img width="48px" data-src="illustrations/cli.png" style="float:left;padding-left:22%;margin:30px">
<pre style="width:800px"><code data-trim class="bash">
export OTB_TF_NSOURCES=3 # Number of sources
211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280
otbcli_PatchesExtraction -vec "myvec.gpkg" \
-source1.patchsizex 16 -source1.patchsizey 16 \
-source1.il "raster_x.tif" -source1.out "x1.tif" \
-source2.patchsizex 64 -source2.patchsizey 64 \
-source2.il "raster_y.tif" -source2.out "y1.tif" \
-source3.patchsizex 64 -source3.patchsizey 64 \
-source3.il "raster_z.tif" -source3.out "z1.tif"
</code></pre>
<img width="60%" data-src="illustrations/patches2.png">
<p><small>Patches are stacked in rows and stored in well known raster formats</small></p>
</section>
<section>
<h2>Python API</h2>
<p>Say that you have generated some patches images with PatchesExtraction:</p>
<img width="40%" data-src="illustrations/patches2.png">
<br>
<p>The <g>otbtf.DatasetFromPatchesImages</g> creates a dataset ready to use in TF/Keras:</p>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:20%;margin:30px">
<pre style="width:900px"><code data-trim class="python">
import otbtf
files = {"x": ["x1.tif", ..., "xN.tif"],
"y": ["y1.tif", ..., "yN.tif"],
"z": ["z1.tif", ..., "zN.tif"]}
ds = otbtf.DatasetFromPatchesImages(filenames_dict=files)
# This is a TensorFlow dataset
tf_ds = ds.get_tf_dataset(batch_size=8)
</code></pre>
</section>
<section>
<h3>TFRecords</h3>
<p>Any <g>otbtf.Dataset</g> can be exported in the <g>TFRecords</g> format:</p>
<pre style="width:800px"><code data-trim class="python">
# Create .records files in the directory
ds.to_tfrecords("/path/to/tfrecords_dir")
</code></pre>
<p>Which is convenient in distributed training setup, and easy to use in TF/Keras:</p>
<pre style="width:800px"><code data-trim class="python">
# This is a TensorFlow dataset!
tf_ds = TFRecords("/path/to/tfrecords_dir").read()
</code></pre>
<img width="40%" data-src="illustrations/jean-zay.png">
<p><small>TFRecords guarantee optimal performances on HPC architectures like <h>clusters</h> or <h>clouds</h>.
<br>
OTBTF was extensively tested on the <h>Jean-Zay</h> supercomputer (french institute for development and resources in intensive scientific computing).</small></p>
</section>
<section data-transition="fade">
<h3>ModelBase</h3>
<h4><y>New</y> in v3.3.0!</h4>
<br>
<ul>
<li>Ease the implementation of deep nets</li>
<li>Provides all the necessary to work smoothly with TensorflowModelServe</li>
</ul>
<br>
<img width="50%" data-src="illustrations/modelbase.png">
</section>
<section data-transition="fade">
<h4>Model implementation example</h4>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:18%;margin:30px">
<pre style="width:1000px"><code data-trim class="python">
from otbtf import ModelBase
from tf.keras.layers import Conv2D, Softmax
from functools import partial
281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350
class MyModel(ModelBase):
def normalize_inputs(self, inputs):
return {"x": something(inputs["x"])}
def get_outputs(self, normalized_inputs):
conv = partial(Conv2D, kernel_size=3, activation="relu")
out1 = conv(filters=32)(normalized_inputs["x"])
out2 = conv(filters=32)(out1)
out3 = conv(filters=5)(out2)
return {"y": Softmax(name="y_out")(out3)}
</code></pre>
<img width="50%" data-src="illustrations/modelbase_1.png">
</section>
<section data-transition="fade">
<h4>Dataset preparation example</h4>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:18%;margin:30px">
<pre style="width:1000px"><code data-trim class="python">
def dataset_preprocessing_fn(examples):
# Cast UInt8 to Int32
y_patches = tf.cast(examples["y_patches"], tf.int32)
# One hot encoding
y_onehot = tf.one_hot(tf.squeeze(y_patches, axis=-1), depth=5)
# Return dict of preprocessed inputs
return {"x": examples["x_patches"], "y": y_onehot}
</code></pre>
<img width="50%" data-src="illustrations/modelbase_2.png">
</section>
<section data-transition="fade">
<h4>Training</h4>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:18%;margin:30px">
<pre style="width:1000px"><code data-trim class="python">
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
model = MyModel(dataset_element_spec=ds_train.element_spec)
model.compile(loss=tf.keras.losses.CategoricalCrossentropy(),
optimizer=tf.keras.optimizers.Adam(),
metrics=[tf.keras.metrics.Precision(),
tf.keras.metrics.Recall()])
# Train
model.fit(ds_train, epochs=100, validation_data=ds_valid)
# Save
model.save("/tmp/model")
</code></pre>
<img width="50%" data-src="illustrations/modelbase_3.png">
</section>
<section data-transition="fade">
<h4>Inference (CLI)</h4>
<img width="48px" data-src="illustrations/cli.png" style="float:left;padding-left:18%;margin:30px">
<pre style="width:1000px"><code data-trim class="bash">
otbcli_TensorflowModelServe
-source1.il "input_x.tif"
-source1.rfieldx 7 -source1.rfieldy 7 # receptive field
-source1.placeholder "x" # input name
-model.dir "/tmp/model"
-model.fullyconv on # fully convolutional
-model.outputs "y_out" # output node name
-out "output_y.tif"
</code></pre>
<img width="50%" data-src="illustrations/modelbase_4.png">
351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420
</section>
<section data-transition="fade">
<h4>Inference (python)</h4>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:18%;margin:30px">
<pre style="width:1000px"><code data-trim class="python">
import pyotb
params = {"source1.il": "input_x.tif",
"source1.rfieldx": 7,
"source1.rfieldy": 7,
"model.dir": "/tmp/model",
"model.fullyconv": "on",
"model.outputs": "y_out"}
app = pyotb.TensorflowModelServe(params)
app.write("output_y.tif")
</code></pre>
<img width="50%" data-src="illustrations/modelbase_4.png">
</section>
<section data-transition="fade">
<h4>Default outputs postprocessing</h4>
<img width="50%" data-src="illustrations/fig12.10_new.png">
<br>
<p><h>Example</h>: simple U-Net like model for dense pixel classification</p>
</section>
</section>
<!------------------------------------------------------------------------
OTB EXAMPLES (DEEP LEARNING)
------------------------------------------------------------------------->
<section>
<section>
<h1>Examples</h1>
</section>
<section>
<h3>Rocks mapping</h3>
<img width="40%" data-src="illustrations/telescop_cailloux.jpg">
<p><small>Source:
https://www.soslrc.com/2020/05/15/82-scientifiques-franc-comtois-signent-contre-le-casse-cailloux</small>
</p>
</section>
<section>
<img width="80%" data-src="illustrations/telescop_cap1.jpg">
<p><small>Copyright <h>LaTeleScop</h> and <h>Damien MARAGE</h> (DREAL Bourgogne-Franche-Comté)</small></p>
</section>
<section>
<h3>Large scale land cover mapping</h3>
<h4>Semantic segmentation of buildings footprint over france mainland at 1.5m spacing</h4>
<img width="65%" data-src="illustrations/tosca.png">
<p><a href="https://www.theia-land.fr/en/product/buildings-footprint"
target="_blank">https://www.theia-land.fr/en/product/buildings-footprint/</a></p>
</section>
<section data-background-image='illustrations/gif_2160.gif'></section>
<section>
<img width="60%" data-src="illustrations/masque_ssthrs.png">
<p><small><h>Forest mapping</h> at 1.5m spacing from ~1200 spot 6/7 images acquired between 2016 and 2020. Copyright <h>INRAE</h>/<h>UMR TETIS</h>, <h>OSM</h></small></p>
</section>
<section>
<h3>Super-resolution</h3>
<h>https://github.com/remicres/sr4rs</h>
421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490
<br>
<img width="40%" data-src="illustrations/sr4rs_logos.jpg">
</section>
<section data-background-image='illustrations/sr4rs_tours.jpg'></section>
<section data-background-image='illustrations/sr4rs_ams.jpg'></section>
<section data-background-image='illustrations/sr4rs_lisboa.jpg'></section>
<section data-background-image='illustrations/sr4rs_okla.jpg'></section>
<section data-background-image='illustrations/sr4rs_baotou.jpg'></section>
<section>
<h4>Easy to run</h4>
<pre style="width:1000px"><code data-trim class="bash">
# Download pre-trained model
wget https://tinyurl.com/sr4rsmodelv2
unzip sr4rsmodelv2
# Inference over a Sentinel-2 image (Bands 4, 3, 2, 8)
python sr4rs/code/sr.py \
--savedmodel sr4rs_sentinel2_bands4328_france2020_savedmodel \
--input S2_image_channels_4328_10m.tif \
--output S2_SR.tif
</code></pre>
</section>
<section>
<h4>Under the hood: Inference using OTB, fully streamable (OTB Python API)</h4>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:18%;margin:30px">
<pre style="width:1000px"><code data-trim class="python">
if __name__ == "__main__":
...
# The important stuff
infer = otbApplication.Registry.CreateApplication("TensorflowModelServe")
infer.SetParameterStringList("source1.il", [params.input])
infer.SetParameterInt("source1.rfieldx", rfield)
infer.SetParameterInt("source1.rfieldy", rfield)
infer.SetParameterString("source1.placeholder", constants.lr_input_name)
infer.SetParameterString("model.dir", params.savedmodel)
infer.SetParameterString("model.fullyconv", "on")
infer.SetParameterStringList("output.names", [ph])
infer.SetParameterInt("output.efieldx", efield)
infer.SetParameterInt("output.efieldy", efield)
infer.SetParameterFloat("output.spcscale", ratio)
infer.SetParameterInt("optim.tilesizex", efield)
infer.SetParameterInt("optim.tilesizey", efield)
infer.SetParameterInt("optim.disabletiling", 1)
infer.SetParameterString("out", out_fn)
infer.SetParameterOutputImagePixelType("out", encoding)
infer.ExecuteAndWriteOutput()
</code></pre>
</section>
<section>
<h4>Same thing written in <h>pyotb</h></h4>
<img width="48px" data-src="illustrations/python.png" style="float:left;padding-left:18%;margin:30px">
<pre style="width:1000px"><code data-trim class="python">
if __name__ == "__main__":
...
# Using pyotb
infer = pyotb.TensorflowModelServe({
"source1.il": [params.input],
"source1.rfieldx": rfield,
"source1.rfieldy": rfield,
"source1.placeholder": constants.lr_input_name,
"model.dir": params.savedmodel,
"model.fullyconv": "on",
"output.names": [ph],
491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560
"output.efieldx": efield,
"output.efieldy": efield,
"output.spcscale": ratio,
"optim.tilesizex": efield,
"optim.tilesizey": efield,
"optim.disabletiling": "on"})
# if you need to write:
infer.write("out", out_fn)
</code></pre>
<img width="20%" data-src="illustrations/pyotbadvert.png">
</section>
<section>
<h3>Soil moisture mapping (Theia product)</h3>
<h4>Approx. 20 Theia products/month. More than 6k available products around the globe.</h4>
<p>SAR backscattering model inversion using ANN from S1 and S2</p>
<h>https://gitlab.irstea.fr/loic.lozach/AgriSoilMoisture</h>
<br>
<img width="40%" data-src="illustrations/soil_moisture.jpg">
</section>
<section>
<h3>Cloud removal in optical images</h3>
<h>https://github.com/cnes/decloud</h>
<br>
<img width="40%" data-src="illustrations/tetiscneslogo.png">
</section>
<section>
<h4>From joint SAR/Optical images</h4>
<img width="60%" data-src="illustrations/decloud_meraner.gif">
<p><small>The deep net inputs <h>one SAR image</h> and <h>one cloudy optical image</h></small></p>
</section>
<section>
<img width="80%" data-src="illustrations/decloud_cap2.jpg">
<p><small>Left: <h>original</h> Sentinel-2 image. Right: <h>reconstructed</h></p></small>
</section>
<section>
<img width="80%" data-src="illustrations/decloud_cap1.jpg">
<p><small>Left: <h>original</h> Sentinel-2 image. Right: <h>reconstructed</h>
</p></small>
</section>
<section>
<h4>From joint SAR/Optical time series</h4>
<img width="40%" data-src="illustrations/anim_crga.gif">
<p><small>The deep net inputs <h>time series</h> of SAR and (potentially) cloudy optical images</small></p>
</section>
<section>
<img width="65%" data-src="illustrations/decloud_anim.gif">
<p><small>
<h>4 years</h> of reconstructed optical time series (Occitanie area)</p></small>
</section>
<section>
<h4>Read more</h4>
<br>
<ul>
<li><h>Blog</h>: https://mdl4eo.irstea.fr/2022/04/09/bye-bye-clouds/</li>
<li><h>Paper</h>: https://doi.org/10.5194/isprs-archives-XLIII-B3-2022-1317-2022</li>
<li><h>Code</h> https://github.com/cnes/decloud</li>
</ul>
<img width="50%" data-src="illustrations/crga_os2_unet_slide3.png">
<p><small>Cresson, R., Narçon, N., Gaetano, R., Dupuis, A., Tanguy, Y., May, S., and Commandré, B.: COMPARISON OF CONVOLUTIONAL NEURAL NETWORKS FOR CLOUDY OPTICAL IMAGES RECONSTRUCTION FROM SINGLE OR MULTITEMPORAL JOINT SAR AND OPTICAL IMAGES, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLIII-B3-2022, 1317–1326</small></p>
561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630
</section>
</section>
<section>
<section>
<h1>Conclusion</h1>
</section>
<section>
<h3>What you can do with OTBTF</h3>
<ul>
<li>Use the OTB applications to <h>create datasets</h> from vector/raster</li>
<li><h>Build models</h> in python</li>
<li><h>Train models </h>
<ul>
<li>Beginners: from CLI from patches images</li>
<li>Developers: from python, using <i>otbtf.dataset</i> and otbtf.TFRecords classes</li>
<li>Distributed training with Tensorflow 2</li>
</ul>
<li><h>Run models</h> in OTB pipelines</li>
</ul>
</section>
<section>
<h3>Future work</h3>
<ul>
<li>Improve <h>python API</h></li>
<li>More <h>examples and applications</h> out of the box</li>
<li>Lighter <h>docker</h> images</li>
<li>Fix OTB x TensorFlow compilation limitations</li>
<li>Integration in <h>pyotb</h></li>
</ul>
</section>
</section>
</div>
</div>
<script src="revealjs/dist/reveal.js"></script>
<script src="revealjs/plugin/notes/notes.js"></script>
<script src="revealjs/plugin/markdown/markdown.js"></script>
<script src="revealjs/plugin/highlight/highlight.js"></script>
<script>
// More info about initialization & config:
// - https://revealjs.com/initialization/
// - https://revealjs.com/config/
Reveal.initialize({
hash: true,
controlsLayout: 'edges',
// Visibility rule for backwards navigation arrows; "faded", "hidden"
// or "visible"
controlsBackArrows: 'visible',
slideNumber: true,
// Learn about plugins: https://revealjs.com/plugins/
plugins: [RevealMarkdown, RevealHighlight, RevealNotes],
// The "normal" size of the presentation, aspect ratio will
// be preserved when the presentation is scaled to fit different
// resolutions. Can be specified using percentage units.
width: 1920,
height: 1080,
631632633634635636637638639640641642643644645646647648
// Factor of the display size that should remain empty around
// the content
margin: 0.01,
// Bounds for smallest/largest possible scale to apply to content
minScale: 0.2,
maxScale: 2.0,
drawer: {
colors: ["#fa1e0e", "#8ac926", "#1982c4", "#ffca3a"], // (optional) list of colors avaiable (hex color codes)
color: "#FF0000", // (optional) color of a cursor, first color from `codes` is a default
pathSize: 2, // (optional) path size in px, default 4
}
});
</script>
</body>
</html>