diff --git a/README.md b/README.md index 051e86f13cb85172694383d2ff82423ae96376e9..e2d95fd57240014b321c12810e842a4d482988fd 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,24 @@ -#  OTBTF +#  OTBTF: Orfeo ToolBox meets TensorFlow [](https://opensource.org/licenses/Apache-2.0) -## Orfeo ToolBox meets TensorFlow - This remote module of the [Orfeo ToolBox](https://www.orfeo-toolbox.org) provides a generic, multi purpose deep learning framework, targeting remote sensing images processing. It contains a set of new process objects that internally invoke [Tensorflow](https://www.tensorflow.org/), and a bunch of user-oriented applications to perform deep learning with real-world remote sensing images. Applications can be used to build OTB pipelines from Python or C++ APIs. -### Highlights - - Sampling, - - Training, supporting save/restore/import operations (a model can be trained from scratch or fine-tuned), - - Serving models with support of OTB streaming mechanism. Meaning (1) not limited by images sizes, (2) can be used as a "lego" in any OTB pipeline and preserve streaming, (3) MPI support available (use multiple processing unit to generate one single output image) +## Features + +### OTB Applications + +- Sample patches in remote sensing images with `PatchesExtraction`, +- Model training, supporting save/restore/import operations (a model can be trained from scratch or fine-tuned) with `TensorflowModelTrain`, +- Inference with support of OTB streaming mechanism with `TensorflowModelServe`. The streaming mechanism means (1) no limitation with images sizes, (2) inference can be used as a "lego" in any OTB pipeline (using C++ or Python APIs) and preserving streaming, (3) MPI support available (use multiple processing unit to generate one single output image) + +### Python API + +This is a work in progress. For now, `tricks.py` provides a set of helpers to build deep nets, and `otbtf.py` provides datasets which can be used in Tensorflow pipelines to train networks from python. -### Portfolio +## Portfolio Below are some screen captures of deep learning applications performed at large scale with OTBTF. - Image to image translation (Spot-7 image --> Wikimedia Map using CGAN) @@ -29,9 +34,9 @@ You can read more details about these applications on [this blog](https://mdl4eo ## How to install -For now you have two options: either use the existing **docker image**, or build everything yourself **from source**. +For now you have two options: either use the existing **docker image**, or build everything **from source**. -### Docker image +### Docker Use the latest image from dockerhub: ``` @@ -39,35 +44,15 @@ docker pull mdl4eo/otbtf2.4:cpu docker run -u otbuser -v $(pwd):/home/otbuser mdl4eo/otbtf2.4:cpu otbcli_PatchesExtraction -help ``` -Available docker images: - -| Name | Os | TF | OTB | Description | -| --------------------------- | ------------- | ------ | ----- | ---------------------- | -| **mdl4eo/otbtf1.6:cpu** | Ubuntu Xenial | r1.14 | 7.0.0 | CPU, no optimization | -| **mdl4eo/otbtf1.7:cpu** | Ubuntu Xenial | r1.14 | 7.0.0 | CPU, no optimization | -| **mdl4eo/otbtf1.7:gpu** | Ubuntu Xenial | r1.14 | 7.0.0 | GPU | -| **mdl4eo/otbtf2.0:cpu** | Ubuntu Xenial | r2.1 | 7.1.0 | CPU, no optimization | -| **mdl4eo/otbtf2.0:gpu** | Ubuntu Xenial | r2.1 | 7.1.0 | GPU | -| **mdl4eo/otbtf2.4:cpu** | Ubuntu Focal | r2.4 | 7.2.0 | CPU, no optimization | -| **mdl4eo/otbtf2.4:cpu-mkl** | Ubuntu Focal | r2.4 | 7.2.0 | CPU, with Intel MKL | -| **mdl4eo/otbtf2.4:gpu** | Ubuntu Focal | r2.4 | 7.2.0 | GPU | - -(You can also find plenty of OTBTF flavored images [here](https://gitlab.com/latelescop/docker/otbtf/container_registry/)). - -All GPU docker images are suited for **NVIDIA GPUs**. -They use CUDA/CUDNN support and are built with compute capabilities 5.2, 6.1, 7.0, 7.5. -To change the compute capabilities, you can build your own docker image using the provided dockerfile. See the [docker build documentation} (tools/dockerfiles). -You can find more details on the **GPU docker image** and some **docker tips and tricks** on [this blog](https://mdl4eo.irstea.fr/2019/10/15/otbtf-docker-image-with-gpu/). -Also you can check [this document](https://gitlab.irstea.fr/raffaele.gaetano/moringa/-/tree/develop/docker) that also mentions useful stuff. - +Read more in the [docker use documentation](doc/DOCKERUSE.md). ### Build from sources -See [here](doc/HOWTOBUILD.md) to see how to build the remote module from sources. +Read more in the [build from sources documentation](doc/HOWTOBUILD.md). -## How to use it? +## How to use -- Reading [the documentation](doc/APPLICATIONS.md) will help, of course 😉 +- Reading [the applications documentation](doc/APPLICATIONS.md) will help, of course 😉 - A small [tutorial](https://mdl4eo.irstea.fr/2019/01/04/an-introduction-to-deep-learning-on-remote-sensing-images-tutorial/) on MDL4EO's blog - in the `python` folder are provided some [ready-to-use deep networks, with documentation and scientific references](doc/EXAMPLES.md). - A book: *Cresson, R. (2020). Deep Learning for Remote Sensing Images with Open Source Software. CRC Press.* Use QGIS, OTB and Tensorflow to perform various kind of deep learning sorcery on remote sensing images (patch-based classification for landcover mapping, semantic segmentation of buildings, optical image restoration from joint SAR/Optical time series). diff --git a/doc/APPLICATIONS.md b/doc/APPLICATIONS.md index 644fea77bc63570b0c22c3911600331a7f07e63b..69282cd543576f66a790f9619f6f75b31b4e1514 100644 --- a/doc/APPLICATIONS.md +++ b/doc/APPLICATIONS.md @@ -1,10 +1,13 @@ +# Description of applications -## PatchesExtraction +This section introduces the new OTB applications provided in OTBTF. -This application performs the extraction of patches in images from a vector data containing points. -The OTB sampling framework can be used to generate the set of selected points. -After that, you can use the **PatchesExtraction** application to perform the sampling of your images. -We denote _input source_ an input image, or a stack of input images or channels (which will be concatenated: they must have the same size). +## Patches extraction + +The `PatchesExtraction` application performs the extraction of patches in images from a vector data containing points. +Each point locates the **center** of the **central pixel** of the patch. +For patches with even size of *N*, the **central pixel** corresponds to the pixel index *N/2+1* (index starting at 0). +We denote one _input source_, either an input image, or a stack of input images that will be concatenated (they must have the same size). The user can set the `OTB_TF_NSOURCES` environment variable to select the number of _input sources_ that he wants. For example, for sampling a Time Series (TS) together with a single Very High Resolution image (VHR), two sources are required: - 1 input images list for time series, @@ -321,9 +324,11 @@ None Note that you can still set the `OTB_TF_NSOURCES` environment variable. -# How to use +# Basic example + +Below is a minimal example that presents the main steps to train a model, and perform the inference. -## The basics +## Sampling Here we will try to provide a simple example of doing a classification using a deep net that performs on one single VHR image. Our data set consists in one Spot-7 image, *spot7.tif*, and a training vector data, *terrain_truth.shp* that describes sparsely forest / non-forest polygons. @@ -342,6 +347,9 @@ We want to produce one image of 16x16 patches, and one image for the correspondi ``` otbcli_PatchesExtraction -source1.il spot7.tif -source1.patchsizex 16 -source1.patchsizey 16 -vec points.shp -field class -source1.out samp_labels.tif -outpatches samp_patches.tif ``` + +## Training + Now we have two images for patches and labels. We can split them to distinguish test/validation groups (with the **ExtractROI** application for instance). But here, we will just perform some fine tuning of our model. @@ -355,6 +363,8 @@ otbcli_TensorflowModelTrain -model.dir /path/to/oursavedmodel -training.targetno ``` Note that we could also have performed validation in this step. In this case, the `validation.source2.placeholder` would be different than the `training.source2.placeholder`, and would be **prediction**. This way, the program know what is the target tensor to evaluate. +## Inference + After this step, we use the trained model to produce the entire map of forest over the whole Spot-7 image. For this, we use the **TensorflowModelServe** application to produce the **prediction** tensor output for the entire image. ``` diff --git a/doc/DOCKERUSE.md b/doc/DOCKERUSE.md new file mode 100644 index 0000000000000000000000000000000000000000..2cdebe8cacb95978e09bdf57cb326f26ed39d43d --- /dev/null +++ b/doc/DOCKERUSE.md @@ -0,0 +1,230 @@ +# OTBTF docker images overview + +### Available images + +Here is the list of OTBTF docker images hosted on [dockerhub](https://hub.docker.com/u/mdl4eo). + +| Name | Os | TF | OTB | Description | +| --------------------------- | ------------- | ------ | ----- | ---------------------- | +| **mdl4eo/otbtf1.6:cpu** | Ubuntu Xenial | r1.14 | 7.0.0 | CPU, no optimization | +| **mdl4eo/otbtf1.7:cpu** | Ubuntu Xenial | r1.14 | 7.0.0 | CPU, no optimization | +| **mdl4eo/otbtf1.7:gpu** | Ubuntu Xenial | r1.14 | 7.0.0 | GPU | +| **mdl4eo/otbtf2.0:cpu** | Ubuntu Xenial | r2.1 | 7.1.0 | CPU, no optimization | +| **mdl4eo/otbtf2.0:gpu** | Ubuntu Xenial | r2.1 | 7.1.0 | GPU | +| **mdl4eo/otbtf2.4:cpu** | Ubuntu Focal | r2.4.1 | 7.2.0 | CPU, no optimization | +| **mdl4eo/otbtf2.4:cpu-mkl** | Ubuntu Focal | r2.4.1 | 7.2.0 | CPU, Intel MKL, AVX512 | +| **mdl4eo/otbtf2.4:gpu** | Ubuntu Focal | r2.4.1 | 7.2.0 | GPU | + +- `cpu` tagged docker images are compiled without optimization. +- `gpu` tagged docker images are suited for **NVIDIA GPUs**. They use CUDA/CUDNN support and are built with compute capabilities 5.2, 6.1, 7.0, 7.5. +- `cpu-mkl` tagged docker image is experimental, it is optimized for Intel CPUs with AVX512 flags. + +You can also find plenty of interesting OTBTF flavored images at [LaTelescop gitlab registry](https://gitlab.com/latelescop/docker/otbtf/container_registry/). + +### Development ready images + +Until r2.4, all images are development-ready. For instance, you can recompile the whole OTB from `/work/otb/build/OTB/build`. +Since r2.4, only `gpu` tagged image is development-ready, and you can recompile OTB from `/src/otb/build/OTB/build`. + +### Build your own images + +If you want to use optimization flags, change GPUs compute capability, etc. you can build your own docker image using the provided dockerfile. +See the [docker build documentation](tools/dockerfiles). + +# Mounting file systems + +You can mount filesystem in the docker image. +For instance, suppose you have some data in `/mnt/my_device/` that you want to use inside the container: + +The following command shows you how to access the folder from the docker image. + +```bash +docker run -v /mnt/my_device/:/data/ -ti mdl4eo/otbtf2.4:cpu bash -c "ls /data" +``` +Beware of ownership issues! see the last section of this doc. + +# GPU enabled docker + +In Linux, this is quite straightforward. +Just follow the steps described in the [nvidia-docker documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). +You can then use the OTBTF `gpu` tagged docker images with the **NVIDIA runtime**: + +```bash +docker run --runtime=nvidia -ti mdl4eo/otbtf2.4:gpu bash +``` + +You can find some details on the **GPU docker image** and some **docker tips and tricks** on [this blog](https://mdl4eo.irstea.fr/2019/10/15/otbtf-docker-image-with-gpu/). +Be careful though, these infos might be a bit outdated... + +# Docker Installation + +### Installation and first steps on Windows 10 + +1. Install [WSL2](https://docs.microsoft.com/en-us/windows/wsl/install-win10#manual-installation-steps) (Windows Subsystem for Linux) +2. Install [docker desktop](https://www.docker.com/products/docker-desktop) +3. Start **docker desktop** and **enable WSL2** from *Settings* > *General* then tick the box *Use the WSL2 based engine* +3. Open a **cmd.exe** or **PowerShell** terminal, and type `docker create --name otbtf-cpu --interactive --tty mdl4eo/otbtf2.4:cpu` +4. Open **docker desktop**, and check that the docker is running in the **Container/Apps** menu + +5. From **docker desktop**, click on the icon highlighted as shown below, and use the bash terminal that should pop up! + + +Troubleshooting: +- [Docker for windows WSL documentation](https://docs.docker.com/docker-for-windows/wsl) +- [WSL2 installation steps](https://docs.microsoft.com/en-us/windows/wsl/install-win10) + +### Use the GPU with Windows 10 + WSL2 + +*Work in progress* + +Some users have reported to use OTBTF with GPU in windows 10 using WSL2. +How to install WSL2 with Cuda on windows 10: +https://docs.nvidia.com/cuda/wsl-user-guide/index.html +https://docs.docker.com/docker-for-windows/wsl/#gpu-support + + +### Debian and Ubuntu + +See here how to install docker on Ubuntu [here](https://docs.docker.com/engine/install/ubuntu/). + +# Docker Usage + +This section is largely inspired from the [moringa docker help](https://gitlab.irstea.fr/raffaele.gaetano/moringa/-/raw/develop/docker/README.md). Big thanks to them. + +## Useful diagnostic commands + +Here are some usefull commands. + +```bash +docker info # System info +docker images # List local images +docker container ls # List containers +docker ps # Show running containers +``` + +On Linux, control state with systemd: +```bash +sudo systemctl {status,enable,disable,start,stop} docker +``` + +### Run some commands + +Run a simple command in a one-shot container: + +```bash +docker run mdl4eo/otbtf2.4:cpu otbcli_PatchesExtraction +``` + +You can also use the image in interactive mode with bash: +```bash +docker run -ti mdl4eo/otbtf2.4:cpu bash +``` + +### Persistent container + +Persistent (named) container with volume, here with home dir, but it can be any directory. +Beware of ownership issues, see the last section of this doc. + +```bash +docker create --interactive --tty --volume /home/$USER:/home/otbuser/ \ + --name otbtf mdl4eo/otbtf2.4:cpu /bin/bash +``` + +### Interactive session + +```bash +docker start -i otbtf +``` + +### Background container + +```bash +docker start otbtf +docker exec otbtf ls -alh +docker stop otbtf +``` + +### Running commands with root user + +Background container is the easiest way: + +```bash +docker start otbtf +# Example with apt update (you can't use &&, one docker exec is required for each command) +docker exec --user root otbtf apt-get update +docker exec --user root otbtf apt-get upgrade -y +``` + +### Container-specific commands, especially for background containers: + +```bash +docker inspect otbtf # See full container info dump +docker logs otbtf # See command logs and outputs +docker stats otbtf # Real time container statistics +docker {pause,unpause} otbtf # Freeze container +``` + +### Stop a background container + +Don't forget to stop the container after you have done. + +```bash +docker stop otbtf +``` + +### Remove a persistent container + +```bash +docker rm otbtf +``` + +# Fix volume ownership issue (required if host's UID > 1000) + +When mounting a volume, you may experience errors while trying to write files from within the container. +Since the default user (**otbuser**) is UID 1000, you won't be able to write files into your volume +which is mounted with the same UID than your linux host user (may be UID 1001 or more). +In order to address this, you need to edit the container's user UID and GID to match the right numerical value. +This will only persist in a named container, it is required every time you're creating a new one. + + +Create a named container (here with your HOME as volume), Docker will automatically pull image + +```bash +docker create --interactive --tty --volume /home/$USER:/home/otbuser \ + --name otbtf mdl4eo/otbtf2.4:cpu /bin/bash +``` + +Start a background container process: + +```bash +docker start otbtf +``` + +Exec required commands with user root (here with host's ID, replace $UID and $GID with desired values): + +```bash +docker exec --user root otbtf usermod otbuser -u $UID +docker exec --user root otbtf groupmod otbuser -g $GID +``` + +Force reset ownership with updated UID and GID. +Make sure to double check that `docker exec otbtf id` because recursive chown will apply to your volume in `/home/otbuser` + +```bash +docker exec --user root otbtf chown -R otbuser:otbuser /home/otbuser +``` + +Stop the background container and start a new interactive shell: + +```bash +docker stop otbtf +docker start -i otbtf +``` + +Check if ownership is right + +```bash +id +ls -Alh /home/otbuser +touch /home/otbuser/test.txt +``` diff --git a/doc/images/docker_desktop_1.jpeg b/doc/images/docker_desktop_1.jpeg new file mode 100644 index 0000000000000000000000000000000000000000..9a03bd58cae40dfb428de083531c687020572883 Binary files /dev/null and b/doc/images/docker_desktop_1.jpeg differ diff --git a/doc/images/docker_desktop_2.jpeg b/doc/images/docker_desktop_2.jpeg new file mode 100644 index 0000000000000000000000000000000000000000..e393cdb40d45c1fbcfd69c8983a47aef8731b899 Binary files /dev/null and b/doc/images/docker_desktop_2.jpeg differ