Merge branch 'master' of gitlab-ssh.irstea.fr:remi.cresson/otbtf into develop

README.md

@@ -10,27 +10,175 @@ It contains a set of new process objects that internally invoke [Tensorflow](htt
 
 
 # How to install
-This remote module has been tested successfully on Ubuntu 16.04 and CentOs 7 with latest CUDA drivers.
+This remote module has been tested successfully on Ubuntu 18 and CentOs 7 with CUDA drivers.
 
 ## Build OTB
 First, **build the latest *develop* branch of OTB from sources**. You can check the [OTB documentation](https://www.orfeo-toolbox.org/SoftwareGuide/SoftwareGuidech2.html) which details all the steps, if fact it is quite easy thank to the SuperBuild.
 
-## Build TensorFlow
-Then you have to **build Tensorflow from source** except if you want to use only the sampling applications of OTBTensorflow (in this case, skip this section).
-Follow [the instructions](https://www.tensorflow.org/install/install_sources) to build Tensorflow.
+Basically, you have to create a folder for OTB, clone sources, configure OTB SuperBuild, and build it.
+The following has been validated with an OTB 6.6.0.
+```
+sudo apt-get update
+sudo apt-get upgrade
+sudo apt-get install aptitude
+sudo aptitude install make cmake-curses-gui build-essential libtool automake git libbz2-dev python-dev libboost-dev libboost-filesystem-dev libboost-serialization-dev libboost-system-dev zlib1g-dev libcurl4-gnutls-dev swig
+sudo mkdir /work
+sudo chown $USER /work
+mkdir /work/otb
+cd /work/otb
+mkdir build
+git clone https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb.git OTB
+cd build
+ccmake /work/otb/OTB/SuperBuild
+make -j $(grep -c ^processor /proc/cpuinfo)
+```
+
+## Build TensorFlow with shared libraries
+During this step, you have to **build Tensorflow from source** except if you want to use only the sampling applications of OTBTensorflow (in this case, skip this section).
+The following has been validated with TensorFlow r1.12
+First, I advise you to use GCC 6 rather than 5 or 7 to compile TensorFlow from sources (I encountered several problem with other GCC versions).
+
+### Bazel
+First, install Bazel.
+```
+sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python
+wget https://github.com/bazelbuild/bazel/releases/download/0.20.0/bazel-0.20.0-installer-linux-x86_64.sh
+chmod +x bazel-0.20.0-installer-linux-x86_64.sh
+./bazel-0.20.0-installer-linux-x86_64.sh --user
+export PATH="$PATH:$HOME/bin"
+```
+
+If you fail to install properly Bazel, you can read the beginning of [the instructions](https://www.tensorflow.org/install/install_sources) that present alternative methods for this.
+
+### Required packages
+There is a few required packages that you need to install:
+```
+sudo apt install python-dev python-pip python3-dev python3-pip
+sudo pip install pip six numpy wheel mock keras
+sudo pip3 install pip six numpy wheel mock keras
+```
+
+### Build TensorFlow the right way
+Now, let's build TensorFlow with all the stuff required by OTBTF.
+Make a directory for TensorFlow.
+For instance `mkdir /work/tf`.
+
+Clone TensorFlow.
+```
+cd /work/tf
+git clone https://github.com/tensorflow/tensorflow.git
+```
+
+Now configure the project. If you have CUDA and other NVIDIA stuff installed in your system, remember that you have to tell the script that it is in `/usr/` (no symlink required!).
+
+```
+cd tensorflow
+./configure
+```
+Then, you have to build TensorFlow with the most important instructions sets of your CPU (For instance here is AVX, AVX2, FMA, SSE4.1, SSE4.2 that play fine on a modern intel CPU). You have to tell Bazel to build:
+
+ 1. The TensorFlow python pip package
+ 2. The libtensorflow_cc.so library
+ 3. The libtensorflow_framework.so library
+```
+bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 //tensorflow:libtensorflow_framework.so //tensorflow:libtensorflow_cc.so //tensorflow:libtensorflow.so //tensorflow/tools/pip_package:build_pip_package
+```
+### Prepare the right stuff to use TensorFlow in external (cmake) projects
+This is the most important!
+First, build and deploy the pip package.
+```
+bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
+pip install /tmp/tensorflow_pkg/tensorflow-1.12.0rc0-cp27-cp27mu-linux_x86_64.whl
+```
+For the C++ API, it's a bit more tricky.
+Let's begin.
+First, download dependencies.
+```
+/work/tf/tensorflow/tensorflow/contrib/makefile/download_dependencies.sh
+```
+Then, build Google Protobuf
+```
+mkdir /tmp/proto
+cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/protobuf/
+./autogen.sh
+./configure --prefix=/tmp/proto/
+make -j $(grep -c ^processor /proc/cpuinfo)
+make install
+```
+Then, "build" eigen (header only...)
+```
+mkdir /tmp/eigen
+cd ../eigen
+mkdir build_dir
+cd build_dir
+cmake -DCMAKE_INSTALL_PREFIX=/tmp/eigen/ ../
+make install -j $(grep -c ^processor /proc/cpuinfo)
+```
+Then, build NSync
+```
+mkdir /tmp/proto
+cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/protobuf/
+./autogen.sh
+./configure --prefix=/tmp/proto/
+make -j $(grep -c ^processor /proc/cpuinfo)
+make install
+```
+Then, build absl
+```
+mkdir /tmp/absl
+cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/absl/
+mkdir build_dir
+cd build_dir
+cmake -DCMAKE_INSTALL_PREFIX=/tmp/absl ../
+make -j $(grep -c ^processor /proc/cpuinfo)
+```
+Now, you have to copy the useful stuff in a directory
+
+```
+# Create folders
+mkdir /work/tf/installdir
+mkdir /work/tf/installdir/lib
+mkdir /work/tf/installdir/include
+
+# Copy libs
+cp /work/tf/tensorflow/bazel-bin/tensorflow/libtensorflow_cc.so /work/tf/installdir/lib/
+cp /work/tf/tensorflow/bazel-bin/tensorflow/libtensorflow_framework.so /work/tf/installdir/lib/
+cp /tmp/proto/lib/libprotobuf.a /work/tf/installdir/lib/
+cp /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/nsync/builds/default.linux.c++11/*.a /work/tf/installdir/lib/
+
+# Copy headers
+mkdir /work/tf/installdir/include/tensorflow
+cp -r /work/tf/tensorflow/bazel-genfiles/* /work/tf/installdir/include
+cp -r /work/tf/tensorflow/tensorflow/cc /work/tf/installdir/include/tensorflow
+cp -r /work/tf/tensorflow/tensorflow/core /work/tf/installdir/include/tensorflow
+cp -r /work/tf/tensorflow/third_party /work/tf/installdir/include
+cp -r /tmp/proto/include/* /work/tf/installdir/include
+cp -r /tmp/eigen/include/eigen3/* /work/tf/installdir/include
+cp /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/nsync/public/* /work/tf/installdir/include/
+find /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/absl/absl/ -name '*.h' -exec cp --parents \{\} /work/tf/installdir/include/ \;
+
+# Cleaning
+find /work/tf/installdir/ -name "*.cc" -type f -delete
+```
+Well done. Now you have a working copy of TensorFlow located in `/work/tf/installdir` that is ready to use in external C++ cmake projects :)
 
 ## Build this remote module
 Finally, we can build this module.
-Clone the repository in your the OTB sources directory for remote modules (something like `otb/Modules/Remote/`).
+Clone the repository in your the OTB sources directory for remote modules (something like `/work/otb/OTB/Modules/Remote/`).
 Re configure OTB with cmake of ccmake, and set the following variables
 
  - **Module_OTBTensorflow** to **ON**
  - **OTB_USE_TENSORFLOW** to **ON** (if you set to OFF, you will have only the sampling applications)
- - **TENSORFLOW_CC_LIB** to `/path/to/lib/libtensorflow_cc.so`
- - **TENSORFLOW_FRAMEWORK_LIB** to `/path/to/lib/libtensorflow_framework.so`
- - **tensorflow_include_dir** to `/path/to/include`
+ - **TENSORFLOW_CC_LIB** to `/work/tf/installdir/lib/libtensorflow_cc.so`
+ - **TENSORFLOW_FRAMEWORK_LIB** to `/work/tf/installdir/lib/libtensorflow_framework.so`
+ - **tensorflow_include_dir** to `/work/tf/installdir/include`
 
-Re build and install OTB.
+Re build and re install OTB.
+```
+cd /work/otb/build/OTB/build
+ccmake
+make -j $(grep -c ^processor /proc/cpuinfo)
+```
 Done !
 
 # New applications
@@ -343,8 +491,10 @@ Then, we use the **TensorflowModelServe** application to produce the **predictio
 ```
 otbcli_TensorflowModelServe -source1.il spot7.tif -source1.placeholder x1 -source1.rfieldx 16 -source1.rfieldy 16 -model.dir /tmp/my_new_model -output.names prediction -out map.tif uint8
 ```
-
+# Tutorial
+A complete tutorial is available at [MDL4EO's blog](https://mdl4eo.irstea.fr/2019/01/04/an-introduction-to-deep-learning-on-remote-sensing-images-tutorial/)
 # Contact
 You can contact Rémi Cresson if you have any issues with this remote module at remi [dot] cresson [at] irstea [dot] fr
 
 
+