diff --git a/README.md b/README.md index ed0c652d3f43ae2bef6d360d0d6801c78618f7ea..92272b59085bc394765d4675cfebebf6e0de4de8 100644 --- a/README.md +++ b/README.md @@ -10,27 +10,175 @@ It contains a set of new process objects that internally invoke [Tensorflow](htt # How to install -This remote module has been tested successfully on Ubuntu 16.04 and CentOs 7 with latest CUDA drivers. +This remote module has been tested successfully on Ubuntu 18 and CentOs 7 with CUDA drivers. ## Build OTB First, **build the latest *develop* branch of OTB from sources**. You can check the [OTB documentation](https://www.orfeo-toolbox.org/SoftwareGuide/SoftwareGuidech2.html) which details all the steps, if fact it is quite easy thank to the SuperBuild. -## Build TensorFlow -Then you have to **build Tensorflow from source** except if you want to use only the sampling applications of OTBTensorflow (in this case, skip this section). -Follow [the instructions](https://www.tensorflow.org/install/install_sources) to build Tensorflow. +Basically, you have to create a folder for OTB, clone sources, configure OTB SuperBuild, and build it. +The following has been validated with an OTB 6.6.0. +``` +sudo apt-get update +sudo apt-get upgrade +sudo apt-get install aptitude +sudo aptitude install make cmake-curses-gui build-essential libtool automake git libbz2-dev python-dev libboost-dev libboost-filesystem-dev libboost-serialization-dev libboost-system-dev zlib1g-dev libcurl4-gnutls-dev swig +sudo mkdir /work +sudo chown $USER /work +mkdir /work/otb +cd /work/otb +mkdir build +git clone https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb.git OTB +cd build +ccmake /work/otb/OTB/SuperBuild +make -j $(grep -c ^processor /proc/cpuinfo) +``` + +## Build TensorFlow with shared libraries +During this step, you have to **build Tensorflow from source** except if you want to use only the sampling applications of OTBTensorflow (in this case, skip this section). +The following has been validated with TensorFlow r1.12 +First, I advise you to use GCC 6 rather than 5 or 7 to compile TensorFlow from sources (I encountered several problem with other GCC versions). + +### Bazel +First, install Bazel. +``` +sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python +wget https://github.com/bazelbuild/bazel/releases/download/0.20.0/bazel-0.20.0-installer-linux-x86_64.sh +chmod +x bazel-0.20.0-installer-linux-x86_64.sh +./bazel-0.20.0-installer-linux-x86_64.sh --user +export PATH="$PATH:$HOME/bin" +``` + +If you fail to install properly Bazel, you can read the beginning of [the instructions](https://www.tensorflow.org/install/install_sources) that present alternative methods for this. + +### Required packages +There is a few required packages that you need to install: +``` +sudo apt install python-dev python-pip python3-dev python3-pip +sudo pip install pip six numpy wheel mock keras +sudo pip3 install pip six numpy wheel mock keras +``` + +### Build TensorFlow the right way +Now, let's build TensorFlow with all the stuff required by OTBTF. +Make a directory for TensorFlow. +For instance `mkdir /work/tf`. + +Clone TensorFlow. +``` +cd /work/tf +git clone https://github.com/tensorflow/tensorflow.git +``` + +Now configure the project. If you have CUDA and other NVIDIA stuff installed in your system, remember that you have to tell the script that it is in `/usr/` (no symlink required!). + +``` +cd tensorflow +./configure +``` +Then, you have to build TensorFlow with the most important instructions sets of your CPU (For instance here is AVX, AVX2, FMA, SSE4.1, SSE4.2 that play fine on a modern intel CPU). You have to tell Bazel to build: + + 1. The TensorFlow python pip package + 2. The libtensorflow_cc.so library + 3. The libtensorflow_framework.so library +``` +bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 //tensorflow:libtensorflow_framework.so //tensorflow:libtensorflow_cc.so //tensorflow:libtensorflow.so //tensorflow/tools/pip_package:build_pip_package +``` +### Prepare the right stuff to use TensorFlow in external (cmake) projects +This is the most important! +First, build and deploy the pip package. +``` +bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg +pip install /tmp/tensorflow_pkg/tensorflow-1.12.0rc0-cp27-cp27mu-linux_x86_64.whl +``` +For the C++ API, it's a bit more tricky. +Let's begin. +First, download dependencies. +``` +/work/tf/tensorflow/tensorflow/contrib/makefile/download_dependencies.sh +``` +Then, build Google Protobuf +``` +mkdir /tmp/proto +cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/protobuf/ +./autogen.sh +./configure --prefix=/tmp/proto/ +make -j $(grep -c ^processor /proc/cpuinfo) +make install +``` +Then, "build" eigen (header only...) +``` +mkdir /tmp/eigen +cd ../eigen +mkdir build_dir +cd build_dir +cmake -DCMAKE_INSTALL_PREFIX=/tmp/eigen/ ../ +make install -j $(grep -c ^processor /proc/cpuinfo) +``` +Then, build NSync +``` +mkdir /tmp/proto +cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/protobuf/ +./autogen.sh +./configure --prefix=/tmp/proto/ +make -j $(grep -c ^processor /proc/cpuinfo) +make install +``` +Then, build absl +``` +mkdir /tmp/absl +cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/absl/ +mkdir build_dir +cd build_dir +cmake -DCMAKE_INSTALL_PREFIX=/tmp/absl ../ +make -j $(grep -c ^processor /proc/cpuinfo) +``` +Now, you have to copy the useful stuff in a directory + +``` +# Create folders +mkdir /work/tf/installdir +mkdir /work/tf/installdir/lib +mkdir /work/tf/installdir/include + +# Copy libs +cp /work/tf/tensorflow/bazel-bin/tensorflow/libtensorflow_cc.so /work/tf/installdir/lib/ +cp /work/tf/tensorflow/bazel-bin/tensorflow/libtensorflow_framework.so /work/tf/installdir/lib/ +cp /tmp/proto/lib/libprotobuf.a /work/tf/installdir/lib/ +cp /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/nsync/builds/default.linux.c++11/*.a /work/tf/installdir/lib/ + +# Copy headers +mkdir /work/tf/installdir/include/tensorflow +cp -r /work/tf/tensorflow/bazel-genfiles/* /work/tf/installdir/include +cp -r /work/tf/tensorflow/tensorflow/cc /work/tf/installdir/include/tensorflow +cp -r /work/tf/tensorflow/tensorflow/core /work/tf/installdir/include/tensorflow +cp -r /work/tf/tensorflow/third_party /work/tf/installdir/include +cp -r /tmp/proto/include/* /work/tf/installdir/include +cp -r /tmp/eigen/include/eigen3/* /work/tf/installdir/include +cp /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/nsync/public/* /work/tf/installdir/include/ +find /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/absl/absl/ -name '*.h' -exec cp --parents \{\} /work/tf/installdir/include/ \; + +# Cleaning +find /work/tf/installdir/ -name "*.cc" -type f -delete +``` +Well done. Now you have a working copy of TensorFlow located in `/work/tf/installdir` that is ready to use in external C++ cmake projects :) ## Build this remote module Finally, we can build this module. -Clone the repository in your the OTB sources directory for remote modules (something like `otb/Modules/Remote/`). +Clone the repository in your the OTB sources directory for remote modules (something like `/work/otb/OTB/Modules/Remote/`). Re configure OTB with cmake of ccmake, and set the following variables - **Module_OTBTensorflow** to **ON** - **OTB_USE_TENSORFLOW** to **ON** (if you set to OFF, you will have only the sampling applications) - - **TENSORFLOW_CC_LIB** to `/path/to/lib/libtensorflow_cc.so` - - **TENSORFLOW_FRAMEWORK_LIB** to `/path/to/lib/libtensorflow_framework.so` - - **tensorflow_include_dir** to `/path/to/include` + - **TENSORFLOW_CC_LIB** to `/work/tf/installdir/lib/libtensorflow_cc.so` + - **TENSORFLOW_FRAMEWORK_LIB** to `/work/tf/installdir/lib/libtensorflow_framework.so` + - **tensorflow_include_dir** to `/work/tf/installdir/include` -Re build and install OTB. +Re build and re install OTB. +``` +cd /work/otb/build/OTB/build +ccmake +make -j $(grep -c ^processor /proc/cpuinfo) +``` Done ! # New applications @@ -343,8 +491,10 @@ Then, we use the **TensorflowModelServe** application to produce the **predictio ``` otbcli_TensorflowModelServe -source1.il spot7.tif -source1.placeholder x1 -source1.rfieldx 16 -source1.rfieldy 16 -model.dir /tmp/my_new_model -output.names prediction -out map.tif uint8 ``` - +# Tutorial +A complete tutorial is available at [MDL4EO's blog](https://mdl4eo.irstea.fr/2019/01/04/an-introduction-to-deep-learning-on-remote-sensing-images-tutorial/) # Contact You can contact Rémi Cresson if you have any issues with this remote module at remi [dot] cresson [at] irstea [dot] fr +