diff --git a/doc/HOWTOBUILD.md b/doc/HOWTOBUILD.md index ededd5e5a3f43a0d576362bc74730f2ee95cc56d..82d3b115fbae05c5dfdb824e10bc32318e009006 100644 --- a/doc/HOWTOBUILD.md +++ b/doc/HOWTOBUILD.md @@ -1,44 +1,58 @@ # How to build OTBTF from sources -This remote module has been tested successfully on Ubuntu 18 and CentOs 7 with last CUDA drivers, TensorFlow r1.14 and OTB develop (0df44b312d64d6c3890b65d3790d4a17d0fd5f23). +This remote module has been tested successfully on Ubuntu 18 with last CUDA drivers, TensorFlow r2.1 and OTB 7.1.0. ## Build OTB -First, **build the latest *develop* branch of OTB from sources**. You can check the [OTB documentation](https://www.orfeo-toolbox.org/SoftwareGuide/SoftwareGuidech2.html) which details all the steps, if fact it is quite easy thank to the SuperBuild. +First, **build the *release-7.1* branch of OTB from sources**. You can check the [OTB documentation](https://www.orfeo-toolbox.org/SoftwareGuide/SoftwareGuidech2.html) which details all the steps. It is quite easy thank to the SuperBuild, a cmake script that automates the build. + +Create a folder for OTB, clone sources, configure OTB SuperBuild, and build it. + +Install required packages: -Basically, you have to create a folder for OTB, clone sources, configure OTB SuperBuild, and build it. -The following has been validated with an OTB 6.7.0. ``` sudo apt-get update sudo apt-get upgrade -sudo apt-get install aptitude -sudo aptitude install make cmake-curses-gui build-essential libtool automake git libbz2-dev python-dev libboost-dev libboost-filesystem-dev libboost-serialization-dev libboost-system-dev zlib1g-dev libcurl4-gnutls-dev swig libkml-dev +sudo apt-get install sudo ca-certificates curl make cmake g++ gcc git libtool swig xvfb wget autoconf automake pkg-config zip zlib1g-dev unzip freeglut3-dev libboost-date-time-dev libboost-filesystem-dev libboost-graph-dev libboost-program-options-dev libboost-system-dev libboost-thread-dev libcurl4-gnutls-dev libexpat1-dev libfftw3-dev libgdal-dev libgeotiff-dev libglew-dev libglfw3-dev libgsl-dev libinsighttoolkit4-dev libkml-dev libmuparser-dev libmuparserx-dev libopencv-core-dev libopencv-ml-dev libopenthreads-dev libossim-dev libpng-dev libqt5opengl5-dev libqwt-qt5-dev libsvm-dev libtinyxml-dev qtbase5-dev qttools5-dev default-jdk python3-pip python3.6-dev python3.6-gdal python3-setuptools libxmu-dev libxi-dev qttools5-dev-tools bison software-properties-common dirmngr apt-transport-https lsb-release gdal-bin +``` + +Build OTB from sources: + +``` sudo mkdir /work sudo chown $USER /work mkdir /work/otb cd /work/otb mkdir build -git clone https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb.git OTB +git clone -b release-7.1 https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb.git OTB cd build +``` + +From here you can tell the interactively SuperBuild to use system boost, curl, zlib, libkml for instance. + +``` ccmake /work/otb/OTB/SuperBuild ``` -From here you can tell the SuperBuild to use system boost, curl, zlib, libkml for instance. -Then you can build it: +If you don't know how to configure options, you can use the following: + +``` +cmake /work/otb/OTB/SuperBuild -DUSE_SYSTEM_BOOST=ON -DUSE_SYSTEM_CURL=ON -DUSE_SYSTEM_EXPAT=ON -DUSE_SYSTEM_FFTW=ON -DUSE_SYSTEM_FREETYPE=ON -DUSE_SYSTEM_GDAL=ON -DUSE_SYSTEM_GEOS=ON -DUSE_SYSTEM_GEOTIFF=ON -DUSE_SYSTEM_GLEW=ON -DUSE_SYSTEM_GLFW=ON -DUSE_SYSTEM_GLUT=ON -DUSE_SYSTEM_GSL=ON -DUSE_SYSTEM_ITK=ON -DUSE_SYSTEM_LIBKML=ON -DUSE_SYSTEM_LIBSVM=ON -DUSE_SYSTEM_MUPARSER=ON -DUSE_SYSTEM_MUPARSERX=ON -DUSE_SYSTEM_OPENCV=ON -DUSE_SYSTEM_OPENTHREADS=ON -DUSE_SYSTEM_OSSIM=ON -DUSE_SYSTEM_PNG=ON -DUSE_SYSTEM_QT5=ON -DUSE_SYSTEM_QWT=ON -DUSE_SYSTEM_TINYXML=ON -DUSE_SYSTEM_ZLIB=ON -DUSE_SYSTEM_SWIG=OFF -DOTB_WRAP_PYTHON=OFF +``` + +Then you can build OTB: ``` make -j $(grep -c ^processor /proc/cpuinfo) ``` ## Build TensorFlow with shared libraries During this step, you have to **build Tensorflow from source** except if you want to use only the sampling applications of OTBTensorflow (in this case, skip this section). -The following has been validated with TensorFlow r1.14 and gcc 5.3.1. ### Bazel First, install Bazel. ``` -sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python -wget https://github.com/bazelbuild/bazel/releases/download/0.20.0/bazel-0.20.0-installer-linux-x86_64.sh -chmod +x bazel-0.20.0-installer-linux-x86_64.sh -./bazel-0.20.0-installer-linux-x86_64.sh --user +wget https://github.com/bazelbuild/bazel/releases/download/0.29.1/bazel-0.29.1-installer-linux-x86_64.sh +chmod +x bazel-0.29.1-installer-linux-x86_64.sh +./bazel-0.29.1-installer-linux-x86_64.sh --user export PATH="$PATH:$HOME/bin" ``` @@ -47,9 +61,8 @@ If you fail to install properly Bazel, you can read the beginning of [the instru ### Required packages There is a few required packages that you need to install: ``` -sudo apt install python-dev python-pip python3-dev python3-pip python3-mock -sudo pip install pip six numpy wheel mock keras -sudo pip3 install pip six numpy wheel mock keras +sudo python3 -m pip install --upgrade pip +sudo python3 -m pip install pip six numpy wheel mock keras future setuptools ``` For a pure python3 install, you might need to workaround a bazel bug the following way: @@ -57,9 +70,8 @@ For a pure python3 install, you might need to workaround a bazel bug the followi sudo ln -s /usr/bin/python3 /usr/bin/python ``` -### Build TensorFlow the right way -Now, let's build TensorFlow with all the stuff required by OTBTF. -Make a directory for TensorFlow. +### Build TensorFlow +Create a directory for TensorFlow. For instance `mkdir /work/tf`. Clone TensorFlow. @@ -68,104 +80,74 @@ cd /work/tf git clone https://github.com/tensorflow/tensorflow.git ``` -Now configure the project. If you have CUDA and other NVIDIA stuff installed in your system, remember that you have to tell the script that it is in `/usr/` (no symlink required!). +Now configure the project. If you have CUDA and other NVIDIA stuff installed in your system, remember that you have to tell the script that it is in `/usr/` (no symlink required!). If you have CPU-only hardware, building Intel MKL is a good choice since it provides a significant speedup in computations. ``` cd tensorflow ./configure ``` + Then, you have to build TensorFlow with the instructions sets supported by your CPU (For instance here is AVX, AVX2, FMA, SSE4.1, SSE4.2 that play fine on a modern intel CPU). You have to tell Bazel to build: 1. The TensorFlow python pip package 2. The libtensorflow_cc.so library 3. The libtensorflow_framework.so library + ``` -bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 //tensorflow:libtensorflow_framework.so //tensorflow:libtensorflow_cc.so //tensorflow:libtensorflow.so //tensorflow/tools/pip_package:build_pip_package +bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 //tensorflow:libtensorflow_framework.so //tensorflow:libtensorflow_cc.so //tensorflow:libtensorflow.so //tensorflow/tools/pip_package:build_pip_package --noincompatible_do_not_split_linking_cmdline ``` *You might fail this step (e.g. missing packages). In this case, it's recommended to clear the bazel cache, using something like `rm $HOME/.cache/bazel/* -rf` before configuring and building everything!* -### Prepare the right stuff to use TensorFlow in external (cmake) projects -This is the most important! -First, build and deploy the pip package. +### Pip package +Build and deploy the pip package. + ``` bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg -pip install /tmp/tensorflow_pkg/tensorflow-1.12.0rc0-cp27-cp27mu-linux_x86_64.whl +pip3 install $(find /tmp/tensorflow_pkg/ -type f -iname "tensorflow*.whl") ``` -For the C++ API, it's a bit more tricky. -Let's begin. -First, download dependencies. + +### C++ API +First, download and build TensorFlow dependencies. + ``` -/work/tf/tensorflow/tensorflow/contrib/makefile/download_dependencies.sh +/work/tf/tensorflow/tensorflow/lite/tools/make/download_dependencies.sh +/work/tf/tensorflow/tensorflow/lite/tools/make/build_lib.sh ``` + Then, build Google Protobuf + ``` -mkdir /tmp/proto -cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/protobuf/ -./autogen.sh -./configure --prefix=/tmp/proto/ -make -j $(grep -c ^processor /proc/cpuinfo) -make install -``` -Then, "build" eigen (header only...) -``` -mkdir /tmp/eigen -cd ../eigen -mkdir build_dir -cd build_dir -cmake -DCMAKE_INSTALL_PREFIX=/tmp/eigen/ ../ +mkdir -p /work/tf/installdir +cd /work/tf/ +wget https://github.com/google/protobuf/releases/download/v3.8.0/protobuf-cpp-3.8.0.tar.gz +tar -xvf protobuf-cpp-3.8.0.tar.gz +cd protobuf-3.8.0 +./configure --prefix=/work/tf/installdir/ make install -j $(grep -c ^processor /proc/cpuinfo) ``` -Then, build NSync -``` -/work/tf/tensorflow/tensorflow/contrib/makefile/compile_nsync.sh -``` -Then, build absl + +Then, prepare a folder with everything (include, libs) + ``` -mkdir /tmp/absl -cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/absl/ -mkdir build_dir -cd build_dir -cmake -DCMAKE_INSTALL_PREFIX=/tmp/absl ../ -make -j $(grep -c ^processor /proc/cpuinfo) +mkdir -p /work/tf/installdir/lib +mkdir -p /work/tf/installdir/include +cp bazel-bin/tensorflow/libtensorflow_cc.so* /work/tf/installdir/lib +cp bazel-bin/tensorflow/libtensorflow_framework.so* /work/tf/installdir/lib +cp -r bazel-genfiles/* /work/tf/installdir/include +cp -r tensorflow/cc /work/tf/installdir/include/tensorflow +cp -r tensorflow/core /work/tf/installdir/include/tensorflow +cp -r third_party /work/tf/installdir/include +cp -r bazel-tensorflow/external/eigen_archive/unsupported /work/tf/installdir/include +cp -r bazel-tensorflow/external/eigen_archive/Eigen /work/tf/installdir/include +cp -r tensorflow/lite/tools/make/downloads/absl/absl /work/tf/installdir/include ``` -Now, you have to copy the useful stuff in a directory - -``` -# Create folders -mkdir /work/tf/installdir -mkdir /work/tf/installdir/lib -mkdir /work/tf/installdir/include - -# Copy libs -cp /work/tf/tensorflow/bazel-bin/tensorflow/libtensorflow_cc.so /work/tf/installdir/lib/ -cp /work/tf/tensorflow/bazel-bin/tensorflow/libtensorflow_framework.so /work/tf/installdir/lib/ -cp /tmp/proto/lib/libprotobuf.a /work/tf/installdir/lib/ -cp /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/nsync/builds/default.linux.c++11/*.a /work/tf/installdir/lib/ -ln -s /work/tf/installdir/lib/libtensorflow_framework.so /work/tf/installdir/lib/libtensorflow_framework.so.1 -ln -s /work/tf/installdir/lib/libtensorflow_cc.so /work/tf/installdir/lib/libtensorflow_cc.so.1 - -# Copy headers -mkdir /work/tf/installdir/include/tensorflow -cp -r /work/tf/tensorflow/bazel-genfiles/* /work/tf/installdir/include -cp -r /work/tf/tensorflow/tensorflow/cc /work/tf/installdir/include/tensorflow -cp -r /work/tf/tensorflow/tensorflow/core /work/tf/installdir/include/tensorflow -cp -r /work/tf/tensorflow/third_party /work/tf/installdir/include -cp -r /tmp/proto/include/* /work/tf/installdir/include -cp -r /tmp/eigen/include/eigen3/* /work/tf/installdir/include -cp /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/nsync/public/* /work/tf/installdir/include/ -cd /work/tf/tensorflow/tensorflow/contrib/makefile/downloads/absl -find absl/ -name '*.h' -exec cp --parents \{\} /work/tf/installdir/include/ \; -find absl/ -name '*.inc' -exec cp --parents \{\} /work/tf/installdir/include/ \; - -# Cleaning -find /work/tf/installdir/ -name "*.cc" -type f -delete -``` -Well done. Now you have a working copy of TensorFlow located in `/work/tf/installdir` that is ready to use in external C++ cmake projects :) - -## Build this remote module -Finally, we can build this module. -Clone the repository in your the OTB sources directory for remote modules (something like `/work/otb/OTB/Modules/Remote/`). + +Now you have a working copy of TensorFlow located in `/work/tf/installdir` that is ready to use in external C++ cmake projects :) + +## Build the OTBTF remote module +Finally, we can build the OTBTF module. +Clone the repository inside the OTB sources directory for remote modules: `/work/otb/OTB/Modules/Remote/`. Re configure OTB with cmake of ccmake, and set the following variables - **Module_OTBTensorflow** to **ON** @@ -175,10 +157,12 @@ Re configure OTB with cmake of ccmake, and set the following variables - **tensorflow_include_dir** to `/work/tf/installdir/include` Re build and re install OTB. + ``` cd /work/otb/build/OTB/build ccmake make -j $(grep -c ^processor /proc/cpuinfo) +make install ``` Done ! @@ -186,7 +170,7 @@ Don't forget to add some important environment variables, and this is finished. ``` export PATH="$PATH:/work/otb/superbuild_install/bin/" -export PYTHONPATH="$PYTHONPATH:/work/otb/superbuild_install/lib/otb/python" +export PYTHONPATH="$PYTHONPATH:/work/otb/superbuild_install/lib/otb/python:/work/otb/otb/Modules/Remote/otbtf/python" export OTB_APPLICATION_PATH="$OTB_APPLICATION_PATH:/work/otb/superbuild_install/lib/otb/applications" export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/work/otb/superbuild_install/lib/:/work/tf/installdir/lib/" ```