otbtf issueshttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues2022-07-28T07:35:49+02:00https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/29Add a Model class in OTBTF2022-07-28T07:35:49+02:00Narcon NicolasAdd a Model class in OTBTFOTBTF python now contains a Dataset and TFRecords classes, the next step could be to create a Model class (inside `model.py`?)OTBTF python now contains a Dataset and TFRecords classes, the next step could be to create a Model class (inside `model.py`?)Narcon NicolasNarcon Nicolashttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/46Add decloud in testing2023-08-26T21:34:11+02:00Cresson RemiAdd decloud in testinghttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/44Add helpers for common preprocessing functions2023-08-22T20:11:23+02:00Cresson RemiAdd helpers for common preprocessing functionse.g.
```python
tf.one_hot(
tf.squeeze(tf.cast(sample["labels_patches"], tf.int32), axis=-1),
depth=nb_cls
)
```
```python
class DilatedMask(keras.layers.Layer):
def __init__(self, nodata_value, dilatation_radius, name=Non...e.g.
```python
tf.one_hot(
tf.squeeze(tf.cast(sample["labels_patches"], tf.int32), axis=-1),
depth=nb_cls
)
```
```python
class DilatedMask(keras.layers.Layer):
def __init__(self, nodata_value, dilatation_radius, name=None):
self.nodata_value = nodata_value
self.dilatation_radius = dilatation_radius
super().__init__(name=name)
def call(self, inp):
"""
:param inp: input layer
"""
# Compute a binary mask from the input
nodata_mask = tf.cast(tf.math.equal(inp, self.nodata_value), tf.uint8)
dilatation_size = 1 + 2 * self.dilatation_radius
# Create a morphological kernel suitable for binary dilatation, cf https://stackoverflow.com/q/54686895/13711499
kernel = tf.zeros((dilatation_size, dilatation_size, 1), dtype=tf.uint8)
return tf.cast(tf.nn.dilation2d(input=nodata_mask, filters=kernel, strides=[1, 1, 1, 1], padding="SAME",
data_format="NHWC", dilations=[1, 1, 1, 1], name="dilatation_tf"), tf.uint8)
class ApplyMask(keras.layers.Layer):
def __init__(self, out_nodata, name=None):
super().__init__(name=name)
self.out_nodata = out_nodata
def call(self, inputs):
"""
:param inputs: [mask, input]. Mask is a binary mask, where 1 indicate the values to be masked on the input.
"""
mask, inp = inputs
return tf.where(mask == 1, float(self.out_nodata), inp)
class ScalarsTile(keras.layers.Layer):
"""
Duplicate some scalars in an whole array.
Simple example with only one scalar = 0.152: output [[0.152, 0.152, 0.152],
[0.152, 0.152, 0.152],
[0.152, 0.152, 0.152]]
"""
def __init__(self, name=None):
super().__init__(name=name)
def call(self, inputs):
"""
:param inputs: [reference, scalar inputs]. Reference is the tensor whose shape has to be matched
"""
ref, scalar_inputs = inputs
inp = tf.stack(scalar_inputs, axis=-1)
inp = tf.expand_dims(tf.expand_dims(inp, axis=1), axis=1)
return tf.tile(inp, [1, tf.shape(ref)[1], tf.shape(ref)[2], 1])
class Argmax(keras.layers.Layer):
"""
Compute the argmax of a tensor. For example, for a vector A=[0.1, 0.3, 0.6], the output is 2 (A[2] is the max)
Useful to transform a probability multibands map into a categorical map
"""
def __init__(self, name=None):
super().__init__(name=name)
def call(self, inputs):
return tf.expand_dims(tf.math.argmax(inputs, axis=-1), axis=-1)
class Max(keras.layers.Layer):
"""
Compute the max of a tensor. For example, for a vector [0.1, 0.3, 0.6], the output is 0.6
Useful to transform a probability multibands map into a "confidence" map
"""
def __init__(self, name=None):
super().__init__(name=name)
def call(self, inputs):
return tf.expand_dims(tf.math.reduce_max(inputs, axis=-1), axis=-1)
```https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/14Add TFRecords creation to OTBTF2022-05-10T12:13:39+02:00Narcon NicolasAdd TFRecords creation to OTBTFWe would need to define a new class TFRecord that handles all the dataset <-> tfrecord conversions. Inside `otbtf.py` ? Or we split `otbtf.py` into a python module:
```
otbtf
|───dataset.py
|───tfrecord.py
└───utils.py
```
Maybe add a m...We would need to define a new class TFRecord that handles all the dataset <-> tfrecord conversions. Inside `otbtf.py` ? Or we split `otbtf.py` into a python module:
```
otbtf
|───dataset.py
|───tfrecord.py
└───utils.py
```
Maybe add a method `to_tfrecords(out_dir)` inside Dataset class ?https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/18Add toy example to train a model with keras/otbtf dataset2022-07-28T07:35:49+02:00Cresson RemiAdd toy example to train a model with keras/otbtf datasetAdd an example how to train a model with keras, using some otbtf.datasets and saving the trained model as SavedModel.Add an example how to train a model with keras, using some otbtf.datasets and saving the trained model as SavedModel.https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/20Add warning message when no output_shape or output_type is missing2022-09-14T09:06:30+02:00Cresson RemiAdd warning message when no output_shape or output_type is missinghttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/48Bug with non overlapping multisources2023-10-06T12:41:07+02:00Cresson RemiBug with non overlapping multisourcespan="https://minio-api-dinamis.apps.okd.crocc.meso.umontpellier.fr/catalog/spot67/SPOT7_MS_202207121015304_SPOT7_P_202207121015304/COG_SPOT7_P_202207121015304_ORT_SPOT7_20220713_1331561rgekx3ul1x1g_1.tif"
xs="https://minio-api-dinamis.a...pan="https://minio-api-dinamis.apps.okd.crocc.meso.umontpellier.fr/catalog/spot67/SPOT7_MS_202207121015304_SPOT7_P_202207121015304/COG_SPOT7_P_202207121015304_ORT_SPOT7_20220713_1331561rgekx3ul1x1g_1.tif"
xs="https://minio-api-dinamis.apps.okd.crocc.meso.umontpellier.fr/catalog/spot67/SPOT7_MS_202207121015304_SPOT7_P_202207121015304/COG_SPOT7_MS_202207121015304_ORT_SPOT7_20220713_1331511awwhqclkre47_1.tif"
- rf 64
- rf 16
- ef 32
- spcsc 1https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/12Bugs in v3.0-rc12022-03-07T19:19:12+01:00Cresson RemiBugs in v3.0-rc1- add exception when returned number of components is -1 (CopyUtils)
- resolve userplaceholders tensors names- add exception when returned number of components is -1 (CopyUtils)
- resolve userplaceholders tensors nameshttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/16Build docker images after release2022-04-22T16:26:06+02:00Cresson RemiBuild docker images after releaseUse a custom runner to build & push docker images on the registry
- gpu basic dev
- cpu basic dev
- gpu basic
- cpu basic
- gpu dev (SSE, AVX optimized)
- cpu dev (SSE, AVX optimized)Use a custom runner to build & push docker images on the registry
- gpu basic dev
- cpu basic dev
- gpu basic
- cpu basic
- gpu dev (SSE, AVX optimized)
- cpu dev (SSE, AVX optimized)https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/9Cannot mix OTBTF application and Tensorflow in Python2023-04-04T12:25:13+02:00Narcon NicolasCannot mix OTBTF application and Tensorflow in PythonThis piece of script raises an error on the last line
```python
import tensorflow as tf
import otbApplication
# Create a dummy Tensorflow model and save it
x1 = tf.keras.Input(shape=[None, None, 4]) # [h, w, N]
x2 = tf.keras.Input(shap...This piece of script raises an error on the last line
```python
import tensorflow as tf
import otbApplication
# Create a dummy Tensorflow model and save it
x1 = tf.keras.Input(shape=[None, None, 4]) # [h, w, N]
x2 = tf.keras.Input(shape=[None, None, 4]) # [h, w, N]
y = tf.math.multiply(x1, x2)
model = tf.keras.Model(inputs=[x1, x2], outputs=y, name="multiply")
model.save("my_model_multiply")
# Create an OTB application
serve = otbApplication.Registry.CreateApplication('TensorflowModelServe')
```
Was run in 4 differents environments and raised different errors.
On `mdl4eo/otbtf2.4:cpu` and `mdl4eo/otbtf2.4:gpu` :
```
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: tensorflow/core/profiler/profiler_service_monitor_result.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1379] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
Traceback (most recent call last):
File "pyotb.py", line 12, in <module>
serve = otbApplication.Registry.CreateApplication('TensorflowModelServe')
```
on `mdl4eo/otbtf2.5:gpu` and `mdl4eo/otbtf2.5:cpu` :
```
2021-10-28 13:31:51.933796: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/op_expansion/node_counter
2021-10-28 13:31:51.933889: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/op_expansion/graph_counter
2021-10-28 13:31:51.933922: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/core/op_expansion/op_counter
2021-10-28 13:31:51.933988: F tensorflow/core/framework/device_factory.cc:90] Duplicate registration of device factory for type XLA_CPU with the same priority 50
Aborted (core dumped)
```https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/28Cannot use otbtf TFrecords without gdal2022-09-19T22:32:00+02:00Narcon NicolasCannot use otbtf TFrecords without gdalWhen on a system without gdal, the following fails:
```python
from otbtf.tfrecords import TFRecords
# import otbtf.tfrecords.TFRecords # this also fails
```
Rasing:
```
raceback (most recent call last):
File "<stdin>", line 1, in <...When on a system without gdal, the following fails:
```python
from otbtf.tfrecords import TFRecords
# import otbtf.tfrecords.TFRecords # this also fails
```
Rasing:
```
raceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/linkhome/rech/gentet01/ube33sw/otbtf/otbtf/__init__.py", line 24, in <module>
from otbtf.dataset import Buffer, PatchesReaderBase, PatchesImagesReader, IteratorBase, RandomIterator, Dataset, \
File "/linkhome/rech/gentet01/ube33sw/otbtf/otbtf/dataset.py", line 30, in <module>
from otbtf.utils import read_as_np_arr, gdal_open
File "/linkhome/rech/gentet01/ube33sw/otbtf/otbtf/utils.py", line 23, in <module>
from osgeo import gdal
ModuleNotFoundError: No module named 'osgeo'
```
This is because in `__init__.py`, there is some pieces of code that rely on gdal:
```python
from utils import read_as_np_arr, gdal_open
from otbtf.dataset import ...
```
To fix this issue, we would need an empty __init__.py. However, there is a drawback: all functions/classes would be split depending on their file (`dataset`, `utils` or `tfrecords`). Thus, backward comp would break, because stuff like import `otbtf.Dataset` would move to `otbtf.dataset.Dataset`https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/23CI: docker build doesn't reuse cache from 2nd stage and after2022-05-10T12:13:39+02:00Cresson RemiCI: docker build doesn't reuse cache from 2nd stage and afterhttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/5Clang-format2022-07-21T15:58:09+02:00Cresson RemiClang-formatClang-format permet de formater le code c++.
On peur le lancer sur les fichiers `.hxx`, `.txx`, `.cxx`, `.h` de la façon suivante:
```
find . -regex '.*\.\(cxx\|hxx\|txx\|h\)' -exec clang-format -style=file -i {} \;
```
Il faut ajouter...Clang-format permet de formater le code c++.
On peur le lancer sur les fichiers `.hxx`, `.txx`, `.cxx`, `.h` de la façon suivante:
```
find . -regex '.*\.\(cxx\|hxx\|txx\|h\)' -exec clang-format -style=file -i {} \;
```
Il faut ajouter le fichier `.clang-format` qui définit les rêgles de formatage à la racine de otbtf (cf [celui d'OTB](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/blob/develop/.clang-format)).
J'ai essayé chez moi c'est pas mal du tout!https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/1Complementary informations for GPU users2023-04-04T12:25:12+02:00Gaetano RaffaeleComplementary informations for GPU usersGreat job with the otftf docker suite, I grabbed the otbtf2.0:gpu docker and it really eases things up!
By the way, some additional information for gpu users might be included in the documentation, basically:
- the need to install the n...Great job with the otftf docker suite, I grabbed the otbtf2.0:gpu docker and it really eases things up!
By the way, some additional information for gpu users might be included in the documentation, basically:
- the need to install the nvidia-container-runtime package at the host level
- the need to specify the '--gpus' option when calling the container to make system gpus visible in the docker
Cheers!
Rafhttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/38Docker: push "latest" tag2022-10-04T14:35:46+02:00Cresson RemiDocker: push "latest" taghttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/33Documentation: how to build and train a model with keras2023-04-04T12:25:13+02:00Cresson RemiDocumentation: how to build and train a model with kerasUse the FOSS4G-2022 slidesUse the FOSS4G-2022 slideshttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/39Documentation: readthedocs?2023-04-04T12:25:13+02:00Cresson RemiDocumentation: readthedocs?https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/17gdal import (python)2022-04-05T19:55:32+02:00Cresson Remigdal import (python)see https://github.com/remicres/otbtf/issues/71see https://github.com/remicres/otbtf/issues/71Cresson RemiCresson Remihttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/31GDAL not built with GEOS support2022-09-13T09:12:01+02:00Cresson RemiGDAL not built with GEOS supportsee https://stackoverflow.com/questions/70666700/compile-gdal-with-geossee https://stackoverflow.com/questions/70666700/compile-gdal-with-geosCresson RemiCresson Remihttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/41Improve documentation2023-04-24T16:15:07+02:00Cresson RemiImprove documentation- [ ] pip install otbtf (standalone)- [ ] pip install otbtf (standalone)