otbtf issueshttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues2023-10-06T12:41:07+02:00https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/48Bug with non overlapping multisources2023-10-06T12:41:07+02:00Cresson RemiBug with non overlapping multisourcespan="https://minio-api-dinamis.apps.okd.crocc.meso.umontpellier.fr/catalog/spot67/SPOT7_MS_202207121015304_SPOT7_P_202207121015304/COG_SPOT7_P_202207121015304_ORT_SPOT7_20220713_1331561rgekx3ul1x1g_1.tif"
xs="https://minio-api-dinamis.a...pan="https://minio-api-dinamis.apps.okd.crocc.meso.umontpellier.fr/catalog/spot67/SPOT7_MS_202207121015304_SPOT7_P_202207121015304/COG_SPOT7_P_202207121015304_ORT_SPOT7_20220713_1331561rgekx3ul1x1g_1.tif"
xs="https://minio-api-dinamis.apps.okd.crocc.meso.umontpellier.fr/catalog/spot67/SPOT7_MS_202207121015304_SPOT7_P_202207121015304/COG_SPOT7_MS_202207121015304_ORT_SPOT7_20220713_1331511awwhqclkre47_1.tif"
- rf 64
- rf 16
- ef 32
- spcsc 1https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/47Release 4.2.02023-09-12T22:12:48+02:00Cresson RemiRelease 4.2.0- [ ] release notes
- [ ] doc
- [ ] setup.py- [ ] release notes
- [ ] doc
- [ ] setup.pyhttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/46Add decloud in testing2023-08-26T21:34:11+02:00Cresson RemiAdd decloud in testinghttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/45Update docker image2023-08-22T20:11:23+02:00Cresson RemiUpdate docker imageIt looks like today's "CRC book" test failed, even we don't have changed OTB/TF versions.
To be investigated, maybe some change in python packages?It looks like today's "CRC book" test failed, even we don't have changed OTB/TF versions.
To be investigated, maybe some change in python packages?https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/44Add helpers for common preprocessing functions2023-08-22T20:11:23+02:00Cresson RemiAdd helpers for common preprocessing functionse.g.
```python
tf.one_hot(
tf.squeeze(tf.cast(sample["labels_patches"], tf.int32), axis=-1),
depth=nb_cls
)
```
```python
class DilatedMask(keras.layers.Layer):
def __init__(self, nodata_value, dilatation_radius, name=Non...e.g.
```python
tf.one_hot(
tf.squeeze(tf.cast(sample["labels_patches"], tf.int32), axis=-1),
depth=nb_cls
)
```
```python
class DilatedMask(keras.layers.Layer):
def __init__(self, nodata_value, dilatation_radius, name=None):
self.nodata_value = nodata_value
self.dilatation_radius = dilatation_radius
super().__init__(name=name)
def call(self, inp):
"""
:param inp: input layer
"""
# Compute a binary mask from the input
nodata_mask = tf.cast(tf.math.equal(inp, self.nodata_value), tf.uint8)
dilatation_size = 1 + 2 * self.dilatation_radius
# Create a morphological kernel suitable for binary dilatation, cf https://stackoverflow.com/q/54686895/13711499
kernel = tf.zeros((dilatation_size, dilatation_size, 1), dtype=tf.uint8)
return tf.cast(tf.nn.dilation2d(input=nodata_mask, filters=kernel, strides=[1, 1, 1, 1], padding="SAME",
data_format="NHWC", dilations=[1, 1, 1, 1], name="dilatation_tf"), tf.uint8)
class ApplyMask(keras.layers.Layer):
def __init__(self, out_nodata, name=None):
super().__init__(name=name)
self.out_nodata = out_nodata
def call(self, inputs):
"""
:param inputs: [mask, input]. Mask is a binary mask, where 1 indicate the values to be masked on the input.
"""
mask, inp = inputs
return tf.where(mask == 1, float(self.out_nodata), inp)
class ScalarsTile(keras.layers.Layer):
"""
Duplicate some scalars in an whole array.
Simple example with only one scalar = 0.152: output [[0.152, 0.152, 0.152],
[0.152, 0.152, 0.152],
[0.152, 0.152, 0.152]]
"""
def __init__(self, name=None):
super().__init__(name=name)
def call(self, inputs):
"""
:param inputs: [reference, scalar inputs]. Reference is the tensor whose shape has to be matched
"""
ref, scalar_inputs = inputs
inp = tf.stack(scalar_inputs, axis=-1)
inp = tf.expand_dims(tf.expand_dims(inp, axis=1), axis=1)
return tf.tile(inp, [1, tf.shape(ref)[1], tf.shape(ref)[2], 1])
class Argmax(keras.layers.Layer):
"""
Compute the argmax of a tensor. For example, for a vector A=[0.1, 0.3, 0.6], the output is 2 (A[2] is the max)
Useful to transform a probability multibands map into a categorical map
"""
def __init__(self, name=None):
super().__init__(name=name)
def call(self, inputs):
return tf.expand_dims(tf.math.argmax(inputs, axis=-1), axis=-1)
class Max(keras.layers.Layer):
"""
Compute the max of a tensor. For example, for a vector [0.1, 0.3, 0.6], the output is 0.6
Useful to transform a probability multibands map into a "confidence" map
"""
def __init__(self, name=None):
super().__init__(name=name)
def call(self, inputs):
return tf.expand_dims(tf.math.reduce_max(inputs, axis=-1), axis=-1)
```https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/40TensorflowModelServe nodata value2023-05-23T22:39:28+02:00Cresson RemiTensorflowModelServe nodata value# Possible improvement
- if no-data value is not provided: do as usual
- if no-data value is provided: test if all pixels are nodata in current chunk. If yes, skip inference.# Possible improvement
- if no-data value is not provided: do as usual
- if no-data value is provided: test if all pixels are nodata in current chunk. If yes, skip inference.https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/39Documentation: readthedocs?2023-04-04T12:25:13+02:00Cresson RemiDocumentation: readthedocs?https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/38Docker: push "latest" tag2022-10-04T14:35:46+02:00Cresson RemiDocker: push "latest" taghttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/37Release 3.3.22022-09-22T13:08:23+02:00Cresson RemiRelease 3.3.2Cresson RemiCresson Remihttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/36Move to OTB 8.1.02022-09-21T17:35:21+02:00Cresson RemiMove to OTB 8.1.0https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/34Upgrade pip in the docker image2022-09-19T20:51:43+02:00Cresson RemiUpgrade pip in the docker imageadd a `pip install pip --upgrade`add a `pip install pip --upgrade`https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/33Documentation: how to build and train a model with keras2023-04-04T12:25:13+02:00Cresson RemiDocumentation: how to build and train a model with kerasUse the FOSS4G-2022 slidesUse the FOSS4G-2022 slideshttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/31GDAL not built with GEOS support2022-09-13T09:12:01+02:00Cresson RemiGDAL not built with GEOS supportsee https://stackoverflow.com/questions/70666700/compile-gdal-with-geossee https://stackoverflow.com/questions/70666700/compile-gdal-with-geosCresson RemiCresson Remihttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/30Improve plot script2022-09-20T10:27:38+02:00Cresson RemiImprove plot scriptImprove the script (see deepmapper)Improve the script (see deepmapper)https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/29Add a Model class in OTBTF2022-07-28T07:35:49+02:00Narcon NicolasAdd a Model class in OTBTFOTBTF python now contains a Dataset and TFRecords classes, the next step could be to create a Model class (inside `model.py`?)OTBTF python now contains a Dataset and TFRecords classes, the next step could be to create a Model class (inside `model.py`?)Narcon NicolasNarcon Nicolashttps://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/28Cannot use otbtf TFrecords without gdal2022-09-19T22:32:00+02:00Narcon NicolasCannot use otbtf TFrecords without gdalWhen on a system without gdal, the following fails:
```python
from otbtf.tfrecords import TFRecords
# import otbtf.tfrecords.TFRecords # this also fails
```
Rasing:
```
raceback (most recent call last):
File "<stdin>", line 1, in <...When on a system without gdal, the following fails:
```python
from otbtf.tfrecords import TFRecords
# import otbtf.tfrecords.TFRecords # this also fails
```
Rasing:
```
raceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/linkhome/rech/gentet01/ube33sw/otbtf/otbtf/__init__.py", line 24, in <module>
from otbtf.dataset import Buffer, PatchesReaderBase, PatchesImagesReader, IteratorBase, RandomIterator, Dataset, \
File "/linkhome/rech/gentet01/ube33sw/otbtf/otbtf/dataset.py", line 30, in <module>
from otbtf.utils import read_as_np_arr, gdal_open
File "/linkhome/rech/gentet01/ube33sw/otbtf/otbtf/utils.py", line 23, in <module>
from osgeo import gdal
ModuleNotFoundError: No module named 'osgeo'
```
This is because in `__init__.py`, there is some pieces of code that rely on gdal:
```python
from utils import read_as_np_arr, gdal_open
from otbtf.dataset import ...
```
To fix this issue, we would need an empty __init__.py. However, there is a drawback: all functions/classes would be split depending on their file (`dataset`, `utils` or `tfrecords`). Thus, backward comp would break, because stuff like import `otbtf.Dataset` would move to `otbtf.dataset.Dataset`https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/26Wrong count of geometries in PatchesExtraction2022-05-31T22:25:01+02:00Cresson RemiWrong count of geometries in PatchesExtractionGeometries are not properly counted, when a field in missing in some features.
see https://github.com/remicres/otbtf/issues/81#issuecomment-1141372073Geometries are not properly counted, when a field in missing in some features.
see https://github.com/remicres/otbtf/issues/81#issuecomment-1141372073https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/25Multi-stage cached docker build issues2023-04-04T12:25:13+02:00Vincent DelbarMulti-stage cached docker build issuesFollowing discussions [here](https://gitlab.irstea.fr/remi.cresson/otbtf/-/merge_requests/36), I want to suggest a new arch for the docker thing (without multistage build).
Thus allowing us to rebuild TF only when necessary.
The gene...Following discussions [here](https://gitlab.irstea.fr/remi.cresson/otbtf/-/merge_requests/36), I want to suggest a new arch for the docker thing (without multistage build).
Thus allowing us to rebuild TF only when necessary.
The general idea is to use a base image of TF which is stored in subdir of the repo (like gitlab-registry.irstea.fr/remi.cresson/otbtf/base ) where every base of different TF build are stored (tags like tf-2.8.0-cpu-avx2).
Then there is a second Dockerfile where the arg BASE_IMG is one of the TF base image. Finally it is pushed at the root of the repo registry.
This should ease the process of building and caching during a CI job.https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/24Update/fix docker images list in documentation2022-05-10T12:13:39+02:00Cresson RemiUpdate/fix docker images list in documentationsee
https://github.com/remicres/otbtf/issues/79see
https://github.com/remicres/otbtf/issues/79https://gitlab.irstea.fr/remi.cresson/otbtf/-/issues/23CI: docker build doesn't reuse cache from 2nd stage and after2022-05-10T12:13:39+02:00Cresson RemiCI: docker build doesn't reuse cache from 2nd stage and after