scenes issueshttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues2023-06-06T19:32:29+02:00https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/40MàJ documentation2023-06-06T19:32:29+02:00Cresson RemiMàJ documentationhttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/39Installation2023-06-06T11:07:00+02:00Cresson RemiInstallationAdd doc instructions how to installAdd doc instructions how to installhttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/38Cache improvements2023-05-24T15:40:13+02:00Cresson RemiCache improvementscreer dossier cache si il n'existe pascreer dossier cache si il n'existe pashttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/37Partial Sentinel-2 downloads2023-04-22T20:33:03+02:00Cresson RemiPartial Sentinel-2 downloadspermettre d'instancier des scenes sentinel-2 à partir de téléchargements incomplets, typiquement un subset des fichiers comme le fait theia-pickerpermettre d'instancier des scenes sentinel-2 à partir de téléchargements incomplets, typiquement un subset des fichiers comme le fait theia-pickerhttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/36pydocstyle (CI)2023-03-23T07:19:53+01:00Cresson Remipydocstyle (CI)https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/35Logging2023-03-22T20:46:38+01:00Cresson RemiLoggingutiliser un logger digne de ce nom à la place des affreux `print`utiliser un logger digne de ce nom à la place des affreux `print`https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/34Move to pyotb 2.0.02023-06-29T20:44:29+02:00Cresson RemiMove to pyotb 2.0.0https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/33AssertionError when creating instance of Spot67Scene2023-03-23T17:12:02+01:00Giffard-Carlet JosselinAssertionError when creating instance of Spot67SceneWith SPOT image from https://ids-dinamis.data-terra.org/web/guest/catalog1
image selected with coordinates N : 45.4924, E : 6.6502, S : 45.3072, W : 6.3426, year : 2020, "Image brute" :
- 2020-07-08_10:09:55.539_SPOT6_SENSOR_MS
- 2020-0...With SPOT image from https://ids-dinamis.data-terra.org/web/guest/catalog1
image selected with coordinates N : 45.4924, E : 6.6502, S : 45.3072, W : 6.3426, year : 2020, "Image brute" :
- 2020-07-08_10:09:55.539_SPOT6_SENSOR_MS
- 2020-07-08_10:09:55.538_SPOT6_SENSOR_P
When creating instance of Spot67Scene
```
from scenes.spot import Spot67Scene
sc = Spot67Scene(dimap_file_xs="/path/to/DIM_SPOT6_MS_..._1.XML",
dimap_file_pan="/path/to/DIM_SPOT6_P_..._1.XML")
```
Getting assertion error :
```
Traceback (most recent call last):
File "/usr/lib/python3.8/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/home/josselingiffard/.local/lib/python3.8/site-packages/scenes/spot.py", line 323, in __init__
epsg_xs, extent_wgs84_xs, self.bbox_wgs84_xs = get_epsg_extent_bbox(dimap_file_xs)
File "/home/josselingiffard/.local/lib/python3.8/site-packages/scenes/raster.py", line 108, in get_epsg_extent_bbox
epsg = get_epsg(gdal_ds)
File "/home/josselingiffard/.local/lib/python3.8/site-packages/scenes/raster.py", line 24, in get_epsg
assert str(epsg).isdigit()
AssertionError
```https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/32Lecture MDs Spot-6/7 via IMD2022-10-11T19:40:53+02:00Cresson RemiLecture MDs Spot-6/7 via IMD...plutôt que parser le XML...plutôt que parser le XMLhttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/31Source._new_source to find output parameter key2022-10-09T20:38:01+02:00Cresson RemiSource._new_source to find output parameter keytake the 1st output key instead of default "out"take the 1st output key instead of default "out"https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/30Possibilité de sélectionner les bandes à stacker dans Sentinel2Source2023-03-23T17:12:02+01:00Cresson RemiPossibilité de sélectionner les bandes à stacker dans Sentinel2Sourcehttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/29Pipelines cache2022-09-21T16:47:36+02:00Cresson RemiPipelines cacheQuand on aura la super fonctionnalité de serialization des pyotb objects (cf [cette issue](https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/issues/43)) on pourra faire du cache.
Par ex, considérons un traitement gourmand en calcul...
...Quand on aura la super fonctionnalité de serialization des pyotb objects (cf [cette issue](https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/issues/43)) on pourra faire du cache.
Par ex, considérons un traitement gourmand en calcul...
```python
pxs = sc.get_pxs(method='bayes').cld_msk_drilled()
```
... qu'on est amené à refaire souvent dans un pipeline:
```python
my_pipeline = f(pxs)
```
Alors on pourrait le mettre en cache en remplaçant le code ci-dessus de cette manière:
```python
pxs_cached = scenes.Cache(pxs)
my_pipeline = f(pxs_cached)
```https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/28Le pipeline de develop --> master plante pour "Spot67_test"2023-04-23T20:20:33+02:00Cresson RemiLe pipeline de develop --> master plante pour "Spot67_test"Et je ne comprends pas pourquoi.
Par ex.
https://gitlab.irstea.fr/umr-tetis/scenes/-/pipelines/38910Et je ne comprends pas pourquoi.
Par ex.
https://gitlab.irstea.fr/umr-tetis/scenes/-/pipelines/38910https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/27Index: Chercher plus proche voisin2022-09-08T12:44:37+02:00Cresson RemiIndex: Chercher plus proche voisinSee https://rtree.readthedocs.io/en/latest/tutorial.html#nearest-neighborsSee https://rtree.readthedocs.io/en/latest/tutorial.html#nearest-neighborshttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/26ajouter un test pour les classes Sentinel22023-03-23T17:12:02+01:00Cresson Remiajouter un test pour les classes Sentinel2- tester les classes héritant de `Sentinel2Source`
- vérifier que la filiation est OK- tester les classes héritant de `Sentinel2Source`
- vérifier que la filiation est OKhttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/25Les classes sentinel ne retournent que des sources Sentinel22A2022-09-07T21:59:31+02:00Cresson RemiLes classes sentinel ne retournent que des sources Sentinel22ACresson RemiCresson Remihttps://gitlab.irstea.fr/umr-tetis/scenes/-/issues/24Support de STAC2023-03-23T17:12:02+01:00Cresson RemiSupport de STAC# Base
- Publication scene --> STAC Item (cf. https://pystac.readthedocs.io/en/stable/tutorials/how-to-create-stac-catalogs.html):
- [x] rendre bbox similaire à rasterio
- [x] ajouter footprint (doit être similaire à rasterio)
- Ins...# Base
- Publication scene --> STAC Item (cf. https://pystac.readthedocs.io/en/stable/tutorials/how-to-create-stac-catalogs.html):
- [x] rendre bbox similaire à rasterio
- [x] ajouter footprint (doit être similaire à rasterio)
- Instanciation de scenes à partir de catalogue STAC
- [x] instancier un object `Scene` à partir d'un STAC Item
# A investiguer
- [x] Genericité de l'API STAC pour faire des recherches par bbox et date
- [x] Mapping générique STAC Item --> SensorScene permettant l'ajout de multiples fournisseurs pour un capteur donné
# Refonte des constructeurs de classes `Scene`
## Avant
```mermaid
classDiagram
Scene <|-- Spot67Scene
Scene <|-- Sentinel2SceneBase
Sentinel2SceneBase <|-- Sentinel22AScene
Sentinel2SceneBase <|-- Sentinel23AScene
Scene : __init__(acquisition_date, epsg, extent_wgs84)
Scene : datetime acquisition_date
Scene : list extent_wgs84
Scene : int epsg
Scene : BoundingBox bbox_wgs84
Spot67Scene : __init__(dim_xs, dim_pan)
Spot67Scene : Spot67Source get_xs(reflectance)
Spot67Scene : Spot67Source get_pan(reflectance)
Spot67Scene : list extent_wgs84_xs
Spot67Scene : list extent_wgs84_pan
Spot67Scene : float sun_az
Spot67Scene : float sun_elev
Spot67Scene : str cld_msk_file_xs
Spot67Scene : str cld_msk_file_pan
Spot67Scene : str roi_msk_file_xs
Spot67Scene : str roi_msk_file_pan
Sentinel2SceneBase : __init__(archive, tag)
Sentinel2SceneBase : str archive
Sentinel2SceneBase : str tag
Sentinel22AScene : __init__(archive)
Sentinel22AScene : Sentinel22ASource get_10m_bands()
Sentinel22AScene : Sentinel22ASource get_20m_bands()
Sentinel22AScene: str clm_r1_msk_file
Sentinel22AScene: str edg_r1_msk_file
Sentinel22AScene: str clm_r2_msk_file
Sentinel22AScene: str edg_r2_msk_file
Sentinel23AScene : __init__(archive)
Sentinel23AScene : Sentinel23ASource get_10m_bands()
Sentinel23AScene : Sentinel23ASource get_20m_bands()
Sentinel23AScene: str flg_r1_msk_file
Sentinel23AScene: str flg_r2_msk_file
```
## Après
### Scenes
```mermaid
classDiagram
Scene <|-- Spot67Base
Spot67Base <|-- Spot67DRS
Spot67Base <|-- Spot67IGN
Scene <|-- Sentinel2Base
Sentinel2Base <|-- Sentinel2Theia2A
Sentinel2Base <|-- Sentinel2Theia3A
Sentinel2Base <|-- Sentinel2MPC2A
Scene : __init__(acquisition_date, epsg, extent_wgs84)
Scene : datetime acquisition_date
Scene : list extent_wgs84
Scene : int epsg
Scene : BoundingBox bbox_wgs84
Scene : dict[str, str] sources_paths
Spot67Base : __init__(src_xs, src_pan)
Spot67Base : list extent_wgs84_xs
Spot67Base : list extent_wgs84_pan
Sentinel2Base : __init__(assets_dict)
Sentinel2Base : class SourceClass
```
### Sources
```mermaid
classDiagram
Source <|-- Spot67DRSSource
Source <|-- Sentinel2Source
Sentinel2Source <|-- Sentinel2Theia2ASource
Sentinel2Source <|-- Sentinel2Theia3ASource
Spot67DRSSource: reflectance(kwargs)
Spot67DRSSource: cld_msk_drilled(nodata)
Sentinel2Source: msk_drilled(msk_dict, exp, nodata)
Sentinel2Theia2ASource: get_cld_drilled(nodata)
Sentinel2Theia3ASource: get_flg_drilled(keep_flags_values, nodata)
```
## Use cases
### Local ADS-DRS Spot-6/7 products (DIMAP format)
<details>
```python
sc = scenes.spot.from_dimaps(dim_xs='/path/to/DIM_SPOT6_XS_..._1.XML',
dim_pan='/path/to/DIM_SPOT6_P_..._1.XML')
print(sc.sources)
```
```bash
['xs', 'p', 'cld_msk', 'xs_cld_msk', 'pan_cld_msk']
```
</details>
### Remote ADS-DRS Spot-6/7 products from Dinamis STAC catalog
<details>
```python
scs = scenes.stac.search(url = "https://stacapi.147.100.200.143.nip.io"
collections = ["Spot-6-7-DRS-collection"],
bbox_wgs84=bbox)
print(scs[0].sources)
```
```bash
['xs', 'p', 'cld_msk', 'xs_cld_msk', 'pan_cld_msk']
```
</details>
### Remote IGN Spot-6/7 orthos from Dinamis STAC catalog
<details>
```python
scs = scenes.stac.search(url = "https://stacapi.147.100.200.143.nip.io"
collections = ["Spot-6-7-IGN-collection"],
bbox_wgs84=bbox)
print(scs[0].sources)
```
```bash
['xs', 'p']
```
</details>
### Sentinel-2 Level 2A from Microsoft Planetary Computer
<details>
```python
mpc_scs = scenes.stac.search(url="https://planetarycomputer.microsoft.com/api/stac/v1",
collections=["sentinel-2-l2a"],
bbox_wgs84=bbox)
print(mpc_scs[0].sources)
```
```bash
['10m_bands', '20m_bands', '60m_bands']
```
</details>
### Sentinel-2 Level 2A from Theia
<details>
```python
theia_scs = scenes.theia.search(level="2A",
bbox_wgs84=bbox)
print(theia_scs[0].sources)
```
```bash
['10m_bands', '20m_bands', '60m_bands', '10m_CLM', '20m_CLM', '10m_EDG', '20m_EDG']
```
</details>
### Sentinel-2 Level 3A from Theia
<details>
```python
theia_scs = scenes.theia.search(level="3A",
bbox_wgs84=bbox)
print(theia_scs[0].sources)
```
```bash
['10m_bands', '20m_bands', '60m_bands', '10m_FLG', '20m_FLG']
```
</details>
## Howto
### approche pragmatique
- [ ] add `get_10m_CLM`, `get_20m_CLM`, `get_10m_FLG`, `get_20m_FLG`, `get_10m_EDG`, `get_20m_EDG` in `Sentinel2Source`
- [ ] add `get_cld_msk` and `get_xs_cld_msk`, `get_pan_cld_msk` in `Spot67DRSSource`
- [x] add optional band indices in `get_10m_bands` and `get_20m_bands` in `Sentinel2Scene`
- [x] utiliser `sources_paths` pour garder les paths (urls, uris, filenames) des assets disponibles, summarizer cela dans les metadatas automatiquement (suppression des surcharges de `get_metadata` dans les classes héritant de `Scene`)
### idee, trop compliqué à mettre en place
- single Source class
- add `capabilities` attribute to `Source`: list of all inherited capabilities (e.g. `['drilled', 'masked', 'resample_over', 'clip_over_img', 'clip_over_vec', 'reproject']` for any basic `Source`)
- implement dynamic sources get (use a dict to store sources)https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/23Typo in reproject function2022-07-21T14:54:43+02:00Narcon NicolasTypo in reproject functionThe use of `reproject` function from `Source` class raises:
```
AttributeError: module 'pyotb' has no attribute 'Orthorectification'
```
It is a typo: it should be OrthoRectification instead of Orthorectification
Second error: it shou...The use of `reproject` function from `Source` class raises:
```
AttributeError: module 'pyotb' has no attribute 'Orthorectification'
```
It is a typo: it should be OrthoRectification instead of Orthorectification
Second error: it should be 'map.epsg.code' instead of 'map.epsg'
Third issue:
When initializing Source, we should be able to specify the output parameter key (https://gitlab.irstea.fr/umr-tetis/scenes/-/blob/develop/scenes/core.py#L66)
```diff
- def __init__(self, root_scene: Scene, out: str | pyotb.core.otbObject, parent: Source = None):
- ...
- super().__init__(app=pyotb.Input(out).pyotb_app if isinstance(out, str) else out, output_parameter_key="out")
+ def __init__(self, root_scene: Scene, out: str | pyotb.core.otbObject, parent: Source = None, output_parameter_key='out'):
+ ...
+ super().__init__(app=pyotb.Input(out).pyotb_app if isinstance(out, str) else out, output_parameter_key=output_parameter_key)
```https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/22Pyotb 1.4.1 has introduced a bug2022-07-13T10:28:00+02:00Narcon NicolasPyotb 1.4.1 has introduced a bug```python
from scenes import spot
sc = spot.Spot67Scene(dimap_file_xs='DIM_SPOT6_MS_201503261014386_ORT_SPOT6_20170524_1422391k0ha487979cy_1.XML', dimap_file_pan='DIM_SPOT6_P_201503261014386_ORT_SPOT6_20170524_1422391k0ha487979cy_1.XML'...```python
from scenes import spot
sc = spot.Spot67Scene(dimap_file_xs='DIM_SPOT6_MS_201503261014386_ORT_SPOT6_20170524_1422391k0ha487979cy_1.XML', dimap_file_pan='DIM_SPOT6_P_201503261014386_ORT_SPOT6_20170524_1422391k0ha487979cy_1.XML')
xs = sc.get_xs()
print(hasattr(xs, 'shape')) # prints False
xs.shape # Raises exception
```
Exception raised:
```
Traceback (most recent call last):
File "/home/otbuser/.local/lib/python3.8/site-packages/pyotb/core.py", line 245, in __getattr__
res = getattr(self.app, name)
File "/opt/otbtf/lib/otb/python/otbApplication.py", line 2833, in __getattr__
raise AttributeError("'%s' object has no attribute '%s'" % (Application.__name__, name))
AttributeError: 'Application' object has no attribute 'shape'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "tmp_bug_pyotb_dans_scenes.py", line 20, in <module>
print(xs.shape) ## Ceci fait planter le code
File "/home/otbuser/.local/lib/python3.8/site-packages/pyotb/core.py", line 248, in __getattr__
raise AttributeError(f'{self.name}: Could not find attribute `{name}`') from e
AttributeError: Output out from ExtractROI: Could not find attribute `shape`
```https://gitlab.irstea.fr/umr-tetis/scenes/-/issues/21Un peu rigide sur les dates...2022-07-11T16:58:50+02:00Cresson RemiUn peu rigide sur les dates...quand on crée une date avec any2datetime, on doit enlever les zéros devant les nombres. un peu contraignant...
par ex "01/01/2022" n'est pas accepté (il faut mettre "1/1/2022")
Il faudrait assouplir çaquand on crée une date avec any2datetime, on doit enlever les zéros devant les nombres. un peu contraignant...
par ex "01/01/2022" n'est pas accepté (il faut mettre "1/1/2022")
Il faudrait assouplir ça