Question about using local backend with multiple STAC collections

openEO.org
1

My goal is to download the data via the STAC collections, and via this backend, “process” it locally and then download/create it in .nc format.
I’m using pypi’s openeo 0.27.0 python client.

My code is:

import openeo # !pip install openeo
import matplotlib.pyplot as plt  # !pip install ipympl
import xarray   #!pip install xarray netCDF4 h5netcdf      !pip install "xarray[complete]"
import pprint #pip install pprint , pprint36
import numpy as np

connection = openeo.connect("http://localhost:8080/openeo/1.2/")
connection.authenticate_oidc_device()

def calculate_square_coordinates(lat_centro, lon_centro, lato_km):
    # Costanti
    RAGGIO_TERRA_KM = 6371  # Raggio approssimativo della Terra in km
    LAT_KM = 111  # Distanza in km per 1 grado di latitudine

    # Calcolo della variazione di latitudine e longitudine
    delta_lat = (lato_km / 2) / LAT_KM
    delta_lon = (lato_km / 2) / (RAGGIO_TERRA_KM * np.cos(np.radians(lat_centro)) * np.pi / 180)

    # Calcolo delle coordinate del quadrato
    return lon_centro - delta_lon, lat_centro - delta_lat, lon_centro + delta_lon, lat_centro + delta_lat

# Esempio di utilizzo
lat_centro, lon_centro = 40.16529635539171, 8.950717990455276
lato_km = 10.0

west, south, east, north = calculate_square_coordinates(lat_centro, lon_centro, lato_km)

s2_cube = connection.load_collection(
    "SENTINEL2_L2A_CREO",
    temporal_extent=("2022-05-01", "2022-05-30"),
    spatial_extent={
        "west": west ,  # longitudine minimaa
        "south": south , # latitudine minima
        "east": east,  # longitudine massimama
        "north":  north, # latitudine massimama
        #"crs": "EPSG:4326",
    },
    bands=["B04", "B03", "B02", "SCL"],
    max_cloud_cover=100,
)

scl_band = s2_cube.band("SCL")
cloud_mask= (scl_band == 3) | (scl_band == 7) | (scl_band == 8) | (scl_band == 9) | (scl_band == 10) 

cloud_mask = cloud_mask.resample_cube_spatial(s2_cube)
cube_masked = s2_cube.mask(cloud_mask)
composite_masked = cube_masked.min_time()
composite_masked.download("composite-masked.nc")

ds = xarray.load_dataset( 'composite-masked.nc')
# Convert xarray DataSet to a (bands, x, y) DataArray
data = ds[["B04", "B03", "B02"]].to_array(dim="bands")

fig, ax = plt.subplots(ncols=1, figsize=(4, 4), dpi=90)
data.plot.imshow(vmin=0, vmax=2000, ax=ax);

And give me this error:

{
	"name": "OpenEoApiError",
	"message": "[500] Internal: Server error: java.io.IOException: Exception while evaluating catalog request https://finder.creodias.eu/api/collections/Sentinel2/search.json?box=8.891653321417749%2C40.12016403425433%2C9.009636046744415%2C40.21039972384912&sortParam=startDate&sortOrder=ascending&page=1&maxRecords=100&status=ONLINE&dataset=ESA-DATASET&cloudCover=%5B0%2C100%5D&startDate=2022-05-01T00%3A00%3A00Z&completionDate=2022-05-30T00%3A00%3A00Z:  (ref: r-2402069dc62a478a811db7770dea3da9)",
	"stack": "---------------------------------------------------------------------------
OpenEoApiError                            Traceback (most recent call last)
Cell In[20], line 50
     46 cube_masked = s2_cube.mask(cloud_mask)
     48 composite_masked = cube_masked.min_time()
---> 50 composite_masked.download('composite-masked.nc')
     52 ds = xarray.load_dataset( 'composite-masked.nc')
     53 # Convert xarray DataSet to a (bands, x, y) DataArray

File ~/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/datacube.py:2102, in DataCube.download(self, outputfile, format, options, validate)
   2100     format = guess_format(outputfile)
   2101 cube = self._ensure_save_result(format=format, options=options)
-> 2102 return self._connection.download(cube.flat_graph(), outputfile, validate=validate)

File ~/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/connection.py:1559, in Connection.download(self, graph, outputfile, timeout, validate)
   1557 pg_with_metadata = self._build_request_with_process_graph(process_graph=graph)
   1558 self._preflight_validation(pg_with_metadata=pg_with_metadata, validate=validate)
-> 1559 response = self.post(
   1560     path=\"/result\",
   1561     json=pg_with_metadata,
   1562     expected_status=200,
   1563     stream=True,
   1564     timeout=timeout or DEFAULT_TIMEOUT_SYNCHRONOUS_EXECUTE,
   1565 )
   1567 if outputfile is not None:
   1568     with Path(outputfile).open(mode=\"wb\") as f:

File ~/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/connection.py:230, in RestApiConnection.post(self, path, json, **kwargs)
    222 def post(self, path: str, json: Optional[dict] = None, **kwargs) -> Response:
    223     \"\"\"
    224     Do POST request to REST API.
    225 
   (...)
    228     :return: response: Response
    229     \"\"\"
--> 230     return self.request(\"post\", path=path, json=json, allow_redirects=False, **kwargs)

File ~/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/connection.py:769, in Connection.request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    762     return super(Connection, self).request(
    763         method=method, path=path, headers=headers, auth=auth,
    764         check_error=check_error, expected_status=expected_status, **kwargs,
    765     )
    767 try:
    768     # Initial request attempt
--> 769     return _request()
    770 except OpenEoApiError as api_exc:
    771     if api_exc.http_status_code in {401, 403} and api_exc.code == \"TokenInvalid\":
    772         # Auth token expired: can we refresh?

File ~/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/connection.py:762, in Connection.request.<locals>._request()
    761 def _request():
--> 762     return super(Connection, self).request(
    763         method=method, path=path, headers=headers, auth=auth,
    764         check_error=check_error, expected_status=expected_status, **kwargs,
    765     )

File ~/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/connection.py:168, in RestApiConnection.request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    166 expected_status = ensure_list(expected_status) if expected_status else []
    167 if check_error and status >= 400 and status not in expected_status:
--> 168     self._raise_api_error(resp)
    169 if expected_status and status not in expected_status:
    170     raise OpenEoRestError(\"Got status code {s!r} for `{m} {p}` (expected {e!r}) with body {body}\".format(
    171         m=method.upper(), p=path, s=status, e=expected_status, body=resp.text)
    172     )

File ~/.pyenv/versions/3.10.13/envs/venv/lib/python3.10/site-packages/openeo/rest/connection.py:188, in RestApiConnection._raise_api_error(self, response)
    186     error_message = info.get(\"message\")
    187     if error_code and isinstance(error_code, str) and error_message and isinstance(error_message, str):
--> 188         raise OpenEoApiError(
    189             http_status_code=status_code,
    190             code=error_code,
    191             message=error_message,
    192             id=info.get(\"id\"),
    193             url=info.get(\"url\"),
    194         )
    196 # Failed to parse it as a compliant openEO API error: show body as-is in the exception.
    197 text = response.text

OpenEoApiError: [500] Internal: Server error: java.io.IOException: Exception while evaluating catalog request https://finder.creodias.eu/api/collections/Sentinel2/search.json?box=8.891653321417749%2C40.12016403425433%2C9.009636046744415%2C40.21039972384912&sortParam=startDate&sortOrder=ascending&page=1&maxRecords=100&status=ONLINE&dataset=ESA-DATASET&cloudCover=%5B0%2C100%5D&startDate=2022-05-01T00%3A00%3A00Z&completionDate=2022-05-30T00%3A00%3A00Z:  (ref: r-2402069dc62a478a811db7770dea3da9)"
}

How do I solve this problem?
In case I want, via this backend, how to do them correctly (without these errors):

  1. I want to authenticate to a specific STAC collection(e.g. copernicus), as I specify in my code
  2. Download raw images from the STAC collection
  3. “Process” it locally and then create the processed file in .nc format.

INFO : win11, vs code, wsl2, ubuntu

2

what backend are you running locally here?

There error you get is an “[500] Internal” error, so an unexpected back-end-side problem. You probably have to check your local server logs for more information

3

The corresponding code gives(in the terminal) me this log in the backend(openeo-geopyspark-driver),I remove the first messages, otherwise it won’t fit in the message:

{"message": "apply_process mask with {'data': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}, 'mask': {'from_node': 'resamplecubespatial1', 'node': {'process_id': 'resample_cube_spatial', 'arguments': {'data': {'from_node': 'reducedimension1', 'node': {'process_id': 'reduce_dimension', 'arguments': {'data': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}, 'dimension': 'bands', 'reducer': {'process_graph': {'arrayelement1': {'process_id': 'array_element', 'arguments': {'data': {'from_parameter': 'data'}, 'index': 3}}, 'eq1': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 3}}, 'eq2': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 7}}, 'or1': {'process_id': 'or', 'arguments': {'x': {'from_node': 'eq1'}, 'y': {'from_node': 'eq2'}}}, 'eq3': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 8}}, 'or2': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or1'}, 'y': {'from_node': 'eq3'}}}, 'eq4': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 9}}, 'or3': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or2'}, 'y': {'from_node': 'eq4'}}}, 'eq5': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 10}}, 'or4': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or3'}, 'y': {'from_node': 'eq5'}}, 'result': True}}}}}}, 'method': 'near', 'target': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}}}}}", "levelname": "DEBUG", "name": "openeo_driver.ProcessGraphDeserializer", "created": 1707313859.884568, "filename": "ProcessGraphDeserializer.py", "lineno": 1531, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "data_mask: convert_node(mask_node): {'from_node': 'resamplecubespatial1', 'node': {'process_id': 'resample_cube_spatial', 'arguments': {'data': {'from_node': 'reducedimension1', 'node': {'process_id': 'reduce_dimension', 'arguments': {'data': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}, 'dimension': 'bands', 'reducer': {'process_graph': {'arrayelement1': {'process_id': 'array_element', 'arguments': {'data': {'from_parameter': 'data'}, 'index': 3}}, 'eq1': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 3}}, 'eq2': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 7}}, 'or1': {'process_id': 'or', 'arguments': {'x': {'from_node': 'eq1'}, 'y': {'from_node': 'eq2'}}}, 'eq3': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 8}}, 'or2': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or1'}, 'y': {'from_node': 'eq3'}}}, 'eq4': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 9}}, 'or3': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or2'}, 'y': {'from_node': 'eq4'}}}, 'eq5': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 10}}, 'or4': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or3'}, 'y': {'from_node': 'eq5'}}, 'result': True}}}}}}, 'method': 'near', 'target': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}}}}", "levelname": "DEBUG", "name": "openeo_driver.ProcessGraphDeserializer", "created": 1707313859.884758, "filename": "ProcessGraphDeserializer.py", "lineno": 1538, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "apply_process resample_cube_spatial with {'data': {'from_node': 'reducedimension1', 'node': {'process_id': 'reduce_dimension', 'arguments': {'data': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}, 'dimension': 'bands', 'reducer': {'process_graph': {'arrayelement1': {'process_id': 'array_element', 'arguments': {'data': {'from_parameter': 'data'}, 'index': 3}}, 'eq1': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 3}}, 'eq2': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 7}}, 'or1': {'process_id': 'or', 'arguments': {'x': {'from_node': 'eq1'}, 'y': {'from_node': 'eq2'}}}, 'eq3': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 8}}, 'or2': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or1'}, 'y': {'from_node': 'eq3'}}}, 'eq4': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 9}}, 'or3': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or2'}, 'y': {'from_node': 'eq4'}}}, 'eq5': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 10}}, 'or4': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or3'}, 'y': {'from_node': 'eq5'}}, 'result': True}}}}}}, 'method': 'near', 'target': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}}", "levelname": "DEBUG", "name": "openeo_driver.ProcessGraphDeserializer", "created": 1707313859.884823, "filename": "ProcessGraphDeserializer.py", "lineno": 1531, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "apply_process reduce_dimension with {'data': {'from_node': 'loadcollection1', 'node': {'process_id': 'load_collection', 'arguments': {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}}}, 'dimension': 'bands', 'reducer': {'process_graph': {'arrayelement1': {'process_id': 'array_element', 'arguments': {'data': {'from_parameter': 'data'}, 'index': 3}}, 'eq1': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 3}}, 'eq2': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 7}}, 'or1': {'process_id': 'or', 'arguments': {'x': {'from_node': 'eq1'}, 'y': {'from_node': 'eq2'}}}, 'eq3': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 8}}, 'or2': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or1'}, 'y': {'from_node': 'eq3'}}}, 'eq4': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 9}}, 'or3': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or2'}, 'y': {'from_node': 'eq4'}}}, 'eq5': {'process_id': 'eq', 'arguments': {'x': {'from_node': 'arrayelement1'}, 'y': 10}}, 'or4': {'process_id': 'or', 'arguments': {'x': {'from_node': 'or3'}, 'y': {'from_node': 'eq5'}}, 'result': True}}}}", "levelname": "DEBUG", "name": "openeo_driver.ProcessGraphDeserializer", "created": 1707313859.8848789, "filename": "ProcessGraphDeserializer.py", "lineno": 1531, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "apply_process load_collection with {'bands': ['B04', 'B03', 'B02', 'SCL'], 'id': 'SENTINEL2_L2A', 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754}, 'temporal_extent': ['2022-05-01', '2022-05-30']}", "levelname": "DEBUG", "name": "openeo_driver.ProcessGraphDeserializer", "created": 1707313859.884921, "filename": "ProcessGraphDeserializer.py", "lineno": 1531, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Using process 'load_collection' from namespace 'backend'.", "levelname": "INFO", "name": "openeo_driver.ProcessGraphDeserializer", "created": 1707313859.884959, "filename": "ProcessGraphDeserializer.py", "lineno": 1585, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "load_collection: start 2024-02-07 14:50:59.886146", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313859.886194, "filename": "util.py", "lineno": 341, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Creating layer for SENTINEL2_L2A with load params {'temporal_extent': ('2022-05-01', '2022-05-30'), 'spatial_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754, 'crs': 'EPSG:4326'}, 'global_extent': {'west': 8.891876254269395, 'south': 40.12025131034667, 'east': 9.009559726641157, 'north': 40.210341400436754, 'crs': 'EPSG:4326'}, 'bands': ['B04', 'B03', 'B02', 'SCL'], 'properties': {}, 'aggregate_spatial_geometries': None, 'sar_backscatter': None, 'process_types': {<ProcessType.GLOBAL_TIME: 4>, <ProcessType.FOCAL_SPACE: 6>}, 'custom_mask': {}, 'data_mask': None, 'target_crs': None, 'target_resolution': None, 'resample_method': 'near', 'pixel_buffer': None}", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313859.8874626, "filename": "layercatalog.py", "lineno": 136, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Resolving 'merged_by_common_name' collection SENTINEL2_L2A", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313859.8895512, "filename": "layercatalog.py", "lineno": 168, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Check missing products for TERRASCOPE_S2_TOC_V2 using {'method': 'terrascope', 'creo_catalog': {'mission': 'Sentinel2', 'level': 'LEVEL1C'}}", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313859.893609, "filename": "layercatalog.py", "lineno": 1028, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Expected tiles (0): set()", "levelname": "DEBUG", "name": "openeogeotrellis.layercatalog", "created": 1707313860.409731, "filename": "layercatalog.py", "lineno": 1071, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Oscar tiles (19): {('32TNK', '20220523'), ('32TMK', '20220508'), ('32TMK', '20220513'), ('32TNK', '20220503'), ('32TMK', '20220526'), ('32TNK', '20220508'), ('32TNK', '20220513'), ('32TMK', '20220528'), ('32TMK', '2...", "levelname": "DEBUG", "name": "openeogeotrellis.layercatalog", "created": 1707313864.749411, "filename": "layercatalog.py", "lineno": 1085, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "(common_name) 'SENTINEL2_L2A': using 'terrascope'.", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313864.7496417, "filename": "layercatalog.py", "lineno": 737, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Resolved 'merged_by_common_name' to collection TERRASCOPE_S2_TOC_V2", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313864.74974, "filename": "layercatalog.py", "lineno": 175, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Layer source type: 'file-s2'", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313864.749877, "filename": "layercatalog.py", "lineno": 187, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "band_indices: [3, 2, 1, 11]", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313864.7504988, "filename": "layercatalog.py", "lineno": 198, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Correlation ID is 'r-240207d5ab484e12b3d0e2e27cd0e418'", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313864.7508347, "filename": "layercatalog.py", "lineno": 219, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Detected process types:{<ProcessType.GLOBAL_TIME: 4>, <ProcessType.FOCAL_SPACE: 6>}", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313864.75095, "filename": "layercatalog.py", "lineno": 221, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Using load_params.data_mask None", "levelname": "DEBUG", "name": "openeogeotrellis.layercatalog", "created": 1707313864.7795308, "filename": "layercatalog.py", "lineno": 101, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "loading pyramid file-s2", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313864.781818, "filename": "layercatalog.py", "lineno": 621, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "load_collection: fail 2024-02-07 14:51:08.787166, elapsed 0:00:08.901020", "levelname": "INFO", "name": "openeogeotrellis.layercatalog", "created": 1707313868.787294, "filename": "util.py", "lineno": 347, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "exception chain classes: java.io.IOException caused by java.io.FileNotFoundException", "levelname": "DEBUG", "name": "openeogeotrellis.backend", "created": 1707313868.7897794, "filename": "backend.py", "lineno": 1326, "process": 4960, "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "Py4JJavaError('An error occurred while calling o524.datacube_seq.\\n', JavaObject id=o526)", "levelname": "ERROR", "name": "openeo_driver.views.error", "created": 1707313868.7904215, "filename": "views.py", "lineno": 278, "process": 4960, "exc_info": "Traceback (most recent call last):\n  File \"/home/dio/openeo-geopyspark-driver/venv/lib/python3.8/site-packages/flask/app.py\", line 1516, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/home/dio/openeo-geopyspark-driver/venv/lib/python3.8/site-packages/flask/app.py\", line 1502, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)\n  File \"/home/dio/openeo-python-driver/openeo_driver/users/auth.py\", line 88, in decorated\n    return f(*args, **kwargs)\n  File \"/home/dio/openeo-python-driver/openeo_driver/views.py\", line 655, in result\n    result = backend_implementation.processing.evaluate(process_graph=process_graph, env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 301, in evaluate\n    return evaluate(process_graph=process_graph, env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 373, in evaluate\n    result = convert_node(result_node, env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 398, in convert_node\n    process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}),\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in apply_process\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in <dictcomp>\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 412, in convert_node\n    return convert_node(processGraph['node'], env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 398, in convert_node\n    process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}),\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in apply_process\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in <dictcomp>\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 412, in convert_node\n    return convert_node(processGraph['node'], env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 398, in convert_node\n    process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}),\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1539, in apply_process\n    the_mask = convert_node(mask_node, env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 412, in convert_node\n    return convert_node(processGraph['node'], env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 398, in convert_node\n    process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}),\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in apply_process\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in <dictcomp>\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 412, in convert_node\n    return convert_node(processGraph['node'], env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 398, in convert_node\n    process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}),\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in apply_process\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1558, in <dictcomp>\n    args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())}\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 412, in convert_node\n    return convert_node(processGraph['node'], env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 398, in convert_node\n    process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}),\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 1590, in apply_process\n    return process_function(args=ProcessArgs(args, process_id=process_id), env=env)\n  File \"/home/dio/openeo-python-driver/openeo_driver/ProcessGraphDeserializer.py\", line 581, in load_collection\n    return env.backend_implementation.catalog.load_collection(collection_id, load_params=load_params, env=env)\n  File \"/home/dio/openeo-python-client/openeo/util.py\", line 361, in wrapper\n    return f(*args, **kwargs)\n  File \"/home/dio/openeo-geopyspark-driver/openeogeotrellis/layercatalog.py\", line 132, in load_collection\n    return self._load_collection_cached(collection_id, load_params, WhiteListEvalEnv(env,WHITELIST))\n  File \"/home/dio/openeo-geopyspark-driver/openeogeotrellis/layercatalog.py\", line 624, in _load_collection_cached\n    pyramid = file_s2_pyramid()\n  File \"/home/dio/openeo-geopyspark-driver/openeogeotrellis/layercatalog.py\", line 330, in file_s2_pyramid\n    return file_pyramid(pyramid_factory)\n  File \"/home/dio/openeo-geopyspark-driver/openeogeotrellis/layercatalog.py\", line 386, in file_pyramid\n    return create_pyramid(factory)\n  File \"/home/dio/openeo-geopyspark-driver/openeogeotrellis/layercatalog.py\", line 350, in create_pyramid\n    return factory.datacube_seq(\n  File \"/home/dio/openeo-geopyspark-driver/venv/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1321, in __call__\n    return_value = get_return_value(\n  File \"/home/dio/openeo-geopyspark-driver/venv/lib/python3.8/site-packages/py4j/protocol.py\", line 326, in get_return_value\n    raise Py4JJavaError(\npy4j.protocol.Py4JJavaError: An error occurred while calling o524.datacube_seq.\n: java.io.IOException: Exception while determining data type of collection urn:eop:VITO:TERRASCOPE_S2_TOC_V2 and item /data/MTDA/TERRASCOPE_Sentinel2/TOC_V2/2022/05/01/S2A_20220501T101601_32TMK_TOC_V210/S2A_20220501T101601_32TMK_TOC-B04_10M_V210.tif. Detailed message: /data/MTDA/TERRASCOPE_Sentinel2/TOC_V2/2022/05/01/S2A_20220501T101601_32TMK_TOC_V210/S2A_20220501T101601_32TMK_TOC-B04_10M_V210.tif (No such file or directory)\n\tat org.openeo.geotrellis.layers.FileLayerProvider.determineCelltype(FileLayerProvider.scala:728)\n\tat org.openeo.geotrellis.layers.FileLayerProvider.readKeysToRasterSources(FileLayerProvider.scala:758)\n\tat org.openeo.geotrellis.layers.FileLayerProvider.readMultibandTileLayer(FileLayerProvider.scala:957)\n\tat org.openeo.geotrellis.file.PyramidFactory.datacube(PyramidFactory.scala:128)\n\tat org.openeo.geotrellis.file.PyramidFactory.datacube_seq(PyramidFactory.scala:91)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)\n\tat py4j.ClientServerConnection.run(ClientServerConnection.java:106)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: java.io.FileNotFoundException: /data/MTDA/TERRASCOPE_Sentinel2/TOC_V2/2022/05/01/S2A_20220501T101601_32TMK_TOC_V210/S2A_20220501T101601_32TMK_TOC-B04_10M_V210.tif (No such file or directory)\n\tat java.base/java.io.FileInputStream.open0(Native Method)\n\tat java.base/java.io.FileInputStream.open(FileInputStream.java:219)\n\tat java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)\n\tat geotrellis.util.FileRangeReader.readClippedRange(FileRangeReader.scala:34)\n\tat geotrellis.util.RangeReader.readRange(RangeReader.scala:42)\n\tat geotrellis.util.RangeReader.readRange$(RangeReader.scala:41)\n\tat geotrellis.util.FileRangeReader.readRange(FileRangeReader.scala:30)\n\tat geotrellis.util.StreamingByteReader.readChunk(StreamingByteReader.scala:99)\n\tat geotrellis.util.StreamingByteReader.ensureChunk(StreamingByteReader.scala:112)\n\tat geotrellis.util.StreamingByteReader.get(StreamingByteReader.scala:130)\n\tat geotrellis.raster.io.geotiff.reader.GeoTiffInfo$.read(GeoTiffInfo.scala:127)\n\tat geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readMultiband(GeoTiffReader.scala:211)\n\tat geotrellis.raster.geotiff.GeoTiffResampleRasterSource.$anonfun$tiff$1(GeoTiffResampleRasterSource.scala:45)\n\tat scala.Option.getOrElse(Option.scala:189)\n\tat geotrellis.raster.geotiff.GeoTiffResampleRasterSource.tiff$lzycompute(GeoTiffResampleRasterSource.scala:42)\n\tat geotrellis.raster.geotiff.GeoTiffResampleRasterSource.tiff(GeoTiffResampleRasterSource.scala:39)\n\tat geotrellis.raster.geotiff.GeoTiffResampleRasterSource.$anonfun$cellType$1(GeoTiffResampleRasterSource.scala:49)\n\tat scala.Option.getOrElse(Option.scala:189)\n\tat geotrellis.raster.geotiff.GeoTiffResampleRasterSource.cellType(GeoTiffResampleRasterSource.scala:49)\n\tat org.openeo.geotrellis.layers.BandCompositeRasterSource.$anonfun$cellType$1(FileLayerProvider.scala:92)\n\tat cats.data.NonEmptyList.map(NonEmptyList.scala:87)\n\tat org.openeo.geotrellis.layers.BandCompositeRasterSource.cellType(FileLayerProvider.scala:92)\n\tat org.openeo.geotrellis.layers.FileLayerProvider.determineCelltype(FileLayerProvider.scala:722)\n\t... 16 more\n", "req_id": "r-240207d5ab484e12b3d0e2e27cd0e418", "user_id": "22615bb01984f64a614f19cdfe73de14250148c287c72deed89ed1ec73040149@egi.eu"}
{"message": "127.0.0.1 - - [07/Feb/2024:14:51:08 +0100] \"POST /openeo/1.2/result HTTP/1.1\" 500 294 \"-\" \"openeo-python-client/0.27.0 cpython/3.10.13 linux\"", "levelname": "INFO", "name": "gunicorn.access", "created": 1707313868.8135061, "filename": "glogging.py", "lineno": 363, "process": 4960, "req_id": "no-request", "user_id": null}
4

I’m writing another message to better describe the latest messages where the errors are found:

  • Message: “Using load_params.data_mask None”
  • Level: DEBUG
  • Meaning: This message indicates that during the data loading process, no data mask (data_mask) was specified in the load parameters (load_params). In other words, no filter mask was applied to the data while it was being loaded. This may be the default behavior or it may indicate that the user did not specify a mask.
  • Message: “loading pyramid file-s2”
  • Level: INFO
  • Meaning: This message informs you that the system is beginning the process of loading a pyramid of files, specifically Sentinel-2 data (file-s2). File pyramids are a way to efficiently organize and access raster data at different levels of detail. This step is crucial for preparing the data for subsequent analysis operations.
  • Message: “load_collection: fail 2024-02-07 14:51:08.787166, elapsed 0:00:08.901020”
  • Level: INFO
  • Meaning: This message reports the failure of the data collection loading process, specifying the exact moment of failure and the time taken since the operation began. The fact that it is recorded as INFO and not as ERROR could indicate that the system handled the failure correctly, recording the event for further analysis.
  • Message: “exception chain classes: java.io.IOException caused by java.io.FileNotFoundException”
  • Level: DEBUG
  • Meaning: This message provides technical details about the error that caused the data load to fail, specifying that the root exception is java.io.FileNotFoundException, which in turn triggered a java. io.IOException. This confirms that the system was unable to find or access one or more files needed to complete the upload process.
  • Message: “Py4JJavaError(‘An error occurred while calling o798.datacube_seq.\n’, JavaObject id=o800)”
  • Level: ERROR
  • Meaning: This message indicates an error during a call to the datacube_seq method using Py4J, a library that allows communication between Python and Java. The Py4JJavaError reports that there was a problem invoking a method on a Java object, specifically the object identified as o798. The root cause of this error is related to an attempt to determine the data type of a collection and a specific item, which failed because the requested file was not found. The detailed error message provides the path to the missing file, which is similar to the error discussed previously, indicating that the main problem is the absence of the specified file.
5

this is duplicate of [500] Internal: Server error: java.io.IOException when a run download command. · Issue #680 · Open-EO/openeo-geopyspark-driver · GitHub
not sure where to continue the discussion

6

delete it if you want