Atmospheric Correction Force

openEO Platform
1

I am trying to apply the atmospheric_correction_method = force but it says "Process with identifier ‘ard_surface_reflectance’ is not available in namespace ‘backend’."

datacube = connection.load_collection(
    'SENTINEL2_L1C_SENTINELHUB',
    spatial_extent  = {'west':bbox[0],'east':bbox[2],'south':bbox[1],'north':bbox[3]},
    temporal_extent = ["2022-01-21", "2022-01-21"],
    bands = ['B04', 'B03', 'B02', 'B09', 'B8A', 'B11', 'sunAzimuthAngles', 'sunZenithAngles', 'viewAzimuthMean', 'viewZenithMean']
)


l2a = datacube.ard_surface_reflectance(atmospheric_correction_method = 'FORCE', cloud_detection_method = 'Fmask')
s2_GTiff = l2a.save_result(format="GTiff")

job_s2 = s2_GTiff.send_job()
job_id_s2 = job_s2.job_id
results = job_s2.start_and_wait().get_results()

I am getting this error:

0:00:00 Job 'vito-5bf34afc-9447-4123-9a15-f8e221a84c8f': send 'start'
---------------------------------------------------------------------------
OpenEoApiError                            Traceback (most recent call last)
Input In [61], in <module>
      4 job_s2 = s2_GTiff.send_job()
      5 job_id_s2 = job_s2.job_id
----> 6 results = job_s2.start_and_wait().get_results()

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\job.py:164, in RESTJob.start_and_wait(self, print, max_poll_interval, connection_retry_interval, soft_error_max)
    162 # TODO: make `max_poll_interval`, `connection_retry_interval` class constants or instance properties?
    163 print_status("send 'start'")
--> 164 self.start_job()
    166 # TODO: also add  `wait` method so you can track a job that already has started explicitly
    167 #   or just rename this method to `wait` and automatically do start if not started yet?
    168 
    169 # Start with fast polling.
    170 poll_interval = min(5, max_poll_interval)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\job.py:71, in RESTJob.start_job(self)
     69 """ Start / queue a job for processing."""
     70 # POST /jobs/{job_id}/results
---> 71 self.connection.post("/jobs/{}/results".format(self.job_id), expected_status=202)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:172, in RestApiConnection.post(self, path, json, **kwargs)
    164 def post(self, path, json: dict = None, **kwargs) -> Response:
    165     """
    166     Do POST request to REST API.
    167 
   (...)
    170     :return: response: Response
    171     """
--> 172     return self.request("post", path=path, json=json, allow_redirects=False, **kwargs)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:120, in RestApiConnection.request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    118 expected_status = ensure_list(expected_status) if expected_status else []
    119 if check_error and status >= 400 and status not in expected_status:
--> 120     self._raise_api_error(resp)
    121 if expected_status and status not in expected_status:
    122     raise OpenEoRestError("Got status code {s!r} for `{m} {p}` (expected {e!r})".format(
    123         m=method.upper(), p=path, s=status, e=expected_status)
    124     )

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:151, in RestApiConnection._raise_api_error(self, response)
    149     else:
    150         exception = OpenEoApiError(http_status_code=status_code, message=text)
--> 151 raise exception

Process with identifier 'ard_surface_reflectance' is not available in namespace 'backend'.
2

Hi Andrea,
to use force, you’ll need to start from an EODC layer, like: SENTINEL2_L1C.

The layer you’re trying is hosted by the Terrascope backend, and only supports SMAC and iCor.
This is also illustrated here:

best regards!

3

I have tried also with that ENTINEL2_L1C :

datacube = connection.load_collection(
    'SENTINEL2_L1C',
    spatial_extent = {'west':bbox[0],'east':bbox[2],'south':bbox[1],'north':bbox[3]},
    temporal_extent = ["2022-01-01", "2022-01-31"],
    bands = ['B01', 'B02', 'B03'])

l2a = datacube.ard_surface_reflectance(atmospheric_correction_method = 'FORCE', cloud_detection_method = 'Fmask')
s2_GTiff = l2a.save_result(format="GTiff")

but getting this error:

0:00:00 Job 'eodc-jb-eaeac586-3b90-4fbe-b5c6-cc4c0cdcf4dd': send 'start'
---------------------------------------------------------------------------
OpenEoApiError                            Traceback (most recent call last)
Input In [26], in <module>
      1 job_s2 = s2_GTiff.send_job()
      2 job_id_s2 = job_s2.job_id
----> 3 results = job_s2.start_and_wait().get_results()

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\job.py:164, in RESTJob.start_and_wait(self, print, max_poll_interval, connection_retry_interval, soft_error_max)
    162 # TODO: make `max_poll_interval`, `connection_retry_interval` class constants or instance properties?
    163 print_status("send 'start'")
--> 164 self.start_job()
    166 # TODO: also add  `wait` method so you can track a job that already has started explicitly
    167 #   or just rename this method to `wait` and automatically do start if not started yet?
    168 
    169 # Start with fast polling.
    170 poll_interval = min(5, max_poll_interval)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\job.py:71, in RESTJob.start_job(self)
     69 """ Start / queue a job for processing."""
     70 # POST /jobs/{job_id}/results
---> 71 self.connection.post("/jobs/{}/results".format(self.job_id), expected_status=202)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:172, in RestApiConnection.post(self, path, json, **kwargs)
    164 def post(self, path, json: dict = None, **kwargs) -> Response:
    165     """
    166     Do POST request to REST API.
    167 
   (...)
    170     :return: response: Response
    171     """
--> 172     return self.request("post", path=path, json=json, allow_redirects=False, **kwargs)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:120, in RestApiConnection.request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    118 expected_status = ensure_list(expected_status) if expected_status else []
    119 if check_error and status >= 400 and status not in expected_status:
--> 120     self._raise_api_error(resp)
    121 if expected_status and status not in expected_status:
    122     raise OpenEoRestError("Got status code {s!r} for `{m} {p}` (expected {e!r})".format(
    123         m=method.upper(), p=path, s=status, e=expected_status)
    124     )

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:151, in RestApiConnection._raise_api_error(self, response)
    149     else:
    150         exception = OpenEoApiError(http_status_code=status_code, message=text)
--> 151 raise exception

OpenEoApiError: [500] unknown: [500] 500: file_dem must be of type <class 'str'>. Received file_dem: <class 'str'>.
4

When I have use the commands FORCE toolbox (provided by EODC)

datacube = connection.load_collection(
    'SENTINEL2_L1C',
    spatial_extent = {'west': 3.75, 'east': 4.08, 'south': 51.29, 'north': 51.39},
    temporal_extent = ['2017-03-07', '2017-03-07'],
)
datacube.ard_surface_reflectance(atmospheric_correction_method = 'FORCE', cloud_detection_method = 'Fmask').download('force.tif', format = 'GTiff')

does not work so probably it will be good to have a quick look as well

**OpenEoApiError: [500] Internal: Failed to process synchronously on backend eodc: OpenEoApiError("[500] 500: 'bands'")**

I have also inserted the bands:

datacube = connection.load_collection(
    'SENTINEL2_L1C',
    spatial_extent  = {'west':bbox[0],'east':bbox[2],'south':bbox[1],'north':bbox[3]},
    temporal_extent = ("2022-01-21", "2022-01-22"),
    bands           = ["B02","B03","B04"],
)
job = datacube.ard_surface_reflectance(atmospheric_correction_method = 'FORCE', cloud_detection_method = 'Fmask')



job_s2 = job.send_job()
job_id_s2 = job_s2.job_id
results = job_s2.start_and_wait().get_results()

but it shows then another error:

0:00:00 Job 'eodc-jb-2b8e5f10-85d1-42cb-a711-8fc5f1ad631e': send 'start'
---------------------------------------------------------------------------
OpenEoApiError                            Traceback (most recent call last)
Input In [40], in <module>
     10 job_s2 = job.send_job()
     11 job_id_s2 = job_s2.job_id
---> 12 results = job_s2.start_and_wait().get_results()

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\job.py:164, in RESTJob.start_and_wait(self, print, max_poll_interval, connection_retry_interval, soft_error_max)
    162 # TODO: make `max_poll_interval`, `connection_retry_interval` class constants or instance properties?
    163 print_status("send 'start'")
--> 164 self.start_job()
    166 # TODO: also add  `wait` method so you can track a job that already has started explicitly
    167 #   or just rename this method to `wait` and automatically do start if not started yet?
    168 
    169 # Start with fast polling.
    170 poll_interval = min(5, max_poll_interval)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\job.py:71, in RESTJob.start_job(self)
     69 """ Start / queue a job for processing."""
     70 # POST /jobs/{job_id}/results
---> 71 self.connection.post("/jobs/{}/results".format(self.job_id), expected_status=202)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:172, in RestApiConnection.post(self, path, json, **kwargs)
    164 def post(self, path, json: dict = None, **kwargs) -> Response:
    165     """
    166     Do POST request to REST API.
    167 
   (...)
    170     :return: response: Response
    171     """
--> 172     return self.request("post", path=path, json=json, allow_redirects=False, **kwargs)

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:120, in RestApiConnection.request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    118 expected_status = ensure_list(expected_status) if expected_status else []
    119 if check_error and status >= 400 and status not in expected_status:
--> 120     self._raise_api_error(resp)
    121 if expected_status and status not in expected_status:
    122     raise OpenEoRestError("Got status code {s!r} for `{m} {p}` (expected {e!r})".format(
    123         m=method.upper(), p=path, s=status, e=expected_status)
    124     )

File ~\Miniconda3\envs\openeo\lib\site-packages\openeo\rest\connection.py:151, in RestApiConnection._raise_api_error(self, response)
    149     else:
    150         exception = OpenEoApiError(http_status_code=status_code, message=text)
--> 151 raise exception

OpenEoApiError: [500] unknown: [500] 500: file_dem must be of type <class 'str'>. Received file_dem: <class 'str'>.
5

@sean.hoyal Can you have a look?

6

@jeroen.dries Yeah, will do!

7

Hi @sulova.andrea!

Below is a working example (I have just tested on the EODC backend directly) of what I think you are trying to achieve, and following some explanation as to why!

datacube = eodc.load_collection(
    'SENTINEL2_L1C',
    spatial_extent = {'west': 3.75, 'east': 4.08, 'south': 51.29, 'north': 51.39},
    temporal_extent = ['2017-03-07', '2017-03-08'],
    bands = [
            "B02",
            "B03",
            "B04"
    ]
)

force_do_topo = datacube.ard_surface_reflectance(atmospheric_correction_method = 'FORCE', cloud_detection_method = 'Fmask', elevation_model = "cop-dem-30m")

ard_tif = force_do_topo.save_result(format="GTiff")

job_eodc_ard = eodc.create_job(ard_tif.graph)

The initial example you posted tries to post the Level 1 job in an asynchronous manner, unfortunately we aren’t able to handle these jobs quite so speedily yet! So, you need to first create the process graph on the client side, and then send it to the backend for processing. For future reference to carry out force processing, I recommend using the 20MGRS notebook. (I will look to get the reference you used in the cloud docs updated!)

Do you have any feedback in terms of how we could make the force processing a little clearer?

If the example doesn’t help, do let me know!