SENTINEL2_L1C ARD processing - No data found

[mitro.muller]

Greetings

I am trying to ARD process SENTINEL2_L1C data for a small area in Southern Sweden surrounding an eddy covariance tower. The bounding box information defined in the Swedish national CRS and stored in a list:

> bbox_h
$west
[1] 401150
$south
[1] 6217572
$east
[1] 402124.3
$north
[1] 6218560
$crs
[1] 3006

I have tried to process a small test batch of SENTINEL2_L1C data to analysis ready format as:

# Connect and log in
con = connect(host = "https://openeo.cloud")
login()

# Available processing options and output formats
p = processes()
formats <- list_file_formats()

# Load data 
data_h <- p$load_collection(
    id = "SENTINEL2_L1C",
    spatial_extent = bbox_h,
    temporal_extent = c("2018-06-01", "2018-06-20"),
    bands=c("B04","B08","B8A","B11"))

ard_h = p$ard_surface_reflectance(
    data = data_h,
    elevation_model = NULL,
    atmospheric_correction_method = 'FORCE', 
    atmospheric_correction_options = c(res_merge='NONE',do_brdf =T, do_topo=T), 
    cloud_detection_method = 'Fmask',
    cloud_detection_options = c(erase_clouds= T, max_cloud_cover_frame=85))

ard_data = p$save_result(data=ard_h, format=formats$output$GTiff)
job <- create_job(graph = ard_data, title = "ard_hyltemossa")
start_job(job = job)

This prompts an error message:

HTTP 500 Internal Server Error.
• SERVER-ERROR: Failed to create job on backend ‘eodc’: OpenEoApiError(“[404] 404: {‘Error’: ‘No data found for this spatiotemporal extent’} (ref: a97d98cf-ef3b-416f-bf2c-768a43a4c892)”)Warning messages:
1: In is.environment(value) || !is.na(value) :
‘length(x) = 2 > 1’ in coercion to ‘logical(1)’
2: In is.environment(value) || !is.na(value) :
‘length(x) = 4 > 1’ in coercion to ‘logical(1)’
3: In is.environment(value) || !is.na(value) :
‘length(x) = 3 > 1’ in coercion to ‘logical(1)’
4: In is.environment(value) || !is.na(value) :
‘length(x) = 2 > 1’ in coercion to ‘logical(1)’
5: In is.environment(value) || !is.na(value) :
‘length(x) = 5 > 1’ in coercion to ‘logical(1)’
6: In !is.environment(self$getValue()) && is.na(self$getValue()) :
‘length(x) = 5 > 1’ in coercion to ‘logical(1)’
7: In !is.environment(self$getValue()) && is.na(self$getValue()) :
‘length(x) = 2 > 1’ in coercion to ‘logical(1)’
8: In length(self$getValue()) == 0 || is.na(self$getValue()) :
‘length(x) = 2 > 1’ in coercion to ‘logical(1)’
9: In !is.environment(self$getValue()) && is.na(self$getValue()) :
‘length(x) = 4 > 1’ in coercion to ‘logical(1)’
10: In length(self$getValue()) == 0 || is.na(self$getValue()) :
‘length(x) = 4 > 1’ in coercion to ‘logical(1)’

I have tried changing the temporal extent, defining the bbox in latlon and using a simple feature as input for the spatial_extent. These did not have any affect. I am running on Windows with R version 4.2.2 and openeo_1.2.2.

NEWS for R version 4.2.2 (2022-10-31) states that:

Calling if() or while() with a condition of length greater than one gives an error rather than a warning

Having hard time to interpret this error.
How should the SENTINEL2_1C collection be correctly loaded when using R?

[daniel.thiex]

@sean.hoyal Maybe you can help here. I checked and in general there should be data for the time and location specified.

[m.mohr]

Thanks for the report!
The warnings from R are an issue in the R client, which we are looking into, see coercion issue in 1.2.0 · Issue #123 · Open-EO/openeo-r-client · GitHub
The warnings are not the main issue here though, the communication with the backend happens flawlessly so you can ignore them for now.
The issue regarding the data availability on EODC needs to be checked on their side, e.g. by @sean.hoyal …

[sean.hoyal]

Aplogies, I will take a look to see what is returned by CSW (the current metadata provider for this collection).

Though, you are welcome to try and run this on the development backend, as we are more likely to be able to get this running for you, the only caveat is there is no data for this spatiotemporal extent in the eodc.stac catalogue, but we can update that to make this processing possible for you.

[mitro.muller]

@sean.hoyal

eodc.stac

Code presented above was created to test and evaluate the data processed for a short time period.
Aim is to do the processing over the whole catalogue (from 2015 to present) at three sites in Sweden. Likely there is no data available at any of these sites at the development backend?

Thank you for your time and help!

PS.
As an additional note, there was some problems when moving my 30-day trial to an Early adopter membership. For a while my new 'role was not recognized. Thus, I could not submit jobs. I believe that these problems are now fixed and should not be related to current problem at hand.

[sean.hoyal]

@mitro.muller

Aim is to do the processing over the whole catalogue (from 2015 to present) at three sites in Sweden.

Yes that’s correct there currently is only minimal data indexed in our stac catalogue as we have only recently started migrating away from CSW. I will coordinate internally with my colleague responsible for the STAC catalogue and check the feasibility of how speedily we can add data for the given regions.

Additionally, I am currently checking the list of scenes for the provided AOI and will check both the queried and total time period you’re after. Could you provide the coordinates of the other sites you are after?

P.s
Yes, I remember this, the error that is being returned is the CSW catalogue not yielding results for the AOI, which means you are at least authenticated!

[mitro.muller]

@sean.hoyal

dditionally, I am currently checking the list of scenes for the provided AOI and will check both the queried and total time period you’re after. Could you provide the coordinates of the other sites you are after?

P.s

The two other sites in question and their center point coordinates are:

Norunda (Norunda)
Lat/Lon 60.0865, 17.479504

Svartberget (Svartberget)
Lat/Lon 64.25611, 19.7745

Data is needed for ~500meter buffer around the stations.

[sean.hoyal]

Wonderful, I’ll respond once I have got some preliminary items ingested for you to begin working with!

[sean.hoyal]

Hi Mitro!

I have added S2 L1C data for the three AOI’s for the year 2018, there is no definitive date I can give for when the additional years will be added, but I hope that this will suffice for now. You are welcome try out the eodc dev backend to run ard processing for these areas.

I am yet to test these new additions with our ARD, and there may be a couple of caveats to running the ARD in dev. If you wish to try dev I can prep a brief example notebook for you if that would help? Otherwise the new ARD will become available in production in the next week or two.

[mitro.muller]

@sean.hoyal

have added S2 L1C data for the three AOI’s for the year 2018, there is no definitive date I can give for when the additional years will be added, but I hope that this will suffice for now. You are welcome try out the eodc dev backend to run ard processing for these areas

Thank you for the information. I would gladly test the dev backend if its not too cumbersome. Example notebook would be handy.

Did I understand correctly that the data availability will be limited still after launch of the new ARD?

[michele.claus]

In the meanwhile that @sean.hoyal comes back with more details, this is the development endpoint for testing:
https://openeo-dev.eodc.eu

you have to use this one instead of openeo.cloud for the connection part, the rest should be the same.

[mitro.muller]

Hi,

I can successfully connect to the endpoint

con = connect(host = "https://openeo-dev.eodc.eu")

However, trying to log into this endpoint returns:

> login()
Login failed. Reason: argument is of length zero

No log in window is opened in the browser. Is this a missing rights issue or something else?

[florian.lahn]

Hi,

I can confirm the issue with the current development version of the R client. I will look into it. Right now, I need to check if this is a client issue or if the meta data for the login coming from the back-end cannot be interpreted. But it looks like the client is misinterpreting something, because the login with the WebEditor looks fine.

I also raised an issue in the Github Repository for this:

Login Issues at EODC

[florian.lahn]

I think I have found the problem, which prevents selecting the correct authentication provider with default clients automatically. I will tend to this. Also the Google Authentication does not work at the moment in the R client.

But if you are registered at EGI, you can login, when you select the provider manually like this:
login(provider = "egi")

[mitro.muller]

Thank you for the information. Log in using EGI works fine.
Error messages related to missing data did also disappear.

I used the same code as before, expect defining the elevation model. When I ran:

ard_h = p$ard_surface_reflectance(
   data = data_h,
   elevation_model = "cop-dem-30m",
   atmospheric_correction_method = 'FORCE', 
   atmospheric_correction_options = c(res_merge='NONE',do_brdf =T, do_topo=T), 
   cloud_detection_method = 'Fmask',
   cloud_detection_options = c(erase_clouds= T, max_cloud_cover_frame=80)
)

Checking the graph states:

Parameter 'elevation_model': invalid 'pattern' argument

Regardless do I define the elevation_model parameter or not, when creating a job I got:

HTTP 500 Internal Server Error.
• SERVER-ERROR: {'id': ['Field may not be null.']}

Is this purely a server side problem or is there a parameter missing I have to define?

[sean.hoyal]

Hi Mitro!

Apologies for the amount of time it has take me to reply to again here!

I’ve attached an example python notebook with an example of checking some spatial extent again the eodc-stac collection to be sure of data. This is just in case you want to highlight some new area in the future for us to ingest.

** i pressed enter before finishing the message, example file here file on github

I am unfamiliar with the R client, but you should be fine creating the elevation model as you are, or even leaving it blank, both of these the server on the eodc side will default for. @florian.lahn does the R client omit generating IDs for the process graph? That could cause an issue if it’s the case on our side!

P.S. I have spent time putting the ARD into prod, so you can run this notebook against the prod environment as normal. I’ll make sure to respond quicker to any further feedback!

[m.mohr]

This shouldn’t be an issue on the R side, but it would help to see the process graph for verification.

[florian.lahn]

I had a look at this again, and it seems that the Pattern issue is indeed a client issue, which I have fixed in the development branch on Github, but this is just an issue with internal validation. The Process object to run the job is created regardless by the client.

HTTP 500 Internal Server Error.
• SERVER-ERROR: {'id': ['Field may not be null.']}

That error message indicates a server error that is caused with some field “id”. I noticed that you referenced to the collection cop-dem-30m in the elevation_model parameter. Reading the documentation of “ard_surface_reflectance” by process_viewer("ard_surface_reflectance"), I noticed that the parameter elevation_model is of type collection ID which refers to a collection stored at the back-end. This data set is not listed at that backend, which might cause the problem.

Also you used c() to create the options parameters. This creates vectors of only one type - meaning for example all fields are characters, numerics or logicals. For creating multitype objects, please use list().