Backend fails when using "between" function

Hi,
when trying to create a cloud mask from the Sentinel SCL band using the “between” function, the backend throws an error. Code example below:


collections = "SENTINEL2_L2A_SENTINELHUB"
bands = c("B02", "B03", "B04", "B08", "SCL")
period = c("2021-04-01", "2021-06-30")
ext = list(11.12692, 49.74992, 11.15708, 49.76938)
names(ext) = c("west", "south", "east", "north")

# change to login with your credentials
# con = openeo::connect("https://openeo.cloud")
# login(
#     login_type = "oidc"
#     , provider = "egi"
#     , config = list(
#         client_id = "<client_id>"
#         , secret = "<secret>"
#     )
# )

procs = processes()

cube = procs$load_collection(
    id = collections
    , spatial_extent = ext
    , temporal_extent = period
    , bands = bands
)

## apply mask
mask = procs$reduce_dimension(
    cube
    , dimension = "bands"
    , reducer = \(data, context) {
        scl = data[5]
        res = p$between(scl, min = 3, max = 6)
        return(res)
    }
)

## apply mask
cube_masked = procs$mask(cube, mask)

## create process graph
graph = procs$save_result(
    data = cube_masked
    , format = "GTiff"
)

## create and start job
job = openeo::create_job(graph)
openeo::start_job(job)

id = as.character(job$id)
jobs = openeo::list_jobs()
jobs[[id]]

# Job ID:		vito-e838f3f2-c95e-4f5e-b953-d28136616a4e
# Title:		---
# Description:	---
# Status:		error
# Created:	2021-12-03T14:53:39Z
# Updated:	2021-12-03T14:55:02Z
# Plan:		---
# Costs:
# Budget:		---

After some testing our best guess is that this is related to the “between” function.
Alongside with that maybe a silly question: Is it possible to get more informative responses from the backends rather than just “error”?

Hi Hendrik,
the job normally also has logs that you can access, but don’t directly know the call in R for that.

In any case, the error was:
Unsupported operation: between (arguments: [min, max, x, exclude_max])
In Python, to construct a mask like this, I would try >= 3 & scl <=6
But supporting between in there seems to make sense as well!

Hi Hendrik,
FYI, support for between is underway, will take some time to get into production, let me know if it should go faster.

Hi Jeroen,
thanks for letting me know. No immediate need from our end. We were just confused because the function is listed in the available processes (openeo::processes()) for the backend but did not work.
Great that a fix is on the way!
Best,
Hendrik

I agree it’s confusing, that’s why I’m trying to support the process more broadly.
The problem is that the process discovery only allows us to say if we support a process somewhere in our backend, it’s not granular (which is also quite hard to do).

Hi Hendrik,

thanks for the reporting. I am the developer of the R package, and now that I have write access I can finally also support via the forum.

Regarding the “error” statement on list_jobs, this is rather intentional. The job list shall simply provide an overview about the status each of your job are in. The log functionality Jeroen is referring to exists also for R - at least in the state I left it in since end of the H2020 project. log_job would be the function you are looking for. Usually it prints updates to the console unless you terminate it or you don’t get new lines for a certain amount of time.
I mentioned that the function was developed using openEO API v1.0.0, so it might not work propely with a newer version. That will be something I’m going to look into, when I catch up on the API changes.

Support for ‘between’ in a reducer has been released. Please try again and let us know if there are still issues!