Issues about downloading CGLS LAI V2

xshan · 4 May 2025 11:35

Hi guys,

I tried to download CGLS LAI from openEO. I managed to do it few years ago using following codes:

session = openeo.connect("https://openeo-dev.vito.be").authenticate_oidc("egi")
# using eci sso !!!
#session.describe_collection("CGLS_LAI300_V1_GLOBAL")
#LAI300 = session.load_collection("CGLS_LAI300_V1_GLOBAL")

LAIV2 = session.load_collection("CGLS_LAI_V2_GLOBAL")
# CGLS_LAI_V2_GLOBAL

year = [i+2006 for i in range(14)]
month = [i+1 for i in range(12)]

ts = pd.date_range("2006-01-01", "2019-12-31")


for iyear in year:
    for imonth in month:
        ts_tmp = ts[(ts.year == iyear) & (ts.month == imonth)]
#        LAIV2.filter_temporal(ts_tmp.strftime("%Y-%m-%d")[0],
#                              ts_tmp.strftime("%Y-%m-%d")[-1]).filter_bbox([-4.875,41.125,10.125,52.125]).download("/Users/xushan/research/TUD/test_jackknife/LAI/LAI_1km_V2/LAI_"+ts_tmp.strftime("%Y-%m-%d")[0]+"_"+ts_tmp.strftime("%Y-%m-%d")[-1]+".nc",format="NetCDF")
        LAIV2.filter_temporal(ts_tmp.strftime("%Y-%m-%d")[0],
                              ts_tmp.strftime("%Y-%m-%d")[-1]).filter_bbox([-11.75,36,29.5,58]).download("/Users/xushan/research/TUD/test_jackknife/LAI/westEurope_LAI_1km_V2/LAI_"+ts_tmp.strftime("%Y-%m-%d")[0]+"_"+ts_tmp.strftime("%Y-%m-%d")[-1]+".nc",format="NetCDF")
#        LAIV2.filter_temporal(ts_tmp.strftime("%Y-%m-%d")[0],
#                              ts_tmp.strftime("%Y-%m-%d")[-1]).filter_bbox([-4.875,41.125,10.125,52.125]).download("/Users/xushan/research/TUD/test_jackknife/LAI/LAI_1km_V2/FAPAR_"+ts_tmp.strftime("%Y-%m-%d")[0]+"_"+ts_tmp.strftime("%Y-%m-%d")[-1]+".nc",format="NetCDF")
        print(ts_tmp.strftime("%Y-%m-%d")[0]+' to '+ts_tmp.strftime("%Y-%m-%d")[-1] + " done!")

However, now I found this script does not work anymore…and right now I am using a new laptop, so I do understand I need to authentic myself again. However, I found it is hard to find a way to do it.
I checked from the openEOhub that two backends have CGLS LAI: openEO and Vito. So I tried to use the URL link from https://hub.openeo.org: https://openeocloud.vito.be/openeo/1.0.0 and https://openeo.vito.be/openeo/1.2. However, I found both URL cannot authentic. For example, if I run

session = openeo.connect("https://openeocloud.vito.be/openeo/1.0.0").authenticate_oidc()

there is no output under this command…but just waiting for its returned values…

if I tried to use other URL like “”

it returns:

session = openeo.connect("https://earthengine.openeo.org").authenticate_oidc()
Traceback (most recent call last):

  Cell In[23], line 1
    session = openeo.connect("https://earthengine.openeo.org").authenticate_oidc()

  File ~/miniforge3/envs/py310/lib/python3.10/site-packages/openeo/rest/connection.py:564 in authenticate_oidc
    provider_id, client_info = self._get_oidc_provider_and_client_info(

  File ~/miniforge3/envs/py310/lib/python3.10/site-packages/openeo/rest/connection.py:310 in _get_oidc_provider_and_client_info
    raise OpenEoClientException("No client_id found.")

OpenEoClientException: No client_id found.

So how can I do it? Thanks!

stefaan.lippens · 5 May 2025 10:04

If you execute authenticate_oidc() in a fresh/new environment (where you don’t have valid refresh tokens), you should see some output print’ed like

Visit https://example.com/device?user_code=RAFT-GJOL to authenticate.

you are expected to visit that URL in a browser of your preference and follow the login/authentication flow there. Your python script is currently hanging for some time, waiting for you to complete that authentication flow. After some time (5 minutes by default) it will timeout and fail.

If you don’t see that line printed: are you redirecting the output of your script in some way?

xshan · 5 May 2025 12:31

Thanks for your reply!

Yes, I create a new environment for openeo (mamba create -n openeo_lai python=3.11 xarray netCDF4 openeo pandas), and managed to authentic using following codes:

connection = openeo.connect("https://openeocloud.vito.be/openeo/1.0.0").authenticate_oidc()

However, following for-loop downloading script yields error like this:

In [10]: connection = openeo.connect("https://openeocloud.vito.be/openeo/1.0.0")
       ⋮ .authenticate_oidc()
Authenticated using refresh token.

    ...: ts = pd.date_range("2000-01-01", "2019-12-31")
    ...: 
    ...: lat, lon = 51.0792, 10.4522
    ...: buffer = 0.00003
    ...: 
    ...: lat_min = lat - buffer
    ...: lat_max = lat + buffer
    ...: lon_min = lon - buffer
    ...: lon_max = lon + buffer
    ...: bbox = [lon_min, lat_min, lon_max, lat_max]  # order: [west, south, eas
       ⋮ t, north]
    ...: 
    ...: 
    ...: for iyear in year:
    ...:     for imonth in month:
    ...:         ts_tmp = ts[(ts.year == iyear) & (ts.month == imonth)]
    ...:         LAIV2.filter_temporal(ts_tmp.strftime("%Y-%m-%d")[0],
    ...:                               ts_tmp.strftime("%Y-%m-%d")[-1]).filter_b
       ⋮ box(bbox).download("/Users/xushan/research/TUD/test_jackknife/LAI/westE
       ⋮ urope_LAI_1km_V2/LAI_"+ts_tmp.strftime("%Y-%m-%d")[0]+"_"+ts_tmp.strfti
       ⋮ me("%Y-%m-%d")[-1]+".nc",format="NetCDF")
    ...:         print(ts_tmp.strftime("%Y-%m-%d")[0]+' to '+ts_tmp.strftime("%Y
       ⋮ -%m-%d")[-1] + " done!")
    ...: 
---------------------------------------------------------------------------
OpenEoApiError                            Traceback (most recent call last)
Cell In[11], line 23
     20 for imonth in month:
     21     ts_tmp = ts[(ts.year == iyear) & (ts.month == imonth)]
     22     LAIV2.filter_temporal(ts_tmp.strftime("%Y-%m-%d")[0],
---> 23                           ts_tmp.strftime("%Y-%m-%d")[-1]).filter_bbox(bbox).download("/Users/xushan/research/TUD/test_jackknife/LAI/westEurope_LAI_1km_V2/LAI_"+ts_tmp.strftime("%Y-%m-%d")[0]+"_"+ts_tmp.strftime("%Y-%m-%d")[-1]+".nc",format="NetCDF")
     24     print(ts_tmp.strftime("%Y-%m-%d")[0]+' to '+ts_tmp.strftime("%Y-%m-%d")[-1] + " done!")

File ~/miniforge3/envs/openeo_lai/lib/python3.11/site-packages/openeo/rest/datacube.py:2455, in DataCube.download(self, outputfile, format, options, validate, auto_add_save_result, additional, job_options, on_response_headers)
   2453 else:
   2454     res = self
-> 2455 return self._connection.download(
   2456     res.flat_graph(),
   2457     outputfile=outputfile,
   2458     validate=validate,
   2459     additional=additional,
   2460     job_options=job_options,
   2461     on_response_headers=on_response_headers,
   2462 )

File ~/miniforge3/envs/openeo_lai/lib/python3.11/site-packages/openeo/rest/connection.py:1607, in Connection.download(self, graph, outputfile, timeout, validate, chunk_size, additional, job_options, on_response_headers)
   1603 pg_with_metadata = self._build_request_with_process_graph(
   1604     process_graph=graph, additional=additional, job_options=job_options
   1605 )
   1606 self._preflight_validation(pg_with_metadata=pg_with_metadata, validate=validate)
-> 1607 response = self.post(
   1608     path="/result",
   1609     json=pg_with_metadata,
   1610     expected_status=200,
   1611     stream=True,
   1612     timeout=timeout or DEFAULT_TIMEOUT_SYNCHRONOUS_EXECUTE,
   1613 )
   1614 if on_response_headers:
   1615     on_response_headers(response.headers)

File ~/miniforge3/envs/openeo_lai/lib/python3.11/site-packages/openeo/rest/_connection.py:205, in RestApiConnection.post(self, path, json, **kwargs)
    197 def post(self, path: str, json: Optional[dict] = None, **kwargs) -> Response:
    198     """
    199     Do POST request to REST API.
    200 
   (...)    203     :return: response: Response
    204     """
--> 205     return self.request("post", path=path, json=json, allow_redirects=False, **kwargs)

File ~/miniforge3/envs/openeo_lai/lib/python3.11/site-packages/openeo/rest/connection.py:664, in Connection.request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    657     return super(Connection, self).request(
    658         method=method, path=path, headers=headers, auth=auth,
    659         check_error=check_error, expected_status=expected_status, **kwargs,
    660     )
    662 try:
    663     # Initial request attempt
--> 664     return _request()
    665 except OpenEoApiError as api_exc:
    666     if api_exc.http_status_code in {401, 403} and api_exc.code == "TokenInvalid":
    667         # Auth token expired: can we refresh?

File ~/miniforge3/envs/openeo_lai/lib/python3.11/site-packages/openeo/rest/connection.py:657, in Connection.request.<locals>._request()
    656 def _request():
--> 657     return super(Connection, self).request(
    658         method=method, path=path, headers=headers, auth=auth,
    659         check_error=check_error, expected_status=expected_status, **kwargs,
    660     )

File ~/miniforge3/envs/openeo_lai/lib/python3.11/site-packages/openeo/rest/_connection.py:132, in RestApiConnection.request(self, method, path, params, headers, auth, check_error, expected_status, **kwargs)
    130 expected_status = ensure_list(expected_status) if expected_status else []
    131 if check_error and status >= 400 and status not in expected_status:
--> 132     self._raise_api_error(resp)
    133 if expected_status and status not in expected_status:
    134     raise OpenEoRestError(
    135         "Got status code {s!r} for `{m} {p}` (expected {e!r}) with body {body}".format(
    136             m=method.upper(), p=path, s=status, e=expected_status, body=resp.text
    137         )
    138     )

File ~/miniforge3/envs/openeo_lai/lib/python3.11/site-packages/openeo/rest/_connection.py:154, in RestApiConnection._raise_api_error(self, response)
    152     error_message = info.get("message")
    153     if error_code and isinstance(error_code, str) and error_message and isinstance(error_message, str):
--> 154         raise OpenEoApiError(
    155             http_status_code=status_code,
    156             code=error_code,
    157             message=error_message,
    158             id=info.get("id"),
    159             url=info.get("url"),
    160         )
    162 # Failed to parse it as a compliant openEO API error: show body as-is in the exception.
    163 text = response.text

OpenEoApiError: [401] TokenExchangeFailure: Authentication failed. Please login on our marketplace web https://portal.terrascope.be first, and then try authenticate again. (ref: r-25050512274041ad9cb6a5189c69f1d9)

I tried to log into the page https://portal.terrascope.be, then exit ipython under this environment, then reopen python to run the codes, it still yields the above errors…could you please help me with this? Thanks!

stefaan.lippens · 5 May 2025 14:41

And did it work? Did you successfully log in there to connect your account on Terrascope?

After connecting on Terrascope, I would also recommend to clear your current tokens, with one of the options listed at Authentication and Account Management — openEO Python Client 0.41.0a2 documentation, e.g.

from openeo.rest.auth.config import RefreshTokenStore
RefreshTokenStore().remove()

xshan · 5 May 2025 14:53

Thanks for your reply!

Yes, I loginfo the page…and after that I don’t know what to do next step…i.e. how to connect my account on Terrascope? didn’t find a doc for this…

stefaan.lippens · 5 May 2025 15:34

Connecting your account should happen automatically if you sign on on portal.terrascope.be with The same account you used for authenticating on openeo.cloud, which means you have to pick “EduGAIN & social logins” on this step:

xshan · 5 May 2025 15:50

Thanks!
Yes I am using the same account (I guess), both are of x.shan-2@tudelft.nl
Here is the snapshot from oidc:

Then if I clicked EduGAIN & social logins, it says:

Then if I proceeded to link it to my existed account, it becomes:

stefaan.lippens · 6 May 2025 08:07

Ok I checked with the people in charge of portal.terrascope.be to find a way to get you out of this dead end.

What you should do as far as I understand:

First log in on portal.terrascope.be, not with the “EduGAIN & social logins” option on the right as mentioned earlier, but instead with your direct Terrascope credentials, using the “Username or email + password” option on the left. As you got an “account already exists” error, you apparently already have a Terrascope account associated with the email address you mentioned. If you don’t know your password anymore, you should be able to reset it with the “forgot password?” link on the login form.
once logged in this way:
1. Go to the profile page https://portal.terrascope.be/profile and click on ‘Manage on Terrascope’ button. This will open a new tab.
2. Click on the federated identity menu, and then click on ‘Remove’ button which is aligned with the ‘EduGAIN & social logins` input.
3. Click on the Add button, and then log in with your EduGain and social logins. This will connect your Marketplace account with your new EGI account.

xshan · 8 May 2025 13:21

now it works. Thanks for your help! @stefaan.lippens

xshan · 8 May 2025 14:05

but I suddenly found I don’t have enough credits to download the data…wondering do we have any support for universities or institutions? Thanks!