Terrascope backend upgrade

[jeroen.dries]

Anther major update of the Terrascope backend:

• merge_cubes and mask will now try to automatically resample input data cubes if required, reducing the need to always use resample_cube_spatial
• When catalogs contain 2 versions of the same product, we now detect this better and try to use the most recent version. This improves performance, but also reproducability.
• sar_backscatter based on sentinelhub has improved defaults for orthorectification

The majority of improvements in this release are smaller performance improvements and bugfixes.

[jeroen.dries]

The Terrascope production backend received another upgrade.
Main features listed below!

2023-02-27 (0.6.7a1)

• GeoParquet support to allow loading large vector files
• Improved specific log messages
• Better support for multiple filter_spatial prcesses in same process graph (Support sample_by_features when filter_spatial is used more than once · Issue #147 · Open-EO/openeo-geopyspark-driver · GitHub)
• Bugfix for sampling sentinelhub based collections (regression: decision to use SHub batch process is based on bbox area instead of geometry area · Issue #279 · Open-EO/openeo-geopyspark-driver · GitHub)
• vector_buffer: Throw an error when a negative buffer size resuls in invalid geometries (return error when negative buffering results in an empty geometry · Issue #164 · Open-EO/openeo-python-driver · GitHub)
• batch jobs now also report usage of credits (reporting of credit usage in batch jobs · Issue #272 · Open-EO/openeo-geopyspark-driver · GitHub)
• non-utm collections should now have a better alignment to the original rasters, if the process graph does not apply an explicit resampling (Image-shifts through bbox, filter_spatial, or spatial_extent · Issue #69 · Open-EO/openeo-geotrellis-extensions · GitHub)

2023-02-07 (0.6.7a1)

• Added initial support for the inspect process. It can be used on datacubes and in callbacks.
• The size of a single chunk is now automatically increased for larger jobs, to improve IO performance.
• resample_cube_spatial is no longer needed in all cases when using merge_cubesor mask
• Better detection of duplicate products in source catalogs
• The ‘if’ process will no longer evaluate the branch that is not accepted implement lazy `if` branching · Issue #109 · Open-EO/openeo-python-driver · GitHub

[jeroen.dries]

The previous update of Terrascope had to be rolled back for a short time, but is now back in effect.
In addition, this is included:

• improvements to messages that are logged, as part of an ongoing effort to include quality of messages
• area calculated and reported by batch job metadata should now be more accurate
• derived-from links when using Sentinelhub based collections have been improved
• more robustness for functionality that depends on external services, by adding more retries before failing
• a new ‘filename_prefix’ format option allows more control over the filename of generated assets, which is otherwise set to default values like ‘openEO.tif’.

[jeroen.dries]

2023-03-20
The Terrascope production backend received another upgrade:

• a bug in aggregate_temporal is fixed, it could cause the first aggregation to be incorrectly set to nodata
• logs are being improved, for instance to also show error messages generated by user code in a UDF

[jeroen.dries]

2023-03-23
In today’s upgrade the Terrascope backend started running on Spark 3.3.1. Generally speaking this should not result in very visible changes, but do let us know if you suddenly see issues!

[jeroen.dries]

2023-09-18
While some intermediate upgrades with smaller fixes were released over summer, we now again have a pretty major upgrade, featuring new processes for vector cubes and many small fixes.

One major change is the interpretation of time intervals as half-open, which was previously announced. This aligns the Terrascope backend with the specification and ensures compatibility with other backends.

New collections were also added:

• SENTINEL3_SYNERGY_VG1
• SENTINEL3_SYNERGY_VG10
• CGLS_FAPAR300_V1_GLOBAL
• CGLS_GDMP300_V1_GLOBAL

The detailed list of changes can be found in our changelog, and contains many fixes and features that were requested by openEO platform users on this forum.

Open-EO/openeo-geopyspark-driver/blob/28a2acbcc107e60fea703fa203a9386f895a3030/CHANGELOG.md

[jeroen.dries]

We’re happy to announce a new set of improvements and features that were integrated in the Terrascope backend over the last months. In general, there has been a strong focus on small bugfixes and stability improvements, but we also were able to add new features, usually requested by our users.

The full changelog can be found in the link below, but we highlight a few noteworthy items:

• A number of processes now use floating point operations automatically, avoiding unexepected results for instance when subtracting between unsigned data types.
• Using ‘filter_labels’, it is now possible to load data for multiple time intervals rather than a single continuous one.
• The most commonly used UDF signature can now use xarray.DataArray directly, rather than using an openEO wrapper class. This makes your UDF’s look more simple.
• The size of netCDF files that can be generated by openEO has increased. Do note that a 20GB netCDF file can be hard to handle and takes a long time to download.
• Improved support for reading and generating GeoParquet files. This relatively new cloud-native format has some key advantages compared to traditional formats.
• STAC metadata for results has gotten a few minor fixes to be compliant with latest versions of the extensions. There’s no big changes, but you may need to adjust if you depend on very specific STAC properties.
• load_stac is getting more and more versatile. Allowing you to integrate your own datasets.

Open-EO/openeo-geopyspark-driver/blob/907135bbe50ae2e40995f285064b7c2029774b2a/CHANGELOG.md