Terrascope backend upgrade

[jeroen.dries]

Anther major update of the Terrascope backend:

• merge_cubes and mask will now try to automatically resample input data cubes if required, reducing the need to always use resample_cube_spatial
• When catalogs contain 2 versions of the same product, we now detect this better and try to use the most recent version. This improves performance, but also reproducability.
• sar_backscatter based on sentinelhub has improved defaults for orthorectification

The majority of improvements in this release are smaller performance improvements and bugfixes.

[jeroen.dries]

The Terrascope production backend received another upgrade.
Main features listed below!

2023-02-27 (0.6.7a1)

• GeoParquet support to allow loading large vector files
• Improved specific log messages
• Better support for multiple filter_spatial prcesses in same process graph (Support sample_by_features when filter_spatial is used more than once · Issue #147 · Open-EO/openeo-geopyspark-driver · GitHub)
• Bugfix for sampling sentinelhub based collections (regression: decision to use SHub batch process is based on bbox area instead of geometry area · Issue #279 · Open-EO/openeo-geopyspark-driver · GitHub)
• vector_buffer: Throw an error when a negative buffer size resuls in invalid geometries (return error when negative buffering results in an empty geometry · Issue #164 · Open-EO/openeo-python-driver · GitHub)
• batch jobs now also report usage of credits (reporting of credit usage in batch jobs · Issue #272 · Open-EO/openeo-geopyspark-driver · GitHub)
• non-utm collections should now have a better alignment to the original rasters, if the process graph does not apply an explicit resampling (Image-shifts through bbox, filter_spatial, or spatial_extent · Issue #69 · Open-EO/openeo-geotrellis-extensions · GitHub)

2023-02-07 (0.6.7a1)

• Added initial support for the inspect process. It can be used on datacubes and in callbacks.
• The size of a single chunk is now automatically increased for larger jobs, to improve IO performance.
• resample_cube_spatial is no longer needed in all cases when using merge_cubesor mask
• Better detection of duplicate products in source catalogs
• The ‘if’ process will no longer evaluate the branch that is not accepted implement lazy `if` branching · Issue #109 · Open-EO/openeo-python-driver · GitHub

[jeroen.dries]

The previous update of Terrascope had to be rolled back for a short time, but is now back in effect.
In addition, this is included:

• improvements to messages that are logged, as part of an ongoing effort to include quality of messages
• area calculated and reported by batch job metadata should now be more accurate
• derived-from links when using Sentinelhub based collections have been improved
• more robustness for functionality that depends on external services, by adding more retries before failing
• a new ‘filename_prefix’ format option allows more control over the filename of generated assets, which is otherwise set to default values like ‘openEO.tif’.

[jeroen.dries]

2023-03-20
The Terrascope production backend received another upgrade:

• a bug in aggregate_temporal is fixed, it could cause the first aggregation to be incorrectly set to nodata
• logs are being improved, for instance to also show error messages generated by user code in a UDF

[jeroen.dries]

2023-03-23
In today’s upgrade the Terrascope backend started running on Spark 3.3.1. Generally speaking this should not result in very visible changes, but do let us know if you suddenly see issues!

[jeroen.dries]

2023-09-18
While some intermediate upgrades with smaller fixes were released over summer, we now again have a pretty major upgrade, featuring new processes for vector cubes and many small fixes.

One major change is the interpretation of time intervals as half-open, which was previously announced. This aligns the Terrascope backend with the specification and ensures compatibility with other backends.

New collections were also added:

• SENTINEL3_SYNERGY_VG1
• SENTINEL3_SYNERGY_VG10
• CGLS_FAPAR300_V1_GLOBAL
• CGLS_GDMP300_V1_GLOBAL

The detailed list of changes can be found in our changelog, and contains many fixes and features that were requested by openEO platform users on this forum.

Open-EO/openeo-geopyspark-driver/blob/28a2acbcc107e60fea703fa203a9386f895a3030/CHANGELOG.md