Memory - overhead problem

sulova.andrea · 4 July 2022 08:13

Thanks Jeroen, it works fine with ‘if-else’.

sulova.andrea · 5 July 2022 08:39

Hey again,
I can see the same memory problem in other jobs. FX: the calculation of NDVI and NDWI for a couple of scenes. Is there more efficient approach how to calculate this? The current approach:

s2_cube = append_index(s2_cube,"NDWI") ## index 8
s2_cube = append_index(s2_cube,"NDVI") ## index 9

You can check jobs based on these IDs:

j-f134e2de601942cba7ace33ceccb3689' ,
j-ad7c818280b34b5b9b5e7fb7c1372e8d

jeroen.dries · 6 July 2022 07:03

Hi Andrea,

it’s already quite efficient. One trick we can add is to change the data type to something smaller. I’m guessing you now get float32 output, and we can scale this for instance to shorts. If your current range is between 0 and 1, you could do:
cube.linear_scale_range(0.0,1.0,0,10000)

It is however nog guaranteed to work, so simply increasing memory is also an option. You can try the settings below to add 1GB to each worker:

jeroen.dries:

job_options = {
        "executor-memory": "3G",
        "executor-memoryOverhead": "3G",
        "executor-cores": "2"
}
cube.execute_batch(  out_format="GTiff",
        job_options=job_options)

sulova.andrea · 13 July 2022 09:23

Hey Jeroen,

Could you please advice me if the job_option is applied correctly in send_job:

job_options = {
        "executor-memory": "3G",
        "executor-memoryOverhead": "3G",
        "executor-cores": "2"}

s2_cube_save = s2_cube_swf.save_result(format='netCDF') #GTiff #netCDF
my_job  = s2_cube_save .send_job(title="s2_cube", job_options=job_options)
results = my_job.start_and_wait().get_results()
results.download_files("s2_cube")

jeroen.dries · 14 July 2022 10:54

Hi Andrea,
yes, this looks correct. You can increase memory values if jobs still give a memory related error.
If you have a job id, I can always check deeper if the settings are really picked up.

sulova.andrea · 15 August 2022 09:09

I have tried to scale it for a bigger spatial extent, however it doesn not work:

start_date           = '2021-06-01'
spatial_extent  = {'west': -74, 'east': -73, 'south': 4, 'north': 5, 'crs': 'epsg:4326'} #colombia

## Get the Sentinel-2 data for a 3 month window.
start_date_dt_object = datetime.strptime(start_date, '%Y-%m-%d')
end_date             = (start_date_dt_object + relativedelta(months = +1)).date() ## End date, 1 month later (1st Feb. 2021)
start_date_exclusion = (start_date_dt_object + relativedelta(months = -2)).date() ## exclusion date, to give a 3 month window.

bands                = ['B02', 'B03', 'B04', 'B08', 'CLP', 'SCL' , 'sunAzimuthAngles', 'sunZenithAngles'] 

s2_cube_scale = connection.load_collection(
    'SENTINEL2_L2A_SENTINELHUB',
    spatial_extent  = spatial_extent,
    temporal_extent = [start_date_exclusion, end_date],
    bands           = bands)

job_options = {
        "executor-memory": "5G",
        "executor-memoryOverhead": "5G",
        "executor-cores": "3"}

s2_cube_scale_save = s2_cube_scale.save_result(format='netCDF') #GTiff #netCDF
my_job  = s2_cube_scale_save .send_job(title="s2_cube_scale", job_options=job_options)
results = my_job.start_and_wait().get_results()
results.download_files("s2_cube_scale")

This was error message:
Traceback (most recent call last): File “/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py”, line 1727, in get_log_entries with (self.get_job_output_dir(job_id) / “log”).open(‘r’) as f: File “/usr/lib64/python3.8/pathlib.py”, line 1221, in open return io.open(self, mode, buffering, encoding, errors, newline, File “/usr/lib64/python3.8/pathlib.py”, line 1077, in _opener return self._accessor.open(self, flags, mode) FileNotFoundError: [Errno 2] No such file or directory: ‘/data/projects/OpenEO/j-1bf5a22fd5674494b1e295c3a9654d16/log’

sulova.andrea · 15 August 2022 09:27

Also I have tried with a bit smaller area and bigger memory but not suceess, can you advice me what should be improved?

start_date           = '2021-06-01'
spatial_extent  = {'west': -74, 'east': -73.5, 'south': 4.5, 'north': 5, 'crs': 'epsg:4326'} #colombia

## Get the Sentinel-2 data for a 3 month window.
start_date_dt_object = datetime.strptime(start_date, '%Y-%m-%d')
end_date             = (start_date_dt_object + relativedelta(months = +1)).date() ## End date, 1 month later (1st Feb. 2021)
start_date_exclusion = (start_date_dt_object + relativedelta(months = -2)).date() ## exclusion date, to give a 3 month window.

bands                = ['B02', 'B03', 'B04', 'B08', 'CLP', 'SCL' , 'sunAzimuthAngles', 'sunZenithAngles'] 

s2_cube_scale = connection.load_collection(
    'SENTINEL2_L2A_SENTINELHUB',
    spatial_extent  = spatial_extent,
    temporal_extent = [start_date_exclusion, end_date],
    bands           = bands)

job_options = {
        "executor-memory": "100G",
        "executor-memoryOverhead": "100G",
        "executor-cores": "4"}

s2_cube_scale_save = s2_cube_scale.save_result(format='netCDF') #GTiff #netCDF
my_job  = s2_cube_scale_save .send_job(title="s2_cube_scale", job_options=job_options)
results = my_job.start_and_wait().get_results()
results.download_files("s2_cube_scale")

sulova.andrea · 15 August 2022 09:51

I have tried with: s2_cube_scale = s2_cube_scale.linear_scale_range(0.0,1.0,0,10000) but still no positive answer.

Printing logs:
[{'id': 'error', 'level': 'error', 'message': 'Traceback (most recent call last):\n  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py", line 1727, in get_log_entries\n    with (self.get_job_output_dir(job_id) / "log").open(\'r\') as f:\n  File "/usr/lib64/python3.8/pathlib.py", line 1221, in open\n    return io.open(self, mode, buffering, encoding, errors, newline,\n  File "/usr/lib64/python3.8/pathlib.py", line 1077, in _opener\n    return self._accessor.open(self, flags, mode)\nFileNotFoundError: [Errno 2] No such file or directory: \'/data/projects/OpenEO/j-3c44cd000ba648a8b6350a0765d8e48c/log\'\n'}]

sulova.andrea · 18 August 2022 12:15

@stefaan.lippens if you have time could you please have a look on this

stefaan.lippens · 18 August 2022 13:23

From our internal logs I see this happened Mon 15th of Aug around 11/12 AM. I remember we had “full disk” problems earlier this week. Do still get this same error if you rerun now or something else?
Can you also share the job id of your latest attempt?

sulova.andrea · 18 August 2022 18:39

The same error than before:

FileNotFoundError: [Errno 2] No such file or directory: '/data/projects/OpenEO/j-9d9a934535844b4eace8926e06307e18/log'

stefaan.lippens · 18 August 2022 20:48

for j-9d9a934535844b4eace8926e06307e18, I find this in the logs:

Exception in thread “main” java.lang.IllegalArgumentException:
Required executor memory (102400 MB), offHeap memory (0) MB, overhead (102400 MB), and PySpark memory (0 MB)
is above the max threshold (52224 MB) of this cluster! Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’.

apparently these job options are way too high

We should improve the error message for this (see Better error for user when requesting too much batch job memory or cpu · Issue #203 · Open-EO/openeo-geopyspark-driver · GitHub)

sulova.andrea · 19 August 2022 09:07

Will try that
A general question: I was running simultaneously a few scripts yesterday but now I am getting this message **OpenEoApiError** : [429] unknown: max connections reached: 1 . Does it mean that I can run only one script?

sulova.andrea · 22 August 2022 07:04

The job options were changed to the smaller numbers and It has processed data after ca. 6h.

I am wondering what are the most correct values in job_options if I want to run (1deg x 1 deg ):
spatial_extent = {'west': -74, 'east': -73, 'south': 4, 'north': 5, 'crs': 'epsg:4326'}?

sulova.andrea · 22 August 2022 08:10

Additionally, I would like to obtain satellite data for some country using a vector layer (.shp, .geojson) and then process that spatial extent further so is it possible to insert a geojson/shp file for a spatial_extent or it always needs to be in this format: spatial_extent ={'west':bbox[0],'east':bbox[2],'south':bbox[1],'north':bbox[3],'crs':4326}?

sulova.andrea · 22 August 2022 11:10

Could you please confirm that there is the size limitation?

This message was obtained after requiring S2 scenes for the Columbia site:

OpenEoApiError: [400] Internal: Requested area 48653514887301.67 m² for collection SENTINEL2_L2A_SENTINELHUB exceeds maximum of 1000000000000.0 m². (ref: r-6bd4565ad8c14288a33570bcd8d3dc87)

stefaan.lippens · 22 August 2022 12:00

No, it should be possible to run multiple scripts/notebooks in parallel.
However, depending on your usage patterns you might hit a “rate limiting” threshold like that.

stefaan.lippens · 22 August 2022 12:02

The answer to that is “it depends” unfortunately.
It depends on how many collections and bands you load in tha AOI, whether you use some more heavy operations (e.g. UDFs), etc.

stefaan.lippens · 22 August 2022 12:09

I’m not sure I understand what you mean here. Some points that might be an answer:

instead of the filter_bbox process (which is equivalent to the spatial_extent in load_collection) to pass a west-south-east-nort bounding box, you can also use the filter_spatial process with for example a geojson structure to spatially filter with this geometry directly
if you use operations like mask and aggregate_spatial that require a limiting geometry by design, you can leave out a spatial extent in load_collection, as the VITO backend will infer that from the geometry used in mask/aggregate_spatial. Be careful that this feature is not supported on other backends necessarily.

sulova.andrea · 25 August 2022 09:11

Thanks for a good overview!

1. filter_spatial
It seems that it does not want to accept my geojson file.

Code:

s2_cube = connection.load_collection(
    'SENTINEL2_L2A_SENTINELHUB',
    # spatial_extent  = spatial_extent,
    temporal_extent = [start_date_exclusion, end_date],
    bands           = bands)

aoi = "../../aoi/DK/DK.geojson"

# aoi = "https://github.com/DHI-GRAS/worldwater-toolbox/blob/main/openeo_platform_notebooks/DK.geojson"
bbox = dict(zip(["west", "south", "east", "north"], gpd.read_file(aoi).total_bounds))
print(bbox)

s2_cube = s2_cube.filter_spatial(aoi)

s2_cube_save = s2_cube.save_result(format='netCDF') #GTiff #netCDF
my_job  = s2_cube_save.send_job(title="s2_cube")
results = my_job.start_and_wait().get_results()
results.download_files("s2_cube")

Error message: OpenEoApiError: [500] Internal: Server error: NotImplementedError("filter_spatial only supports dict but got DelayedVector('../../aoi/DK/DK.geojson')") (ref: r-1aaf592938bc41c69d022ee03bfb26e0)

The geojson file is uploaded on GitHub . The file is in EPSG:4326.

2. mask_polygon (another option which I have tried)

I have followed this instructions but it seems there is a problem to work with my geojson file. The bbox can be print {‘west’: 8.07638931, ‘south’: 54.55902863, ‘east’: 15.19305611, ‘north’: 57.75152588} so I believe that my file is fine but it can not be used with mask_ polygon or in filter_spatial

Code:

s2_cube = connection.load_collection(
    'SENTINEL2_L2A_SENTINELHUB',
    # spatial_extent  = spatial_extent,
    temporal_extent = [start_date_exclusion, end_date],
    bands           = bands)

aoi = "../../aoi/DK/DK.geojson"
bbox = dict(zip(["west", "south", "east", "north"], gpd.read_file(aoi).total_bounds))


s2_cube = s2_cube.filter_bbox(bbox)
s2_cube = s2_cube.mask_polygon(aoi)

s2_cube_save = s2_cube.save_result(format='netCDF') #GTiff #netCDF
my_job  = s2_cube_save.send_job(title="S2L2A_cube")
results = my_job.start_and_wait().get_results()

Error:
OpenEoApiError: [500] Internal: Server error: FileNotFoundError(2, 'No such file or directory') (ref: r-53be7db1c4934f15a2b6a6dd0c19064e)

Would you advice me what is incorrect in these codes please?