Corrupted Files

openEO Platform
1

Hey everyone,

When I was trying to download a few scenes I have got the corrupted files in multiple cases.

First case:
I hava tried to download just one RGB S2scene but a final tif file looks strange:

image
image975×761 89.8 KB

  • west 11.800511439000047
  • east 13.591601298000057
  • south 54.91902653800008
  • north 55.93727334500005
datacube = connection.load_collection("SENTINEL2_L2A_SENTINELHUB", 
            spatial_extent =  {'west':bbox[0],'east':bbox[2],'south':bbox[1],'north':bbox[3]},
            temporal_extent=["2021-01-01","2021-01-06"],
            bands = ["B04", "B03", "B02"])

my_job = datacube.send_job()
results = my_job.start_and_wait().get_results()

Also, when I have tried to apply the atmospheric correction method:

datacube = connection.load_collection(
    'SENTINEL2_L1C_SENTINELHUB',
    spatial_extent  = {'west':bbox[0],'east':bbox[2],'south':bbox[1],'north':bbox[3]},
    temporal_extent = ["2021-05-01", "2021-05-05"],
    bands = ['B04', 'B03', 'B02', 'B09', 'B8A', 'B11', 'sunAzimuthAngles', 'sunZenithAngles', 'viewAzimuthMean', 'viewZenithMean']
)

l2a = datacube.atmospheric_correction(method = 'iCor')
s2_GTiff = l2a.save_result(format="GTiff")
job_s2 = s2_GTiff.send_job()
job_id_s2 = job_s2.job_id
results = job_s2.start_and_wait().get_results()
results.download_files(out_dir / "7_S2_AC_iCor_scene_2021.tif")

This is how the final scene looks like:

image
image1289×827 148 KB

:

2

Hi Andrea,
I’ve tried to reproduce your problem, but wasn’t able to get the corrupted tif just yet.
Let’s focus on the first case, (without atmospheric correction).
For the given date range, I would expect that the batch job generates multiple tif files. Which one do you visualize?
Or is it possible to reproduce with a script that only looks at a single day?

4

I have tried to run for the S2 tile = 33UUB

Firstly, I’ve tried different temporal extents (2 different months) and I got these:
image

  • I have tried to process just 4 scenes, it did not download all of them completely…A few scenes which managed to download were corrupted as you can see above.

  • I have tried to run for entire Jun- It usually donwloads first 3 scenes and it stops downloading. A job was finished wich I can see from the message but downloding was stopped for unknown reason. I have run this kind of processing multiple times with different process for a month (mask, ndvi, atm.corre) but the same thing happens - it does not manage to download all scenes and plus those scenes, which were downloaded were corrupted

5

Thanks, I now managed to see some of the corrupted files. The geotiffs are assembled incorrectly, which is certainly not supposed to happen. They also do not contain overviews, which makes loading fairly slow for such large files.
I’ll give this high priority for a fix, but may take till monday to get something deployed.
Will let you know if I find a workaround, as it appears that it does work in a number of other cases.

By the way, if it’s download that gets interrupted for some other reason, then you should still be able to download your results in the web editor or with the Python client. Almost all of your jobs have succeeded, so it’s mostly the problem inside the files themselves.

6

Thanks for your update.

Could you elaborate on this topic please :slight_smile: ?
Why when a job is finished then I have a difficulty to download data?

The data are not downloading, and it happens frequently that eventually there are not data. I have been waiting sometime for a half a day or day. Runinng mulitple times but without results

image
image1032×781 64.7 KB

8

Hi Andrea,

the download after the job has finished should certainly not take that long. I’ve now run the same job multiple times, and over here it’s very fast.

Could you perhaps try using the editor at https://editor.openeo.cloud ? The screenshot below shows where the download link is located:

image
image1448×1075 164 KB

I’m still pinpointing the exact cause of the corruption, in a lot of very similar cases, it does work fine, so I’m going through the various options.

best regards,
Jeroen

9

Hi Andrea,
the first part of the problem has been fixed, and I no longer get corrupted geotiffs. You could try this yourself on our development endpoint (openeo-dev.vito.be).
There is however a second part, causing the georeferencing to be wrong (this is what caused the corruption in the first place, but it was hard to see). I’m still working on also resolving that!

best regards,
Jeroen

1 Like
10

Hi Andrea,
apologies for still taking so long, but it’s now fully fixed on openeo-dev.vito.be, so geotiff should not be corrupt and correctly georeferenced.

We’ll continue with rolling this out in production next week.

have a nice weekend,
Jeroen

1 Like