No error for flood mapping

The process below (flood mapping over Pakistan) gives me no error when erroring. Job ID:
vito-j-c6b472c8840747c987be5698e8fa507b

{
  "process_graph": {
    "add3": {
      "arguments": {
        "data": {
          "from_node": "reduce1"
        },
        "label": "2022",
        "name": "t",
        "type": "temporal"
      },
      "process_id": "add_dimension"
    },
    "add5": {
      "arguments": {
        "data": {
          "from_node": "reduce2"
        },
        "label": "2022",
        "name": "t",
        "type": "temporal"
      },
      "process_id": "add_dimension"
    },
    "load1": {
      "arguments": {
        "id": "SENTINEL2_L2A",
        "spatial_extent": {
          "_id": "pakistan",
          "geometry": {
            "coordinates": [
              [
                [
                  75.158028,
                  37.133031
                ],
                [
                  75.896897,
                  36.666806
                ],
                [
                  76.192848,
                  35.898403
                ],
                [
                  77.837451,
                  35.49401
                ],
                [
                  76.871722,
                  34.653544
                ],
                [
                  75.757061,
                  34.504923
                ],
                [
                  74.240203,
                  34.748887
                ],
                [
                  73.749948,
                  34.317699
                ],
                [
                  74.104294,
                  33.441473
                ],
                [
                  74.451559,
                  32.7649
                ],
                [
                  75.258642,
                  32.271105
                ],
                [
                  74.405929,
                  31.692639
                ],
                [
                  74.42138,
                  30.979815
                ],
                [
                  73.450638,
                  29.976413
                ],
                [
                  72.823752,
                  28.961592
                ],
                [
                  71.777666,
                  27.91318
                ],
                [
                  70.616496,
                  27.989196
                ],
                [
                  69.514393,
                  26.940966
                ],
                [
                  70.168927,
                  26.491872
                ],
                [
                  70.282873,
                  25.722229
                ],
                [
                  70.844699,
                  25.215102
                ],
                [
                  71.04324,
                  24.356524
                ],
                [
                  68.842599,
                  24.359134
                ],
                [
                  68.176645,
                  23.691965
                ],
                [
                  67.443667,
                  23.944844
                ],
                [
                  67.145442,
                  24.663611
                ],
                [
                  66.372828,
                  25.425141
                ],
                [
                  64.530408,
                  25.237039
                ],
                [
                  62.905701,
                  25.218409
                ],
                [
                  61.497363,
                  25.078237
                ],
                [
                  61.874187,
                  26.239975
                ],
                [
                  63.316632,
                  26.756532
                ],
                [
                  63.233898,
                  27.217047
                ],
                [
                  62.755426,
                  27.378923
                ],
                [
                  62.72783,
                  28.259645
                ],
                [
                  61.771868,
                  28.699334
                ],
                [
                  61.369309,
                  29.303276
                ],
                [
                  60.874248,
                  29.829239
                ],
                [
                  62.549857,
                  29.318572
                ],
                [
                  63.550261,
                  29.468331
                ],
                [
                  64.148002,
                  29.340819
                ],
                [
                  64.350419,
                  29.560031
                ],
                [
                  65.046862,
                  29.472181
                ],
                [
                  66.346473,
                  29.887943
                ],
                [
                  66.381458,
                  30.738899
                ],
                [
                  66.938891,
                  31.304911
                ],
                [
                  67.683394,
                  31.303154
                ],
                [
                  67.792689,
                  31.58293
                ],
                [
                  68.556932,
                  31.71331
                ],
                [
                  68.926677,
                  31.620189
                ],
                [
                  69.317764,
                  31.901412
                ],
                [
                  69.262522,
                  32.501944
                ],
                [
                  69.687147,
                  33.105499
                ],
                [
                  70.323594,
                  33.358533
                ],
                [
                  69.930543,
                  34.02012
                ],
                [
                  70.881803,
                  33.988856
                ],
                [
                  71.156773,
                  34.348911
                ],
                [
                  71.115019,
                  34.733126
                ],
                [
                  71.613076,
                  35.153203
                ],
                [
                  71.498768,
                  35.650563
                ],
                [
                  71.262348,
                  36.074388
                ],
                [
                  71.846292,
                  36.509942
                ],
                [
                  72.920025,
                  36.720007
                ],
                [
                  74.067552,
                  36.836176
                ],
                [
                  74.575893,
                  37.020841
                ],
                [
                  75.158028,
                  37.133031
                ]
              ]
            ],
            "type": "Polygon"
          },
          "properties": {
            "code": "PAK",
            "group": "Countries",
            "name": "Pakistan"
          },
          "type": "Feature"
        },
        "temporal_extent": [
          "2021-08-01T00:00:00Z",
          "2021-09-30T23:59:59Z"
        ]
      },
      "process_id": "load_collection"
    },
    "load2": {
      "arguments": {
        "id": "SENTINEL2_L2A",
        "spatial_extent": {
          "_id": "pakistan",
          "geometry": {
            "coordinates": [
              [
                [
                  75.158028,
                  37.133031
                ],
                [
                  75.896897,
                  36.666806
                ],
                [
                  76.192848,
                  35.898403
                ],
                [
                  77.837451,
                  35.49401
                ],
                [
                  76.871722,
                  34.653544
                ],
                [
                  75.757061,
                  34.504923
                ],
                [
                  74.240203,
                  34.748887
                ],
                [
                  73.749948,
                  34.317699
                ],
                [
                  74.104294,
                  33.441473
                ],
                [
                  74.451559,
                  32.7649
                ],
                [
                  75.258642,
                  32.271105
                ],
                [
                  74.405929,
                  31.692639
                ],
                [
                  74.42138,
                  30.979815
                ],
                [
                  73.450638,
                  29.976413
                ],
                [
                  72.823752,
                  28.961592
                ],
                [
                  71.777666,
                  27.91318
                ],
                [
                  70.616496,
                  27.989196
                ],
                [
                  69.514393,
                  26.940966
                ],
                [
                  70.168927,
                  26.491872
                ],
                [
                  70.282873,
                  25.722229
                ],
                [
                  70.844699,
                  25.215102
                ],
                [
                  71.04324,
                  24.356524
                ],
                [
                  68.842599,
                  24.359134
                ],
                [
                  68.176645,
                  23.691965
                ],
                [
                  67.443667,
                  23.944844
                ],
                [
                  67.145442,
                  24.663611
                ],
                [
                  66.372828,
                  25.425141
                ],
                [
                  64.530408,
                  25.237039
                ],
                [
                  62.905701,
                  25.218409
                ],
                [
                  61.497363,
                  25.078237
                ],
                [
                  61.874187,
                  26.239975
                ],
                [
                  63.316632,
                  26.756532
                ],
                [
                  63.233898,
                  27.217047
                ],
                [
                  62.755426,
                  27.378923
                ],
                [
                  62.72783,
                  28.259645
                ],
                [
                  61.771868,
                  28.699334
                ],
                [
                  61.369309,
                  29.303276
                ],
                [
                  60.874248,
                  29.829239
                ],
                [
                  62.549857,
                  29.318572
                ],
                [
                  63.550261,
                  29.468331
                ],
                [
                  64.148002,
                  29.340819
                ],
                [
                  64.350419,
                  29.560031
                ],
                [
                  65.046862,
                  29.472181
                ],
                [
                  66.346473,
                  29.887943
                ],
                [
                  66.381458,
                  30.738899
                ],
                [
                  66.938891,
                  31.304911
                ],
                [
                  67.683394,
                  31.303154
                ],
                [
                  67.792689,
                  31.58293
                ],
                [
                  68.556932,
                  31.71331
                ],
                [
                  68.926677,
                  31.620189
                ],
                [
                  69.317764,
                  31.901412
                ],
                [
                  69.262522,
                  32.501944
                ],
                [
                  69.687147,
                  33.105499
                ],
                [
                  70.323594,
                  33.358533
                ],
                [
                  69.930543,
                  34.02012
                ],
                [
                  70.881803,
                  33.988856
                ],
                [
                  71.156773,
                  34.348911
                ],
                [
                  71.115019,
                  34.733126
                ],
                [
                  71.613076,
                  35.153203
                ],
                [
                  71.498768,
                  35.650563
                ],
                [
                  71.262348,
                  36.074388
                ],
                [
                  71.846292,
                  36.509942
                ],
                [
                  72.920025,
                  36.720007
                ],
                [
                  74.067552,
                  36.836176
                ],
                [
                  74.575893,
                  37.020841
                ],
                [
                  75.158028,
                  37.133031
                ]
              ]
            ],
            "type": "Polygon"
          },
          "properties": {
            "code": "PAK",
            "group": "Countries",
            "name": "Pakistan"
          },
          "type": "Feature"
        },
        "temporal_extent": [
          "2022-08-01T00:00:00Z",
          "2022-09-30T23:59:59Z"
        ]
      },
      "process_id": "load_collection"
    },
    "merge6": {
      "arguments": {
        "cube1": {
          "from_node": "add3"
        },
        "cube2": {
          "from_node": "add5"
        }
      },
      "process_id": "merge_cubes"
    },
    "reduce1": {
      "arguments": {
        "data": {
          "from_node": "load1"
        },
        "dimension": "t",
        "reducer": {
          "process_graph": {
            "median1": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                }
              },
              "process_id": "median",
              "result": true
            }
          }
        }
      },
      "process_id": "reduce_dimension"
    },
    "reduce2": {
      "arguments": {
        "data": {
          "from_node": "load2"
        },
        "dimension": "t",
        "reducer": {
          "process_graph": {
            "median1": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                }
              },
              "process_id": "median",
              "result": true
            }
          }
        }
      },
      "process_id": "reduce_dimension"
    },
    "reduce9": {
      "arguments": {
        "data": {
          "from_node": "merge6"
        },
        "dimension": "bands",
        "reducer": {
          "process_graph": {
            "array1": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                },
                "index": 1,
                "return_nodata": true
              },
              "process_id": "array_element"
            },
            "array2": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                },
                "index": 0,
                "return_nodata": true
              },
              "process_id": "array_element"
            },
            "normalized2": {
              "arguments": {
                "x": {
                  "from_node": "array2"
                },
                "y": {
                  "from_node": "array1"
                }
              },
              "process_id": "normalized_difference",
              "result": true
            }
          }
        }
      },
      "process_id": "reduce_dimension"
    },
    "save11": {
      "arguments": {
        "data": {
          "from_node": "reduce9"
        },
        "format": "GTIFF"
      },
      "process_id": "save_result",
      "result": true
    }
  }
}

I’m also having trouble finding that job in our records, I assume you ran it just now?
Did you try running it more than once?

thanks!

1 Like

Ran it just an hour ago, restarted now and I’m getting “‘Cannot create a polygon with exterior with fewer than 4 points: LINEARRING EMPTY’” now. It’s confusing because I provided a bbox, not a polygon. Also trying a simplified version, which doesn’t execute the last reducer, which runs for quite a while now so it might be something in there…

Your last job was pretty close to finishing after 13 hours, but then failed because the (compressed) geotiff size got over 5GB:

In general we test our backend up to 100x100km areas, beyond that it may still work, but not in this case apparently. I recommend using a tile grid for that, which can also be used by the aggregator to automatically split your job.

I also see that the data type of your tiff is float32. If you can convert it to 8 or 16 bit, you may also get more compression and go below this 5GB limit.

We also have to consider that we now loaded a lot of data from sentinelhub. On open-dev.vito.be, that would probably have gone via a batch job, which would be better for such a large area. Or we could try running it on the creo backend.

1 Like

Thank you, @jeroen.dries. I did not expect that it gets so large, I’ll try to reduce the size. The data type is a good point, this can easily be changed to uint8.

A general question is whether now that the data was loaded from SH once, is it cached (and faster) or will it be loaded from SH again if I try again? Is there an alternative collection on Terrascope for this? I must say with all the different collections I’m always a bit confused what’s the best choice.

I don’t quite get what you are saying about running on openeo-dev.vito.be and batch jobs because on Platform as it was also a batch job?

I’m afraid data is not cached for the prod endpoint, and there’s also no alternative collection for that area. Perhaps you can try to run it over a similar area in Europe if it’s a matter of seeing if it is possible.

(The remark about batch jobs was referring to Sentinelhub batch jobs.)

I’m wondering @daniel.thiex wouldn’t this example also run on the new Sentinelhub backend directly? That would at least avoid all this data transfer.

best regards,
Jeroen

Thanks, yes, I actually tried to run it on SH first using the new web service approach, but I got that the Sentinel_L2A collection is not supported or so: https://github.com/openEOPlatform/architecture-docs/issues/307 Maybe that’s a bug, then I’d indeed prefer to run it directly on SH. @daniel.thiex

Eventually, I’d like to run this over Pakistan for summer 2022 so I can experiment over Europe but later I’d need to go back to SH, it seems.

I was also trying to run it on creo now, and noticed that you do not seem to filter on bands. As a result, you first load all bands, and then go on to use only two.
Can you add either filter_bands, or specify a list in load_collection? This has a quite drastic impact.

In a similar way, it’s always good to add a filter on eo:cloud_cover, maybe setting it to 99% as this is about floods.

1 Like

Thanks, yes, I actually tried to run it on SH first using the new web service approach, but I got that the Sentinel_L2A collection is not supported or so: https://github.com/openEOPlatform/architecture-docs/issues/307 Maybe that’s a bug, then I’d indeed prefer to run it directly on SH. @daniel.thiex

I replied in the issue. In short: The Sentinel L2A collection from Sentinel Hub is added under the id SENTINEL2_L2A_SENTINELHUB. We are supporting all the processes but when running it now I got some error. We will look into it.

1 Like