No error for flood mapping

m.mohr · 24 January 2023 11:24

The process below (flood mapping over Pakistan) gives me no error when erroring. Job ID:
vito-j-c6b472c8840747c987be5698e8fa507b

{
  "process_graph": {
    "add3": {
      "arguments": {
        "data": {
          "from_node": "reduce1"
        },
        "label": "2022",
        "name": "t",
        "type": "temporal"
      },
      "process_id": "add_dimension"
    },
    "add5": {
      "arguments": {
        "data": {
          "from_node": "reduce2"
        },
        "label": "2022",
        "name": "t",
        "type": "temporal"
      },
      "process_id": "add_dimension"
    },
    "load1": {
      "arguments": {
        "id": "SENTINEL2_L2A",
        "spatial_extent": {
          "_id": "pakistan",
          "geometry": {
            "coordinates": [
              [
                [
                  75.158028,
                  37.133031
                ],
                [
                  75.896897,
                  36.666806
                ],
                [
                  76.192848,
                  35.898403
                ],
                [
                  77.837451,
                  35.49401
                ],
                [
                  76.871722,
                  34.653544
                ],
                [
                  75.757061,
                  34.504923
                ],
                [
                  74.240203,
                  34.748887
                ],
                [
                  73.749948,
                  34.317699
                ],
                [
                  74.104294,
                  33.441473
                ],
                [
                  74.451559,
                  32.7649
                ],
                [
                  75.258642,
                  32.271105
                ],
                [
                  74.405929,
                  31.692639
                ],
                [
                  74.42138,
                  30.979815
                ],
                [
                  73.450638,
                  29.976413
                ],
                [
                  72.823752,
                  28.961592
                ],
                [
                  71.777666,
                  27.91318
                ],
                [
                  70.616496,
                  27.989196
                ],
                [
                  69.514393,
                  26.940966
                ],
                [
                  70.168927,
                  26.491872
                ],
                [
                  70.282873,
                  25.722229
                ],
                [
                  70.844699,
                  25.215102
                ],
                [
                  71.04324,
                  24.356524
                ],
                [
                  68.842599,
                  24.359134
                ],
                [
                  68.176645,
                  23.691965
                ],
                [
                  67.443667,
                  23.944844
                ],
                [
                  67.145442,
                  24.663611
                ],
                [
                  66.372828,
                  25.425141
                ],
                [
                  64.530408,
                  25.237039
                ],
                [
                  62.905701,
                  25.218409
                ],
                [
                  61.497363,
                  25.078237
                ],
                [
                  61.874187,
                  26.239975
                ],
                [
                  63.316632,
                  26.756532
                ],
                [
                  63.233898,
                  27.217047
                ],
                [
                  62.755426,
                  27.378923
                ],
                [
                  62.72783,
                  28.259645
                ],
                [
                  61.771868,
                  28.699334
                ],
                [
                  61.369309,
                  29.303276
                ],
                [
                  60.874248,
                  29.829239
                ],
                [
                  62.549857,
                  29.318572
                ],
                [
                  63.550261,
                  29.468331
                ],
                [
                  64.148002,
                  29.340819
                ],
                [
                  64.350419,
                  29.560031
                ],
                [
                  65.046862,
                  29.472181
                ],
                [
                  66.346473,
                  29.887943
                ],
                [
                  66.381458,
                  30.738899
                ],
                [
                  66.938891,
                  31.304911
                ],
                [
                  67.683394,
                  31.303154
                ],
                [
                  67.792689,
                  31.58293
                ],
                [
                  68.556932,
                  31.71331
                ],
                [
                  68.926677,
                  31.620189
                ],
                [
                  69.317764,
                  31.901412
                ],
                [
                  69.262522,
                  32.501944
                ],
                [
                  69.687147,
                  33.105499
                ],
                [
                  70.323594,
                  33.358533
                ],
                [
                  69.930543,
                  34.02012
                ],
                [
                  70.881803,
                  33.988856
                ],
                [
                  71.156773,
                  34.348911
                ],
                [
                  71.115019,
                  34.733126
                ],
                [
                  71.613076,
                  35.153203
                ],
                [
                  71.498768,
                  35.650563
                ],
                [
                  71.262348,
                  36.074388
                ],
                [
                  71.846292,
                  36.509942
                ],
                [
                  72.920025,
                  36.720007
                ],
                [
                  74.067552,
                  36.836176
                ],
                [
                  74.575893,
                  37.020841
                ],
                [
                  75.158028,
                  37.133031
                ]
              ]
            ],
            "type": "Polygon"
          },
          "properties": {
            "code": "PAK",
            "group": "Countries",
            "name": "Pakistan"
          },
          "type": "Feature"
        },
        "temporal_extent": [
          "2021-08-01T00:00:00Z",
          "2021-09-30T23:59:59Z"
        ]
      },
      "process_id": "load_collection"
    },
    "load2": {
      "arguments": {
        "id": "SENTINEL2_L2A",
        "spatial_extent": {
          "_id": "pakistan",
          "geometry": {
            "coordinates": [
              [
                [
                  75.158028,
                  37.133031
                ],
                [
                  75.896897,
                  36.666806
                ],
                [
                  76.192848,
                  35.898403
                ],
                [
                  77.837451,
                  35.49401
                ],
                [
                  76.871722,
                  34.653544
                ],
                [
                  75.757061,
                  34.504923
                ],
                [
                  74.240203,
                  34.748887
                ],
                [
                  73.749948,
                  34.317699
                ],
                [
                  74.104294,
                  33.441473
                ],
                [
                  74.451559,
                  32.7649
                ],
                [
                  75.258642,
                  32.271105
                ],
                [
                  74.405929,
                  31.692639
                ],
                [
                  74.42138,
                  30.979815
                ],
                [
                  73.450638,
                  29.976413
                ],
                [
                  72.823752,
                  28.961592
                ],
                [
                  71.777666,
                  27.91318
                ],
                [
                  70.616496,
                  27.989196
                ],
                [
                  69.514393,
                  26.940966
                ],
                [
                  70.168927,
                  26.491872
                ],
                [
                  70.282873,
                  25.722229
                ],
                [
                  70.844699,
                  25.215102
                ],
                [
                  71.04324,
                  24.356524
                ],
                [
                  68.842599,
                  24.359134
                ],
                [
                  68.176645,
                  23.691965
                ],
                [
                  67.443667,
                  23.944844
                ],
                [
                  67.145442,
                  24.663611
                ],
                [
                  66.372828,
                  25.425141
                ],
                [
                  64.530408,
                  25.237039
                ],
                [
                  62.905701,
                  25.218409
                ],
                [
                  61.497363,
                  25.078237
                ],
                [
                  61.874187,
                  26.239975
                ],
                [
                  63.316632,
                  26.756532
                ],
                [
                  63.233898,
                  27.217047
                ],
                [
                  62.755426,
                  27.378923
                ],
                [
                  62.72783,
                  28.259645
                ],
                [
                  61.771868,
                  28.699334
                ],
                [
                  61.369309,
                  29.303276
                ],
                [
                  60.874248,
                  29.829239
                ],
                [
                  62.549857,
                  29.318572
                ],
                [
                  63.550261,
                  29.468331
                ],
                [
                  64.148002,
                  29.340819
                ],
                [
                  64.350419,
                  29.560031
                ],
                [
                  65.046862,
                  29.472181
                ],
                [
                  66.346473,
                  29.887943
                ],
                [
                  66.381458,
                  30.738899
                ],
                [
                  66.938891,
                  31.304911
                ],
                [
                  67.683394,
                  31.303154
                ],
                [
                  67.792689,
                  31.58293
                ],
                [
                  68.556932,
                  31.71331
                ],
                [
                  68.926677,
                  31.620189
                ],
                [
                  69.317764,
                  31.901412
                ],
                [
                  69.262522,
                  32.501944
                ],
                [
                  69.687147,
                  33.105499
                ],
                [
                  70.323594,
                  33.358533
                ],
                [
                  69.930543,
                  34.02012
                ],
                [
                  70.881803,
                  33.988856
                ],
                [
                  71.156773,
                  34.348911
                ],
                [
                  71.115019,
                  34.733126
                ],
                [
                  71.613076,
                  35.153203
                ],
                [
                  71.498768,
                  35.650563
                ],
                [
                  71.262348,
                  36.074388
                ],
                [
                  71.846292,
                  36.509942
                ],
                [
                  72.920025,
                  36.720007
                ],
                [
                  74.067552,
                  36.836176
                ],
                [
                  74.575893,
                  37.020841
                ],
                [
                  75.158028,
                  37.133031
                ]
              ]
            ],
            "type": "Polygon"
          },
          "properties": {
            "code": "PAK",
            "group": "Countries",
            "name": "Pakistan"
          },
          "type": "Feature"
        },
        "temporal_extent": [
          "2022-08-01T00:00:00Z",
          "2022-09-30T23:59:59Z"
        ]
      },
      "process_id": "load_collection"
    },
    "merge6": {
      "arguments": {
        "cube1": {
          "from_node": "add3"
        },
        "cube2": {
          "from_node": "add5"
        }
      },
      "process_id": "merge_cubes"
    },
    "reduce1": {
      "arguments": {
        "data": {
          "from_node": "load1"
        },
        "dimension": "t",
        "reducer": {
          "process_graph": {
            "median1": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                }
              },
              "process_id": "median",
              "result": true
            }
          }
        }
      },
      "process_id": "reduce_dimension"
    },
    "reduce2": {
      "arguments": {
        "data": {
          "from_node": "load2"
        },
        "dimension": "t",
        "reducer": {
          "process_graph": {
            "median1": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                }
              },
              "process_id": "median",
              "result": true
            }
          }
        }
      },
      "process_id": "reduce_dimension"
    },
    "reduce9": {
      "arguments": {
        "data": {
          "from_node": "merge6"
        },
        "dimension": "bands",
        "reducer": {
          "process_graph": {
            "array1": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                },
                "index": 1,
                "return_nodata": true
              },
              "process_id": "array_element"
            },
            "array2": {
              "arguments": {
                "data": {
                  "from_parameter": "data"
                },
                "index": 0,
                "return_nodata": true
              },
              "process_id": "array_element"
            },
            "normalized2": {
              "arguments": {
                "x": {
                  "from_node": "array2"
                },
                "y": {
                  "from_node": "array1"
                }
              },
              "process_id": "normalized_difference",
              "result": true
            }
          }
        }
      },
      "process_id": "reduce_dimension"
    },
    "save11": {
      "arguments": {
        "data": {
          "from_node": "reduce9"
        },
        "format": "GTIFF"
      },
      "process_id": "save_result",
      "result": true
    }
  }
}

jeroen.dries · 24 January 2023 11:58

I’m also having trouble finding that job in our records, I assume you ran it just now?
Did you try running it more than once?

thanks!

m.mohr · 24 January 2023 12:11

Ran it just an hour ago, restarted now and I’m getting “‘Cannot create a polygon with exterior with fewer than 4 points: LINEARRING EMPTY’” now. It’s confusing because I provided a bbox, not a polygon. Also trying a simplified version, which doesn’t execute the last reducer, which runs for quite a while now so it might be something in there…

jeroen.dries · 25 January 2023 07:04

Your last job was pretty close to finishing after 13 hours, but then failed because the (compressed) geotiff size got over 5GB:

In general we test our backend up to 100x100km areas, beyond that it may still work, but not in this case apparently. I recommend using a tile grid for that, which can also be used by the aggregator to automatically split your job.

I also see that the data type of your tiff is float32. If you can convert it to 8 or 16 bit, you may also get more compression and go below this 5GB limit.

We also have to consider that we now loaded a lot of data from sentinelhub. On open-dev.vito.be, that would probably have gone via a batch job, which would be better for such a large area. Or we could try running it on the creo backend.

m.mohr · 25 January 2023 09:04

Thank you, @jeroen.dries. I did not expect that it gets so large, I’ll try to reduce the size. The data type is a good point, this can easily be changed to uint8.

A general question is whether now that the data was loaded from SH once, is it cached (and faster) or will it be loaded from SH again if I try again? Is there an alternative collection on Terrascope for this? I must say with all the different collections I’m always a bit confused what’s the best choice.

I don’t quite get what you are saying about running on openeo-dev.vito.be and batch jobs because on Platform as it was also a batch job?

jeroen.dries · 26 January 2023 08:35

I’m afraid data is not cached for the prod endpoint, and there’s also no alternative collection for that area. Perhaps you can try to run it over a similar area in Europe if it’s a matter of seeing if it is possible.

(The remark about batch jobs was referring to Sentinelhub batch jobs.)

I’m wondering @daniel.thiex wouldn’t this example also run on the new Sentinelhub backend directly? That would at least avoid all this data transfer.

best regards,
Jeroen

m.mohr · 26 January 2023 08:50

Thanks, yes, I actually tried to run it on SH first using the new web service approach, but I got that the Sentinel_L2A collection is not supported or so: https://github.com/openEOPlatform/architecture-docs/issues/307 Maybe that’s a bug, then I’d indeed prefer to run it directly on SH. @daniel.thiex

Eventually, I’d like to run this over Pakistan for summer 2022 so I can experiment over Europe but later I’d need to go back to SH, it seems.

jeroen.dries · 26 January 2023 08:58

I was also trying to run it on creo now, and noticed that you do not seem to filter on bands. As a result, you first load all bands, and then go on to use only two.
Can you add either filter_bands, or specify a list in load_collection? This has a quite drastic impact.

In a similar way, it’s always good to add a filter on eo:cloud_cover, maybe setting it to 99% as this is about floods.

daniel.thiex · 26 January 2023 13:43

Thanks, yes, I actually tried to run it on SH first using the new web service approach, but I got that the Sentinel_L2A collection is not supported or so: https://github.com/openEOPlatform/architecture-docs/issues/307 Maybe that’s a bug, then I’d indeed prefer to run it directly on SH. @daniel.thiex

I replied in the issue. In short: The Sentinel L2A collection from Sentinel Hub is added under the id SENTINEL2_L2A_SENTINELHUB. We are supporting all the processes but when running it now I got some error. We will look into it.