Boolean mask and mask_polygon issue

michele.claus · 27 September 2023 13:04

Dear all,

I am having issues while working with mask_polygon and boolean masks.

I attach a self-explaining notebook, plus the geojson file used in it, plus some screenshots.
Files are here: https://github.com/clausmichele/boolean-mask-openeo-issue/blob/main/boolean_mask_issues.ipynb

I am using CDSE as a back-end, but I am sure the same issue would appear with openeo.cloud using the VITO back-end.

So, if I create a boolean mask and download it, it’s wrong.
If I multiply it by 1.0, it is correct but now the masked area is lost and I get those chunks at zero instead of nans.

I tag someone of the VITO team for visibility:
@stefaan.lippens @jeroen.verstraelen @jeroen.dries @pratichhya.sharma

stefaan.lippens · 27 September 2023 14:18

Just a quick question: in the binary version, could it be that the purple pixel values are 0 and 1, which is too subtle to see in that plot?

michele.claus · 28 September 2023 07:22

Hi @stefaan.lippens, you are right. Basically the main difference is that in the first version, the areas that are at no data of the second have value 129 instead of nan.
Anyway, both results are wrong, apparently when doing the comparison step, the no data values get messed up.

stefaan.lippens · 28 September 2023 09:37

In case of the binary version, you are probably handling a byte-value pixels, where nodata is encoded as byte value 129.

In the float version of the mas, the nodata is encoded as a float nan, which is what you expect right?

I’m not sure I understand what is wrong with the float version of your mask. The nodata values are correctly handled as far as I understand your example and screenshots

michele.claus · 28 September 2023 09:51

Please check the notebook that I’ve shared, it’s pretty clear there. I used mask_polygon and after creating the boolean mask some areas that where nans became either 129 or 0.

github.com

clausmichele/boolean-mask-openeo-issue/blob/main/boolean_mask_issues.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cb2537d1-ff1e-4ac4-b01b-51361945d63a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# platform libraries\n",
    "import openeo\n",
    "\n",
    "# utility libraries\n",
    "from datetime import date\n",
    "import numpy as np\n",
    "import xarray as xr\n",
    "import rioxarray\n",
    "import json\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",

This file has been truncated. show original

michele.claus · 2 October 2023 12:49

@stefaan.lippens did you have time to check the reason of this problem? Is there an internal ticket to follow?

stefaan.lippens · 2 October 2023 14:07

I just had a quick look.

I think the problem is that the VITO backend does not preserve nan-ness in comparisons: like ndsi > 0.4: pixels above 0.4 get value 1 and all other values (below 0.4, and nan) get value 0. This is probably because the data type in the implementation is just binary and there is no room for a third value like nan.

stefaan.lippens · 2 October 2023 14:21

Additionally, there is an optimization to discard (internal) tiles that are completely nan.
As a result you can have multiple outputs where you expect nan:

in tiles not covered by polygon mask: value nan (encoded as 129 in binary output)
in tiles partially covered by polygon: value 0

that’s why you get a mix of 0/nan for float output (or 0/129 for binary output) in those blocky/staircase artifacts

michele.claus · 2 October 2023 14:26

Ok, but how should a normal user get to know about these details? It’s a bit frustrating for an advanced user like me already, I can imagine how it would be for a newbie.

I guess that if a process is implemented in a different way than the specs should be documented somewhere and probably also the exposed process definitions should be aligned: https://openeo.vito.be/openeo/1.1/processes

stefaan.lippens · 2 October 2023 14:27

made it a ticket here: Preserve nan in comparisons · Issue #527 · Open-EO/openeo-geopyspark-driver · GitHub

stefaan.lippens · 2 October 2023 14:49

I don’t think it is intentional this is implemented differently from the spec.
I think it’s even possible that the VITO implementation came before the process spec was fully settled on nan-handling.

cc @jeroen.dries

jeroen.dries · 3 October 2023 05:12

Indeed, it’s not intentional. It’s absolutely correct that we don’t want this to happen, which is one of the reasons for investing in a cross-backend test-suite.