Error when import torch

openEO Platform
1

Hi I have encountered an issue while running my own user defined function regarding pytorch dependencies. The error messages are as follows:

raise ImportError(textwrap.dedent(‘’’ ImportError: Failed to load PyTorch C extensions: It appears that PyTorch has loaded the torch/_C folder of the PyTorch repository rather than the C extensions which are expected in the torch._C namespace. This can occur when using the install workflow. e.g. $ python setup.py install && python -c “import torch” This error can generally be solved using the develop workflow $ python setup.py develop && python -c “import torch” # This should succeed or by running Python from a different directory.

I tried using zipped environment and creating environment remotely, but none of these worked. I wonder what might cause this issue? Thank you in advance!

2

Hi Mingshi,

Can you share the job-id from a task you want to run?

It is not recommended torch neural networks directly from within an UDF. We would recommend to run the network with ONNX instead.
The people involved in that are on vacation this week, but I hear a good example on how to use it can be found here:

(It is a pull request that will be an official example soon)

3

Thanks, I am trying to implement this right now. My job ID was: j-2401303443f54504af76d34ba50f842e. Another problem is when I was trying to connect to OpenEO backend with authenticate_oidc(), there is always a JSONDecodeError: Expecting value: line 1 column 1 (char 0), which seems to have something to do with my refresh token but I have no clue what causes this problem either.

4

Hi Emile,

I tried implementing this onnx example and it turned out to be still the same, I still have dependency error regarding pytorch. Is it possible for you to give a quick look at my python script? If not, is there any other workarounds? Many thanks!

https://github.com/limingshi1994/Onnx_implementation

5

Hi Mingshi,

The JSONDecodeError appears to be related to the environment you are running your notebook in. Are you running this on your local machine?

For the Pytorch issue you will have to rewrite your udf to use the onnxruntime and remove any pytorch imports:
import onnxruntime as ort

Then you can pass the onnxruntime and your model as dependencies to execute_batch:

dependencies_url = "https://artifactory.vgt.vito.be:443/auxdata-public/openeo/onnx_dependencies.zip"
model_url = "https://artifactory.vgt.vito.be:443/auxdata-public/openeo/test_onnx_model.zip"

job_options = {
    "udf-dependency-archives": [
        f"{dependencies_url}#onnx_deps",
        f"{model_url}#onnx_models",
    ],
}
s2_cube.execute_batch("output.nc", job_options=job_options)

Hope that helps,
Jeroen