Hey, when I want to execute a batch job I get the following error and don’t know what to make of it. Can someone help out?
Traceback (most recent call last): File “/data4/hadoop/yarn/local/usercache/openeo/appcache/application_1648799794287_6155/container_e5018_1648799794287_6155_01_000002/venv/lib/python3.8/site-packages/openeogeotrellis/backend.py”, line 1504, in get_log_entries with (self.get_job_output_dir(job_id) / “log”).open(‘r’) as f: File “/usr/lib64/python3.8/pathlib.py”, line 1200, in open return io.open(self, mode, buffering, encoding, errors, newline, File “/usr/lib64/python3.8/pathlib.py”, line 1054, in _opener return self._accessor.open(self, flags, mode) FileNotFoundError: [Errno 2] No such file or directory: ‘/data/projects/OpenEO/e5bd4519-e608-4cc1-b6c5-fe19b480853b/log’
1 Like
when do you get this error? Immediately when starting the job? After some time but before the job reaches status “running”? While running? After the job failed with status “error”?
And do you always get this problem, or just on one particular job?
I get the exact same error, after the job had status “queued” for some time. This happens with jobs that I can execute synchronously. For example id: 1e83f706-863f-49cb-8c43-f721006fde49.
I also get this when I request a logs (in the Web Editor) for an errored job:
Traceback (most recent call last): File "/data3/hadoop/yarn/local/usercache/openeo/appcache/application_1648799794287_13553/container_e5018_1648799794287_13553_01_000001/venv/lib/python3.8/site-packages/openeogeotrellis/backend.py", line 1504, in get_log_entries with (self.get_job_output_dir(job_id) / "log").open('r') as f: File "/usr/lib64/python3.8/pathlib.py", line 1200, in open return io.open(self, mode, buffering, encoding, errors, newline, File "/usr/lib64/python3.8/pathlib.py", line 1054, in _opener return self._accessor.open(self, flags, mode) FileNotFoundError: [Errno 2] No such file or directory: '/data/projects/OpenEO/cc118321-f5e2-4647-aba6-2ff98a6721c0/log'
Sounds like something with the log files broke at VITO? I also get this directly connected to the VITO production back-end. This happens for all jobs that I’ve started recently. Example job id (directly on VITO):
cc118321-f5e2-4647-aba6-2ff98a6721c0
Right now I don’t know how to debug errors, because I can’t retrieve any information about the errors. @jeroen.dries
Note that we’re looking at the root cause for this.
Jobs are not really broken, this is just a case for which no logging exists yet, hence the unhelpful message. We’ll try to get rid of it!
@jeroen.dries Then I’m wondering what’s going on with cc118321-f5e2-4647-aba6-2ff98a6721c0? It runs as sync job, but as batch job it fails with the message reported above. I understood your message as such:
Job errors → log should be available and retrievable (but I can’t figure out what the issue is)
Job finishes → logs can’t be retrieved (i.e. gibe the error above) as no logs are available?
yes, batch and sync are two different paths, as I said, we’re looking at it!