Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
-
None
Description
Thermos is failing to start when log rotation is enabled as described in our documentation
Steps to reproduce:
- start the latest developer vagrant box (or even then released 0.14)
- update to scheduler wide thermos_executor_flags to include --runner-logger-mode=rotate --runner-rotate-log-size-mb=100 --runner-rotate-log-backups=10 as documented (see above)
- launch a task
- observe that the task has failed with LOST : Runner died while task was active.
The thermos stderr then contains this traceback:
ERROR] Caught exception in self.control(): Logger mode rotate is invalid. ERROR] Traceback (most recent call last): File "apache/thermos/core/runner.py", line 572, in control yield File "apache/thermos/core/runner.py", line 916, in run self._run() File "apache/thermos/core/runner.py", line 923, in _run iteration_wait = runner.run() File "apache/thermos/core/runner.py", line 291, in run launched = self.runner._run_plan(self.runner._regular_plan) File "apache/thermos/core/runner.py", line 843, in _run_plan self._set_process_status(process_name, ProcessState.WAITING) File "apache/thermos/core/runner.py", line 687, in _set_process_status self._dispatcher.dispatch(self._state, runner_ckpt, self._recovery) File "apache/thermos/common/ckpt.py", line 384, in dispatch self._run_process_dispatch(process_update.state, process_update) File "apache/thermos/common/ckpt.py", line 215, in _run_process_dispatch getattr(handler, handler_function)(process_update) File "apache/thermos/core/runner.py", line 106, in on_waiting process_update.process, process_update.seq + 1)) File "apache/thermos/core/runner.py", line 723, in _task_process_from_process_name preserve_env=self._preserve_env) File "apache/thermos/core/process.py", line 348, in __init__ ProcessBase.__init__(self, *args, **kw) File "apache/thermos/core/process.py", line 142, in __init__ raise ValueError("Logger mode %s is invalid." % self._logger_mode) ValueError: Logger mode rotate is invalid.
What's kind of obscure here: "rotate" is one of the supported modes.
Attachments
Issue Links
- is related to
-
AURORA-1730 Improve pystachio Logger schema
- Open