[Core] Persist the Driver Console Log When Job Execution Not Through Job API

[MengjinYan]

Why are these changes needed?

Currently, when user submitting a Ray job through interactive shell or running a python script, the job's log will be output to the console and will not be persisted anywhere. This causes loss of log information in the case where the user disconnected from the console.

This PR added the logic to persist the log information at the same time, still output the log information to the console. To be specific:

1. An environment variable RAY_DRIVER_CONSOLE_LOG_TO_FILE will be used as a switch to the behavior. Only when (1) the env var is set to "1" AND (2) The job is not submitted using Job API (meaning both the sys.stderr and sys.stdout are not redirected), will the log be persisted.
2. The log information will be persisted under the current working directory, with file name console_{datetime (%Y-%m-%d_%H-%M-%S_%f)}.log
3. During the redirection, the output of stdour and stderr will be redirected to the above file. At the same time, a ConsoleLogTailer will be executed in a thread to tail the console log file and output the logs to the console. The same function used in job log tailing is reused.
4. Corresponding tests are added as well.

Related issue number

Changes needed on the Ray Core side for https://github.com/anyscale/rayturbo/issues/720

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rynewang]

🤔 instead of manually doing read-stdout-and-write-to-file every 0.1s, can we just open the log file and replace the stdout object to a dual writing code like this?

import sys

class DualOutput:
    def __init__(self, filename):
        self.filename = filename
        self.file = open(filename, "a+")
        self.stream = sys.stdout
        self.old_stdout = sys.stdout
        sys.stdout = self

    def __del__(self):
        self.file.close()
        sys.stdout = self.old_stdout

    def write(self, message):
        self.file.write(message)  # Write to the log file
        self.stream.write(message)  # Write to the original stdout

    def flush(self):
        self.file.flush()
        self.stream.flush()

# Usage example:
dual_output = DualOutput("logfile.log")

# Example output
print("This will be shown to the user and written to the log file.")

# at exit... Restore the original stdout
del dual_output

also this won't get log rotations. cc @dentiny who's working on log rotations on user prints in worker processes.