21:34 <johnsdea> Hi folks, does anyone know of a way to add a log file which has already been written as part of a step? I tried 'addLogFile(name, path)' + adding the logObserver, only to find out that it only watches for *updates* to the file. In this case the file is already written, we just want to add it to Buildbot.

21:36 <johnsdea> I'm considering working around the issue by making our step 'mv <file> /tmp/<file> && mv /tmp/<file> <file>`, but it feels like there should be a better way.

22:07 <p12tic> johnsdea: does logfiles argument to shell command work for you? (the underlying RemoteShellCommand supports it too)

22:08 <p12tic> That gives you log contents back to the master

22:08 <p12tic> You should be able to add a log observer for that

22:09 <p12tic> addLogFile is no longer present since v3.0

22:09 <p12tic> Joe80: The behaviour you're seeing looks like a bug, could you file a bug report on Github with more information?

22:10 <johnsdea> We're using a bit of an older version, but setting `logfiles` is what happens under the hood when we call `addLogFile`. The documentation states: "The logfiles= argument to ShellCommand and its subclasses creates new log files and fills them in realtime by asking the worker to watch an actual file on disk. The worker will look for additions in the target file and report them back to the BuildStep. These

22:10 <johnsdea> additions will be added to the log file by calling addStdout."

22:10 <johnsdea> via http://docs.buildbot.net/current/manual/customization.html#writing-log-files

22:11 <johnsdea> So, it's watching for changes to a file which isn't being changed in that step, and thus creating an empty entry for that file on the step (since we don't set lazyLogFiles)

22:14 <p12tic> Could it be that addLogFile enables follow=True by default? LogFileWatcher.poll() reads file from beginning if not following

22:16 <johnsdea> Hm, not sure if our version of Buildbot supports a follow argument; is that new to v3.0+?

22:19 <johnsdea> That'ss in Buildbot worker -- how would we go about making the master communicate to the worker that the logfile should be poll'd?

22:20 <p12tic> It's documented here: https://docs.buildbot.net/current/manual/configuration/steps/shell_command.html (logfiles argument), though I see that default value is false

22:22 <p12tic> Could it be that you're adding the log observer after you've launched the command?

22:23 <p12tic> In that case you'd get only updates from the point you've added the observer and it could be that the bulk of the file has already been arrived

22:23 <johnsdea> No, because we add the logFile as part of constructing the command.

22:23 <johnsdea> Sorry if I'm miscommunicating. The logfile already exists, and isn't being modified at all.

22:24 <johnsdea> We're basically just trying to upload / attach it to buildbot underneath a step.

22:24 <p12tic> I understand, that should work.

22:25 <p12tic> Could you try to launch a separate toy Buildbot instance and try to reproduce the problem on v3.x using the logfiles argument to shell command?

22:26 <p12tic> If it reproduces, I will look into it. Fixing it on v3.x would likely offer a way to workaround the problem on older versions of Buildbot

22:26 <p12tic> Maybe you'll be able to just apply the same patch

22:29 <johnsdea> Actually, maybe this is the bug? https://github.com/buildbot/buildbot/blob/414312bf410e232d239c9e9b0cfe0eb0042c58ee/worker/buildbot_worker/runprocess.py#L147-L165

22:36 <johnsdea> Namely this spot -- since the step hasn't started (and the file already existed), `poll` would early return: https://github.com/buildbot/buildbot/blob/414312bf410e232d239c9e9b0cfe0eb0042c58ee/worker/buildbot_worker/runprocess.py#L149-L151

22:37 <johnsdea> And because of that early return, polling is never started: https://github.com/buildbot/buildbot/blob/414312bf410e232d239c9e9b0cfe0eb0042c58ee/worker/buildbot_worker/runprocess.py#L164

22:57 <johnsdea> ^ I was able to confirm that by calling `touch` on the file during the script the step runs (which causes MTIME to change, thereby not hitting the early return), it works as expected.