Cannot make to rotate log file of fluentd application on windows

[ISChetverikov]

fluentd or td-agent version.

td-agent-3.4.1-0-x64

Environment information:

Operating system:
Microsoft Windows 10 Enterprise 1703
BuildNumber: 15063
Version: 10.0.15063
OSArchitecture: 64-bit
Kernel version:

Your configuration

C:\opt\td-agent\c

# listen http port
<source>
  @type http
  port 9880
  bind 0.0.0.0
</source>

# send handled messages to elastic
<match **>
	@type copy
	<store>                                                 
		@type stdout
	</store> 
	<store>
	@type elasticsearch_dynamic
	host my.remote.host
	port 9200
	logstash_prefix buffer_test
	logstash_format true
	type_name logevent
	<buffer>
		@type file
		path buffer/Handled
		total_limit_size 10MB
		overflow_action drop_oldest_chunk
		queued_chunks_limit_size 10
		flush_interval 4s
		retry_forever true
		flush_thread_count 2
	</buffer>
	</store>
</match>

Your problem explanation. If you have an error logs, write it together.

Hello! I cannot make to rotate logs file of fluentd application. I use the following command under administrator user to launch program:

fluentd -c c -o C:\opt\td-agent\log --log-rotate-age 3 --log-rotate-size 1000000

But when file C:\opt\td-agent\log reaches the limit of 1000000 bytes, I get messages:

log shifting failed. Permission denied @ rb_file_s_rename - (C:\opt\td-agent\log, C:\opt\td-agent\log.0)
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
log shifting failed. closed stream
log writing failed. closed stream
.... (after that many of these lines)

[repeatedly]

Hmm... Does windows have different permission for file creation and file rename?

[repeatedly]

rename requires delete permission. Could you check the permission?

[ISChetverikov]

Thank you for your response. I have checked file's ("C:\opt\td-agent\log") permissions and it has Full Control permissions including Delete.

[ISChetverikov]

I've tried to set Full Control permissions for Everyone on C:\opt folder, but log rotation still does not work

[Coldln]

I'm facing the same issue with Windows 7

[repeatedly]

@cosmo0920 If you have a time, could you check this? fluentd log depends on serverengine.

[cosmo0920]

Yep, I'll check this.

I guess that Windows API complains that opened files cannot delete from other process or same process.
In Windows, emulating unlink behavior on Unix like OS is very hard.

[cosmo0920]

I checked the serverengine and Ruby logger library code.
ServerEngine delegates logrotate feature to Ruby logger library.

I found that Ruby logger does not use File::SHARE_DELETE flag which had been introduced in Ruby 2.3:

• https://github.com/ruby/ruby/blob/master/lib/logger/log_device.rb#L101
• https://github.com/ruby/ruby/blob/master/lib/logger/log_device.rb#L93

I guess that it's a bug for Ruby logger library.

[repeatedly]

Awesome! On windows, logger should set SHARE_DELETE by default or use parameter at initialize from user code?

[cosmo0920]

On windows, logger should set SHARE_DELETE by default or use parameter at initialize from user code?

Yes!

[Coldln]

Unfourtantly, it isn't helping to add File::SHARE_DELETE in logger.rb

    def open_logfile(filename)
      begin
        open(filename, (File::WRONLY | File::APPEND | File::SHARE_DELETE))
      rescue Errno::ENOENT
        create_logfile(filename)
      end
    end

    def create_logfile(filename)
      begin
        logdev = open(filename, (File::WRONLY | File::APPEND | File::CREAT | File::EXCL | File::SHARE_DELETE))

[cosmo0920]

Oops. Windows also complains lack of Permission in MoveFileEx (inside File.rename):

https://github.com/ruby/ruby/blob/e4be2fda3dbbfdb1f2ace697c96cf6bdd7dfef21/win32/win32.c#L5368

[cosmo0920]

Simple script does not complains permission denied....

require "logger"

logger = Logger.new('test.log', 5, 30*1024)

loop do
  logger.error "lost connection"
  logger.debug "got new connection"
end

It seems that the above script is working as expected.
It's curious....

[Ntzafrir]

It seems that the above script is working as expected.
It's curious....

Try this one instead

require "logger"

logger = Logger.new('test.log', 5, 30*1024)
myOtherLogger = Logger.new('test.log', 5, 30*1024)

loop do
  logger.error "lost connection"
  logger.debug "got new connection"
end

A logger is created 3 times with same file name in the supervisor module