logback.xml file

The logback.xml configuration file can optionally set logging levels for the logs written to system.log and debug.log. The logging levels can also be set using nodetool setlogginglevels.

Options

`appender name="<appender_choice>"…</appender>`

Specify log type and settings. Possible appender names are: SYSTEMLOG, DEBUGLOG, ASYNCDEBUGLOG, and STDOUT. SYSTEMLOG ensures that WARN and ERROR message are written synchronously to the specified file. DEBUGLOG and ASYNCDEBUGLOG ensure that DEBUG messages are written either synchronously or asynchronously, respectively, to the specified file. STDOUT writes all messages to the console in a human-readable format.

Example: <appender name="SYSTEMLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">

`<file> <filename> </file>`

Specify the filename for a log.

Example: <file>$\{cassandra.logdir}/system.log</file>

`<level> <log_level> </level>`

Specify the level for a log. Part of the filter. Levels are: ALL, TRACE, DEBUG, INFO, WARN, ERROR, OFF. TRACE creates the most verbose log, ERROR the least.

Increasing logging levels can generate heavy logging output on a moderately trafficked cluster. You can use the nodetool getlogginglevels command to see the current logging configuration.

Default: INFO

Example: <level>INFO</level>

`<rollingPolicy class="<rolling_policy_choice>" <fileNamePattern><pattern_info></fileNamePattern> … </rollingPolicy>`

Specify the policy for rolling logs over to an archive.

Example: <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

`<fileNamePattern> <pattern_info> </fileNamePattern>`

Specify the pattern information for rolling over the log to archive. Part of the rolling policy.

Example: <fileNamePattern>$\{cassandra.logdir}/system.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>

`<maxFileSize> <size> </maxFileSize>`

Specify the maximum file size to trigger rolling a log. Part of the rolling policy.

Example: <maxFileSize>50MB</maxFileSize>

`<maxHistory> <number_of_days> </maxHistory>`

Specify the maximum history in days to trigger rolling a log. Part of the rolling policy.

Example: <maxHistory>7</maxHistory>

`<encoder> <pattern>…</pattern> </encoder>`

Specify the format of the message. Part of the rolling policy.

Example: <encoder> <pattern>%-5level [%thread] %date\{"yyyy-MM-dd’T’HH:mm:ss,SSS", UTC} %F:%L - %msg%n</pattern> </encoder>

Logging system logs to Cassandra virtual table

It is possible to configure logback.xml in such a way that logs would appear in system_views.system_log table. This is achieved by appender implemented in class VirtualTableAppender which is called CQLLOG in the default configration. When the appender is commented out, no system logs are written to the virtual table.

CQLLOG appender is special as the underlying structure it saves log messages into can not grow without any bound as a node would run out of memory. For this reason, system_log table is limited on its size. By default, it can hold at most 50 000 log messages, it can never hold more than 100 000 log messages.

To specify how many rows you want that virtual table to hold at most, there is a system property called cassandra.virtual.logs.max.rows which takes an integer as value.

You can execute CQL truncate query for system_views.system_log if you want to wipe out all the logs in virtual table to e.g. save some memory.

It is recommended to set filter to at least WARN level so this table holds only important logging messages as each message will occupy memory.

The appender to virtual table is commented out by default so logging to virtual table is not active.

Logging slow queries to Cassandra virual table

It is possible to log slow queries into system_views.slow_queries table. A query is evaluated to be slow if it takes more than slow_query_log_timeout in cassandra.yaml.

To log messages to system_views.slow_queries you need to:

uncomment SLOW_QUERIES_APPENDER log appender
uncomment appender-ref pointing to SLOW_QUERIES_APPENDER in slow_queries logger:

The respective configuration in logback.xml looks like this:

  <!-- Uncomment below configuration and corresponding appender-ref in slow_queries logger to activate
       logging into system_views.slow_queries virtual table. -->
  <!-- <appender name="SLOW_QUERIES_APPENDER" class="org.apache.cassandra.utils.logging.SlowQueriesAppender"/> -->

  <!-- Log slow queries to system_views.slow_queries virtual table -->
  <logger name="slow_queries" additivity="false" level="DEBUG">
    <appender-ref ref="DEBUGLOG"/>
    <!-- uncomment this appender reference together with appender definition above
         to start to put slow queries into system_views.slow_queries virtual table
    <appender-ref ref="SLOW_QUERIES_APPENDER"/>
    -->
  </logger>

By default, slow queries will be logged to debug.log. By uncommenting virtual table appender, it will be logged to debug.log as well as to system_views.slow_queries. If you want to log it to system_views.slow_queries only, you need to comment out DEBUGLOG appender-ref in slow_queries logger declaration.

If you want to log slow queries to a dedicated log file (which is e.g. rotated), that is also possible by pointing slow_queries logger to a respective file appender of a given reference, similar to DEBUGLOG where all logs go by default.

The structure of a table looks like this:

cassandra@cqlsh> DESCRIBE system_views.slow_queries ;

/*
Warning: Table system_views.slow_queries is a virtual table and cannot be recreated with CQL.
Structure, for reference:
VIRTUAL TABLE system_views.slow_queries (
    keyspace_name text,
    table_name text,
    timestamp timestamp,
    query text,
    avg_ms bigint,
    cross_node boolean,
    max_ms bigint,
    min_ms bigint,
    times_reported int,
    PRIMARY KEY (keyspace_name, table_name, timestamp, query)
) WITH CLUSTERING ORDER BY (table_name ASC, timestamp ASC, query ASC)
    AND comment = 'Slow queries';

By having slow queries in a virtual table, an operator can check if there are slow queries for some table, see if some queries violate some time threshold etc. The rows in this table are same data as one would get in debug.log, they are just way more convenient to parse and query.

system_views.slow_queries table is limited on number of rows it can hold, by default 10 000, configurable by cassandra.virtual.slow_queries.max.rows system property. If this table is full, the oldest entry is removed and the newest is inserted. This virtual table can be truncated by CQL and deletion on partition key (keyspace_name column) is allowed.

A reader noticed that by placing custom appender implementation of SLOW_QUERIES_APPENDER appender on a class path and referencing it in logback.xml, it is possible to log slow queries wherever we have an appender for it.

Contents of default `logback.xml`

<configuration scan="true" scanPeriod="60 seconds">

  <!-- No shutdown hook; we run it ourselves in StorageService after shutdown -->

  <!-- SYSTEMLOG rolling file appender to system.log (INFO level) -->

  <appender name="SYSTEMLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
  <level>INFO</level>
    </filter>
    <file>${cassandra.logdir}/system.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <!-- rollover daily -->
      <fileNamePattern>${cassandra.logdir}/system.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
      <!-- each file should be at most 50MB, keep 7 days worth of history, but at most 5GB -->
      <maxFileSize>50MB</maxFileSize>
      <maxHistory>7</maxHistory>
      <totalSizeCap>5GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{"yyyy-MM-dd'T'HH:mm:ss,SSS", UTC} %F:%L - %msg%n</pattern>
    </encoder>
  </appender>

  <!-- DEBUGLOG rolling file appender to debug.log (all levels) -->

  <appender name="DEBUGLOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${cassandra.logdir}/debug.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <!-- rollover daily -->
      <fileNamePattern>${cassandra.logdir}/debug.log.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
      <!-- each file should be at most 50MB, keep 7 days worth of history, but at most 5GB -->
      <maxFileSize>50MB</maxFileSize>
      <maxHistory>7</maxHistory>
      <totalSizeCap>5GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{"yyyy-MM-dd'T'HH:mm:ss,SSS", UTC} %F:%L - %msg%n</pattern>
    </encoder>
  </appender>

  <!-- ASYNCLOG assynchronous appender to debug.log (all levels) -->

  <appender name="ASYNCDEBUGLOG" class="ch.qos.logback.classic.AsyncAppender">
    <queueSize>1024</queueSize>
    <discardingThreshold>0</discardingThreshold>
    <includeCallerData>true</includeCallerData>
    <appender-ref ref="DEBUGLOG" />
  </appender>

  <!-- STDOUT console appender to stdout (INFO level) -->

  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>INFO</level>
    </filter>
    <encoder>
      <pattern>%-5level [%thread] %date{"yyyy-MM-dd'T'HH:mm:ss,SSS", UTC} %F:%L - %msg%n</pattern>
    </encoder>
  </appender>

  <!-- Uncomment bellow and corresponding appender-ref to activate logback metrics
  <appender name="LogbackMetrics" class="com.codahale.metrics.logback.InstrumentedAppender" />
   -->

  <!-- Uncomment below configuration and corresponding appender-ref to activate
  logging into system_views.system_logs virtual table. -->
  <!-- <appender name="CQLLOG" class="org.apache.cassandra.utils.logging.VirtualTableAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>WARN</level>
    </filter>
  </appender> -->

  <root level="INFO">
    <appender-ref ref="SYSTEMLOG" />
    <appender-ref ref="STDOUT" />
    <appender-ref ref="ASYNCDEBUGLOG" /> <!-- Comment this line to disable debug.log -->
    <!--
    <appender-ref ref="LogbackMetrics" />
    -->
    <!--
    <appender-ref ref="CQLLOG"/>
    -->
  </root>

  <logger name="org.apache.cassandra" level="DEBUG"/>
</configuration>

Cassandra Documentation

Version:

logback.xml file

Options

`appender name="<appender_choice>"…</appender>`

`<file> <filename> </file>`

`<level> <log_level> </level>`

`<rollingPolicy class="<rolling_policy_choice>" <fileNamePattern><pattern_info></fileNamePattern> … </rollingPolicy>`

`<fileNamePattern> <pattern_info> </fileNamePattern>`

`<maxFileSize> <size> </maxFileSize>`

`<maxHistory> <number_of_days> </maxHistory>`

`<encoder> <pattern>…</pattern> </encoder>`

Logging system logs to Cassandra virtual table

Logging slow queries to Cassandra virual table

Contents of default `logback.xml`

Cassandra Documentation

Version:

logback.xml file

Options

appender name="<appender_choice>"…​</appender>

<file> <filename> </file>

<level> <log_level> </level>

<rollingPolicy class="<rolling_policy_choice>" <fileNamePattern><pattern_info></fileNamePattern> …​ </rollingPolicy>

<fileNamePattern> <pattern_info> </fileNamePattern>

<maxFileSize> <size> </maxFileSize>

<maxHistory> <number_of_days> </maxHistory>

<encoder> <pattern>…​</pattern> </encoder>

Logging system logs to Cassandra virtual table

Logging slow queries to Cassandra virual table

Contents of default logback.xml

`appender name="<appender_choice>"…</appender>`

`<file> <filename> </file>`

`<level> <log_level> </level>`

`<rollingPolicy class="<rolling_policy_choice>" <fileNamePattern><pattern_info></fileNamePattern> … </rollingPolicy>`

`<fileNamePattern> <pattern_info> </fileNamePattern>`

`<maxFileSize> <size> </maxFileSize>`

`<maxHistory> <number_of_days> </maxHistory>`

`<encoder> <pattern>…</pattern> </encoder>`

Contents of default `logback.xml`