MariaDB MaxScale

Debug & Diagnostic Support

Mark Riddoch

Last Updated: 24th November 2014

[[TOC]]

Change History

Date Comment
20th June 2013 Initial Version
22nd July 2013 Updated with new naming MariaDB MaxScale Addition of description of login process for the debug CLI Updates debug CLI output examples Addition of show users, shutdown maxscale, shutdown service, restart service, set server, clear server, reload users, reload config and add user commands.
23rd July 2013 Rename of show users command to show dbusers and addition of the show users command to show the admin users. Addition of example configuration data.
14th November 2013 Added enable/disable log commands details Added Galera Monitor as an example in show monitors
3rd March 2014 Added show users details for MySQL users
27th May 2014 Document the new debugcli mode switch and command changes in the two modes. Added the new show server command.
29th May 2014 Addition of new list command that gives concise tabular output
4th June 2014 Added new ‘show monitors’ and ‘show servers’ details
31st June 2014 Added NDB monitor in show monitors
29th August 2014 Added new ‘show monitors’ details for MySQL/Galera monitors

Introduction

MariaDB MaxScale is a complex application and as such is bound to have bugs and support issues that occur from time to time. There are a number of things we need to consider for the development stages and long term supportability of MariaDB MaxScale

  • Flexible logging of MariaDB MaxScale activity

  • Support for connecting a debugger to MariaDB MaxScale

  • A diagnostic interface to MariaDB MaxScale

The topic of logging has already been discussed in another document in this series of documents about MariaDB MaxScale and will not be covered further here.

Debugger Support

Beyond the language support for debugging using tools such as gdb, MariaDB MaxScale will also offer convenience functions for the debugger to call and a command line argument that is useful to run MariaDB MaxScale under the debugger.

Command Line Option

Normally when MariaDB MaxScale starts it will place itself in the background and setup the signal masks so that it is immune to the normal set of signals that will cause the process to exit, SIGINT and SIGQUIT. This behavior is normally what is required, however if you wish to run MariaDB MaxScale under the control of a debugger it is useful to suppress this behavior. A command line option, -d is provided to turn off this behavior.

% gdb maxscale

(gdb) run -d

Convenience Functions

A set of convenience functions is provided that may be used within the debugger session to extract information from MariaDB MaxScale.

Printing Services

A service within MariaDB MaxScale provides the encapsulation of the port MariaDB MaxScale listen on, the protocol it uses, the set of servers it may route to and the routing method to use. Two functions exists that allow you to display the details of the services and may be executed from within a debugger session.

The printAllServices() function will print all the defined services within MariaDB MaxScale and is invoked using the call syntax of the debugger.

(gdb) call printAllServices()

Service 0x60da20

Service:        Debug Service

Router:         debugcli (0x7ffff5a7c2a0)

Started:        Thu Jun 20 15:13:32 2013

Backend databases

Total connections:  1

Currently connected:    1

Service 0x60d010

Service:        Test Service

Router:         readconnroute (0x7ffff5c7e260)

Started:        Thu Jun 20 15:13:32 2013

Backend databases

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Total connections:  1

Currently connected:    1

(gdb)

It is possible to print an individual service if you know the memory address of the service.

(gdb) call printService(0x60da20)

Service 0x60da20

Service:        Debug Service

Router:         debugcli (0x7ffff5a7c2a0)

Started:        Thu Jun 20 15:13:32 2013

Backend databases

Total connections:  1

Currently connected:    1

(gdb)

Printing Sessions

Sessions represent the data for a client that is connecting through MariaDB MaxScale, there will be a session for each client and one for each listener for a specific port/protocol combination. Similarly there are two calls to print all or a particular session.

(gdb) call printAllSessions()

Session 0x60fdf0

Service:    Debug Service (0x60da20)

Client DCB: 0x60f6c0

Connected:  Thu Jun 20 15:13:32 2013

Session 0x60f620

Service:    Test Service (0x60d010)

Client DCB: 0x60ead0

Connected:  Thu Jun 20 15:13:32 2013

(gdb) call printSession(0x60fdf0)

Session 0x60fdf0

Service:    Debug Service (0x60da20)

Client DCB: 0x60f6c0

Connected:  Thu Jun 20 15:13:32 2013

(gdb)

Printing Servers

Servers are a representation of the backend database to which MariaDB MaxScale may route SQL statements. Similarly two calls exist to print server details.

(gdb) call printAllServers()

Server 0x60d9a0

Server:         127.0.0.1

Protocol:       MySQLBackend

Port:           3308

Server 0x60d920

Server:         127.0.0.1

Protocol:       MySQLBackend

Port:           3307

Server 0x60d8a0

Server:         127.0.0.1

Protocol:       MySQLBackend

Port:           3306

(gdb) call printServer(0x60d920)

Server 0x60d920

Server:         127.0.0.1

Protocol:       MySQLBackend

Port:           3307

(gdb)

Modules

MariaDB MaxScale makes significant use of modules, shared objects, that are loaded on demand based on the configuration. A routine exists that will print the currently loaded modules.

(gdb) call printModules()

Module Name | Module Type | Version


telnetd | Protocol | V1.0.0

MySQLClient | Protocol | V1.0.0

testroute | Router | V1.0.0

debugcli | Router | V1.0.0

readconnroute | Router | V1.0.0

(gdb)

Descriptor Control Blocks

The Descriptor Control Block (DCB) is an important concept within MariaDB MaxScale since it is this block that is passed to the polling system, when an event occurs it is that structure that is available and from this structure it must be possible to navigate to all other structures that contain state regarding the session and protocol in use.

image alt text

Similar print routines exist for the DCB

(gdb) call printAllDCBs()

DCB: 0x60ead0

DCB state:      DCB for listening socket

Queued write data:  0

Statistics:

    No. of Reads:   0

    No. of Writes:  0

    No. of Buffered Writes: 0

    No. of Accepts: 0

DCB: 0x60f6c0

DCB state:      DCB for listening socket

Queued write data:  0

Statistics:

    No. of Reads:   0

    No. of Writes:  0

    No. of Buffered Writes: 0

    No. of Accepts: 0

(gdb) call printDCB(0x60ead0)

DCB: 0x60ead0

DCB state:      DCB for listening socket

Queued write data:  0

Statistics:

    No. of Reads:   0

    No. of Writes:  0

    No. of Buffered Writes: 0

    No. of Accepts: 0

(gdb)

Diagnostic Interface

It is possible to configure a service to run within MariaDB MaxScale that will allow a user to telnet to a port on the machine and be connected to MariaDB MaxScale. This is configured by creating a service that uses the debugcli routing module and the telnetd protocol with an associated listener. The service does not require any backend databases to be configured since the router never forwards any data, it merely accepts commands and executes them, returning data to the user.

The example below shows the configuration that is required to set-up a debug interface that listens for incoming telnet connections on port 4442.

[Debug Service]

type=service

router=debugcli

[Debug Listener]

type=listener

service=Debug Service

protocol=telnetd

port=4442

The Debug Service section sets up a service with no backend database servers, but with a debugcli module as the router. This module will implement the commands and send the data back to the client.

The debugcli accepts router options of either developer or user, these are used to control the mode of the user interface. If no router options are given then the CLI is in user mode by default.

The Debug Listener section setups the protocol and port combination and links that to the service.

Assuming a configuration that includes the debug service, with the listening port set to 4442, to connect from the machine that runs MariaDB MaxScale you must first install telnet and then simply call telnet to connect.

-bash-4.1$ telnet localhost 4442

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

Welcome the MariaDB MaxScale Debug Interface (V1.1.0).

Type help for a list of available commands.

MaxScale login: admin

Password:

MaxScale>

As delivered MariaDB MaxScale uses a default login name of admin with the password of mariadb for connections to the debug interface. Users may be added to the CLI by use of the add user command.

This places you in the debug command line interface of MariaDB MaxScale, there is a help system that will display the commands available to you

MaxScale> help

Available commands:

add user

clear server

disable log

enable log

list [listeners|modules|services|servers|sessions]

reload [config|dbusers]

remove user

restart [monitor|service]

set server

show [dcbs|dcb|dbusers|epoll|modules|monitors|server|servers|services|service|session|sessions|users]

shutdown [maxscale|monitor|service]

Type help command to see details of each command.

Where commands require names as arguments and these names contain

whitespace either the \ character may be used to escape the whitespace

or the name may be enclosed in double quotes ".

MaxScale>

Different command help is shown in user mode and developer mode, in user mode the help for the show command is;

MaxScale> help show

Available options to the show command:

dcbs       Show all descriptor control blocks (network connections)

dcb        Show a single descriptor control block e.g. show dcb 0x493340

dbusers    Show statistics and user names for a service's user table.

    Example : show dbusers <service name>

epoll      Show the poll statistics

modules    Show all currently loaded modules

monitors   Show the monitors that are configured

server     Show details for a named server, e.g. show server dbnode1

servers    Show all configured servers

services   Show all configured services in MaxScale

service    Show a single service in MaxScale, may be passed a service name

session    Show a single session in MaxScale, e.g. show session 0x284830

sessions   Show all active sessions in MaxScale

users      Show statistics and user names for the debug interface

MaxScale>

However in developer mode the help is;

MaxScale> help show

Available options to the show command:

dcbs       Show all descriptor control blocks (network connections)

dcb        Show a single descriptor control block e.g. show dcb 0x493340

dbusers    Show statistics and user names for a service's user table

epoll      Show the poll statistics

modules    Show all currently loaded modules

monitors   Show the monitors that are configured

server     Show details for a server, e.g. show server 0x485390

servers    Show all configured servers

services   Show all configured services in MaxScale

session    Show a single session in MaxScale, e.g. show session 0x284830

sessions   Show all active sessions in MaxScale

users      Show statistics and user names for the debug interface

MaxScale>

The commands available are very similar to those described above to print things from the debugger, the advantage being that you do not need a debug version or a debugger to use them.

Listing Services

The list services command is designed to give a concise tabular view of the currently configured services within MaxScale along with key data that summarizes the use being made of the service.

MaxScale> list services

Service Name | Router Module | #Users | Total Sessions


Test Service | readconnroute | 1 | 1

Split Service | readwritesplit | 1 | 1

Debug Service | debugcli | 2 | 2

MaxScale>

This provides a useful mechanism to see what is configured and provide the service names that can be passed to a show service command.

Listing Listeners

The list listeners command outputs a table that provides the current set of listeners within the MariaDB MaxScale instance and shows the status of each listener.

MaxScale> list listeners

Service Name | Protocol Module | Address | Port | State


Test Service | MySQLClient | (null) | 4006 | Running

Split Service | MySQLClient | (null) | 4007 | Running

Debug Service | telnetd | localhost | 4242 | Running

MaxScale>

Listing Servers

The list servers command will display a table that contains a row for every server defined in the configuration file. The row contains the server name that can be passed to the show server command, the address and port of the server, its current status and the number of connections to that server from MariaDB MaxScale.

MaxScale> list servers

Server | Address | Port | Status | Connections


server1 | 127.0.0.1 | 3306 | Running | 0

server2 | 127.0.0.1 | 3307 | Slave, Running | 0

server3 | 127.0.0.1 | 3308 | Master, Running | 0

server4 | 127.0.0.1 | 3309 | Slave, Running | 0

MaxScale>

Listing Modules

The list modules command displays a table of all the modules loaded within MariaDB MaxScale.

MaxScale> list modules

Module Name | Module Type | Version


telnetd | Protocol | V1.0.1

MySQLClient | Protocol | V1.0.0

mysqlmon | Monitor | V1.1.0

readconnroute | Router | V1.0.2

readwritesplit | Router | V1.0.2

debugcli | Router | V1.1.1

MaxScale>

Showing Services

The show services command will show all the services configured currently

MaxScale> show services

Service 0xf44c10

Service:        Test Service

Router:         readconnroute (0x7f7fd8afba40)

Number of router sessions:      0

Current no. of router sessions: 0

Number of queries forwarded:    0

Started:        Mon Jul 22 11:24:09 2013

Backend databases

    127.0.0.1:3309  Protocol: MySQLBackend

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Users data:         0xf454b0

Total connections:  1

Currently connected:    1

Service 0xf43910

Service:        Split Service

Router:         readwritesplit (0x7f7fd8f05460)

Number of router sessions:              0

Current no. of router sessions:         0

Number of queries forwarded:            0

Number of queries forwarded to master:  0

Number of queries forwarded to slave:   0

Number of queries forwarded to all:     0

Started:        Mon Jul 22 11:24:09 2013

Backend databases

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Users data:         0xf449b0

Total connections:  1

Currently connected:    1

Service 0xea0190

Service:        Debug Service

Router:         debugcli (0x7f7fd910d620)

Started:        Mon Jul 22 11:24:09 2013

Backend databases

Users data:         0xea2d80

Total connections:  2

Currently connected:    2

MaxScale>

Showing Sessions

There are two options to show sessions, either an individual session or all sessions

MaxScale> show sessions

Session 0x6f8f20

State:          Session Ready

Service:        Debug Service (0x649190)

Client DCB:     0x6f8e20

Client Address:     0.0.0.0

Connected:      Mon Jul 22 11:31:56 2013

Session 0x6f83b0

State:          Session Allocated

Service:        Split Service (0x6ec910)

Client DCB:     0x64b430

Client Address:     127.0.0.1

Connected:      Mon Jul 22 11:31:28 2013

Session 0x6efba0

State:          Listener Session

Service:        Debug Service (0x649190)

Client DCB:     0x64b180

Connected:      Mon Jul 22 11:31:21 2013

Session 0x64b530

State:          Listener Session

Service:        Split Service (0x6ec910)

Client DCB:     0x6ef8e0

Connected:      Mon Jul 22 11:31:21 2013

Session 0x618840

State:          Listener Session

Service:        Test Service (0x6edc10)

Client DCB:     0x6ef320

Connected:      Mon Jul 22 11:31:21 2013

MaxScale> show session 0x6f83b0

Session 0x6f83b0

State:          Session Allocated

Service:        Split Service (0x6ec910)

Client DCB:     0x64b430

Client Address:     127.0.0.1

Connected:      Mon Jul 22 11:31:28 2013

MaxScale>

Show Servers

The configured backend databases can be displayed using the show servers command.

MaxScale> show servers

Server 0x6ec840 (server1)

Server:         127.0.0.1

Status:                 Running

Protocol:           MySQLBackend

Port:               3306

Number of connections:  0

Current no. of connections:0

Server 0x6ec770 (server2)

Server:         127.0.0.1

Status:                 Master, Running

Protocol:           MySQLBackend

Port:               3307

Server Version:     5.5.35-MariaDB-log

Node Id:            1

    Master Id:          -1

    Slave Ids:          2,3

    Repl Depth:         0

    Last Repl Heartbeat:    1417080906

Number of connections:  1

Current no. of connections:1

Server 0x6ec6a0 (server3)

Server:         127.0.0.1

Status:                 Slave, Running

Protocol:           MySQLBackend

Port:               3308

Server Version:     5.5.35-MariaDB-log

Node Id: 2

    Master Id:          1

    Slave Ids:

    Repl Depth:         1

    Slave delay:            8

    Last Repl Heartbeat:    1417080898

Number of connections:  1

Current no. of connections:1

Server 0x6ec5d0 (server4)

Server:         127.0.0.1

Status:                 Down

Protocol:           MySQLBackend

Port:               3309

Server Version:     5.5.35-MariaDB-log

Node Id:            3

    Master Id:          1

    Slave Ids:

    Repl Depth:         1

Number of connections:  0

Current no. of connections:0

MaxScale>

  • Version string is available in the output only if the node is running.

  • Node Id possible values:

  • the value of server-id from MySQL or MariaDB servers in Master/Slave replication setup.

  • the value of ‘wsrep_local_index’ for Galera cluster nodes

  • the value of ‘Ndb_cluster_node_id’ for SQL nodes in MySQL Cluster

  • the -1 value for a failure getting one of these information

  • Repl Depth is the replication depth level found by MaxScale MySQL Monitor

  • Master Id is the master node, if available, for current server

  • Slave Ids is the slave list for the current node, if available

Note, the Master Root Server used for routing decision is the server with master role and with lowest replication depth level. All slaves servers even if they are intermediate master are suitable for read statement routing decisions.

  • Slave Delay is the found time difference used for Replication Lag support in routing decision, between the value read by the Slave (it was inserted into the master) and maxscale current time

    Value of 0 or less than monitor interval means there is no replication delay.

  • Last Repl Heartbeat is the MaxScale timestamp read or inserted (if current server is master)

The Replication Heartbeat table is updated by MySQL replication, starting MariaDB MaxScale when there is a significant slave delay may result that Slave Delay and Last Repl Heartbeat are not available for some time in the slave server details

There may be other status description such us:

Slave of External Server: the node is slave of a server not configured

in MaxScale (by MySQL monitor)

Master Stickiness:       the Master node is kept (by Galera monitor)

Stale Status:        the master node is kept even if the replication

is not working (by MySQL monitor)

Auth Error:          Monitor was not able to login into backend

A few examples:

Server 0x1a5aae0 (server3)

Server:         127.0.0.1

Status:                 Master, Slave of External Server, Running

Protocol:           MySQLBackend

Port:               3308

Server Version:     5.5.35-MariaDB-log

Node Id:            3

Master Id:          1

Slave Ids:          2

Repl Depth:         1

Server 0x2d1b5c0 (server2)

Server:         192.168.122.142

Status:                 Master, Synced, Master Stickiness, Running

Protocol:           MySQLBackend

Port:               3306

Server Version:     5.5.40-MariaDB-wsrep-log

Node Id:            1

Repl Depth:         0

Show Server

Details of an individual server can be displayed by using the show server command. In user mode the show server command is passed the name of the server to display, these names are the section names used in the configuration file.

MaxScale> show server server4

Server 0x6ec5d0 (server4)

Server:         127.0.0.1

Status:                 Down

Protocol:           MySQLBackend

Port:               3309

Number of connections:  0

Current no. of connections:0

MaxScale>

In developer mode the show server command is passed the address of a server structure.

MaxScale> show server 0x6ec5d0

Server 0x6ec5d0 (server4)

Server:         127.0.0.1

Status:                 Down

Protocol:           MySQLBackend

Port:               3309

Number of connections:  0

Current no. of connections:0

MaxScale>

Show DCBS

There are two forms of the show command that will give you DCB information, the first will display information for all DCBs within the system.

MaxScale> show dcbs

DCB: 0x6ef320

DCB state:          DCB for listening socket

Service:            Test Service

Queued write data:  0

Statistics:

    No. of Reads:           0

    No. of Writes:          0

    No. of Buffered Writes: 0

    No. of Accepts:         0

DCB: 0x6ef8e0

DCB state:          DCB for listening socket

Service:            Split Service

Queued write data:  0

Statistics:

    No. of Reads:           0

    No. of Writes:          0

    No. of Buffered Writes: 0

    No. of Accepts:         1

DCB: 0x64b180

DCB state:          DCB for listening socket

Service:            Debug Service

Queued write data:  0

Statistics:

    No. of Reads:           0

    No. of Writes:          0

    No. of Buffered Writes: 0

    No. of Accepts:         1

DCB: 0x64b430

DCB state:          DCB processing event

Service:            Split Service

Connected to:       127.0.0.1

Queued write data:  0

Statistics:

    No. of Reads:           2

    No. of Writes:          3

    No. of Buffered Writes: 0

    No. of Accepts:         0

DCB: 0x6f8400

DCB state:          DCB in the polling loop

Service:            Split Service

Queued write data:  0

Statistics:

    No. of Reads:           3

    No. of Writes:          1

    No. of Buffered Writes: 0

    No. of Accepts:         0

DCB: 0x6f8b40

DCB state:          DCB in the polling loop

Service:            Split Service

Queued write data:  0

Statistics:

    No. of Reads:           2

    No. of Writes:          0

    No. of Buffered Writes: 0

    No. of Accepts:         0

DCB: 0x6f8e20

DCB state:          DCB processing event

Service:            Debug Service

Connected to:       0.0.0.0

Queued write data:  0

Statistics:

    No. of Reads:           8

    No. of Writes:          133

    No. of Buffered Writes: 0

    No. of Accepts:         0

MaxScale>

An individual DCB can be displayed by passing the DCB address to the show dcb command

MaxScale> show dcb 0x64b430

DCB: 0x64b430

DCB state:      DCB processing event

Connected to:       127.0.0.1

Owning Session:     7308208

Queued write data:  0

Statistics:

    No. of Reads:   2

    No. of Writes:  3

    No. of Buffered Writes: 0

    No. of Accepts: 0

MaxScale>

Show Modules

The show modules command will display the list of the currently loaded modules

MaxScale> show modules

Module Name | Module Type | Version


MySQLBackend | Protocol | V2.0.0

telnetd | Protocol | V1.0.1

MySQLClient | Protocol | V1.0.0

mysqlmon | Monitor | V1.0.0

readconnroute | Router | V1.0.2

readwritesplit | Router | V1.0.2

debugcli | Router | V1.1.0

MaxScale>

Show Polling Statistics

Display statistics related to the main polling loop. The epoll cycles is the count of the number of times epoll has returned with one or more event. The other counters are for each individual events that has been detected.

MaxScale> show epoll

Number of epoll cycles: 7928

Number of read events: 2000920

Number of write events: 2000927

Number of error events: 0

Number of hangup events: 0

Number of accept events: 4

MaxScale>

Show Dbusers

The show dbuser command allows data regarding the table that holds the database users for a service to be displayed. It does not give the actual user data, but rather details of the hashtable distribution.

The show dbuser command takes different arguments in the two modes of MariaDB MaxScale, in user mode it may be passed the name of a service rather than an address, whilst in developer mode it needs the address of a user structure that has been extracted from a service.

In developer mode the show users commands must be passed the address of the user table, this can be extracted from the output of a show services command.

MaxScale> show services

Service 0x6ec910

Service:        Split Service

Router:         readwritesplit (0x7ffff1698460)

Number of router sessions:              1

Current no. of router sessions:         0

Number of queries forwarded:            2

Number of queries forwarded to master:  0

Number of queries forwarded to slave:   1

Number of queries forwarded to all:     1

Started:        Mon Jul 22 11:31:21 2013

Backend databases

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Users data:         0x6ed9b0

Total connections:  2

Currently connected:    1

The following example shows the MySQL users.

Users are loaded with the host (IPv4 data) as they are created in the backend.

MaxScale> show dbusers 0x6ed9b0

Users table data

Hashtable: 0x19243a0, size 52

No. of entries:         16

Average chain length:   0.3

Longest chain length:   4

User names: one@%, new@192.168.56.1, new@127.0.0.1, repluser@%, seven@127.0.0.1, four@%

MaxScale>

In user mode the command is simply passed the name of the service

MaxScale> show dbusers "Split Service"

Users table data

Hashtable: 0x19243a0, size 52

No. of entries:         16

Average chain length:   0.3

Longest chain length:   4

User names: one@%, new@192.168.56.1, new@127.0.0.1, repluser@%, seven@127.0.0.1, four@%

MaxScale>

Please note the use of quotes in the name in order to escape the white space character.

Show Users

The show users command lists the users defined for the administration interface. Note that if there are no users defined, and the default admin user is in use, then no users will be displayed.

MaxScale> show users

Administration interface users:

Users table data

Hashtable: 0x25ef5e0, size 52

No. of entries:         2

Average chain length:   0.0

Longest chain length:   1

User names: admin, mark

MaxScale>

Show Monitors

The show monitors show the status of the database monitors. The address of the monitor can be used for the shutdown monitor and restart monitor commands.

MaxScale> show monitors

Monitor: 0x80a510

Name:                   MySQL Monitor
Monitor running
Sampling interval:      10000 milliseconds
Replication lag:        enabled
Detect Stale Master:    disabled
Connect Timeout:        3 seconds
Read Timeout:           1 seconds
Write Timeout:          2 seconds
Monitored servers:      127.0.0.1:3306, 127.0.0.1:3307, 127.0.0.1:3308, 127.0.0.1:3309

Monitor: 0x73d3d0

Name:                   Galera Monitor
Monitor running
Sampling interval:      7000 milliseconds
Master Failback:        off
Connect Timeout:        3 seconds
Read Timeout:           1 seconds
Write Timeout:          2 seconds
Monitored servers:      127.0.0.1:3310, 127.0.0.1:3311, 127.0.0.1:3312

Monitor: 0x387b880

Name:                   NDB Cluster Monitor
Monitor running
Sampling interval:      8000 milliseconds
Connect Timeout:        3 seconds
Read Timeout:           1 seconds
Write Timeout:          2 seconds
Monitored servers:      127.0.0.1:3301, 162.243.90.81:3302

MaxScale>

Monitor timeouts used in monitors follow the rules of mysql_real_connect C API:

  • Connect Timeout is the connect timeout in seconds.

  • Read Timeout is the timeout in seconds for each attempt to read from the server. There are retries if necessary, so the total effective timeout value is three times the option value.

  • Write Timeout is the timeout in seconds for each attempt to write to the server. There is a retry if necessary, so the total effective timeout value is two times the option value.

Shutdown MariaDB MaxScale

The CLI can be used to shutdown the MariaDB MaxScale server by use of the shutdown command, it may be called with the argument either maxscale or gateway.

MaxScale> shutdown maxscale

Shutdown monitor

The shutdown monitor command stops the thread that is used to run the monitor and will stop any update of the server status flags. This is useful prior to manual setting of the states of the server using the set server and clear server commands.

MaxScale> show monitors

Monitor: 0x80a510

Name:           MySQL Monitor

Monitor running

Sampling interval: 10000 milliseconds

Monitored servers:  127.0.0.1:3306, 127.0.0.1:3307, 127.0.0.1:3308, 127.0.0.1:3309

MaxScale> shutdown monitor 0x80a510

MaxScale> show monitors

Monitor: 0x80a510

Name:           MySQL Monitor

Monitor stopped

Sampling interval:  10000 milliseconds

Monitored servers:  127.0.0.1:3306, 127.0.0.1:3307, 127.0.0.1:3308, 127.0.0.1:3309

MaxScale>

It may take some time before a monitor actually stops following the issuing of a shutdown monitor command. Stopped monitors can be restarted by issuing a restart monitor command.

Shutdown service

The shutdown service command can be used to stop the listener for a particular service. This will prevent any new clients from using the service but will not terminate any clients already connected to the service.

The shutdown service command needs the address of a service to be passed as an argument, this can be obtained by running show services.

MaxScale> show services

Service 0x6edc10

Service:        Test Service

Router:         readconnroute (0x7ffff128ea40)

Number of router sessions:      257

Current no. of router sessions: 0

Number of queries forwarded:    1000193

Started:        Mon Jul 22 11:31:21 2013

Backend databases

    127.0.0.1:3309  Protocol: MySQLBackend

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Users data:         0x6ee4b0

Total connections:  258

Currently connected:    1

Service 0x6ec910

Service:        Split Service

Router:         readwritesplit (0x7ffff1698460)

Number of router sessions:              1

Current no. of router sessions:         0

Number of queries forwarded:            2

Number of queries forwarded to master:  0

Number of queries forwarded to slave:   1

Number of queries forwarded to all:     1

Started:        Mon Jul 22 11:31:21 2013

Backend databases

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Users data:         0x6ed9b0

Total connections:  2

Currently connected:    1

Service 0x649190

Service:        Debug Service

Router:         debugcli (0x7ffff18a0620)

Started:        Mon Jul 22 11:31:21 2013

Backend databases

Users data:         0x64bd80

Total connections:  2

Currently connected:    2

MaxScale> shutdown service 0x6edc10

In user mode the shutdown service command may be passed the name of the service as defined in configuration file.

MaxScale> shutdown service Split\ Service

Restart service

The restart service command can be used to restart a previously stopped listener for a service. In developer mode the address of the service must be passed.

MaxScale> restart service 0x6edc10

MaxScale>

In user mode the name of the service may be passed.

MaxScale> restart service Test\ Service

MaxScale>

As with shutdown service the address of the service should be passed as an argument.

Restart Monitor

The restart monitor command will restart a previously stopped monitor.

MaxScale> show monitors

Monitor: 0x80a510

Name:           MySQL Monitor

Monitor stopped

Monitored servers:  127.0.0.1:3306, 127.0.0.1:3307, 127.0.0.1:3308, 127.0.0.1:3309

MaxScale> restart monitor 0x80a510

MaxScale>

Set server

The set server command can be used to set the status flags of a server directly from the user interface. The command should be passed a server address that has been obtained from the output of a show servers command.

MaxScale> show servers

Server 0x6ec840 (server1)

Server:         127.0.0.1

Status:                 Running

Protocol:           MySQLBackend

Port:               3306

Server Version:     10.0.11-MariaDB-log

Node Id:            29

Number of connections:  0

Current n. of conns:    0

Server 0x6ec770 (server2)

Server:         127.0.0.1

Status:                 Master, Running

Protocol:           MySQLBackend

Port:               3307

Server Version:     5.5.35-MariaDB-log

Node Id:            1

Number of connections:  1

Current n. of conns:    0

Server 0x6ec6a0 (server3)

Server:         127.0.0.1

Status:                 Slave, Running

Protocol:           MySQLBackend

Port:               3308

Server Version:     5.5.35-MariaDB-log

Node Id:            2

Number of connections:  258

Current n. of conns:    0

Server 0x6ec5d0 (server4)

Server:         127.0.0.1

Status:                 Down

Protocol:           MySQLBackend

Port:               3309

Node Id:            -1

Number of connections:  0

Current n. of conns:    0

MaxScale> set server 0x6ec840 slave

Valid options that are recognized by the set server command are running, master and slave. Please note that if the monitor is running it will reset the flags to match reality, this interface is really for use when the monitor is disabled.

In user mode there is no need to find the address of the server structure, the name of the server from the section header in the configuration file make be given.

MaxScale> set server server1 slave

Clear server

The clear server command is the complement to the set server command, it allows status bits related to a server to be cleared.

MaxScale> clear server 0x6ec840 slave

Likewise in user mode the server name may be given.

MaxScale> clear server server1 slave

Reload users

The reload users command is used to force a service to go back and reload the table of database users from the backend database. This is the data used in the transparent authentication mechanism in the MySQL protocol. The command should be passed the address of the service as shown in the output of the show services command.

MaxScale> show services

Service 0x6edc10

Service:        Test Service

Router:         readconnroute (0x7ffff128ea40)

Number of router sessions:      257

Current no. of router sessions: 0

Number of queries forwarded:    1000193

Started:        Mon Jul 22 11:31:21 2013

Backend databases

    127.0.0.1:3309  Protocol: MySQLBackend

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Users data:         0x6ee4b0

Total connections:  258

Currently connected:    1

Service 0x6ec910

Service:        Split Service

Router:         readwritesplit (0x7ffff1698460)

Number of router sessions:              1

Current no. of router sessions:         0

Number of queries forwarded:            2

Number of queries forwarded to master:  0

Number of queries forwarded to slave:   1

Number of queries forwarded to all:     1

Started:        Mon Jul 22 11:31:21 2013

Backend databases

    127.0.0.1:3308  Protocol: MySQLBackend

    127.0.0.1:3307  Protocol: MySQLBackend

    127.0.0.1:3306  Protocol: MySQLBackend

Users data:         0x6ed9b0

Total connections:  2

Currently connected:    1

Service 0x649190

Service:        Debug Service

Router:         debugcli (0x7ffff18a0620)

Started:        Mon Jul 22 11:31:21 2013

Backend databases

Users data:         0x64bd80

Total connections:  2

Currently connected:    2

MaxScale> reload users 0x6edc10

Loaded 34 users.

MaxScale>

If user mode is in use then the service name may be given.

MaxScale> reload users "Test Service"

Loaded 34 users.

**MaxScale>**

Reload config

The reload config command can be used to force MariaDB MaxScale to re-read the maxscale.cnf and update itself to the latest configuration defined in that configuration file. It is also possible to force the reading of the configuration file by sending a HangUp signal (SIGHUP) to the maxscale process.

MaxScale> reload config

Reloading configuration from file.

MaxScale>

Note, not all configuration elements can be changed dynamically currently. This mechanism can be used to add new services, servers to services, listeners to services and to update passwords. It can not be used to remove services, servers or listeners currently.

Add user

The add user command is used to add new users to the debug CLI of MariaDB MaxScale. The default behavior of the CLI for MariaDB MaxScale is to have a login name of admin and a fixed password of mariadb. Adding new users will disable this default behavior and limit the login access to the users that are added.

MaxScale> add user admin july2013

User admin has been successfully added.

MaxScale> add user mark hambleden

User mark has been successfully added.

MaxScale>

User names must be unique within the debug CLI, this excludes the admin default user, which may be redefined.

MaxScale> add user mark 22july

User admin already exists.

MaxScale>

If you should forget or lose the the account details you may simply remove the passwd file in /var/cache/maxscale and the system will revert to the default behavior with admin/mariadb as the account.

Enable/disable log

The enable/disable log command is used to enable/disable the log facility of MariaDB MaxScale. The default behavior for MariaDB MaxScale is to have all logs enabled in DEBUG version, and only error log in production release.

Examples:

MaxScale> help enable log

Available options to the enable command:

log        Enable Log options for MaxScale, options trace | error | message E.g. enable log message.

MaxScale> help disable log

Available options to the disable command:

log        Disable Log for MaxScale, Options: debug | trace | error | message E.g. disable log debug

MaxScale> disable log trace

MaxScale>

No output for these commands in the debug interface, but in the affected logs there is a message:

2013 11/14 16:08:33 --- Logging is disabled --

Logging facility

MariaDB MaxScale generates output of its behavior to four distinct logs, error, messages, trace and debug log. Error and message logs are enabled by default but all logs can be dynamically enabled and disabled by using maxadmin utility, debug client interface (telnet) or optionally by using your own application through the client API.

Log contents

By default all log files are located in : /var/log/maxscale and named as :

skygw_errW.log, skygw_msgX.log, skygw_traceY.log and skygw_debugZ.log

where W, X, Y, and Z are sequence numbers for files within the same directory,

Error log

Error log includes errors and warnings; things that occur during runtime and of which the user should at least be aware of. An entry in error log doesn’t necessarily mean a problem - it may be a notification of failed backend server, for example.

Example:

MariaDB Corporation MaxScale /home/jdoe/bin/develop/log/skygw_err1.log Mon Dec 8 13:28:15 2014


--- Logging is enabled.

2014-12-08 13:28:15 Error : Unable to find filter 'testfi' for service 'RW Split Router'

2014-12-08 13:28:26 MaxScale received signal SIGINT. Shutting down.

2014-12-08 13:28:27 MaxScale is shut down.


Message log

The content of message log consists of information about loaded modules, opened listen ports and other information about file locations etc. The amount of data written in message log is typically very small.

Example:

MariaDB Corporation MaxScale /home/jdoe/bin/develop/log/skygw_msg1.log Tue Dec 9 14:47:05 2014


--- Logging is enabled.

2014-12-09 14:47:05 Home directory : /home/jdoe/bin/develop

2014-12-09 14:47:05 Data directory : /home/jdoe/bin/develop/data/data5398

2014-12-09 14:47:05 Log directory : /home/jdoe/bin/develop/log

2014-12-09 14:47:05 Configuration file : /home/jdoe/bin/develop/etc/maxscale.cnf

2014-12-09 14:47:05 Initialise CLI router module V1.0.0.

2014-12-09 14:47:05 Loaded module cli: V1.0.0 from /home/jdoe/bin/develop/modules/libcli.so

2014-12-09 14:47:05 Initialise debug CLI router module V1.1.1.

2014-12-09 14:47:05 Loaded module debugcli: V1.1.1 from /home/jdoe/bin/develop/modules/libdebugcli.so

2014-12-09 14:47:05 Loaded module testroute: V1.0.0 from /home/jdoe/bin/develop/modules/libtestroute.so

2014-12-09 14:47:05 Initialise readconnroute router module V1.1.0.

2014-12-09 14:47:05 Loaded module readconnroute: V1.1.0 from /home/jdoe/bin/develop/modules/libreadconnroute.so

2014-12-09 14:47:05 Initializing statemend-based read/write split router module.

2014-12-09 14:47:05 Loaded module readwritesplit: V1.0.2 from /home/jdoe/bin/develop/modules/libreadwritesplit.so

2014-12-09 14:47:05 Initialise the MySQL Monitor module V1.4.0.

2014-12-09 14:47:05 Loaded module mysqlmon: V1.4.0 from /home/jdoe/bin/develop/modules/libmysqlmon.so

2014-12-09 14:47:05 MariaDB Corporation MaxScale 1.0.1-beta (C) MariaDB Corporation Ab 2013-2014

2014-12-09 14:47:05 MaxScale is running in process 5398

2014-12-09 14:47:05 Encrypted password file /home/jdoe/bin/develop/etc/.secrets can't be accessed (No such file or directory). Password encryption is not used.

2014-12-09 14:47:05 Loaded 6 MySQL Users for service [RW Split Router].

2014-12-09 14:47:05 Loaded module MySQLClient: V1.0.0 from /home/jdoe/bin/develop/modules/libMySQLClient.so

2014-12-09 14:47:05 Loaded 8 MySQL Users for service [Read Connection Router].

2014-12-09 14:47:05 Loaded module telnetd: V1.0.1 from /home/jdoe/bin/develop/modules/libtelnetd.so

2014-12-09 14:47:05 Loaded module maxscaled: V1.0.0 from /home/jdoe/bin/develop/modules/libmaxscaled.so

2014-12-09 14:47:05 Info: A Master Server is now available: 127.0.0.1:3000

2014-12-09 14:47:10 MaxScale is shut down.


Trace log

Trace log includes information about available servers and their states, client sessions, queries being executed, routing decisions and other routing related data. Trace log can be found from the same directory with other logs but it is physically stored elsewhere, to OS's shared memory to reduce the latency caused by logging. The location of physical file is : /dev/shm//skygw_traceX.log where ‘X’ is the same sequence number as in the file name in the /var/log/maxscale directory.

Individual trace log entry looks similar to those in other logs but there is some difference too. Some log entries include a number within square brackets to specify which client session they belong to. For example:

2014-12-09 14:52:36 [6] Session write, routing to all servers.

Writing trace log for each client may produce so much data that it seriously affects on the performance of MariaDB MaxScale. It may also be difficult to follow a specific session if the log is flooded with data from other sessions. While it is possible to dynamically enable and disable trace log as a whole, one can also choose to explicitly enable trace logging for a specific session by first enabling trace log, finding out the session id of the interesting session, disabling trace log and finally enabling trace log only for a given session:

  1. enable log trace (and examine the session id)

  2. disable log trace

  3. enable sessionlog trace

Example of trace log:

MariaDB Corporation MaxScale /dev/shm/5420/skygw_trace1.log Tue Dec 9 14:51:29 2014


--- Logging is disabled.

2014-12-09 14:51:52 --- Logging is enabled --

2014-12-09 14:52:03 Servers and router connection counts:

2014-12-09 14:52:03 current operations : 0 in 127.0.0.1:3003 RUNNING SLAVE

2014-12-09 14:52:03 current operations : 0 in 127.0.0.1:3002 RUNNING SLAVE

2014-12-09 14:52:03 current operations : 0 in 127.0.0.1:3000 RUNNING MASTER

2014-12-09 14:52:03 Selected RUNNING SLAVE in 127.0.0.1:3003

2014-12-09 14:52:03 Selected RUNNING SLAVE in 127.0.0.1:3002

2014-12-09 14:52:03 Selected RUNNING MASTER in 127.0.0.1:3000

...

2014-12-09 14:52:28 [6] > Autocommit: [enabled], trx is [not open], cmd: COM_QUERY, type: QUERY_TYPE_READ, stmt: select count(*) from user

2014-12-09 14:52:28 [6] Route query to slave 127.0.0.1:3003 <

2014-12-09 14:52:36 [6] Disable autocommit : implicit START TRANSACTION before executing the next command.

2014-12-09 14:52:36 [6] > Autocommit: [disabled], trx is [open], cmd: COM_QUERY, type: QUERY_TYPE_GSYSVAR_WRITE|QUERY_TYPE_BEGIN_TRX|QUERY_TYPE_DISABLE_AUTOCOMMIT, stmt: set autocommit=0

2014-12-09 14:52:36 [6] Session write, routing to all servers.

2014-12-09 14:52:36 [6] Route query to slave 127.0.0.1:3003

2014-12-09 14:52:36 [6] Route query to slave 127.0.0.1:3002

2014-12-09 14:52:36 [6] Route query to master 127.0.0.1:3000 <

2014-12-09 14:52:39 [6] > Autocommit: [disabled], trx is [open], cmd: COM_QUERY, type: QUERY_TYPE_BEGIN_TRX, stmt: begin

2014-12-09 14:52:39 [6] Route query to master 127.0.0.1:3000 <

2014-12-09 14:52:51 [6] > Autocommit: [disabled], trx is [open], cmd: COM_QUERY, type: QUERY_TYPE_READ, stmt: select count(*) from user

2014-12-09 14:52:51 [6] Route query to master 127.0.0.1:3000 <

2014-12-09 14:52:59 Servers and router connection counts:

2014-12-09 14:52:59 current operations : 0 in 127.0.0.1:3003 RUNNING SLAVE

2014-12-09 14:52:59 current operations : 0 in 127.0.0.1:3002 RUNNING SLAVE

2014-12-09 14:52:59 current operations : 0 in 127.0.0.1:3000 RUNNING MASTER

2014-12-09 14:52:59 Selected RUNNING SLAVE in 127.0.0.1:3003

2014-12-09 14:52:59 Selected RUNNING SLAVE in 127.0.0.1:3002

2014-12-09 14:52:59 Selected RUNNING MASTER in 127.0.0.1:3000

2014-12-09 14:52:59 Started RW Split Router client session [7] for 'maxuser' from 127.0.0.1

In the log, session’s life cycle is covered by annotating its beginning and the end. During the session, each statement is surrounded with ‘>’ and ‘<’ characters. Entries without ‘[]’ belong to other than client sessions, monitor, for example.

Managing logs

The log files are located in

/var/log/maxscale

by default. If, however, trace and debug logs are enabled, only a soft link is created there. MariaDB MaxScale process creates a directory under

/dev/shm/maxscale.

where it stores the physical trace and debug log files. Link and physical files share the same name. These logs consume the main memory of the host they run on so it is important to archive or remove them periodically to avoid unnecessary main-memory consumption.

Rotating logs

Log files may grow very large over time and that is why users may want to split log files into several smaller ones. That is called log rotation. User can rotate all logs by executing

flush logs

Specific log file can be rotated by executing

flush log [error|message|trace|debug]

The commands above can be executed either by using maxadmin utility or via debug client API with telnet. The sequence number included in the log filename is used to separate files from each other. The logic behind sequence numbering is such that if the log directory is empty when MariaDB MaxScale is started, new log files will be created with sequence number 1 in their names, skygw_err1.log, for example. If files of the same type already exist, the new MariaDB MaxScale process opens the file with largest sequence number and applies into it. If existing log file isn’t writable for the user that MariaDB MaxScale runs on, new log file will be created with bigger sequence number.

More information about log files and administering them can be found from MaxScale Administration Tutorial.