Linux logs while the program is running. Linux log files in order. And other magazines

Introduction

One of the things which makes GNU/Linux a great operating system is that virtually anything and everything happening on and to the system may be logged in some manner. This information is invaluable for using the system in an informed manner, and should be one of the first resources you use to trouble-shoot the system and application issues. The logs can tell you almost anything you need to know, as long as you have an idea where to look first.

Your Ubuntu system provides vital information using various system log files. These log files are typically plain ASCII text in a standard log file format, and most of them sit in the traditional system log subdirectory /var/log . Many are generated by the system log daemon, syslogd on behalf of the system and certain applications, while some applications generate their own logs by writing directly to files in /var/log .

This guide talks about how to read and use several of these system log files, how to use and configure the system logging daemon, syslogd , and how log rotation works. See the Resources section for additional information.

Target Audience

This guide will be simple enough to use if you have any experience using the console and editing text files using a text editor. See the end of this document for some essential commands that may help you find your way around these files if you"re relatively new to the command line.

System Logs

System logs deal primarily with the functioning of the Ubuntu system, not necessarily with additional applications added by users. Examples include authorization mechanisms, system daemons, system messages, and the all-encompassing system log itself, syslog.

Authorization Log

The Authorization Log tracks usage of authorization systems, the mechanisms for authorizing users which prompt for user passwords, such as the Pluggable Authentication Module (PAM) system, the sudo command, remote logins to sshd and so on. The Authorization Log file may be accessed at /var/log/auth.log . This log is useful for learning about user logins and usage of the sudo command.

Use grep to cut down on the volume. For example, to see only information in the Authorization Log pertaining to sshd logins, use this:

Daemon Log

A daemon is a program that runs in the background, generally without human intervention, performing some operation important to the proper running of your system. The daemon log at /var/log/daemon.log and contains information about running system and application daemons such as the Gnome Display Manager daemon gdm , the Bluetooth HCI daemon hcid , or the MySQL database daemon mysqld . This can help you trouble-shoot problems with a particular daemon.

Again, use grep to find specific information, plugging in the name of the daemon you"re interested in.

Debug Log

The debug log at /var/log/debug and provides detailed debug messages from the Ubuntu system and applications which log to syslogd at the DEBUG level.

Kernel Log

The kernel log at /var/log/kern.log provides a detailed log of messages from the Ubuntu Linux kernel. These messages may prove useful for trouble-shooting a new or custom-built kernel, for example.

Kernel Ring Buffer

The kernel ring buffer is not really a log file per se, but rather an area in the running kernel you can query for kernel bootup messages via the dmesg utility. To see the messages, use this:

Or to search for lines that mention the Plug & Play system, for example, use grep like this:

By default, the system initialization script /etc/init.d/bootmisc.sh sends all bootup messages to the file /var/log/dmesg as well. You can view and search this file the usual way.

System Log

The system log typically contains the greatest deal of information by default about your Ubuntu system. It is located at /var/log/syslog , and may contain information other logs do not. Consult the System Log when you can"t locate the desired log information in another log. It also contains everything that used to be in /var/log/messages .

Application Logs

Many applications also create logs in /var/log . If you list the contents of your /var/log subdirectory, you will see familiar names, such as /var/log/apache2 representing the logs for the Apache 2 web server, or /var/log/samba , which contains the logs for the Samba server. This section of the guide introduces some specific examples of application logs, and information contained within them.

Apache HTTP Server Logs

The default installation for Apache2 on Ubuntu creates a log subdirectory: /var/log/apache2 . Within this subdirectory are two log files with two distinct purposes:

/var/log/apache2/access.log - records of every page served and every file loaded by the web server.

/var/log/apache2/error.log - records of all error conditions reported by the HTTP server

By default, every time Apache accesses a file or page, the access logs record the IP address, time and date, browser identification string, HTTP result code and the text of the actual query, which will generally be a GET for a page view. Look at the Apache documentation for a complete rundown; quite a lot can be gleaned from this file, and indeed many statistical packages exist that perform analyzes of these logs.

Also, every time any error occurs, Apache adds a line to the error log. If you run PHP with error and warning messages disabled, this can be your only way to identify bugs.

CUPS Print System Logs

The Common Unix Printing System (CUPS) uses the default log file /var/log/cups/error_log to store informational and error messages. If you need to solve a printing issue in Ubuntu, this log may be a good place to start.

Rootkit Hunter Log

The Rootkit Hunter utility (rkhunter) checks your Ubuntu system for backdoors, sniffers and rootkits, which are all signs of compromise of your system. The log rkhunter uses is located at /var/log/rkhunter.log .

Samba SMB Server Logs

The Server Message Block Protocol (SMB) server, Samba is popularly used for sharing files between your Ubuntu computer and other computers which support the SMB protocol. Samba keeps three distinct types of logs in the subdirectory /var/log/samba:

log.nmbd - messages related to Samba"s NETBIOS over IP functionality (the network stuff)

log.smbd - messages related to Samba"s SMB/CIFS functionality (the file and print sharing stuff)

log. - messages related to requests for services from the IP address contained in the log file name, for example, log.192.168.1.1 .

X11 Server Log

The default X11 Windowing Server in use with Ubuntu is the Xorg X11 server, and assuming your computer has only one display defined, it stores log messages in the file /var/log/Xorg.0.log . This log is helpful for diagnosing issues with your X11 environment.

Non-Human-Readable Logs

Some log files found in the /var/log subdirectory are designed to be readable by applications, not necessarily by humans. Some examples of such log files which appear in /var/log follow.

Login Failures Log

The login failures log located at /var/log/faillog is actually designed to be parsed and displayed by the faillog command. For example, to print recent login failures, use this:

Last Logins Log

The last logins log at /var/log/lastlog should not typically be parsed and examined by humans, but rather should be used in conjunction with the lastlog command. For example to see a listing of logins with the lastlog command, displayed one page per screen with the less command, use the following command:

Login Records Log

The file /var/log/wtmp contains login records, but unlike /var/log/lastlog above, /var/log/wtmp is not used to show a list of recent logins, but is instead used by other utilities such as the who command to present a listed of currently logged in users. This command will show the users currently logged in to your machine:

System Logging Daemon (syslogd)

The system logging daemon syslogd , also known as sysklogd , awaits logging messages from numerous sources and routes the messages to the appropriate file or network destination. Messages logged to syslogd usually contain common elements like system hostnames and time-stamps in addition to the specific log information.

Configuration of syslogd

The syslogd daemon"s configuration file is /etc/syslog.conf . Each entry in this file consists of two fields, the selector and the action. The selector field specifies a facility to be logged, such as for example the auth facility which deals with authorization, and a priority level to log such information at, such as info, or warning. The action field consists of a target for the log information, such as a standard log file (i.e. /var/log/syslog), or the hostname of a remote computer to send the log information to.

Echoing Messages to syslogd With Logger

A neat utility exists in the logger tool, which allows one to place messages into the System Log (i.e. /var/log/syslog) arbitrarily. For example, assume your user name is buddha , and you would like to enter a message into the syslog about a particularly delicious pizza you"re eating, you could use a command such as the following at a terminal prompt:

logger This Pizza from Vinnys Gourmet Rocks

and you would end up with a line in the /var/log/syslog file like this:

You can even specify a tag the messages come from, and redirect the output standard error too.

Executing this script as chkdir.sh on the machine butters where Fred does not have a home directory, /home/fred , gives the following results:

So, as you can see, we received the messages both via standard error, at the terminal prompt, and they also appear in our syslog.

Log Rotation

When viewing directory listings in /var/log or any of its subdirectories, you may encounter log files with names such as daemon.log.0 , daemon.log.1.gz , and so on. What are these log files? They are "rotated" log files. That is, they have automatically been renamed after a predefined time-frame, and a new original log started. After even more time the log files are compressed with the gzip utility as in the case of the example daemon.log.1.gz . The purpose of log rotation is to archive and compress old logs so that they consume less disk space, but are still available for inspection as needed. What handles this functionality? Why, the logrotate command of course! Typically, logrotate is called from the system-wide cron script /etc/cron.daily/logrotate , and further defined by the configuration file /etc/logrotate.conf . Individual configuration files can be added into /etc/logrotate.d (where the apache2 and mysql configurations are stored for example).

This guide will not cover the myriad of ways logrotate may be configured to handle the automatic rotation of any log file on your Ubuntu system. For more detail, check the Resources section of this guide.

NOTE: You may also rotate system log files via the cron.daily script /etc/cron.daily/sysklogd instead of using logrotate. Actually, the utility savelog may produce unexpected results on log rotation which configuring logrotate seems to have no effect on. In those cases, you should check the cron.daily sysklogd script in /etc/cron.daily/sysklogd and read the savelog manual page to see if savelog is not in fact doing the rotation in a way that is not what you are specifying with logrotate.

Essential Commands

If you"re new to the console and the Linux command line, these commands will get you up and running to the point where you can work with log files at a basic level.

Getting Started

To change to the log directory, where most of these files sit, use the cd command. This saves having to type out a full path name for every subsequent command:

Editing Files

You can view and edit files in GEdit or Kate, the simple text editors that come with Ubuntu and Kubuntu respectively, but these can be overkill when all you want to do is look at a file or make simple changes. The easiest editor to use from the console is nano, which is less powerful but also less complicated than vim or emacs. The command to edit a particular logfile /var/log/example.log using nano is:

nano example.log

Press Ctrl+X to exit. It will ask if you want to save your changes when you exit, but unless you run it with the sudo command the files won"t be writable. In general, you won"t want to save your changes to log files, of course.

Viewing Files

To simply look at a file, an editor is overkill. Use the less command, which pages through a file one screen at a time:

You don"t need sudo to look at a file. Press h for help, or q to quit. The cursor keys and page up/down keys will work as expected, and the slash key ("/") will do a case- sensitive search; the n key repeats the last search.

Viewing the Beginning of Files

To see the first ten lines of a file, use the head command:

head example.log

To see some other number of lines from the beginning of the file, add the -n switch, thus:

head -n 20 example.log

Viewing the End of Files

To see the final ten lines of a file, the analogous command is tail:

Again, the -n switch gives you control over how many lines it displays:

tail -n 20 example.log

Watching a Changing File

Also, the -f ("follow") switch puts tail into a loop, constantly waiting for new additions to the file it"s displaying. This is useful for monitoring files that are being updated in real time:

Press Ctrl+C to quit the loop.

Searching Files

Because log files can be large and unwieldy, it helps to be able to focus. The grep command helps you strip out only the content you care about. To find all the lines in a file containing the word "system", for example, use this:

To find all the lines containing "system" at the beginning of the line, use this:

Note the caret symbol, a regular expression that matches only the start of a line. This is less useful for standard log files, which always start with a date and time, but it can be handy otherwise. Not all files have a standard format.

Any time the result of a grep is still too long, you can pipe it through less:

Resources

Additional information on system and application logs and syslogd is available via the following resources:

Local System Resources

If you take up Linux administration, be prepared for the fact that viewing and analyzing log files will take up the lion's share of the time you spend in the console. Log analysis is the main (and most often the only) way to understand the server’s behavior.

Often, the log contains thousands of lines; moreover, it can increase by several more entries every second. And it is advisable to watch it live, tracking the reaction to certain actions. Here two utilities tail and less will help us.

Viewing logs in Linux using tail .

tail (translated as tail) allows you to view 10 last lines file. If you need more, just say so. For example, 25:

# tail -n 25 /var/log/httpd/error_log

Or you can simply:

# tail -25 /var/log/httpd/error_log

The second useful option that can be passed is -f . With this parameter, tail prints the specified number of last lines and continues to read added lines until you press Ctrl+C - i.e. you can monitor changes live in the log file:

# tail -5f /home/sites/web/site.ru/logs/site.ru.log 178.45.0.0 - - "GET / HTTP/1.0" 200 17401 "http://site.ru/allsorts/" "Mozilla /5.0 (iPad; CPU OS 8_0_2 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12A405 Safari/600.1.4" 217.118.0.0 - - "GET /allsorts HTTP/1.0" 301 - "http://site.ru/" 217.118.0.0 - - "GET /allsorts/ HTTP/1.0" 200 16663 "http://site.ru/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75 Safari/601.1" 85.95.0.0 - - "GET / HTTP/1.0" 200 17401 "http://vk.com/away.php " "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 OPR/34.0.2036.50" 85.95.0.0 - - "GET /allsorts/ HTTP/1.0" 200 16663 "http://site.ru/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 OPR/34.0.2036.50"

But the biggest convenience of tail is that several files can be passed as parameters, i.e. you can monitor multiple related log files at the same time:

# tail -f -n 5 -s 3 /home/sites/web/site.ru/logs/site.ru.log /var/log/httpd/domains/site.ru.error.log ==> /home/ sites/web/site.ru/logs/site.ru.log<== 37.113.0.0 - - "GET /wp-content/themes/bigroll/img/delivery-feature-6.png HTTP/1.1" 200 18165 "http://site.ru/delivery/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75 Safari/601.1" 37.113.0.0 - - "GET /wp-content/themes/bigroll/img/delivery-truck.png HTTP/1.1" 200 15243 "http://site.ru/delivery/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75 Safari/601.1" 37.113.0.0 - - "GET /wp-content/themes/bigroll/img/scooter-icon-small.png HTTP/1.1" 200 4322 "http://site.ru/delivery/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75 Safari/601.1" 37.113.0.0 - - "GET /wp-content/themes/bigroll/img/link-button-shadow.png HTTP/1.1" 200 7792 "http://site.ru/delivery/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75 Safari/601.1" 37.113.0.0 - - "GET /wp-content/themes/bigroll/img/header-bg.png HTTP/1.1" 200 194475 "http://site.ru/delivery/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75 Safari/601.1" ==>/var/log/httpd/domains/site.ru.error.log<== client denied by server configuration: /home/sites/web/site.ru/public_html/xmlrpc.php client denied by server configuration: /home/sites/web/site.ru/public_html/xmlrpc.php client denied by server configuration: /home/sites/web/site.ru/public_html/xmlrpc.php client denied by server configuration: /home/sites/web/site.ru/public_html/xmlrpc.php client denied by server configuration: /home/sites/web/site.ru/public_html/xmlrpc.php

Please note that I used the -s 3 construction - using this key, the update interval is set in seconds, i.e. in this case, the files will be reread once every 3 seconds.

Viewing logs on Linux using less.

Despite all its simplicity and convenience, tail has one significant drawback - it shows only the last lines and you do not have the opportunity to see what happened before these records or at a certain point in time. For such situations, the less utility is better suited.

Let's open the file:

# less /var/log/httpd/domains/big-roll.ru.error.log

As you can see, we can easily scroll through the contents of the file using the cursor arrows and the PgUp and PgDn keys.
Remember a few more useful keys:
q – output;
g – go to the beginning of the file;
G – go to the end of the file;
Ng – go to line N (for example, line 14 14g);
N% – go to position N% (for example, 35%);
F – switch to file update tracking mode, i.e. analogue of tail -f ;
/pattern – direct (down the file) search using the pattern pattern;
?pattern – reverse (up the file);
n – next match of the search pattern;
N – previous match;
Exit tracking mode Ctrl+C ;
You can see the complete list of all other keys and modes yourself by pressing h – help (I highly recommend pressing this).

A Linux Administrator should be able to read and understand the various types of messages that are generated by all Linux systems in order to troubleshoot an issue. These messages, called logs, are initiated by Linux and the applications running on it. Linux continuously creates, stores and recycles these logs through various configuration files, programs, commands, and daemons. If you know how to read these files and make optimal use of the various commands we will mention in this tutorial, you can troubleshoot your issues like a pro!

It is important to note that Linux keeps its log files in the /var/log directory in text format.

Viewing System Logs on Ubuntu

In order to reach the core of an issue, or to see if your application or system is behaving in the desired manner, you can view the system log files either graphically or through command line in the following ways:

• Gnome Logs utility (Graphic)
• Log File Viewer utility (Graphic)
• Linux Terminal (Command Line)

View Log Files Through Gnome Logs

‘Logs’ is the default utility that comes with the latest versions of Ubuntu e.g., Ubuntu 18.04 LTS (Bionic Beaver). In order to access it,

Type Logs in the Ubuntu dash:

You will be able to see the Logs utility open, with the option to view logs for Applications, System, Security and Hardware.

Click on the System tab to view system logs:

Here you can view all the system logs along with the time they were generated. You can perform the following actions through this window:

• Display the contents of a log by clicking on it.
• Search for a log by clicking the search icon and then providing keywords in the search bar. The search bar also offers a number of filters that you can apply in order to exactly specify What(Select a Journal field to filter the logs according to it) and When(Select the timestamp range of the log entries to be shown) you want to see:

• You can also export logs to a file by clicking the export button located at the top right corner of the Logs window. You can then save the log file by specifying a name and location.

Through Log File Viewer

The Log File Viewer is the default utility that comes with the older versions of Ubuntu. If your edition of Ubuntu does not have this application by default, you can download and install it through Ubuntu Software.

In order to access the Log File Viewer:

• Enter Log View r in Ubuntu Dash

• If you have installed this program through Ubuntu Software, you can launch it by searching for it in the Ubuntu Software as follows and then clicking the Launch button:

The Log File Viewer will appear as follows:

The left panel of the window shows a number of default log categories and the right panel shows a list of logs for the selected category.

Click on the syslog tab to view system logs. You can search for a specific log by using ctrl+F control and then enter the keyword. When a new log event is generated, it is automatically added to the list of logs and you can see it in bolded form. You can also filter your logs through the Filters menu located in the top menu bar.

In order to view a log for a specific application, click the Open option from the File menu. The following Open Log window will open for you to choose the log from:

Click on a log file and click Open. You will now be able to see logs from the selected log file in the Log File Viewer.

View Log Files Through the Terminal

You can also view system logs through the command line, i.e., the Ubuntu Terminal.

Open the Terminal and enter the following command:

This command fetches all the messages from the kernel’s buffer. You can see the output as follows:

You will see that this is a lot of information. This information will only be useful if we apply some filters to view what we want to see.

Customizing dmesg output

• In order to see messages at your own pace, use the following command:

$ dmesg |less

This command will display only a specific number of messages per screen. You can press Enter in order to move to the next message or press Q to exit the command.

• In order to search for a message that contains a specific keyword, use the following command:

For example, if you want to search for all the messages containing the word core, you can use the following command:

$dmesg |grep core

The Terminal will now display only those messages containing the word “core” in red color.

Open a Log File with cat Command

The dmesg command opens all the logs from the /var/log directory. In order to open the log file from some other location, use the following command:

$cat

$ cat /var/log/syslog

This command will print logs from the syslog file to the screen. You will again observe that this command prints all the information and it is not easy to skim through. Here again, you can use the ‘grep’ and ‘less’ filters to display the desired output as follows:

$cat |grep

$ cat |less

Writing To the System Log

Sometimes we need to write custom messages to our system log during the troubleshooting process. Both the Gnome Log and the Log File Viewer programs are built to display a customized message that you can write through the Terminal.

Open the Ubuntu Terminal and type the following command:

$logger “This is a custom message”

You can see the custom log message, at the end of the above log list, displayed in the graphical log file viewer.

You can also use the logger command within a script for providing additional information. In that case, please use the following command within your script:

$ logger -t scriptname “This is a custom message”

By practicing along with this tutorial, you can learn to troubleshoot your system and application issues by accessing and understanding system logs.

How to View System Log Files on Ubuntu 18.04 LTS

System administrators, and regular Linux users, often need to look at log files to troubleshoot problems. In fact, this is the first thing any system administrator should do when any error occurs in the system.

The Linux operating system itself and the running applications generate various types of messages that are logged in various log files. Linux uses special software, files and directories to store log files. Knowing which files contain the logs of which programs will help you save time and solve the problem faster.

In this article we will look at the main parts of the Linux logging system, log files, as well as utilities with which you can view Linux logs.

Most Linux log files are located in the /var/log/ folder. You can list the log files for your system using the ls command:

Rw-r--r-- 1 root root 52198 May 10 11:03 alternatives.log
drwxr-x--- 2 root root 4096 Nov 14 15:07 apache2
drwxr-xr-x 2 root root 4096 Apr 25 12:31 apparmor
drwx------ 2 root root 4096 May 5 10:15 audit
-rw-r--r-- 1 root root 33100 May 10 10:33 boot.log

Below we will look at 20 different Linux log files located in the /var/log/ directory. Some of these logs are only found on certain distributions, for example dpkg.log is only found on Debian based systems.

/var/log/messages- contains global Linux system logs, including those that are recorded at system startup. Several types of messages are recorded in this log: mail, cron, various services, kernel, authentication and others.

/var/log/dmesg- contains messages received from the kernel. Logs many messages during the boot phase, they display information about hardware devices that are initialized during the boot process. You can say this is another log of the Linux system. The number of messages in the log is limited, and when the file is full, with each new message the old ones will be overwritten. You can also view messages from this log using the dmseg command.

/var/log/auth.log- contains information about user authorization in the system, including user logins and authentication mechanisms that were used.

/var/log/boot.log- Contains information that is logged when the system boots.

/var/log/daemon.log- Includes messages from various background daemons

/var/log/kern.log- Also contains messages from the kernel, useful in troubleshooting errors in custom modules built into the kernel.

/var/log/lastlog- Displays information about the last session of all users. This is a non-text file and you must use the lastlog command to view it.

/var/log/maillog /var/log/mail.log- logs of the email server running on the system.

/var/log/user.log- Information from all logs at the user level.

/var/log/Xorg.x.log- X server message log.

/var/log/alternatives.log- Information about the operation of the update-alternatives program. These are symbolic links to default commands or libraries.

/var/log/btmp- Linux log file contains information about failed login attempts. To view the file, it is convenient to use the command last -f /var/log/btmp

/var/log/cups- All messages related to printing and printers.

/var/log/anaconda.log- all messages recorded during installation are saved in this file

/var/log/yum.log- Logs all information about package installations using Yum.

/var/log/cron- Whenever the Cron daemon starts executing a program, it writes a report and messages from the program itself in this file.

/var/log/secure- contains information related to authentication and authorization. For example, SSHd logs everything here, including failed login attempts.

/var/log/wtmp or /var/log/utmp - Linux system logs , contain a log of user logins. Using the wtmp command you can find out who is logged in and when.

/var/log/faillog- Linux system log, contains unsuccessful login attempts. Use the faillog command to display the contents of this file.

/var/log/mysqld.log- Linux log files from the MySQL database server.

/var/log/httpd/ or /var/log/apache2- log files of linux11 Apache web server. Access logs are in the access_log file, and error logs are in the error_log

/var/log/lighttpd/- Linux logs of the lighttpd web server

/var/log/conman/- ConMan client log files,

/var/log/mail/- this directory contains additional mail server logs

/var/log/prelink/- Prelink links libraries and executables to speed up the loading process. /var/log/prelink/prelink.log contains information about .so files that were modified by the program.

/var/log/audit/- Contains information generated by the auditd daemon.

/var/log/setroubleshoot/ - SE Linux uses the setroubleshootd daemon (SE Trouble Shoot Daemon) to report security problems. This log contains messages from this program.

/var/log/samba/- contains information and logs from the Samba file server, which is used to connect to Windows shares.

/var/log/sa/- Contains .cap files collected by the Sysstat package.

/var/log/sssd/- Used by the system security daemon, which manages remote directory access and authentication mechanisms.

Viewing logs in Linux

To view logs on Linux, it is convenient to use several Linux command line utilities. This can be any text editor, or a special utility. Most likely, you will need superuser rights to view logs in Linux. Here are the commands that are most often used for these purposes:

• zgrep
• zmore

I will not go into detail on each of these commands, since most of them have already been discussed in detail on our website. But I will give a few examples. Viewing Linux logs is very simple:

We look at the log /var/log/messages, with the ability to scroll:

less /var/log/messages

View Linux logs in real time:

tail -f /var/log/messages

Open the dmesg log file:

cat /var/log/dmesg

First lines of dmesg:

head /var/log/dmesg

We only output errors from /var/log/messages:

grep -i error /var/log/messages

In addition, you can view logs on Linux using graphical utilities. System Log Viewer can be used to easily view and monitor system logs on a Linux laptop or personal computer.

You can install the program on any system with an X server installed. Also, any graphical test editor can be used to view logs.

conclusions

In the /var/log directory you can find all the necessary information about the operation of Linux. From today's article you have learned enough to know where to look and what to look for. Now viewing logs in Linux will not cause you problems. If you have any questions, ask in the comments!

Placing a pipe symbol (|) in front of the file name will allow you to use fifo (first in - first out, first in, first out) or named pipe as a receiver for messages. Before starting (or restarting) syslogd, you must create a fifo using the mkfifo command. Sometimes fifo are used for debugging.

Terminal and console

A terminal such as /dev/console.

Remote machine

To forward messages to another host, precede the host name with an (@) symbol. Please note that messages are not forwarded from the receiving host. (for this assignment to work on the client and server in the file /etc/services line must be written syslog 514/udp, and UTP port 514 is open)

a list of users

Comma separated list of users receiving messages (if the user is logged in). This often includes the root user.

All registered users

To notify all registered users using the wall command, use the asterisk (*) character.

An example of a simple syslog.conf:

# Print all kernel messages to the console. #kern.* /dev/console # All logs at info level or higher, except email messages, and # do not log authentication messages or cron daemon messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # Log messages containing sensitive # authentication information to a separate file, regardless of their level. authpriv.* /var/log/secure # All messages from the mail system are also written to a separate file. mail.* -/var/log/maillog # Log scheduler messages to /var/log/cron cron.* /var/log/cron # Emergency messages should be immediately received # by all system users *.emerg * # Save messages news of crit level and higher into a separate file. uucp,news.crit /var/log/spooler # Save boot messages to boot.log local7.* /var/log/boot.log

As with many configuration files, the syntax is as follows:

• Lines starting with # and empty lines are ignored.
• The * symbol can be used to indicate all categories or all priorities.
• The special keyword none specifies that logging for this category should not be performed for this action.
• A hyphen before the file name (like -/var/log/maillog in this example) indicates that the log should not be synchronized after each entry. If your system crashes, you may lose information, but disabling synchronization will improve performance.

In the syntax of the configuration file, you can put before the priority sign! to indicate that the action should not be applied, from this level and above. Similarly, priority can be given sign = to indicate that the rule applies only to this level, or != to show that the rule applies to all levels except this one. Below are a few examples (there are many other examples in man syslog.conf):

# Send all kernel messages to /var/log/kernel. # Send all critical and higher level messages to the remote sysloger machine and to the console # Send all info, notice and warning level messages to /var/log/kernel-info # kern.* /var/log/kernel kern.crit @sysloger kern .crit /dev/console kern.info;kern.!err /var/log/kernel-info # Send all mail system messages except the info level to /var/log/mail. mail.*;mail.!=info /var/log/mail

I tried to show the operation of syslogd as clearly as possible in the diagram:

Starting the syslogd daemon

Starting the logging daemon is started during system initialization using a script /etc/rc.d/init.d/syslog, however, in order to set launch parameters, there is no need to adjust this script - starting from version 7.2, launch options are read from a separate configuration file /etc/sysconfig/syslog (/etc/default/syslogin debian).

Here are some possible syslogd daemon launch parameters:

• -a /folder/socket- specifying an additional listening socket (don't forget to create the socket first)
• -d- debugging mode. In this case, the daemon does not go into the background and issues all messages to the current terminal;
• -f config-file-name. Specifies the name of an alternative configuration file that will be used instead of the default /etc/syslog.conf;
• -l host-list- specifying a list of hosts whose names should not be written with the full domain name (FQDN - Full Qualified Domain Name);
• -m minutes- sysklogd running without this option writes messages of the mark category to the log every 20 minutes. Using the -m option, you can either change the interval between marks or completely cancel the issuance of such messages;
• -p socket- setting an alternative UNIX socket (instead of the default listening /dev/log);
• -r- permission to receive messages from remote hosts;
• -x- prohibition of determining a host name by its address to prevent freezing when working on the same host with a DNS server.
• -v- show version and finish work

After the syslogd daemon starts, a status file is created /var/lock/subsys/syslog zero length, and a file with a process identification number /var/run/syslogd.pid.

Using the command
kill -SIGNAL `cat /var/run/syslogd.pid`

can you send syslogd daemon one of the following signals: SIGHUP- restart the daemon; SIGTERM- completion of work; SIGUSR1- enable/disable debugging mode.

Actually, there are two logging daemons running on the system - syslogd And klogd. Both daemons are included in the package sysklogd.

Demon klogd is responsible for logging events occurring in system core. The need for a separate klogd daemon is because the kernel cannot use the standard syslog function. The fact is that the standard C libraries (including the library in which the syslog function is located) are intended to be used only regular applications. Since the kernel also needs logging functions, it includes its own libraries that are not available to applications. Therefore, the kernel uses its own message generation mechanism.

Demon klogd is designed to organize the processing of these messages. In principle, it could do this processing completely independently and independently of syslogd, for example by logging these messages to a file, but in most cases the default setting of klogd is used, in which all messages from the kernel are forwarded to the same syslogd daemon.

Automatic rotation (updating of completed files) and archiving of logs

Over time, the log file tends to grow, especially when a service is running intensively. Accordingly, it is necessary to be able to control the size of the logs. This is done using logrotate commands, which is usually performed cron daemon. I will talk about the work of cron in the following articles. the main objective logrotate commands is to periodically back up the logs and create new clean logs. Multiple generations of logs are saved and when the last generation log reaches the end of its lifespan, it can be archived (compressed). The result can be sent by mail, for example, to the person responsible for maintaining archives.

To determine the order of rotation and archiving of logs, use configuration file /etc/logrotate.conf . You can set different frequency for different logs, such as daily, weekly or monthly, you can also control the number of generations accumulated, and you can also specify whether and when copies of the archives will be sent to the archive manager. Shown below example file /etc/logrotate.conf:

# first the default parameters are set (global options) # update log files weekly weekly # store an archive of logs for the last 4 weeks rotate 4 # create a new (empty) file after rotation (update) create # uncomment if you want the saved ones files were compressed #compress # enable rotation settings from the specified directory include /etc/logrotate.d # do not store wtmp, or btmp -- log data rotation settings are as follows: /var/log/wtmp ( missingok monthly create 0664 root utmp rotate 1 ) / var/log/btmp ( missingok monthly create 0664 root utmp rotate 1 ) # specific syslogs can be configured below

Global options are placed at the beginning logrotate.conf file. They are used by default unless something more specific is specified elsewhere. In the example, the logs are rotated weekly and backups are retained for four weeks As soon as the journal is rotated, a new one is automatically created in place of the old journal. logrotate.conf file may contain specifications from other files. So, it includes all the files from the directory /etc/logrotate.d.

This example also contains special rules for /var/log/wtmp And /var/log/btmp(storing information about successful and unsuccessful attempts to log into the system), which are rotated monthly. If files are missing, no error message is generated. A new file is created and only one backup is saved.

In this example, when the backup reaches the last generation, it is deleted because it is not determined what to do with it.

Log Backups can also be created when logs reach a certain size, and scripts can be created from sets of commands to run before or after a backup operation. Example:

/var/log/messages ( rotate 5 mail logadmin@sysloger size 100k postrotate /usr/bin/killall -HUP syslogd endscript )

In this example the rotation /var/log/messages produced when it reaches 100 KB in size. Five backups are accumulated, and when the oldest backup expires, it is emailed to logadmin@sysloger. The postrotate command word enables a script that restarts the syslogd daemon after rotation has completed by sending a HUP signal. The endscript command word is required to terminate the script, and also if there is a prerotate script. See the man pages for logrotate for more information.

Options specified in the configuration file logrotate.conf:

• compress| nocompress(older versions may or may not be compressed using gzip)
• compresscmd(specifies the compression program, default is gzip)
• uncompresscmd(specifies the decompression program, default is ungzip)
• compressext(specifies the suffix for compressed files)
• compressoptions(sets the parameters of the compression program; default is "-9", i.e. maximum compression for gzip)
• copytruncate| nocopytruncate(usually the old version is renamed and a new version of the log is created; when this parameter is set, logrotate copies the log to a new file and then truncates the old one; used if the program that created the log does not know how to close it; entries made in the interval between copying and truncating are lost ; but would it help if the program creating the log, instead of the append mode, simply writes to a file using an internal pointer?)
• create[permissions owner group] | nocreate(immediately after renaming the old version of the log and before calling postrotate, a new log is created with the specified attributes - access rights are set in octal form, as in chmod.2; if the attributes are not specified, then they are taken from the old log)
• daily(versions in the series change daily)
• delaycompress| nodelaycompress(some programs do not immediately close the journal, in which case compression must be postponed until the next cycle)
• errorsemail(who to send bug reports to)
• extensionsuffix(sets the suffix added to file names during rotation before the compression suffix)
• ifempty| notifempty(change versions even if the file is empty; default)
• includefile name| directory-name (textually substitute a file or all files from the specified directory; subdirectories, special files and files with suffixes from the exclusion list are not included; cannot be used inside a section)
• mailaddress| nomail(when changing versions leads to the need to delete the old journal, then send it to the specified address)
• mailfirst(send not the deleted version of the log, but the first one)
• maillast(send deleted log version; default)
• missingok| nomissingok(do not send error messages if log is missing)
• monthly(versions change monthly)
• olddirdirectory| noolddir(during a version change, the log is moved to the specified directory; it must be on the same physical device)
• postrotate(all further lines up to the endscript line are executed as shell commands after the version change process)
• prerotate(all further lines up to the endscript line are executed before the version change process)
• rotatenumber(how many old versions to store; if 0, then none)
• sizebyte(a version change occurs if the log size exceeds the specified number; you can use the suffixes "k" - kilobyte - and "M" - megabyte)
• sharedscripts| nosharedscripts(execute the prerotate and postrotate commands only once for all files described in the section)
• tabooext[+] suffix-list(specifying a list of exclusion suffixes for include; if a plus sign is specified, then addition, otherwise replacement; default: .rpmorig, .rpmsave, .rpmnew, ",v", .swp and "~")
• weekly(versions change weekly)

Examining and monitoring logs

Log entries typically contain a timestamp, the name of the host on which the process being described is running, and the name of the process. View logs you can use a pager program, for example, less, you can search for specific entries (for example, kernel messages from a specific daemon) using the command grep:

# less /var/log/messages # grep "ppp" /var/log/messages | tail Dec 17 16:34:25 proxy pppd: Connection terminated. Dec 17 16:34:25 proxy pppd: Exit. Dec 17 16:35:57 proxy pppd: LCP terminated by peer (^P]kV^@

The computer may not work constantly and turn off, say, at night. Therefore, entries in /var/log/messages are stored cyclically from computer startup to shutdown, this can be seen in the messages:

Dec 17 08:32:56 syslog-server syslogd 1.4-0: restart. Dec 17 08:32:56 syslog-server syslog: starting syslogd succeeded Dec 17 08:32:56 syslog-server kernel: klogd 1.4-0, log source = /proc/kmsg started. Dec 17 08:32:56 syslog-server syslog: starting klogd succeeded

Dec 17 08:32:56 syslog-server kernel: Kernel command line: auto BOOT_IMAGE=linux ro root=303 BOOT_FILE=/boot/vmlinuz-2.4.2-2 Dec 17 08:32:56 syslog-server kernel: Memory: 125652k/130560k available (1365k kernel code, 4200k reserved, 92k data, 236k init, 0k highmem) Dec 17 08:32:56 syslog-server kernel: CPU: Intel(R) Pentium(R) 4 CPU 1.60GHz stepping 02

Also, in this file you can find information about disk memory (including information about disk geometry, partition structure and interrupts used), information about peripheral devices, about the launch of individual services, information about connecting file systems and messages about user logins , as well as error messages.

Sometimes it may be necessary monitoring system logs to search for current events. For example, you can try to catch a rarely occurring event at the moment when it happens. In this case, you can use the command tail with option -f to monitor the contents of the system log. Example:

# tail -f /var/log/messages | grep syslog-server Dec 17 16:46:09 syslog-server pppd: pptpd-logwtmp.so ip-up ppp0 maikop 94.77.0.150 Dec 17 16:46:09 syslog-server pppd: Script /etc/ppp/ip-up finished (pid 12552), status = 0x0 Dec 17 16:46:49 syslog-server pptpd: CTRL: Client 85.175.197.65 control connection started Dec 17 16:46:49 syslog-server pptpd: CTRL: Starting call (launching pppd, opening GRE) Dec 17 16:46:49 syslog-server pppd: Plugin /usr/lib/pptpd/pptpd-logwtmp.so loaded.

In addition to the log files specified in /etc/syslog.conf, there are also other files, for example a file that stores information about the system boot process before syslogd was started, as well as files that have a binary format and store information about the last login user into the system, about all successful user logins and all unsuccessful user logins, respectively. Also in the /var/log/ directory there may be log files of such daemons as a web server or a proxy server. The format of these files is similar to syslogd logs.

Finally, I would like to emphasize that this protocol is not very secure, because syslog does not contain any protection against message forgery. Even worse, using the UDP protocol allows attackers to send messages on behalf of any host. Your local network should be shielded from receiving packets with spoofed local addresses (although this will not prevent you from sending spoofed messages from inside the local network) and from receiving packets from outside on port 514/udp. There are known cases of disk overflow with false messages.

The syslog and UDP protocols do not provide guaranteed delivery (messages may be lost due to network congestion or intercepted, corrupted messages are deleted without warning), correct delivery sequence (a message about the completion of a process may arrive before the message about its start), or priority delivery.

Confidentiality of messages is not ensured, since they are transmitted in clear text.

If, when setting up a message generator, you specify an incorrect collector or relay address, then there will be no error messages - the messages will be deleted (or written to someone else's log).

Several projects have been proposed to improve the syslog protocol. For example, RFC 3195 proposes a TCP-based logging system (syslog-conn) that ensures messages are delivered in the correct sequence. The syslog-sign project proposes to provide authentication, ordering, message integrity, and detection of missing messages by generating special messages containing a digital signature of a block of previous messages, preserving the standard syslog protocol and format, and using UDP.

Let's summarize:

Linux has a single daemon responsible for logging events on the local system and remote systems. All events are collected from the /dev/log socket, UDP port - 514, as well as from the “helper” - the klogd daemon, which sends messages from the kernel. All collected messages are filtered by the syslogd daemon through rules in the /etc/syslog.conf file and distributed according to the rules to the appropriate destinations. Log files are periodically truncated. The frequency is determined by the logrotate.conf file and the logrotate command, which is launched by the system scheduler - cron.

That's all for today. I hope I described everything as clearly as possible. I will add to the article over time!

Best regards, McSim!