Expert GuideLog file is larger than the file system?

Andreas Schwab
Reading time: 4:53

On some RedHat Enterprise and Fedora Linux VMs on ESX/VMware it can happen that the log file /var/log/lastlog becomes extremely large. What to do?

For testing purposes, some VMs are additionally backed up with the ISP B/A client via a so-called in-guest backup. Usually, backup tools like IBM Spectrum Protect Plus or Veeam are used for the backup of a VMware environment. In rare cases, however, it can make sense to back up a VM with an In-Guest Client, as this gives you more options when it comes to retention times, e.g. of individual files, or when you need to back up VMs with RAW devices.

The following problem occurs with both IBM Spectrum Protect B/A Client (In-Guest) and IBM Spectrum Protect Plus.

Why in RedHat Enterprise and Fedora Linux VMs the log file /var/log/lastlog becomes large

On some RedHat Enterprise and Fedora Linux VMs, the /var/log/lastlog log file can become extremely large.

In our case, the file was apparently larger than the file system itself at about 500GB. 😊

We noticed this when sporadic "full backups" of various RHEL systems always backed up in about 500GB. The backup results looked somewhat the same on all systems:

Sicherungsergebnisse Last Log Datei

We quickly found it in the ISP backup log file dsmsched.log:

The culprit was called /var/log/lastlog.

In the following nine incremental backups, the file was backed up on 11/27, 11/28, and 11/02/2021.

Through research, it became clear why this file is not backed up daily.

The /var/log/lastlog file stores all successful logins. So it changes only when someone has logged on to the system.

And now for the strange part of the story.

The file is in reality not as big as it is displayed by the ls. We have already revealed this secret in this article, and yet it looks strange when you issue the following commands.

According to the backup and the ls command, this file would have a size of 494.88 GB.

How does a 494.88 GB file fit into a file system that is only 40GB in size?

If you check the file lastlog with the du -h command, you get the following result:

In fact, the file is only 48K in size, but 494.88 GB are backed up!

This could also be easily checked in IBM Spectrum Protect with a QUERY OCCUPANCY or AUDITOCCUP. Because all affected systems occupied much too much space.

The solution

Since others had this error before us, we quickly found it. 😊

The solution is simple at first: simply recreate or empty the corrupt log with the following command.

However, this step alone does not solve the problem conclusively, because the file immediately has a size of about 500 GB again at the next login.

And to avoid problems with this file in the future, you could simply exclude the file from the backup with an exclude statement.

exclude /var/log/lastlog

But if you absolutely have to backup the file, you could try to work with Client Compression.

It worked very well for one of our customers with this.

Effects on the backup

After deleting the data from the backup and a subsequent EXPIRE INVENTORY, we noticed a significant reduction in the backup volume.

In the following example, 2 TB could be saved:

After cleanup

Depending on the license model and the size of the company, this problem could of course have an impact on license costs.

In this example, the backup was performed with IBM Spectrum Protect Backup/Archive Client version 8.1.12.000.

Source

The following VMware Knowledge Base article exists on this issue:

To the VMware article