~/.xsession-errors Consumed My Hard Drive: My Best Solution

In my last two posts, I described the ~/.xsession-errors problem and proposed an OK solution using logrotate. I also made a case for logrotate not being a good solution. In this post I give my best solution. As before, I’ll post the code for those who just want to cut and paste without knowing what’s going on, and I’ll follow that with a more detailed explanation of the code.
note: <user> should be replaced with your username on your host.

Code Only (assuming you’ve already implemented the logrotate solution)

# apt-get install incron
# echo root >> /etc/incron.allow
# incrontab -e
/home/<user>/.xsession-errors IN_MODIFY sudo logrotate /etc/logrotate.d/xsession-errors
# nano /etc/logrotate.d/xsession-errors
/home/<user>/.xsession-errors
{
rotate 1
size 64k
copytruncate
missingok
compress
}
# incrond

or restart.

Full Solution
The first thing you need to do is install incron, a program which works like cron, but is triggered by file events rather than time. I did not need to add any new repos to my sources, but you may so I’ve included the code to do that as well. First try to install incron and if it fails due to apt not being able to find the package, add the repo.

# nano /etc/apt/sources.list
[...]
# Debian backports, added to install incron
deb http://backports.debian.org/debian-backports squeeze-backports main
[...]
# wget -O - http://backports.org/debian/archive.key | apt-key add - 
# apt-get update
# apt-get -t squeeze-backports install incron

The backports repo is inactive by default, so you’ll need that -t flag and argument. Also note that my version of crunchbang is based on debian squeeze. Replace squeeze with your distro’s nickname if it is based on some other version of debian. If you’re not using a debian derivative, you’ll need to search for incron in the repos applicable to your distro.

Now you add root to incron allowed users list.

# echo root >> /etc/incron.allow

The reason you need to allow root and not your user is that the logrotate command requires superuser privileges.

Now you create root’s table for incron.

# incrontab -e
/home/<user>/.xsession-errors IN_MODIFY sudo logrotate /etc/logrotate.d/xsession-errors

I’m not sure if that sudo needs to be there or not, but it’s not hurting anything. Someone could try omitting it and report back the results.
If you read the code only solution, hopefully you noticed that the /etc/logrotate.d/xsession-errors file is different from the previous solution. I changed the rotate number to 1 because I only want one compressed ~/.xsession-errors file not two. The size is obvious. Now for the key: copytruncate. Without it, logrotate just compresses the file to ~/.xsession-errors.1.gz, so we’re left with only the compressed log. Since ~/.xsession-errors doesn’t exist, the file descriptor in the processes that had access to it are now invalid. So why not just create a new ~/.xsession-errors? Processes don’t use file names to access files, they use file descriptors, and creating a file with the same name does not automatically update the file descriptor in the processes file descriptor tables. So, even though ~/.xsession-errors exists, the processes don’t have a way to access it. copytruncate solves this problem by sending the contents of ~/.xsession-errors to ~/.xsession-errors.1.gz without deleting ~/.xsession-errors, so the file descriptor to ~/.xsession-errors in each process’s file descriptor table is still valid! This problem could also be solved by immediately rebooting after logrotate runs since X will create a ~/.xsession-errors on startup if one doesn’t exist, but there are two issues to consider:

  • You may not want to reboot, e.g. the machine is a server
  • The ~/.xsession-errors spam may be occurring on reboot

The first case is obvious. The second case should be obvious. Consider that during boot, X starts up some process foo which immediately spams ~/.xsession-errors with garbage until incron gets the signal, calls logrotate, and then reboots. This results in an infinite loop of frustration! So just put copytruncate in /etc/logrotate.d/xsession-errors and be done with it.
Lastly, you need to start the daemon that services incron. You may either invoke it on the command line or just reboot your system.

# incrond

That’s it. I tested this solution, and all of the non-solutions leading up to it, by manually spamming ~/.xsession-errors from the terminal.

$ for i in {1..256}; do cat <file> ~/.xsession-errors; done

Where <file> is the path to some file with size greater than 256B -> logrotate limits ~/.xsession-errors to 64kB and 256 x 256kB = 64kB.
Yes, this solution is really just using incron as a pass-through to logrotate, and while that seems inefficient, consider that the most time consuming operation here is the compression called from logrotate. Another performance consideration is that incron runs each time ~/.xsession-errors is modified. If you have some broken process spamming ~/.xsession-errors with errors 64B at a time, incron and logrotate will have to run a lot of times before the 64kB limit is reached. I justify the expense of such a low probability situation two ways:

  • If you want to make sure ~/.xsession-errors never grows larger than some limit, you must check it each time it is modified
  • Chances are, the broken process is wreaking more havoc than incron and logrotate

So there you have it. Tomorrow, or um sometime this week, I’ll show how I got the TomTom Multisport Cardio to work on Linux.

.xsession-errors Consumed my Hard Drive!

If you’re a Windows or Mac user please ignore. Maybe Mac users should at least read this since they use XServer, but I have it on good authority that “Macs just work” so chances are your error logs won’t chew up all available space on your hard drive.

I recently installed Android Studio, QTCreator, Sublime Text 2, and probably some other developer tools so the increased disk usage didn’t raise any red flags. In hindsight, I should have noticed that there was no way I just installed 7 GB of new software. The proverbial poop hit the fan yesterday. I attempted to use my terminal’s auto-complete feature and received an error message informing me that a new file could not be created due to low disk space. I started deleting things in order of lowest priority + easiest to find. Unfortunately, my disk kept filling up! The ~/.xsession-error file was 7.01 GB!

I’ll skip to my solution and follow it with a detailed walk through where I address some specific errors that showed up in the log file upon startup.
note: <user> should be replaced with your username on your host.

Quick Solution

$ rm ~/.xsession-errors
# nano /etc/logrotate.d/xsession

Insert:

/home/<user>/.xsession-errors
{
rotate 2
size 1M
missingok
not if empty
compress
}

Full Solution
I first checked .xsession-error to make sure there wasn’t anything in there that might help me sort out the problem. First, I attempted to open it in nano.

$ nano ~/.xsession-error

This was a mistake. I already had no disk space, now my kernel was thrashing trying to swap stuff. Even if my disk had gobs of space, I think it would take a long time to open a 7 GB file. So, I took a quick look at the first few lines of the file.

$ more ~/.xsession-error

The timestamps indicated that the first few messages were months old so I tried to look at the last few lines of the file where the useful stuff usually resides.

$ tail ~/.xsession-error

I gave up after two minutes of no response. I don’t know the implementation details of tail, but I suspect it walks through the file to find the end. In fact, I can’t think of any other way it would do it. Anyways, if you know the line numbers you want to view, you can use sed, or if you know you want the last X bytes you can use dd to copy just the last X bytes of the file. I made the assumption that any errors unrelated to disk space would continue to be generated once I solved my disk space issue, and deleted the error file.

$ rm ~/.xsession-errors

Note that rm does not delete the file, it merely deletes the inode mapping, fortunately a quick reboot solves this problem since Linux will see no mapping exists, and reclaim the space. If you’re working on a machine that shouldn’t be rebooted, you will need to delete the file by inode using find. Here is a nice article with instructions for deleting by inode.

My disk space issues resolved, at least temporarily, I checked to see if the error file had been created on boot and then took a look at its contents.

$ ls -al
$ cat ~/.xsession-errors

After the basic, non-error stuff:

Xsession: X session started for [redacted] at [redacted]
...

I noticed something about gnome-keyring-daemon: insufficient process capabilities, unsecure memory might get used. It turns out this is a result of the user (me) not having the proper privileges to perform a CAP_IPC_LOCK on memory. I used a solution by iMBeCil on crunchbang.org:

$ nano ~/.profile

and then inserted at the end:

eval `/usr/bin/gnome-keyring-daemon --start --components=pkcs11,secrets,ssh,gpg`
export SSH_AUTH_SOCK
export GPG_AGENT_INFO

Any time you change something, check the effects of that change before you move on. I deleted .xsession-errors and rebooted. I opened a terminal and saw .xsession-errors had been created, so I opened it up and saw that the gkd errors went away, and as a bonus the

(clipit:2299): Gdk-CRITICAL **: IA__gdk_window_thaw_toplevel_updates_libgtk_only: assertion `private->update_and_descendants_freeze_count > 0' failed

error did not return either. I now had only non-error type stuff in .xsession-errors on startup, and it was only 445 B to boot! Here’s what it looked like:

Xsession: X session started for [redacted] at [redacted]
localuser:[redacted] being added to access control list
tint2 : nb monitor 1, nb monitor used 1, nb desktop 2
** Message: applet now removed from the notification area
** Message: applet now embedded in the notification area

Now the last thing I wanted to fix was the growth of .xsession-errors. Granted, it would take a lot of reboots to generate another 7 GB error file appending only 445 B each time, but startup is not the only operation that appends stuff to .xsession-errors. I was thinking “Why not use a cron to monitor .xsession-errors?” It turns out Linux already provides something to do this: logrotate. I created a configuration file for .xsession-errors:

# nano /etc/logrotate.d/xsession
/home/<user>/.xsession-errors
{
rotate 2
size 1M
missingok
not if empty
compress
}

This rotates two .xsession-errors files. When the current error file grows to 1 MB, logrotate compresses it, deletes the old compressed error file, and starts a new error file. Notice that I needed to be root to modify stuff in /etc/logrotate.d/.

This has worked well so far, though I’ve hardly tested the robustness of this solution. A discussion on the details of this problem and possible root-causes will follow.