Every once in a while you encounter this edge scenario that requires some specialized knowledge. And let’s face it, it’s always nice to find a quick cheat sheet with “recipes” for common (and sometimes less common) problems, than having to RTFM for hours, trying to find the solution.

So I decided to list the common problems I’ve encountered, and the solutions I found for those problems that have actually worked.

 

Deleting a massive number of files

Sometimes a directory full of old session files (maybe left behind by some php script) can be a real pain, especially when you’re trying to rsync or backup that site. Sometimes there are so many files in that directory, that an rm -rf * will return an error about the argument list being too long. The problem here is the usage of the asterisk (globing), which produces a huge list of files. To understand the effect of * just type this in a directory with some files: echo *

The following command will help you:

find /session/directory/ -type f -delete

Or if you want to erase only sessions that are older than 5 days:

find /session/directory/ -type f -mtime +5 -delete

Note 1: if you add -name sess_* you will stumble into the same globing pitfall mentioned above, so make sure to enclose it with quotes: ‘sess_*’.

Note 2: If you are doing this on a production server, then do this instead with a bash for loop:

 for i in *; do 
 rm -f $i; 
 done

This will take longer, but will keep the server i/o load extremely low and won’t affect your production systems.

 

Converting from ext2 to ext3 and back

Nobody will judge you for choosing to use the ext2 filesystem, a real expert uses the tools for the job. In this case, the expert would make a calculated decision to trade safety for performance. Not having to write a journal could boost performance, but not having a journal could pose a problem during data recovery. However data recovery might not always be a concern.

This conversion process will not result in any data loss, and should be fairly quick even on large drives.

The process for ext2 -> ext3:

1. Unmount the ext2 partition
2. tune2fs -j /dev/sdb1
3. Edit /etc/fstab and change ext2 to ext3 for /dev/sdb1
4. Remount your ext3 partition

 

Hard Reboot on a semi-dead machine

Yesterday I found myself still logged into a machine that was stuck in a semi dead state, due to data corruption on one of the drives. It was impossible to reboot the machine using the conventional methods, and I had no IPMI or KVM for this old server, which meant I had to rely on a technician to physically press the reset button. But it turns out there is another way, and if you find yourself in a similar situation with root access to a barely functioning machine, you might want to consider the REISUB process, which uses the SysReq kernel feature as described in this wonderful Wikipedia article about the Magic SysRq key.

The short version is this:

echo 1 > /proc/sys/kernel/sysrq  # Enable the SysRq feature (just in case it's disabled)
echo s > /proc/sysrq-trigger     # Sync: Write all buffers to disks
echo b > /proc/sysrq-trigger     # Reboot the machine

 

MySQL Dump with Broken Views

So you decided to dump a database, either as backup, or to transfer it to another machine. But you get one of the below errors:

mysqldump: Couldn't execute 'SHOW FIELDS FROM `funny_table`': View 'silly_db.funny_table' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them (1356)

mysqldump: Got error: 1449: The user specified as a definer ('silly_db'@'localhost') does not exist when using LOCK TABLES

Fear not, you can dump it with this command:

mysqldump --force -uBob -pSecret --single-transaction silly_db > silly_db.sql

You will still get some warnings or errors but you can probably safely ignore them. What’s happening here is that something or someone left behind some Views that refer to tables that no longer exist, or to tables that exist but with fields that no longer exist. Some wordpress plugins have bad hygiene habits, and do not properly clean up behind them (hey, just like some people!).

Reading BIOS Events

Sometimes there’s a problem with your server harware. But if your machine lacks IPMI or DRAC, how do you know what’s wrong with the hardware?

Luckily, the BIOS will in some cases log some of the hardware faults in its BIOS Event Log. But how do you get to it from the comfort of your couch? dmidecode to the rescut.

In debian / ubuntu, just apt-get install dmidecode and then run:

dmidecode -t 15

This will show you the last few errors in your event log. Maybe an ECC Memory error, which might indicate your memory modules are faulty (as happened to me).