Stale NFS Mounts
If you use NFS for file sharing, the odds are you've come by some stale NFS mounts now and then.
I just had this issue on a semi-important box that I didn't want to reboot.
I'd had a directory from my laptop mounted on this box (TITAN) and turned the laptop off. Oops. I rarely turn the laptop off, so when I ssh'd in from work and attempted to cd to the laptop, all hell broke loose.
At this point, I'd tried everything - I had a few ls processes open, from trying to poke around. The mount point was symlinked to my homedir, so I couldn't even ls home without it hanging. I attempted to clean up some of the hanging processes. Interestingly, even lsof and fuser hung. Trying to umount the share gave the usual:
TITAN:/home/jantman # umount /mnt/antmanLaptop Cannot MOUNTPROG RPC: RPC: Port mapper failure - RPC: Unable to receive umount: /mnt/antmanLaptop: device is busy Cannot MOUNTPROG RPC: RPC: Port mapper failure - RPC: Unable to receive umount: /mnt/antmanLaptop: device is busy
Even more interestingly, looking at
mount, I found something interesting: two exactly identical lines. As if the share was mounted twice to the same mount point.
I tried a series of
umount -f and
umount -r, seemingly to no avail - though
umount -r did remount ONE of the instances ro.
I let the system sit for 10 minutes or so, then came back. The ro mount had disappeared, leaving me with only one mount (the way it should be).
umount -r again got me a single ro mount. Still,
umount -f and
-r were giving device busy errors. And
fuser were still hanging.
At this point, even trying
service nfs stop gave me:
Cannot MOUNTPROG RPC: RPC: Port mapper failure - RPC: Unable to receive umount: /mnt/antmanLaptop: device is busy
and umount gave the same error.
I tried the advice here and actually got somewhere. After trying to kill -9 the PID of
rpciod, the mount actually disappeared. And lsof worked perfectly.
So, the bottom line was trying lots of killing processes that I thought were using the mount, then trying a few umounts, stopping nfs, trying more umounts, killing rpciod, and then bingo, no more stale mount.
There has to be an easier way to do this...