So you are trying to wean your users off of using FTP for obvious reasons, but your FTP server was at least doing chroot’ing which was nice for keeping them isolated from one another. You flip your users over to SFTP and now they’re not chrooted, AND, there’s not even a log of what they are doing like you had with your old FTP server.

So, first, fix the logging. You’re probably running OpenSSH, so simply make this line in your /etc/ssh/sshd_confg file:

Subsystem    sftp /usr/libexec/openssh/sftp-server

look like this:

Subsystem    sftp /usr/libexec/openssh/sftp-server -l INFO

That tells the sftp-server binary to log level INFO. You can add a -f FACILITY as well if you want to change the facility to something other than the default of AUTH. Now, when your users log in and do stuff, you’ll have a record of it, including uploads, downloads, directory changes, removals, etc.

Well, let’s say you try to lock security down a bit via chrooting your users into their home directories. Uh oh, your logging just disappeared didn’t it? This is because the sftp-server binary writes to syslog via the socket file /dev/log which is unreachable from your chrooted environment.

My first idea at a workaround was let’s set the syslog server to listen for network connections and get sftp-server to send its events via udp to the syslog server.  Nope, sftp-server does not support logging in that manner, although someone else had the same idea and actually wrote a patch to OpenSSH to implement udp-based syslogging:

https://github.com/mwarner1/openssh-udp-logging

I liked the idea but didn’t even think of going that route because OpenSSH receives patches fairly regularly and I don’t want to have those patches need to be downloaded, patched for logging, recompiled and then deployed to all of my servers whenever a security change occurs since time is critical at that point.

My second idea, maybe I can get my syslog server, RSYSLOG (http://www.rsyslog.com/) which is the default in RHEL/CentOS 6, to just put a socket file in each user’s chrooted directory.  This actually worked fine; basically what you do is for each chrooted environment, you add the following to a file in /etc/rsyslog.d/; I chose customer-logging.conf, and got the right syntax from http://www.rsyslog.com/doc/imuxsock.html Even better, you can specify a hostname to be logged with each entry:

$InputUnixListenSocketHostName jail1.example.net
$AddUnixListenSocket /jail/1/dev/log
$InputUnixListenSocketHostName jail2.example.net
$AddUnixListenSocket /jail/2/dev/log

With that config in place, entries logged in jail1 will be tagged with jail1.example.net, your first customer, and entries generated by customer #2 will be tagged jail2.example.net.

Okay, so everything is working great, until you add more than 49 websites to the server.  Unfortunately there is a compiled in limit of 50 total sockets in rsyslogd.  On a shared hosting environment, you probably have hundreds of sites per server so that doesn’t scale well, plus it’s not very efficient.  You could download the SRPM, up the limit, recompile, redeploy, but just like the patch to OpenSSH, I didn’t want to do that because I didn’t want to have to worry about doing that every time an update comes down the pipe.

So, my final idea was maybe a hard link would work.  There is of course the restriction that hard links can’t cross mount points, and they also do not work on NFS mounts, so this solution to my problem may not work for you if your customer chrooted area resides on an NFS volume.  In my case, my /var is a separate mount but is not NFS, so I needed a second rsyslogd socket under /var to test my idea.  I added the following in /etc/rsyslog.d/customer-logging.conf:

$AddUnixListenSocket /var/customer-syslog.sock

Then, I restarted rsyslogd and went into each customer’s chrooted area and ran the following (customer’s already had a dev directory FYI):

ln /var/customer-syslog.sock dev/log

Now each customer chroot area has a hard link to the /var/customer-syslog.sock file created by rsyslogd, and logging works great with just that one additional socket.  This adds a minor maintenance hassle of having to create the hard link as each new customer is added, and if you forget it, you won’t get any logging for that customer, but it’s far less maintenance than the other options.  Unfortunately I don’t have any ideas for those of you mounting your customer areas on NFS other than what was previously presented which was not ideal.

Of course, there is a caveat to my solution like there always seems to be; the logging only includes the sftp-server PID and the path information from a chrooted perspective, so you’ll see things like this in your syslog:

Mar 14 18:08:43 www sftp-server[4783]: session opened for local user userA from [192.0.2.1]
Mar 14 18:08:44 www sftp-server[4783]: opendir "/"
Mar 14 18:08:44 www sftp-server[4783]: closedir "/"
Mar 14 18:08:50 www sftp-server[4783]: opendir "/cgi-bin"
Mar 14 18:08:50 www sftp-server[4783]: closedir "/cgi-bin"
Mar 14 18:08:50 www sftp-server[4783]: opendir "/cgi-bin"
Mar 14 18:08:50 www sftp-server[4783]: closedir "/cgi-bin"
Mar 14 18:08:56 www sftp-server[5168]: session opened for local user userA from [192.0.2.2]
Mar 14 18:08:57 www sftp-server[5168]: opendir "/cgi-bin"
Mar 14 18:08:57 www sftp-server[5168]: closedir "/cgi-bin"
Mar 14 18:08:57 www sftp-server[5168]: sent status No such file
Mar 14 18:08:57 www sftp-server[5168]: open "/cgi-bin/formmail.cgi" flags WRITE,CREATE,TRUNCATE mode 0666
Mar 14 18:08:59 www sftp-server[5168]: close "/cgi-bin/formmail.cgi" bytes read 0 written 436470
Mar 14 18:08:59 www sftp-server[5168]: opendir "/cgi-bin"
Mar 14 18:08:59 www sftp-server[5168]: closedir "/cgi-bin"
Mar 14 18:09:09 www sftp-server[4783]: remove name "/cgi-bin/formmail.cgi"
Mar 14 18:09:13 www sftp-server[5168]: session closed for local user userA from [192.0.2.2]

Annoying right?  You’ve got userA logging in from two different IP’s and doing different things at the same time, and just the PIDs to separate the activity.  And of course PIDs get recycled, so if you’re trying to figure out what a specific customer did, you just need to be very careful about watching the order events are logged and tying the right PIDs back to the correct sessions.