How do you monitor your low end virtual servers?

My personal setup needs to meet the following requirements:

• Centralised monitoring on multiple boxes, as I have servers with various sizes — from 32MB to 1.5GB memory. Need something that work for them all, and monitor from one single location.
• Simple RRD of memory, bandwidth utilisation and load average will do.
• Light weight and use minimum amount of memory. Don’t want some crap coded in Ruby or Python that takes up 10MB memory just to monitor a cheap VPS with only 64MB to play around.

So I ended up using Cacti + a few hand coded scripts. Cacti does need a bit hacks to do notification stuff when servers are down, but it’s in PHP so relatively easy to modify.

What do you guys use?

Today I received an email from Arthur Thornton from LxCenter, the new home for the now-open-sourced HyperVM and Kloxo.

They (HyperVM and Kloxo) will be open sourced when we release the vulnerability fix versions of both panels and we will continue to support them from then on.

The reason I want you to post this is because your site seems to be popular and I think there are many people still worried about the products and this could reassure them.

Another note is that they will be free to use completely. But the most important part is that they will live on, most likely in an AGPL license (off the record), rather than dying with Ligesh.

Sounds like a great news to me. For the last month and half, the VPS scene has been hectic. Tons of half-baked control panel programmes flooded the market, and there are some open sourced ones trying to grab some shares as well. Although I doubt any is offering the functionality of HyperVM at its price point (otherwise it would be a clear winner by now). It is also a time to find out which hosts are one-man show and which ones are capable — from their strategies in developing custom in-house panel, or in migrating customers to new backend, etc.

Also note the clever choice of AGPL which means those made customisations/enhancements and then deploy them onto public network must also contribute back. That would only mean good things to the community.

Well. I guess we just have to wait to find out.

A seedbox is a private dedicated server used for the uploading and downloading of digital files. Seedboxes generally make use of the BitTorrent protocol for uploading and downloading, although they have also been used on the eDonkey2000 network. Seedboxes are usually connected to a high speed network, often with a throughput of 100 Mbit/s or more. Files are uploaded to a seedbox from other BitTorrent users, and from there they can be downloaded at high speeds to a user’s personal computer via the HTTP, FTP, SFTP, or rsync protocols.

In another word, a server with fast and stable bandwidth to the Internet so owners can use it as step-stone to download files from P2P network, as traditionally ADSL or cable connection (and their modems) on a residential link is too unreliable to bring back hundreds and thousands of pieces of files you wish to download in a reasonable speed.

Now, are low end virtual servers good candidates for seedbox hosting? Before we get into that, I need to make it clear that downloading copyrighted digital content without copyright holder’s consent is ILLEGAL in many places in the world so don’t do it. I personally do not use bittorrent because I much prefer downloading the minimum-install Linux CD ISO from my local mirror, which is usually only around 150MB :)

Anyway. If you want to use a $5 VPS as a cheap seedbox, I guess the answer is YES. When I need to download stuff using bittorrent I’ll use rTorrent because (1) it’s in command line and you can run it inside screen (2) it uses relatively less memory than the Python reference client. If you are on OpenVZ/Virtuozzo-based VPS you’ll need to watch out for the TCP socket limit though.

$ sudo cat user_beancounters | grep numtcpsock | awk '{print $4;}'
7999992

BT creates a massive number of TCP sockets and some providers might limit the TCP sockets you can have on your VE. Actually, it is a good idea to check your Terms and Conditions of Use before running a P2P client, because some providers explicitly forbid P2P traffic in their network.

HOWEVER, I would hate to share a server with someone running bittorrent though. Physical server IO is shared by all the virtual servers on it and apparently it is harder to limit IO than to throttle CPU usage. Due to the way bittorrent clients are designed, they are seeking, reading and writing everywhere on a big file. End result is that VPS running as seedbox hosting usually uses a lot of IO requests which affects other VPS on the same box (and we are not even talking about all the calculation needed to do checksum). So beware — you might be hated by your neighbours :)

However, interactive shell is one thing, and scripting shell is another. If your scripts are compatible with the original bourne shell, then you should try dash instead of that big fat Bash! This morning I just discovered one of my Debian 5 VPS is not having /bin/sh linked to dash by default, so I “fixed” the problem, and here is the memory consumption result:

Before

$ ps aux | grep /bin/sh
root      7473  0.0  0.1   2416  1156 ?        S    Feb12   0:00 /bin/sh /usr/bin/mysqld_safe

After

$ ps aux | grep /bin/sh
root     15528  0.0  0.0   1632   520 ?        S    10:03   0:00 /bin/sh /usr/bin/mysqld_safe

Yup. 1/2 MB reclaimed. Just make sure your /bin/sh is linked to dash. If it’s not already the case, then for Debian/Ubuntu based systems

# apt-get install dash
# dpkg-reconfigure dash

You also need to make sure all your scripts are compatible with sh/dash (which they ought to anyway).

Another idea is to replace bash (~4M/instance) with pdksh (~1M/instance). It doesn’t sound like much, but if you’re like me and you’re have a half-dozen or more concurrent ssh sessions, plus you’re su’ed on a couple of them, etc… it can be a significant help.

Indeed. Bash, which is the default interactive shell on most (if not all) Linux distributions, has pretty much become THE user interface of the command line users. When I first used it in mid-90′s (on SunOS/Solaris), it was a great relief from the plain vanilla bourne shell, and was much better than the default shell back then, korn shell, IMHO. It was love at first sight.

However over the years she has just gone fatter, heavier and was getting more and more CPU intensive to start. It did not really bother me though, as my main computer has also grown in grunt… Not until are trying to run it on a small low end box!

pdksh seems to be quite a good replacement. It uses relatively less resource, has simple tab completion, has history (what else do you need from an interactive shell?) and has bourne shell style syntax (if csh/tcsh is not your thing).

If you are really extreme and thinking tab completion and command line history are for n00bs, also give dash a try. Here’s a memory usage comparison:

USER   PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root  8658  0.0  2.8   3788  1848 pts/0    S    20:42   0:00 bash
root  8732  0.2  0.7   1604   464 pts/0    S    20:44   0:00 dash
root  8733  0.0  0.8   2844   552 pts/0    S    20:44   0:00 pdksh

For me, I’ll stick with bash for now. You know, user interface is the thing that once you got used to, it’s hard to unattach yourself from (i.e. too old to learn new things). Good luck young padawans for picking up new light weight shells!

So is it possible to host a low traffic PHP site on a low end VPS? Yes, but the first thing you need to do is getting rid of the A in LAMP.

While infinitely configurable, pre-forked Apache (which is required for mod_php) is not really scalable to suit low memory requirement. I won’t get into all the details, but there are several free/open source alternatives.

At LowEndBox, I am running Lighttpd + PHP in FastCGI mode. I am running 2x slave FastCGI process and it works quite well for 3x very low usage sites on this box, using less than 15MB of memory all together.

# ps -u www-data u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data 27509  0.0  3.1   6672  2092 ?        S    11:50   0:00 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
www-data 27510  0.0  7.3  16656  4848 ?        S    11:50   0:00 /usr/bin/php5-cgi
www-data 27512  1.1 11.9  16652  7868 ?        S    11:50   0:04 /usr/bin/php5-cgi

Installing them is easy –

# apt-get install lighttpd php5-cgi

Then just install other PHP modules (MySQL, GD, etc). Here is a comprehensive guide on running PHP on Lighttpd, but on Debian/Ubuntu, you usually just need do:

# ln -s /etc/lighttpd/conf-available/10-fastcgi.conf /etc/lighttpd/conf-enabled

Make sure you edit the 10-fastcgi.conf and (1) leave max-procs at 2 (2) reduce PHP_FCGI_CHILDREN to 1. After starting the daemon, you’ll see 2x php5-cgi process running. With PHP 5.2.x it would not spawn a manager process. However, if you intend to use PHP op-code cache/accelerator, then it would be a good idea to change max-procs to 1 and PHP_FCGI_CHILDREN to 2 so that your 2 worker processes can share the cache (but you’ll see a 3rd php5-cgi process running as manager/dispatcher).

While it’s lighter and faster, Lighttpd does not offer as much functionality as Apache, nor as configurable. For applications that depend on Apache’s mod_rewrite capability, there is usually hacks and work-around for Lighttpd. For WordPress to get clean URL, just add

server.error-handler-404 = "/index.php"

to your configuration file for your WP sites.

Another alternative web server that I’ve used for my other projects is Nginx. It uses slightly more memory (at least 7MB on 32bit Ubuntu) and does not support spawning CGI processes (so all FastCGI procs have to be external). A great piece software if you have very hight traffic site.

The easiest way is to start from an existing configuration file to bring down the memory usage, and then tune it to bring back adequate performance. On Debian/Ubuntu you might wish to replace your installed /etc/mysql/my.cnf with my-small.cnf that can be found in /usr/share/doc/my-server-5.0/examples. I am also attaching a my.cnf here that is taken out from CentOS 4 installation of LxAdmin (which has been famous for its low-memory foot print). It is pretty much based on my-small.cnf. Some important notes:

• skip-bdb and skip-innodb are added, so you don’t get BSD DB nor InnoDB support. BSD DB support in MySQL is pretty much obsolete, and many open source scripts don’t rely on the presence of InnoDB. A low end VPS is not likely to enjoy the concurrency InnoDB is offering anyway. Transaction and referential integrity? Real Programmers^TM write their own rollback routines :)

• key_buffer is only 16K which is far from enough. key_buffer is pretty much one of the most important parameter for MyISAM tables and I usually bump it up to at least 1MB. The same can be said about table_cache — 4 is way too small. A WordPress page will likely touch 10 tables, and much more for apps like Drupal or MediaWiki.

• Query cache might be a good thing if you intend to run a busy site on such a low end VPS (provided that it has small data set, mostly read, like blogs, news sites, etc). I have my query_cache_limit set to 256K and query_cache_size set to 4M here.

After a few adjustment you should be able to start mysqld at around 5-6MB RSS. You might need to check the runtime variables to monitor the performance and how everything works out.

On a Ubuntu 7.10 x86 server running OpenSSH:

$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      2052  0.0  0.2   5288   996 ?        Ss   Mar25   0:00 /usr/sbin/sshd
root     21819  0.2  0.7   8168  2700 ?        Ss   10:28   0:00 sshd: user@pts/0
user     21836  0.0  0.4   8044  1748 ?        S    10:29   0:00 sshd: user@pts/0
...

On a Debian 4 x86 server running Dropbear:

$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      1344  0.0  1.3   2648   864 ?        Ss   10:28   0:00 /usr/sbin/xinetd
root      1347  0.2  1.8   4216  1220 ?        Ss   10:29   0:00 dropbear -i -g
...

As you can see it is quite a bit of saving, especially if you operates a shell service where lots of people logging in (or use sftp/scp) at the same time. Also note that:

• While Dropbear can be used as a standalone server, I am using it with xinetd so it uses even less memory (I am also running xinetd for a few other services).
• If you log in as a non-root user, OpenSSH spawns two processes but Dropbear spawns only one.

Dropbear does have a few issues (terrible performance with SftpDrive, fewer features and options, etc), but it feels as good as OpenSSH for server administration but uses half amount of memory.