$ free -m
             total       used       free     shared    buffers     cached
Mem:           512         65        446          0          0          0
-/+ buffers/cache:         65        446
Swap:            0          0          0

So — 512MB total memory and 0 swap? After all I am only using 65MB of memory so well below my “guaranteed” amount. Zero swap because it is just how OpenVZ does its memory account. But is it really the case that all my processes reside in physical RAM?

Let’s dig out the good ol’ user_beancounters.

$ sudo cat /proc/user_beancounters | grep -E '(uid|physpages|oomguarpages)'
uid  resource              held       maxheld       barrier         limit       failcnt
     physpages             6662         11570             0    2147483647             0
     oomguarpages          8085         12750         65536    2147483647             0

I am trying to highlight the “physpages” and “oomguarpages” amount. According to OpenVZ Wiki:

physpages

Total number of RAM pages used by processes in this container.

oomguarpages

oomguarpages parameter accounts the total amount of memory and swap space used by the processes of a particular container.

Sounds like one is the total number of physical pages used by the container, and the other one is the total number of pages (physical + swap) used by the container. That means when you take oomguarpages – physpages you get — the number of swapped pages from processes in this container! Now back to my UBC readouts, I have

(8085 – 6662) * 4096 / 1024 / 1024 = 5.56MB!

out of 31.58MB (8,085 pages) that has been swapped out. And the 65MB figure returned by “free” is merely how much my processes have been allocating — so only 1/2 are actually used, and 18% of that is already swapped out.

So, has your low end provider been overselling the memory and caused your pages to swap out?

At the same time I would like to have a bug/enhancement tracking system to track and to prioritize my work. A wiki for documentation would be a bonus (if I ever have time to write docos). Trac fits the bill. I know there is also RedMine but I am a Python developer it’s just easier for me to write plugins for Trac.

This will (hopefully) be a step-by-step guide to install both Git and Trac 0.12 on a Ubuntu 10.04 system. I used my 6 pound/year RackVM VPS in UK to write this HOWTO. 256MB burstable memory is more than enough.

Do note that this is not a tutorial on how to use git or any other distributed version control system. See Git’s website if you wish to learn more about it.

1. Setting up a Low End Box

To get the basics running,

1. Get an empty VPS, but feel free to use one that is already deployed for something else — in this case feel free to skip this method and just make sure Nginx is installed.
2. Refer to the lowendscript article to set up a low end VPS with Exim4 and Nginx running.

# wget http://github.com/lowendbox/lowendscript/raw/master/setup-debian.sh
...
# bash setup-debian.sh system
...
# bash setup-debian.sh exim4
...
# bash setup-debian.sh nginx
...

2. Install Git and Set up Master Repository

Make sure git is installed. We are also creating a git user under /home/git as the home of all repositories.

# apt-get install -y git-core
...
# useradd -m -K UMASK=027 git

Now you have a “git” user. You could set it up with git-shell as restricted login shell (which also needs to be added to /etc/shells), however I found it’s too restrictive, especially when I also want to be able to copy the repositories around with scp. You will also need to decide how to authenticate it — with password or with pubkeys so you can log in to push/pull the git repositories in there.

For example I want to create copy my lowendscript repository there. I will do this on my home computer (assuming running Linux).

$ cd ~/proj/lowendscript
$ git clone --bare . /tmp/lowendscript.git
$ scp -r /tmp/lowendscript.git git@my-gitbox:
$ git remote add origin git@my-gitbox:lowendscript.git

Replace “my-gitbox” with the hostname of your VPS. scp would only work after you have set up authentication for user “git” (password or pubkey).

You should now be able to push/pull from the central shared repository.

3. Install Trac

Ubuntu 10.04 LTS only comes with Trac 0.11 but I really wanted to use Trac 0.12 because of better multi-repositories support. So instead of apt-get install trac, we are going the long way and downloading and installing it manually. The Trac installation guide is a good reference, as well as Git plugin for Trac.

Let’s get some Python dependencies installed first, and then use easy_install to install Trac and Git plugin. Note that Genshi 0.6 is required for Trac so we won’t be using the 0.5 from Ubuntu repo.

# apt-get install python python-setuptools python-pygments python-tz subversion
...
# easy_install Trac
...
# easy_install http://trac-hacks.org/svn/gitplugin/0.12
...

I will now create a trac user to run the standalone instance, and initialize our first Trac project. Note that I am adding user trac into git group so it can have read access to the git repositories.

# useradd -m -K UMASK=077 -G git trac
# sudo -u trac trac-admin ~trac/project initenv
...

Just put in your new project’s name (that can be changed later), and use the default for the rest. Done! You have a Trac project created, and it’s time to get the Trac standalone server up and running.

# sudo -u trac tracd -s -p 8000 --pidfile ~trac/project/tracd.pid -d ~trac/project

You should be able to connect to port 8000 on your VPS via your browser to see it up and running. However, a plain vanilla Trac installation is not very useful. We need to get authentication working first so you can administer your Trac install from its web interface. You can find more info on setting up authentication for Trac standalone server here. Need to get apache2-utils installed first so we can use htpasswd.

I’ll also create a “lowendadmin” user with password “12345″.

# apt-get install apache2-utils
...
# sudo -u trac htpasswd -bc ~trac/project/htpasswd lowendadmin 12345
Adding password for user lowendadmin

We will also need to grant WebAdmin permission to the user “lowendadmin”. Now restart Trac to use the htpasswd file we have just created.

# [ -e ~trac/project/tracd.pid ] && kill `cat ~trac/project/tracd.pid`
# sudo -u trac trac-admin ~trac/project permission add lowendadmin TRAC_ADMIN
# sudo -u trac tracd -s -p 8000 --pidfile ~trac/project/tracd.pid -d --basic-auth="*,/home/trac/project/htpasswd,Project" ~trac/project

When you log back into Trac (port 8000 on your VPS), you should be able to see a “Login” link. Click on that to bring out the HTTP Basic Authentication dialog box that prompts for username/password. Once authenticated, you’ll be back at Trac with “Admin” link at the top navigation bar.

4. Adding Git Repository to Trac

Once you have Admin enabled on Trac, it can be all GUI work from now. First you need to enable GitPlugin for Trac.

1. Click on “Admin” at top right.
2. Click on “Plugins” on the left navigation menu, under “General”
3. Click on “TracGit” in the middle of the page, enable “GitConnector” and Save Change

Next, you need to add your repositories using the Admin interface.

1. Click on “Repositories” on the left navigation menu, under “Version Control”
2. In my lowendscript example, I put
   • Name=lowendscript
   • Type=git
   • Directory=/home/git/lowendscript.git
3. Click on Add! And you should now be able to browse your source under “Browse”

That’s pretty much it for Trac. It would be a good idea to put that long tracd command in /etc/rc.local so it starts everytime you reboot the VPS.

5. SSL + Proxy via Nginx

Now, you do want your latest source code browser + bug tracking system behind SSL, right? I found the easiest way to do it is by putting a Nginx proxy in front. Since we already have Nginx installed in step 1, we just need to

• Get a SSL certificate.
• Add Nginx configuration to do the proxy.

For “free” SSL certificates, I use StartSSL, which works for most modern browsers. However to make it simple, I am going to use self-signed certificate in this tutorial.

# openssl genrsa -out /etc/nginx/ssl.key 1024
Generating RSA private key, 1024 bit long modulus
..............................................++++++
........................................++++++
e is 65537 (0x10001)
# openssl req -new -key /etc/nginx/ssl.key -out /tmp/ssl.csr
...
Country Name (2 letter code) [US]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:my-gitbox
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
# openssl x509 -req -days 365 -in /tmp/ssl.csr -signkey /etc/nginx/ssl.key -out /etc/nginx/ssl.crt
Signature ok
subject=/C=US/ST=Some-State/O=Internet Widgits Pty Ltd/CN=my-gitbox
Getting Private key
# rm -f /tmp/ssl.csr

Done. Your self signed SSL key and certificate are now installed. Time to get Nginx to use it. Save the following file to /etc/nginx/sites-enabled/my-gitbox.conf:

server {
    server_name _;
    listen 443;
    ssl on;
    ssl_certificate /etc/nginx/ssl.crt;
    ssl_certificate_key /etc/nginx/ssl.key;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
    location /trac/chrome/site {
        rewrite /trac/chrome/site/(.*) /htdocs/$1 break;
        root    /home/trac/project;
    }
}

Restart Nginx, and now browse to your VPS’s HTTPS port. You’ll see Trac in its SSL glory (well, after bypass the browser’s security check at least).

Trac also happens to have pretty good Wiki engine, and it has heaps of plugins for customization. With a bit of optimization (most notably reducing the stack size for multi-threaded tracd), you should be able to run the entire stack with less than 80MB on an OpenVZ VPS (and much less on a Xen).

Now with code repository, bug tracking system and documentation wiki ready, I better get back to do more coding :)

It is based on my previous script here, and it has been tested on both Debian 5 “Lenny” and Ubuntu 10.04 “Lucid Lynx”. Sorry no other distributions are supported. It is currently hosted at github so feel free to branch it and tune it to your heart’s desire.

Usage

Here is a quick step by step on how to set up a WordPress blog running on Nginx, PHP/FastCGI and MySQL on a low end box. Let’s assume that you are going to set up a blog on hostname “blog.example.com“.

1. Go and buy a cheap VPS. Yes, you know where to find cheap virtual servers under $7/month, don’t you? :) I am using my 6 pound/year RackVM VPS with 128MB memory to test this script, and the performance turns out to be quite good. YMMV.

   Make sure it is either a Debian or a Ubuntu distro though.

2. After gaining root access to your VPS,

   # wget --no-check-certificate https://github.com/lowendbox/lowendscript/raw/master/setup-debian.sh
   ...
   # bash setup-debian.sh system
   ...
   # bash setup-debian.sh exim4
   ...
   # bash setup-debian.sh nginx
   ...
   # bash setup-debian.sh mysql
   ...
   # bash setup-debian.sh php
   ...
   # bash setup-debian.sh wordpress blog.example.com
   ...
   

3. Now altering your DNS entry to point blog.example.com to your new VPS (or alter your local /etc/hosts entry). Navigate to it from your browser, and you should be presented with WordPress set up page. Fill in your blog’s name, admin’s password and contact email, and your new WordPress is online!

   Just repeat the last step (running bash setup-debian.sh wordpress <hostname>) to set up more blogs on the same box.

That’s it!

Software Stack

It’s pretty similar to my own stack that runs LowEndBox.com.

• dropbear to replace openssh. Invoked from xinetd.
• inetutils-syslogd to replace rsyslog.
• exim4 to replace sendmail (if installed). Re-configured to allow Internet delivery.
• cron
• nginx
• mysql. Re-configured to remove innodb support, remove query cache and reduce key buffer size.
• php with built-in FastCGI interface. Running only 1 child process and respawn after 5,000 requests.

The lowendscript is also

• Removing some commonly bundled applications that should not be there in the first place for a minimal distro (apache2, sendmail, bind9, samba, nscd, etc).
• MySQL root is given a new password (which can be found in ~root/.my.cnf)
• Installing each WordPress site under /var/www/<hostname>. It will create appropriate database, users and password for the site.

I have to say that it is still “work in progress”, but if you want to figure out the internals please check the source code of that bash script.

Memory Usage

I am testing the same script on the following 3 systems:

• Ubuntu 10.04 64bit (RackVM)
• Debian 5 64bit (RackVM)
• Debian 5 32bit (VPSGuy)

They are all showing different memory usage. I am taking the snapshot of memory usage AFTER 10 page loads on the WordPress blog that has been created to ensure PHP process is fully loaded.

Ubuntu 10.04 64bit

root@ubuntu64:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.5  23300  1536 ?        Ss   06:58   0:00 init
root     10124  0.0  0.5  23576  1496 ?        Ss   07:34   0:00 dropbear -i
root     15742  0.0  0.7  17880  1944 pts/0    Ss   07:35   0:00 -bash
root     15754  0.0  0.4  14960  1084 pts/0    R+   07:35   0:00 ps aux
root     24290  0.0  0.3  12508   788 ?        S    07:22   0:00 /usr/sbin/syslo
root     24524  0.0  0.3  21064  1004 ?        Ss   07:22   0:00 cron
root     25689  0.0  0.3  19512   972 ?        Ss   07:22   0:00 /usr/sbin/xinet
mysql    26346  0.0  4.2  61812 11248 ?        Ssl  07:23   0:00 /usr/sbin/mysql
www-data 27920  0.0  2.6  87768  7048 ?        Ss   07:23   0:00 /usr/bin/php-cg
www-data 27922  0.5 15.0 122460 39432 ?        S    07:23   0:04 /usr/bin/php-cg
103      30162  0.0  0.3  44356  1000 ?        Ss   07:24   0:00 /usr/sbin/exim4
root     30267  0.0  0.3  28104   932 ?        Ss   07:24   0:00 nginx: master p
www-data 30268  0.0  0.8  28744  2196 ?        S    07:24   0:00 nginx: worker p

root@ubuntu64:~# free
             total       used       free     shared    buffers     cached
Mem:        262144     100500     161644          0          0          0
-/+ buffers/cache:     100500     161644
Swap:            0          0          0

Memory Usage: 98.1MB

Debian 5 64bit

debian64:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.2  10304   740 ?        Ss   12:22   0:00 init [2]
106       1414  0.0  0.3  42708  1004 ?        Ss   12:26   0:00 /usr/sbin/exim4
www-data  1655  0.0  2.5  72028  6560 ?        Ss   12:26   0:00 /usr/bin/php-cg
www-data  1657  0.9 11.7  98500 30928 ?        S    12:26   0:03 /usr/bin/php-cg
www-data  3126  0.0  0.7  28380  1912 ?        S    12:29   0:00 nginx: worker p
root      7668  0.1  0.5  23340  1528 ?        Rs   12:31   0:00 dropbear -i
root      7831  0.0  0.6  17472  1716 pts/0    Ss   12:32   0:00 -bash
root      7836  0.0  0.3  14720   980 pts/0    R+   12:32   0:00 ps aux
root     20169  0.0  0.3  18540   936 ?        Ss   12:22   0:00 /usr/sbin/cron
root     24462  0.0  0.2  12236   732 ?        S    12:25   0:00 /usr/sbin/syslo
root     25637  0.0  0.3  19256   952 ?        Ss   12:25   0:00 /usr/sbin/xinet
root     25694  0.0  0.5  28380  1552 ?        Ss   12:25   0:00 nginx: master p
root     32427  0.0  0.2   3872   588 ?        S    12:26   0:00 /bin/sh /usr/bi
mysql    32466  0.0  3.0  50420  8124 ?        Sl   12:26   0:00 /usr/sbin/mysql
root     32468  0.0  0.2   3772   592 ?        S    12:26   0:00 logger -p daemo

debian64:~# free
             total       used       free     shared    buffers     cached
Mem:        262144      61008     201136          0          0          0
-/+ buffers/cache:      61008     201136
Swap:            0          0          0

Memory Usage: 59.6MB

Debian 5 32bit

debian32:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1   1980   688 ?        Ss   23:34   0:00 init [2]
root     14064  0.0  0.1   1704   512 ?        S    23:52   0:00 /bin/sh /usr/bi
mysql    14119  0.0  1.7  20056  6968 ?        Sl   23:52   0:00 /usr/sbin/mysql
root     14120  0.0  0.1   1628   536 ?        S    23:52   0:00 logger -p daemo
www-data 15968  0.0  1.2  16352  4996 ?        Ss   23:53   0:00 /usr/bin/php-cg
www-data 15971  0.5  4.7  29436 18576 ?        S    23:53   0:01 /usr/bin/php-cg
root     16040  0.0  0.3   2700  1464 ?        Rs   23:53   0:00 dropbear -i
root     16119  0.0  0.1   1880   684 ?        S    23:44   0:00 /usr/sbin/syslo
root     16133  0.0  0.3   2780  1536 pts/0    Ss   23:54   0:00 -bash
root     17728  0.0  0.2   2352   896 ?        Ss   23:45   0:00 /usr/sbin/xinet
www-data 17758  0.0  0.3   5004  1488 ?        S    23:54   0:00 nginx: worker p
root     27834  0.0  0.3   4548  1236 ?        Ss   23:47   0:00 nginx: master p
root     28146  0.0  0.2   2296   896 pts/0    R+   23:57   0:00 ps aux
102      32065  0.0  0.2   6104   900 ?        Ss   23:49   0:00 /usr/sbin/exim4

debian32:~# free
             total       used       free     shared    buffers     cached
Mem:        262144      44272     217872          0          0          0
-/+ buffers/cache:      44272     217872
Swap:            0          0          0

Memory Usage: 43.2MB

So to put that in summary form

• Ubuntu 10.04 uses significantly more memory than Debian 5 (PHP5.3/MySQL5.1 vs. PHP5.2/MySQL5.0). So pick your distro wisely and maybe you don’t need the latest and greatest if your application does not need it.
• 32bit also uses quite a bit less than 64bit, running exactly the same version of stack. That pretty much confirms what I posted before.

And hopefully the above script is useful for your next WordPress splog farm :P

That’s something I have discussed over 2 years ago, but for all the new readers out there — go for 32bit OS for your next low end VPS. The “memory management” perspect is the same between 32bit and 64bit — it’s the same OS with the same virtual memory manager (well, not quite, as with 64bit there are a lot more tricks for OS to play with). But the main reason for going 32bit is, your applications are going to use far less memory due to a lot of data is only stored at 1/2 the space as 64bit. Modern scripting language VMs are full of pointers (I am talking about PHP, Python VMs etc) and when all those values are double in size, it can really add up.

I will only go 64bit for my low end VPS if

• That’s the only OS image the providers offer. Let’s just call it a conspiracy to get you upgrade earlier.
• 64bit testing. I have one small 64bit VPS around so I can do compilation + testing of some of my open sourced code under 64bit.
• Application requirement. Probably not that relevant if you are just running an web app, but some other apps might require 64bit.

Or your MySQL database that needs more than 4GB for key buffers. But in those case you should not be reading LowEndBox anyway :) These days I found 32bit is usually good enough for big boxes when you use PAE, as the performance penalty is really insignificant with today’s CPU/memory speed.

Personally I code quite a few multi-threaded applications in Python (as some things are just too hard to do asynchronisely). Very often a 6MB RSS application would eat up close to 100MB of VSZ. As one of the most important metrics on OpenVZ — not “guaranteed memory” but “burstable memory”, i.e. privvmpages — is counted on how much your processes have allocated, you’ll soon find your application, which actually only use 6MB of “real memory”, quickly died on your 64MB VPS with 128MB of burstable memory.

The same applies to many other multi-threaded applications that use the default stack size when creating threads. I use pdnsd as a local DNS resolver/cache. A fresh start on Debian would show to use around 1MB RSS but 51MB VSZ!

$ ps -u pdnsd u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
pdnsd    32521  0.0  0.3  51436   976 ?        Sl   23:12   0:00 /usr/sbin/pdnsd

It turns out that for multi-threaded applications, each thread will be allocated 8MB of stack (although not actually used). The allocated stack actually counts against privvmpages in user beancounters, i.e. against your “burstable memory”. That sucks. Which is why I much prefer a Xen VPS which has a proper memory accounting!

So the trick is reducing the stack size so not much will be allocated for each of the threads. In many cases, you don’t need that much stack (applications should malloc instead of assuming a big stack is available). So let’s reduce the stack size (with ulimit -s) and see how much memory you can save.

# ulimit -s 256
# /etc/init.d/pdnsd restart
...
# ps -u pdnsd u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
pdnsd     3261  0.0  0.3   3308   916 ?        Sl   23:24   0:00 /usr/sbin/pdnsd

There we go. From 51MB to 3.3MB. You should see a huge drop under the held column of privvmpages as well under /proc/user_beancounters!

You can pretty much use ulimit -s to limit the stack size of any application but I found it’s only really effectively on multi-threaded applications on OpenVZ. To find out which apps are multi-threaded, you can use `ps auxm` to list not just the processes but also threads. Add the limit before you start the service, or put the ulimit line inside the RC file. You’ll find that you can now squeeze more into your low end box :)

Well. It’s still fun the second time around when I moved hosting, but it wasn’t the same. 3^rd time? Hmm. By now I probably have set up 30+ virtual servers due to expansion or migration, and it feels a lot like work than fun. In the effort to possess the 3 great virtues of a programmer according to Larry Wall, laziness, impatience, and hubris, I decided to write some simple scripts to quickly get my low end virtual servers up running and configured in no time!

Here is one that I want to share today.

• Download lowend-debian.sh

Usage:

1. Buy a new low end VPS from any provider
2. Build a standard Debian 5 minimum install
3. Got the root password
4. ssh into your new low end box, and

   $ ssh root@my-new-box
   Linux lowendbox 2.6.18-blah #1 SMP Wed Aug 26 15:47:17 MSD 2009 i686
   ...
   Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
   permitted by applicable law.
   # wget -q http://www.lowendbox.com/scripts/lowend-debian.sh
   # bash ./lowend-debian.sh
   ...
   

What it does is:

• Remove rsyslogd (memory hog on OpenVZ system) and portmap (mostly useless).
• Run apt-get update && apt-get upgrade to get the server up to date.
• Install and configure dash as /bin/sh.
• Install and configure inetutils-syslog as default system logger.
• Install and configure dropbear as OpenSSH replacement.

Once done, it should bring the memory usage to around 4-5MB on a 32bit Debian (where 2.8MB is from bash and dropbear that handles the SSH connection). It should also work on Ubuntu-based distributions (tested on a 64MB Ubuntu 9.04 VPS). Feel free to check the code to make sure I am not doing anything malicious like hijacking the server and install a ssh key or something :)

“I do not believe you can host 18 websites on 64MB of RAM. I’d bump that up to at least 128 or 256.” –nexbyte

“I really wouldn’t advise anything lower than 265MB RAM for website hosting.” –RikeMedia

(Well, there are some more optimistic comments but I mainly list out those “with things to sell”)

So, just trying to prove the point that yes, 64MB is more than enough to host 18 static sites, I decided to add a Link-1 Xen to my account and document the process. Btw, thanks to Dan @ VPSLink for getting my billing issue resolved :) You can get 10% recursive discount here, or 66% off for the first 3 months here.

Setting Up the VPS

After my order has been provisioned, I re-image the server with a Debian 5 “Lenny” image. I normally pick Debian or Ubuntu because apt-get uses much less memory than RedHat/Fedora’s equivalent, and it’s also my personal preference. I named my new VPS “endor” as I usually just name my boxes after Star Wars systems. Re-imaging a VPS is pretty fast — 2 minutes later I have my root password sent to my email address so I can ssh in to set up the new system.

$ ssh root@endor
root@endor's password:
Linux 66671 2.6.18-53.1.13.el5xen #1 SMP Tue Feb 12 14:04:18 EST 2008 i686

endor:~# free
             total       used       free     shared    buffers     cached
Mem:         65704      64008       1696          0       5616      44100
-/+ buffers/cache:      14292      51412
Swap:       131064          0     131064
endor:~# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     E4500  @ 2.20GHz
stepping        : 13
cpu MHz         : 2194.496
cache size      : 2048 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc up pni monitor ds_cpl est tm2 cx16 xtpr lahf_lm
bogomips        : 5558.81

Plenty of free memory and a single core of C2Duo E4500 — although not a high-end Xeon CPU, but should be more than sufficient to do what we need it to do. The next thing I want to do is to make sure every package is up to date.

endor:~# apt-get update && apt-get upgrade
Get:1 http://debrepo.mirror.vpslink.com lenny Release.gpg [386B]
Get:2 http://debrepo.mirror.vpslink.com lenny Release [63.2kB]
Get:3 http://debrepo.mirror.vpslink.com lenny/main Packages [5295kB]
Get:4 http://security.debian.org lenny/updates Release.gpg [197B]
Get:5 http://security.debian.org lenny/updates Release [40.8kB]
Get:6 http://debrepo.mirror.vpslink.com lenny/contrib Packages [76.1kB]
Ign http://security.debian.org lenny/updates/main Packages/DiffIndex
Get:7 http://security.debian.org lenny/updates/contrib Packages [14B]
Get:8 http://security.debian.org lenny/updates/main Packages [50.6kB]
Fetched 5526kB in 4s (1330kB/s)
Reading package lists... Done
...

Setting Up Web Server

Okay. The 64MB VPS is now up and running. What should we do next? Installing a web server of course, so we can start serving our static pages! Which web server? Definitely not Apache as it would be a waste of valuable memory here. Again my personal favourite is Nginx (pronounces Engine X), which currently powers LowEndBox.com. However, in this exercise I will go for Lighttpd because I found it easier to set up for abitary sites.

First of all — get Lighttpd installed.

endor:~# apt-get install lighttpd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
...
Setting up libterm-readkey-perl (2.30-4) ...
Setting up libterm-readline-perl-perl (1.0302-1) ...
Setting up lighttpd (1.4.19-5) ...
Starting web server: lighttpd.
endor:~# ps -u www-data u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data  1690  0.0  1.5   5416  1008 ?        S    07:17   0:00 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf

Plain vanilla stripped down and un-configured 32 bit Lighttpd sits around 1MB RSS — not bad.

Next, we need to get our websites up there and point Lighttpd to them. It’s a good idea to put the web sites in an organised structure inside the file system. I usually just place them this way:

• /var/www/<hostname>/html

So if I have an HTML file at http://www.example.com/testing.html, it will sit on the file system at /var/www/www.example.com/html/testing.html. Unfortunately I do not have 18 static sites. For testing purpose I am only going to display a very basic HTML page at http://test.lowendbox.com/.

endor:~# mkdir -p /var/www/test.lowendbox.com/html
endor:~# echo '<h1>Low End Box Rocks!</h1>' > /var/www/test.lowendbox.com/html/index.html

So now our “website” is ready — how does Lighttpd, our webserver, knows where to find the files corresponding to the website? That’s where Lighttpd’s mod_simple_vhost comes in handy.

endor:~# lighttpd-enable-mod simple-vhost
Available modules: auth cgi fastcgi proxy rrdtool simple-vhost ssi ssl status userdir
Already enabled modules:
Enabling simple-vhost: ok
Run /etc/init.d/lighttpd force-reload to enable changes
endor:~# /etc/init.d/lighttpd force-reload
Stopping web server: lighttpd.
Starting web server: lighttpd.

Now navigate to test.lowendbox.com (which already has an A record to my new VPS’s IP address) — here we have it! Low End Box Rocks!!!

Prerequisite:
You must be already familiar with DNS and know how to create records to point to IP addresses. For free DNS hosting I recommend EveryDNS, which has also been hosting LowEndBox’s domain.

You can now basically just dump static files at /var/www/<hostname>/html, with <hostname> resolved to your VPS’s IP address, and you will have your static websites over there in no time. You do not even need to tell Lighttpd to reload, as mod_simple_vhost automatically maps the hostname to appropriate file name. Repeat it 18 times and problem solved!

At 1 single testing site with no traffic, Lighttpd sits at around 1.5MB RSS, although I doubt it would increase significantly when you increase the number of sites or the traffic. Lighttpd and Nginx are single-threaded poll-based asynchronised web servers so for static file serving, the bottle-neck would be disk/network IO rather than amount of memory or CPU performance.

There are still lots of memory left. Maybe we can have some fun.

Installing WordPress

So you think, “hey Low End Box rocks and it runs on WordPress. So maybe I will have that installed as well!” Wow. But WordPress is a content management system for creating dynamic websites! It simply cannot be possible on a 64MB VPS, the WHT crowd says! Grrr!! Let’s give it a try.

To run WordPress from your static-file serving Lighttpd, you need a few more packages — namely MySQL and PHP in CGI/FastCGI mode.

endor:~# apt-get install mysql-server php5-cgi php5-mysql
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
...
Creating config file /etc/php5/cgi/php.ini with new version
Setting up php5-mysql (5.2.6.dfsg.1-1+lenny2) ...
Setting up sgml-base (1.26) ...
Setting up xml-core (0.12) ...
Setting up mailx (1:20071201-3) ...

I know it installs whole lot of other junks but don’t worry — we will live with them first and will try to optimise later. It also requires you to set up the root password for MySQL server, and I conveniently chose the most obscured password in this exercise — “root” (yes, don’t use that because I am already using it as my root password :)

We then need to configure lighttpd to handle PHP files.

endor:~# cat > /etc/lighttpd/conf-enabled/10-cgi-php.conf
server.modules += ("mod_cgi")
cgi.assign = (".php" => "/usr/bin/php5-cgi")^D
endor:~# /etc/init.d/lighttpd force-reload
Stopping web server: lighttpd.
Starting web server: lighttpd.

Done! It should be able to serve PHP files. Just to test it out:

endor:~# echo '<?php phpinfo(); ?>' > /var/www/test.lowendbox.com/html/phpinfo.php

Now navigate to http://test.lowendbox.com/phpinfo.php — you should be able to see the output of phpinfo() function. What we are going to do next is to set up a WordPress blog under http://test.lowendbox.com/blog/. WordPress.org already provides a great tutorial on installing WordPress, but let’s do it step by step on the command prompt.

My plan:

• Create database “test_blog”
• Download the latest WordPress and unzip to test.lowendbox.com/blog
• Set up configuration file and run the WordPress install
• Update Lighttpd to provide clean URL, aka Pretty Permalinks.

Let’s go!

endor:~# mysqladmin -uroot -proot create test_blog
endor:~# wget http://wordpress.org/latest.tar.gz
Resolving wordpress.org... 72.233.56.138, 72.233.56.139
Connecting to wordpress.org|72.233.56.138|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: `latest.tar.gz'

...

2009-03-17 13:03:15 (1.01 MB/s) - `latest.tar.gz' saved [1624416]

endor:~# tar zxf latest.tar.gz -C /var/www/test.lowendbox.com/html
endor:~# cd /var/www/test.lowendbox.com/html
endor:/var/www/test.lowendbox.com/html# mv wordpress blog
endor:/var/www/test.lowendbox.com/html# mv blog/wp-config-sample.php blog/wp-config.php
endor:/var/www/test.lowendbox.com/html# vi blog/wp-config.php

When you are editing WordPress’ configuration file, set DB_NAME to “test_blog”, DB_USER and DB_PASSWORD to “root” for something quick, dirty and potentially insecure. Here is one final step — navigate to http://test.lowendbox.com/blog/, and WordPress will guide you through the rest of setup.

It is also relatively easy to set up pretty permalinks for WordPress on Lighttpd. In our example,

endor:~# cat > /etc/lighttpd/conf-enabled/lowendbox.conf
$HTTP["host"] == "test.lowendbox.com" {
    $HTTP["url"] =~ "^/blog/" {
        server.error-handler-404 = "/blog/index.php"
    }
}^D
endor:~# /etc/init.d/lighttpd force-reload
Stopping web server: lighttpd.
Starting web server: lighttpd.

That’s it! You can now go into WordPress to configure the desirable Permalink Structure. Do note that the current WordPress dashboard page is very resource intensive, as it fetches development blog, other WP news, incoming links, etc from various sources, concurrently on separate PHP CGI processes. There might be plugins to turn off this server-killing behavior (or just use older version of WordPress like 2.0.x which is still maintained). Likewise some WP caching plugin can be very useful in reducing the load. Google them and you shall find.

Optimisation — Squeeze More Memory!

So now we have a Debian 5 web server box that can handle lots of static sites + a few WordPress blogs, and it fits “fine” on a 64MB Xen VPS. Let’s see what processes are running:

endor:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root       325  0.0  0.4   2032   292 ?        S<s  04:25   0:00 udevd --daem
root       879  0.0  0.4   2788   304 ?        Ss   Mar17   0:00 /bin/bash --
root      1216  0.0  0.0      0     0 ?        S    Mar17   0:00 [pdflush]
root      1649  0.0  0.2   3144   188 ?        Ss   Mar17   0:00 /usr/sbin/famd
root      6427  0.0  4.4   8024  2928 ?        Ss   Mar17   0:00 sshd: root@pts/
root      6429  0.0  2.3   2804  1564 pts/0    Ss   Mar17   0:00 -bash
root      6441  0.0  1.8  33092  1200 ?        Sl   Mar17   0:00 /usr/sbin/rsysl
root      6453  0.0  1.4   5284   976 ?        Ss   Mar17   0:00 /usr/sbin/sshd
root      6470  0.0  1.3   2048   896 ?        Ss   Mar17   0:00 /usr/sbin/cron
daemon    6482  0.0  0.8   1772   560 ?        Ss   Mar17   0:00 /sbin/portmap
www-data  6510  0.0  2.6   5848  1736 ?        S    Mar17   0:00 /usr/sbin/light
root      6572  0.0  1.7   2488  1156 pts/0    S    Mar17   0:00 /bin/sh /usr/bi
mysql     6611  0.0 26.2 143652 17228 pts/0    Sl   Mar17   0:00 /usr/sbin/mysql
root      6613  0.0  0.8   1636   536 pts/0    S    Mar17   0:00 logger -p daemo
103       6973  0.0  1.3   6112   908 ?        Ss   Mar17   0:00 /usr/sbin/exim4
root      6986  0.0  1.3   2308   908 pts/0    R+   00:01   0:00 ps aux
endor:~# free
             total       used       free     shared    buffers     cached
Mem:         65704      51424      14280          0        936      22588
-/+ buffers/cache:      27900      37804
Swap:       131064        864     130200

Note that it’s an idle box. The swap is slightly used and at 37MB free it is actually not too bad. Let’s try to squeeze a little bit more memory out from the factory setup.

MySQL is by far the biggest offender, and I have talked about how to reduce MySQL memory usage here. If you are just running simple CMS, InnoDB is probably not required — it uses more memory and a lot heavier on IO as well. We can simply use the LxAdmin’s mysql.cnf which I linked on the other blog post to get the bare-minimum MySQL running.

endor:~# cat > /etc/mysql/conf.d/lowendbox.cnf
[mysqld]
key_buffer = 16K
max_allowed_packet = 1M
table_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 64K
skip-bdb
skip-innodb^D

As mysqld_safe script uses /bin/sh for scripting, it’s also a good idea to check whether dash is used instead of bash.

endor:~# apt-get install dash
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
...
Unpacking dash (from .../dash_0.5.4-12_i386.deb) ...
Processing triggers for man-db ...
Setting up dash (0.5.4-12) ...
endor:~# dpkg-reconfigure dash
Adding `diversion of /bin/sh to /bin/sh.distrib by dash'
Adding `diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
endor:~# /etc/init.d/mysql restart
Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld.
Checking for corrupt, not cleanly closed and upgrade needing tables..

One thing I don’t like about Debian 5 is its default inclusion of rsyslog. Well — it’s feature rich, but I don’t need MySQL and TCP syslog support. Weight at 1.2MB RSS is just a bit too fat I reckon. I am not game enough to go without a syslog daemon, so I just go for syslog-ng. Probably not the most light weight, but it’s just something I have been using for the last couple of years.

endor:~# ps -C rsyslogd v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 6441 ?        Sl     0:00      0   207 32936  1260  1.9 /usr/sbin/rsyslogd -c3
endor:~# apt-get install syslog-ng && dpkg --purge rsyslog
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
...
Setting up syslog-ng (2.0.9-4.1) ...
Starting system logging: syslog-ng.
(Reading database ... 16517 files and directories currently installed.)
Removing rsyslog ...
Purging configuration files for rsyslog ...
endor:~# ps -C syslog-ng v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 8215 ?        Ss     0:00      0   105  2754   708  1.0 /usr/sbin/syslog-ng -p

Shedding 500kb RSS — not too bad I guess :)

Next — Portmap and FAM got installed when Lighttpd was first installed. Lighttpd does not really need FAM. It’s used for stat cache to reduce seeks, but can live without. Not that I have noticed any performance difference anyway for small traffic anyway. Having both of them removed from the process list would give us extra 750KB.

endor:~# apt-get remove --purge portmap
eading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
...

OpenSSH can be replaced by dropbear to save memory.

endor:~# touch /etc/ssh/sshd_not_to_be_run
endor:~# apt-get install dropbear
...
endor:~# vi /etc/default/dropbear
endor:~# /etc/init.d/dropbear start
Starting Dropbear SSH server: dropbear.

Just remember to set NO_START=0 in /etc/default/dropbear so dropbear can run as a daemon. Dropbear daemon is using around 500KB less than OpenSSH daemon during idle, and for each connection it uses 1.5MB less on this Debian 5 box — that’s quite a saving!

That’s probably it! Vixie cron can be replaced by a light weight DCRON but I can’t seem to be able find it in Debian’s repository. Exim4 is probably one of the most light weight mail daemon you can have, but then again you might want to question — “do I need a mail daemon running”? You can probably bring it down, and just run /usr/sbin/runq once every 2 hours to process the queue, in case the previous delivery failed. That would probably give you another 1MB to play with.

You can also use PDKSH to replace BASH as interactive shell to loose some weight.

endor:~# ps -C bash v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 8409 pts/1    Ss     0:00      2   663  2140  1568  2.3 -bash
endor:~# apt-get install pdksh
endor:~# chsh -s /bin/pdksh
<log out and then SSH back in>
# ps -C pdksh v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 8550 pts/0    Rs     0:00      0   174  1633   588  0.8 -pdksh

That’s 1 full megabyte off the scale! Also note that VPSLink’s /etc/inittab automatically spawn a BASH process on the console — just in case you got locked out from firewall. For me it’s the last line of inittab file. Change it to /bin/sh or /bin/pdksh, run init q to reload init(1), and then kill that bash process.

Here’s the end result:

# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root       325  0.0  0.4   2032   292 ?        S<s  Mar17   0:00 udevd --daem
root      1216  0.0  0.0      0     0 ?        S    Mar17   0:00 [pdflush]
root      6470  0.0  1.3   2048   896 ?        Ss   Mar17   0:00 /usr/sbin/cron
103       6973  0.0  1.3   6112   912 ?        Ss   Mar17   0:00 /usr/sbin/exim4
root      7953  0.0  0.7   1716   524 ?        S    00:23   0:00 /bin/sh /usr/bi
mysql     7992  0.0  8.2  37904  5404 ?        Sl   00:23   0:00 /usr/sbin/mysql
root      7994  0.0  0.8   1636   536 ?        S    00:23   0:00 logger -p daemo
root      8215  0.0  1.1   2860   776 ?        Ss   00:31   0:00 /usr/sbin/syslo
www-data  8313  0.0  2.4   5712  1640 ?        S    00:37   0:00 /usr/sbin/light
root      8418  0.0  0.7   2052   468 ?        Ss   00:51   0:00 /usr/sbin/dropb
root      8527  0.0  0.7   1712   468 ?        Ss   01:19   0:00 /bin/sh --
root      8549  0.0  1.9   2712  1300 ?        Ss   01:21   0:00 /usr/sbin/dropb
root      8550  0.0  0.9   1808   600 pts/0    Rs   01:21   0:00 -pdksh
root      8562  0.0  1.3   2308   908 pts/0    R+   01:26   0:00 ps aux
# free
             total       used       free     shared    buffers     cached
Mem:         65704      58852       6852          0       2180      40344
-/+ buffers/cache:      16328      49376
Swap:       131064        380     130684

That’s 12MB trimmed, which can be used in disk cache to improve static file serving.

Conclusion

So how do we conclude? 64MB is more than enough to serve a few low traffic static websites. You can actually run a few WordPress sites with a few hundred visitors a day — at the price equivalent to many heavily oversold shared hosting and you get root access!

One thing about root access though — in all my examples above I used root account and never bothered to use a normal user account. It is bad from security aspect so don’t do it. Or at least don’t tell anyone that you use nothing but root :)

However, contrary to conventional wisdom, bigger is not necessarily better. Do you really have a need for 64 bit on your 64MB VPS? Maybe (but not likely), and it will actually be more prudent to stick with the good ol’ 32 bit, because for exactly the same application, it actually uses less memory on 32bit than 64bit!

Currently I’ve got two 64MB Xen VPS — one running 32bit Debian Etch at Virtual-Dedicated.net, and the other running 64bit Ubuntu Hardy at RapidXen. Two have very similar stack — syslogh-ng, xinetd with dropbear, Vixie Cron, Postfix, Lighttpd and PHP. The 32bit Debian actually runs a few more — gld for greylisting and 2x PHP/FastCGI processes (which are at 6MB each). Guess what, I actually have more memory left with my 32bit Debian VPS than my 64bit Ubuntu VPS!

It makes sense that 64bit programs use more memory. Integers and pointers are now double the size. So you are not only allocating more, you also need a bigger stack for bigger pointers. Result? Less free memory for other applications or cache.

You can still make a 64bit 64MB VPS do a lot of tricks, but if the only reason to get 64bit is huge address space + ability to do massive matrix manipulation, then you are going to need a lot more memory anyway.

Meanwhile, for a small web head or a redundant MX, 32 bit ought to be enough :)

The easiest way is to start from an existing configuration file to bring down the memory usage, and then tune it to bring back adequate performance. On Debian/Ubuntu you might wish to replace your installed /etc/mysql/my.cnf with my-small.cnf that can be found in /usr/share/doc/my-server-5.0/examples. I am also attaching a my.cnf here that is taken out from CentOS 4 installation of LxAdmin (which has been famous for its low-memory foot print). It is pretty much based on my-small.cnf. Some important notes:

• skip-bdb and skip-innodb are added, so you don’t get BSD DB nor InnoDB support. BSD DB support in MySQL is pretty much obsolete, and many open source scripts don’t rely on the presence of InnoDB. A low end VPS is not likely to enjoy the concurrency InnoDB is offering anyway. Transaction and referential integrity? Real Programmers^TM write their own rollback routines :)

• key_buffer is only 16K which is far from enough. key_buffer is pretty much one of the most important parameter for MyISAM tables and I usually bump it up to at least 1MB. The same can be said about table_cache — 4 is way too small. A WordPress page will likely touch 10 tables, and much more for apps like Drupal or MediaWiki.

• Query cache might be a good thing if you intend to run a busy site on such a low end VPS (provided that it has small data set, mostly read, like blogs, news sites, etc). I have my query_cache_limit set to 256K and query_cache_size set to 4M here.

After a few adjustment you should be able to start mysqld at around 5-6MB RSS. You might need to check the runtime variables to monitor the performance and how everything works out.

I am taking a similar approach tuning my 64MB VPS from Virtual-Dedicated.net, except:

• PHP 5.2 does not need a parent monitoring process if PHP_FCGI_CHILDREN => 0. Therefore I am running 2x PHP/FastCGI processes with maxprocs => 2 and PHP_FCGI_CHILDREN => 0.
• I haven’t bothered to run any query cache on MySQL. Or maybe I should :)
• SSHD is replaced with Dropbear running off Xinetd, which saves me 1MB idle, and 1MB for every SSH connection.