That too is also one of many arguments against OpenVZ, especially when it is compared to Xen. “OpenVZ got oversold resources. Xen VPS has dedicated memory, blah blah blah.” Well. Not entirely true.

2 weeks ago I got a cheap VPS from a provider in Asheville NC. Someone who should probably remain anonymous here :) It is a Xen VPS running Ubuntu 10.04 and 64bit Linux 2.6.32 kernel. 512MB of memory, 25GB of disk space and more bandwidth than I ever need. Pretty good price too. So I logged in and check how much memory it got…

# free
             total       used       free     shared    buffers     cached
Mem:        543776     535360       8416          0      66516     130504
-/+ buffers/cache:     338340     205436
Swap:      1048568        136    1048432

Hmm. 330MB of memory already used for my brand new Xen VPS. Let’s see what processes are running.

# ps aux
...

root       204  0.0  0.1  17028   780 ?        S    Oct27   0:00 upstart-udev-bridge --daemon
102        356  0.0  0.1  23548  1080 ?        Ss   Oct27   0:00 dbus-daemon --system --fork
root       431  0.0  0.1  21068   788 ?        Ss   Oct27   0:00 cron
root      3110  0.0  0.5 253832  2992 ?        Sl   Oct28   0:00 /usr/sbin/console-kit-daemon --no-daemon
root     11037  0.0  0.1  49256  1012 ?        Ss   Oct28   0:00 /usr/sbin/sshd
root     15427  0.0  0.1  12520   772 ?        S    Oct28   0:00 /usr/sbin/syslogd --no-forward
root     31231  0.0  0.0  16748   436 ?        S<s  Nov12   0:00 udevd --daemon
root      5716  0.0  0.6  79100  3772 ?        Ss   01:02   0:00 sshd: root@pts/0
root      5731  0.0  0.3  19400  2148 pts/0    Ss   01:02   0:00 -bash
root      5782  0.0  0.1   6072   724 ?        Ss   01:08   0:00 /sbin/getty -8 38400 hvc0
root      5783  0.0  0.2  15248  1172 pts/0    R+   01:08   0:00 ps --sort=start_time uax

Yes. That’s it — there is nothing memory intensive running on the box. SSH server, syslogd, cron and that’s about it. So where did my 330MB of used memory disappear into?

Before digging any further, here is the fact. You can most definitely oversell memory on Xen VPS. Something that is well known for years, although it is not something Xen providers want to talk about. It uses a technique called ballooning.

Basically a special Linux kernel driver is installed on your system — the “balloon driver”. When dom0 (the Xen server/hypervisor) needs more memory, and wishes to claim some from the guest VPS (domU), it asks the guest VPS’s balloon driver to inflate itself — by asking its Linux kernel for some memory. Kernel memory allocation will be requested from the available memory for that VPS, and cannot be paged out to swap. Once the balloon driver consumes the memory, it then passes to dom0/hypervisor to be used elsewhere (creating a new VPS for example). So the amount of your VPS’s “total memory” will stay the same, but there will be a big increase in “used memory”, as a big chunk has now been used by the balloon driver inside the kernel, and possibly now a part of another VPS. A user-land daemon “xenballoond” (a bash script actually) is also available to allow dynamic ballooning, although I did not see that in my VPS.

I guess it might explain why I have 330MB of memory used, while I only have a very small number of processes running.

I found this file in procfs interesting:

# cat /proc/xen/balloon
Current allocation:   524288 kB
Requested target:     524288 kB
Minimum target:       173056 kB
Maximum target:       532480 kB
Low-mem balloon:        8192 kB
High-mem balloon:          0 kB
Driver pages:            224 kB

I can’t seem to be able to find sufficient document on explaining what those value mean. It looks like my VPS has requested 512MB of memory (Request target), and has currently allocated so (Current allocation). However at the same time it also has “Minimum target” — the minimum amount of memory reserved for this VPS to prevent it from FUBAR when the balloon requested too much — set to 169MB. Would that mean that is how much memory that is really guaranteed to my VPS?

It’s certainly something that I am not familiar with and maybe some providers can enlighten us. However the conclusion remains the same. Overselling on Xen is certainly possible.

And this is the VPS hosting industry we are talking about here!

Here is one article that discussed guaranteed vs. burstable memory in OpenVZ, and why you should not care about the so-called “guaranteed memory”. Basically the real guaranteed parameter, oomguarpages, is only used when the host server hits an OOM (out of memory). VPS that exceeds too much oomguarpages will have its processes killed. However how often does a Linux box hit an OOM? When both physical memory and swap are exhausted.

Not sure whether you have tried to admin a Linux box with all its swap space exhausted (usually set a 2x the physical memory). It is painfully slow. And if it is used as a physical node for OpenVZ VE’s, all the customers would have fleed the server and posting “over-sold! over-loaded!” comments all over the place. Basically it rarely happens, and that implies burstable memory should be all you need to care about.

That would certainly wreck the way OpenVZ VPS providers plan their packages — they probably shouldn’t be using guaranteed memory to calculate how many VPS to fit onto a box. That also means for providers that give significantly more burstable memory might have their servers more oversold, if they did the planning base on guaranteed memory. For example AlienVPS (512MB/2048MB), Virpus (512MB/2048MB), Vixile (256MB/2048MB), Newwebsite (512MB/2048MB) etc.

For example you can allocate almost 2GB of memory on Virpus constantly, and no ill-effect will happen to your VPS, unless the actual server runs out both physical memory and swap (unlikely).

Anyway. Maybe it’s not as bad as I think if the provider can constantly monitor their own servers to ensure no overloading is happening. Virpus is a larger provider so I picked on them (and I think they are in better position of handling these issues). But what about all these one/two men teams?

Or use Xen. Pricy, but you get what you have paid for, and memory accounting is much easier to understand.

What’s your best low end box experience in 2009?

There has been heaps of low end providers profiled over the last 12 months — some has failed, some has raised their price so can no longer stay low end, some are still struggling but some just shine amongst the others. I have used/tested out plenty of low end virtual servers this year, and currently there are still 5 under my inventory doing various tasks (mostly boring idle jobs). And the one that I am mostly happy with is… Surprise surprise…

2HOST.com!

They were first featured here in May, and then I signed up in June to demonstrate how to run Debian 4 Etch on it. I have sinced rebuilt that 512MB Xen 8-core Xeon “beast” with Debian 5 Lenny to run one of my app that takes periodical screenshots of websites using VNC and Firefox. I was a bit worried when they cancelled my PayPal subscription, but it turns out to be a non-event as I have continued to pay manually month after month.

Performance has been rock solid as I would expect from any good Xen host, but it could be just because of lack of work on that $5 VPS. Great stability as well.

$ uptime
 10:20:10 up 173 days,  7:01,  1 user,  load average: 0.02, 0.01, 0.00

Support was a bit slow at the beginning but I never needed them afterwards. I could be lucky — but it’s definitely my best experience from a low end virtual server this year. Unfortunately I think the deal is no longer available.

What about yours?

Adjective

1. evasive and shifty
2. unsound and unreliable
3. dishonest
4. risky
5. deviant
6. weird

Sounds like a good descriptive word for many past low end providers featured on this blog. I have seen many hosts open their business and advertise their cheap low end packages on WebHostingTalk, only to close their door 2 months afterwards. In most cases they’ll just do a runner and leave their customers panicking trying to recover files. I have also trying to do some background check on the providers and make caveat emptor warnings if I suspect the provider is a little bit dodgy.

I have to say that a lot of investigation is based on “gut feelings”. Some just smells plain dodgy, but with some you need to do quite a bit of digging to reveal their shady past. After all you do not want to sign up a low end VPS package (and possibly pay a whole year of fee in full) and later on loss everything on it. What sort of investigation do you use to determine whether a provider is dodgy?

Some checks I use:

• Running both DNS on the same box. Only incompetent sysadmins do that.
• Domain WHOIS privacy turned on. Why, as a registered business, don’t you want your customers/prospects to know who you are? I don’t buy the “prevent email address harvested” excuse either, as a competent hosting provider would know how to do spam filtering.
• Lack of proper SSL certificate on their WHMCS and control panel pages. That usually spells “rush job” to me. SSL certificates can be had for $15/year & gives your customers a lot more “trust”.
• Other posts on WHT — either those about this provider, or early posts made by the owner of the provider.

Any other sanity checks you use to evaluate a low end provider?

1. Provider’s website’s IP address has changed, or
2. Provider’s website becomes uncontactable, or
3. Provider’s website cannot be resolved

Surprisingly, #3 where the provider’s website cannot be resolved from DNS, actually happens pretty frequently! I mean, it’s not that webserver is down, but both DNS for the domain are off-line so not even the IP address can be resolved.

For example,

• AMHNetwork.com.au not resolvable (9 Nov)
• CubicHost.org not resolvable (8 Nov, and is still offline)
• 2Host.com not resolvable (7 Nov)
• NetSpaceLtd.com not resolvable (7 Nov)
• ServerDeals.org not resolvable (6 Nov)
• …

Everyday there are providers down for a brief period of time. WTF? I am talking about hosting providers’ domains, i.e. the ones that provide you servers, network and all that. What are they thinking by putting both of their name servers on the same box, when they are supposed to provide redundancy?! When your webserver is down but A/MX records are still resolvable, then I guess there are still hope. When even ns(1|2).yourdomain.com are unresolvable — it’s as good as dead.

Well. Next time when I see a provider putting both NS on the same box I’ll voice it, as it shows their incompetency.

Just read a link on WHT to this article on India Times — Tecie hangs himself in HSR Layout:

On Monday morning, software company owner K T Ligesh, 32, was found hanging in his house.

He was also deeply upset his company, Lx Labs, in HSR Layout, 6th Sector, had recently lost a project to another company…

Any loss of life is sad. This guy built HyperVM — the virtualization control panel for both OpenVZ and Xen — and most importantly many low end box providers, i.e. those that have been listed under this blog, are using his software (as it costs them only 50 cents/VPS I heard, comparing to many times more Parallels charge for Virtuozzo). That basically puts HyperVM in limbo — which means there will actually have big impact on budget VPS providers. Who will take control of the source code? Or is there an alternative?

My condolences to his family.