Micro Cloud
A 3 node cluster for under $1000.
I have a few projects that I wanted to work on that would be easier if I had multiple machines for. One project was getting Eucalyptus up and running, and that required physical machines. I also have space constraints, so picking up some old servers on ebay wasn’t an option. So I needed something relatively new that had hardware virtualization support, enough RAM to run VM’s, and something cheap enough that I wasn’t breaking the budget.
| Part | Quantity | Price | Total Price |
|---|---|---|---|
| Acer Veriton Nettop Computer - Intel Celeron 887 1.50 GHz - Gray, Black | 3 | 222.99 | 668.97 |
| NETGEAR 8 Port Gigabit Smart Switch - Lifetime Warranty (GS108T) | 1 | 79.99 | 79.99 |
| Crucial 8GB (2 x 4GB) 204-Pin DDR3 SO-DIMM DDR3 1600 | 2 | 89.99 | 179.98 |
| Total | 928.94 |
The little nettop’s are great. Small enough for a couple of them to sit on my desk with out looking overwhelming, quiet, low powered, and easy to upgrade the memory and disk (also includes a mSATA slot). I also picked them up on sale for $180 each, which made it even a better deal. The switch is necessary because I would like to do a “Managed” network setup with Eucalyptus and I need VLAN support. It’s is limited to 64 VLAN’s, but that will be plenty for this setup.
- Install the switch, connecting it to my home router.
- Install the RAM, run memtest.
- 8GB in the first node, and split the memory for the other nodes so they have 6GB each.
- Install CentOS, getting rid of that awful Linpus distro.
- Creating an USB netboot was the easiest thing for me.
- Prep the network
- Configure the router.
- Shorten the DHCP Range to give a few IP’s for our new “public” range.
- I’m using the routers MAC->IP assignment in DHCP to set some “static” IP’s for the switch, and nodes.
- Configure the switch
- Quite possibly one of the hardest steps since the web interface is so painful.
- Go to Switching -> VLAN and add VLAN’s 10-13
- Verify that VLAN 1 is set to untagged for all of the ports.
- Go through VLAN 10 through 13 and set them to “T” for tagged on all of the ports. (Yes, the little triangle is overly difficult to toggle to show the ports).
- Configure the router.
- Install Eucalyptus
- For the most part, just follow the directions.
- Put your CLC, CC, SC, and Walrus on the first node with the most ram.
- Make sure you let Eucalyptus know which VLAN’s it can use
euca-modify-property -p cloud.network.global_min_network=10 euca-modify-property -p cloud.network.global_max_network=13
- The IP’s that you took away from your router’s DHCP range. Add them to your config:
VNET_PUBLICIPS="192.168.1.240-192.168.1.250"
- For the most part, just follow the directions.
A couple of issues needed to be dealt with before I was ready to go.
- I do not run a DNS server on my local network.
- I got tripped up using euca2ools until I figured out there were a bunch of subdomains that it was looking for. I simply added hosts to /etc/hosts on my workstation. I’ll need to do this on any workstation I’m using euca2ools from, or get around to setting up a local dns server.
192.168.1.210 node1 eucalyptus.cluster.<domain> walrus.cluster.<domain> euare.cluster.<domain> tokens.cluster.<domain> autoscaling.cluster.<domain> cloudwatch.cluster.<domain> loadbalancing.cluster.<domain> 192.168.1.211 node2 192.168.1.212 node3
- I got tripped up using euca2ools until I figured out there were a bunch of subdomains that it was looking for. I simply added hosts to /etc/hosts on my workstation. I’ll need to do this on any workstation I’m using euca2ools from, or get around to setting up a local dns server.
- I do not run an email server on my local network.
- The cloud controller likes to send you emails to register accounts. You can probably get around this, but I have an email server on my colo, I figured this wouldn’t be to bad to get going.
- For the most part it was pretty simple. I use self-signed certs on my mail server, and it took my a while to figure out I needed to remove the “noplaintext” option from smtp_sasl_security_options, even though it wasn’t being transported over plain text.
- I created a user on my email server.
- I put the credentials in /etc/postfix/relay_auth on node1 (see docs for smtp_sasl_password_maps for syntax)
- I configured the mail server (postfix) to relay to my server, and use sasl_auth.
relayhost = [mail.example.com]:submission smtp_sasl_auth_enable = yes smtp_sasl_password_maps = hash:/etc/postfix/relay_auth smtp_use_tls = yes smtp_tls_security_level = may smtp_sasl_security_options = noanonymous - I believe with newer versions of postfix, you might be able to whitelist the self-signed cert, and can make sure your TLS settings are a little tighter.
- The cloud controller likes to send you emails to register accounts. You can probably get around this, but I have an email server on my colo, I figured this wouldn’t be to bad to get going.
euca2ools was out of date on my distribution.
- I was running into a strange problem trying to create images from my workstation using euca2ools (from eustore). I was getting this error message:
InvalidAMIName.Malformed: AMI names must be between 3 and 128 characters long, and may contain letters, numbers, '(', ')', '.', '-', '/' and '_'
A little digging, and I realized my euca2ools was version 2.x and it was 3.x on all my nodes (where I successfully imported my first image from eustore). A quick upgrade to the latest version, adjusting my path, and I was importing and uploading my images.
- I was running into a strange problem trying to create images from my workstation using euca2ools (from eustore). I was getting this error message:
It booted! Wait.. where did it go?
I successfully booted my first instance, and everything worked! I SSH'ed into my virtualized CentOS 6.4 instance via the “public” IP 192.168.1.240. I verified that it had a 10.101.x.x IP on the backend, and was using a VLAN. So, things are working, let’s get this instance up to date. “yum upgrade” got me to 6.5, I rebooted an suddenly found I could no longer reach the instance. For some reason, I’ve always thought it was going to be the networking that was the hardest part in all of this, and I jumped to the conclusion that it had failed me. The instance was still up on my node after all. Long story short, after spending waaay too much time on iptable LOG insertions, I realized that this was actually something on the instance. I destroyed it, and started again. This time I paid closer attention, and found some SELinux errors after the upgrade (but before the reboot!). I haven’t found exactly what is wrong, but a “restorecon -Rv /” fixed things for now so that it survives a reboot.
Next steps
Right now to launch an instance it’s a simple:
1
| |
I need to learn a little bit more about the security groups to really get going though, as I’m sure I’ll need something other than port 22. Once I figure that out, it’s off to Ansible, to see if I can deploy my RSS app automatically.
Happy clouding!
