He does go on doesn't he?

Cloud Application Architectures: Building Applications and Infrastructure in the Cloud by George Reese
My rating: 1 of 5 stars

I picked up this book really looking forward to the premise. Sadly the book was a let down on a number of fronts. Specifically: The focus on Amazon AWS really detracted from general discussion of the cloud. Surely the edit could have rounded out the common refrain of you could do this with S3 but I’m not sure about anyone else. There are a large number of missing diagrams in my print. The author tends to lay out the flaws of traditional infrastructure followed by cloud issues with the same design problem. It makes you feel as though the cloud is only ready to be sold, not used in anger. Repetition was common enough to be noticeable. What is the point of Appendix A (a reference guide to the aws command line tools)? I’m trying to architect solutions not write operations manuals!

In fairness there is some good discussion about the approach to availability and some techniques for applying cloud solutions. However O’Reilly and the author could both do better than this rushed and flawed effort.”

View all my reviews

The Joy of Clojure by Michael Fogus
My rating: 5 of 5 stars

This book is lovely. As a second book about Clojure it propels you from basic understanding through to deep understanding. It doesn’t exhaustively cover the library for Clojure but it covers the thought process behind the language and definitely exhorts a better style of development. The information flow doesn’t let up for all the chapters and as such you feel like you’ve learned an immense amount aas you move through the book. Having said that the conclusion is pretty much, we’ve stopped writing here’s a closing paragraph. Which is odd considering the rest of the book.

View all my reviews

So I’m back to hell-town. It seems when I updated my details, particularly the details of my credit card myki had a sad moment and forgot my new card. Now as you probably know from my last blog post it took a while to get this sorted so I ended up buying a metcard the moment it became clear I was the subject of a “myki doom notice”. I couldn’t get through to the myki line on my first attempt (I assume it was clogged with ecstatic travellers describing their delight at the myki system) so it was’t until maybe a week later that I remembered to get into the myki site. I assumed that by now my top-up of death would have rendered the card unusable so I didn’t bother using it. However to my surprise the myki site did not show the notice telling me my myki was toast. So I updated my details and topped up the card.

Now I know a few things about the whole myki debacle I thought, well I could use some knowledge and try and work out what the optimum top-up limits for my card are so that it works. Lo and behold, the following appeared after half a bottle of Prosecco (horrible) and a bottle of Bison Chocolate Stout (awesome).

My assumptions are the following:

• It takes five work days to sort out your bank
• You want a small amount of money on the card (because really how much money do you want to give these people?)
• Fares come from the myki page as of the 21st May 2011

So here’s the graph for Zone 1 concession:

So here’s the graph for Zone 1 full fare: And also the graph for Zone 1+2 for those out in the sticks :)

So that means along with a 30 day myki pass (erm don’t a number of months have 31 days in myki folk? Wouldn’t that also mean that metcard wins on a 31 day month price-wise?) in order to deal with a lost/stolen bank card you will need to provide myki with the follwoing funds:

Myki	Zone 1 Concession	Zone 1	Zone 1 + 2
No concession myki pass is listed on their fares page… sucks to be old huh?
Money	31.14	52.28	81.40
Pass	111.00	111.00	171.60
Total	142.14	163.28	253

All in all, get spending :(

Further research is required, particularly, when does the myki pass kick in?

Also I’d like to thank the creators of the easy chart builder plugin for wordpress. Much more pleasant than getting a spreadsheet to do the same thing :)

It wasn’t my intention to write this as a rant about virtualisation, more I was going to discuss chef and VMWare orchestration. Sadly the problem is there really isn’t much orchestration between these two things and that’s a terrible shame. Also Mike Bailey has been nagging me about giving Xen a go and when someone does that you start thinking. If I were to sit down and duke it out with Mike what would I say? Which led me to this post. In some ways it’s a call to action by VMWare and Xen, in some ways its just a whinge about virtualisation right now. Whatever, it’s a snapshot of a point in time where devops thinking is frustrated by the tools we have.

We use VMWare at work, we did before I got there. It is loaded on our dev servers and we have some 50+ VMs running on it. Initially I kinda liked it, there’s not a lot of fluffing around with the hypervisor so there’s not a lot you have to do either to set it up or to go wrong once you have it set up. Which isn’t the case with Xen or KVM. For both of those you’re going to have to work harder to get up and running. Also because in both cases you’re going to be maintaining an OS rather than an appliance. Since you’ve got an entire OS there are more moving parts to be maintained once you are going.

Let’s start with VMWare’s ridiculous state of client support for their products. Seriously, how hard would a linux and mac vsphere client be? The short effect of that is that you can’t then write things like chef recipes for VMWare. So every time you’re faffing about inside some Windows interface remembering why it is that you hate Windows administration. As a side note Windows is execrable in large parts because of the work you have to do to make the thing behave, at all, in any administrative capacity. Take configuring a service, or juggling ports, why do I have to go into the registry every five minutes? For VMWare you have to click about all over the tool to get things going. Things trip you up like the vSwitch networking not having enough ports. On top of that run a console in a VM for super laggy performance :( Oh and you know what? Every time I switch on vSphere I love to be told what a Virtual Machine is… That is why I work with your product, I have a memory disorder sufficient that I need to be told what a VM is. Every. Time. I. Open. The. Program. Does that sound ridiculous to you? VMWare makes a mint out of their tools that sit on top of the ESX infrastructure so to an extent it’s not in their interest to provide an awesome set of tools to admin it with. However I’d argue that I want the polished tools less, much less than I want the basic tools so I can innovate off them. I will pay you, have no fear, because at some point soon I will need help to deal with the 100+ VMs, at which point VMWare gets a bungload of money. VMWare not the business models of Oracle, Microsoft and a whole heap of others for instruction with regard to this practice. Trust me, as I go you will grow with me VMWare. As a secondary matter, good tools promote great cross-selling opportunities, ones you may not even have thought of right now. Sadly those seeds will never hatch without a conducive environment fed by great tools.

Now Xen, don’t think you’re getting off scott free here either. Your website is great when I’m buying your top-dollar product and erm… vile when I’m not. You know Red Hat? They have a lovely sister project called Fedora. You know what sticks out when you see that? That Red Hat is invested in this thing, they believe in Fedora and its ability to uplift their business. xen.org says that when Xen is free then Citrix stops the meter and tells the geeks it’s on their dime. There is only one outcome from that, the geeks go elsewhere because they can’t work out how your application works. Also why in the name of all that’s geeky has it taken so painfully long to get Dom0 support into the upstream linux kernel? I’m glad you did but I can’t help wondering, will version 5 take so long? Having said that, your API is lovely. You have an array of awesome projects touching management, security and cloud provisioning. If you could get yourselves in order Xen-lets you could burn out VMWare in its current format with the grace of a swan.

If I’m honest, I’m going to try Xen on my next server. VMWare, get off your collectives and sort yourselves out.

I love using technology, there’s a lovely thrill in getting something to work.  However technology should not be brittle to the facts of life.  Which sadly is where I am right now with couchdb.  I don’t develop on couchdb, I’m a user of couchdb as part of chef.  So as a user I did what most users do, install the technology and get on with the thing I’m really interested in.  Sadly I got bitten by one of those oldest of sysadmin mistakes, ignore the stack and you can guarantee that the bit you know least will take a large chunk out of you.

Now this is not a post about NoSQL databases but it did occur to me that with Oracle and Postgresql (at least, I bet MySQL and others do too) is perfectly possible to recover from this kind of issue.  Which is why postgresql and friends use Write Ahead Logging so that you can get the database consistent again. What’s even more galling is that if there’s one set of folks who should be kicking goals here it’s the Apache Software Foundation.  They make quality software and by and large I’d say their recommendation is a good start in selecting an open source package.

Now let me take you back to the point, what happened and what to do about it.  So the layup is simple, what you do is fail to compact the couchdb database.  Couchdb is based on the assumption is that disk is cheap and so couchdb munches along treating your disk like its free beer in the pub. There is an assumption that you’ll run couchdb’s compaction process, which if you use the chef recipe for couchdb will happen automatically, I found that out too late sadly. If you don’t use compaction? Well instead of someone just having a free drink at that bar it’s more like someone drinking from one of those beer hats that seem so beloved of american college students. The consumption goes right on until eventually couchdb is sick and runs out of disk space. Now when when couchdb runs out of disk is critical. If couchdb runs out out during a write (as our instance did) then you’re going to be using your backups. Oh and as a side note I love Amazon S3, just don’t talk to me about dragging down a 70GB file from S3 here in Australia.

Now I bet you’ve just sshed onto your boxen and feverishly dfed to check you’re not reading this just five minutes too late. Of course if you are reading this after the fact… I feel your pain, I really do. However comiserations get us nowhere so what are we to do? Well, the answer is simple start by working out a couchdb setup that’s resilient. So how do you make a resilient couchdb setup? This is a much tougher question to answer but my initial thoughts are:

1. Get a cron job for compaction going early and keep it going
2. Copy your couchdb files to back them up
3. Test recovering your couch db, you’ll be glad you did
4. Replicate, we didn’t do this and it was a major mistake in hindsight
5. Make sure you’re monitoring your boxen, couchdb isn’t going to let you know when it’s hovered up your disk space

Best of luck out there :)

UPDATED: removed dodgy recommendation of the dump utility that Mikeal rightly points out does not exist.