Toronto AWS Users United: Building Unbreakable Software

We met at The Score‘s office, and Nate Smith talked about ways to structure AWS-hosted applications for availability and consistency.

Here are some points that stood out:

  • Your software will probably outlive the VM it lives in. Be ready to deploy to another server on short notice, or have redundant servers ready

  • Build systems that get stronger when they break, like human muscle. Read Antifragile

  • EC2 can have network outages between nodes. Do not trust the network more than you trust an instance

  • One bad outage example – on April 2 2011, parts of the AWS EBS service were down for 80 hours (despite this, Amazon is still better at sysadminning than you)

  • When thinking about CAP theorem and a network of database servers, assume the network will go down. That means the P (partition tolerance) is chosen as 1 of your 2 options, and you are choosing between C and A. Shopping Cart software usually picks Availability as the other option. How to deal with the resulting inConsistency is a business issue

  • Check out Jepsen, tests of different databases to see they react to network partitions.

  • Oversimplified CAP summary:

    • Got 1 MySQL server? You have CP, because your data is consistent (it’s all in 1 place), but not available (1 server goes down = no availability)
    • Got 1 read/write server and replicas to read from? You have AP because your data is available (some read replicas can down and you’ll be OK), but data isn’t consistent across all servers (due to replication lag, network partitions, or other issues)

The crowd at the Toronto AWS Users United meetup

Building Unbreakable Software on Amazon Web Services

Wednesday, Sep 10, 2014, 6:00 PM

500 King Street West, 4th Floor Toronto, ON

60 Members Went

We’re all set to get together again and chat all things AWS.This time we will have Nate Smith from Shopify share his experience and thoughts on designing unbreakable systems on AWS.You’ve turned off the last server in the rack. Your software is running blissfully on a dozen – maybe hundreds – of EC2 instances. Everything’s humming along and then…

Check out this Meetup →


I see these over and over again in debates on Hacker News. Time to write’m down!

ACID Principle

4 letters describing transactions in a database.

Atomicity (like an atom, each transaction can’t be divided into smaller operations. A failed transaction will not partially write to the database)
Consistency (transactions can not leave data in the DB in an invalid state, data is consistent with the rules. Different than the C in CAP)
Isolation (transactions happen in isolation, even if processed concurrently)
Durability (the result of a completed transaction sticks, no matter what)

These become important when thinking about servers losing power or HDs failing while processing transactions with important data. A server going down while you update your login and password on website should not leave your account partially updated with the new login but old password (Atomicity), with an invalid blank password (Consistency) or with a corrupted values for either field(Durability).

The InnoDB storage engine for MySQL follows the ACID model. MySQL running on MyISAM does not. MongoDB does not follow the ACID model either (See Question on StackExchange for discussion). Also a nice description here

SOLID Object Oriented Design Principle

5 letters describing Bob Martin’s first 5 principles of Object Oriented Programming

Single responsibility principle
Open/closed principle
Liskov Substitution principle
Interface Segregation principle
Dependency inversion principle

I think about S, D daily. What does that make me? Dive deep into this one Wikipedia

DRY Principle

3 letters describing a principle of programming (and many other things in life).


I think about this a lot when I work, and it’s probably the strongest force that drives me to refactor code.

CAP Theorem

3 letters outlining a theory about databases. The theory states that database (or cluster of databases) can have only 2 of the following qualities.

Consistency (all nodes in a network of servers see the same data at the same time. Eventual consistency doesn’t count
Availability (a node in the network will always respond to a request)
Partition Tolerance (the network of nodes will continue to work, even if some of the nodes crash or go down)

When I work with simple database deployments, this stuff doesn’t cross my mind. When I think about scaling up, this stuff matters more. Wikipedia has more info on the CAP Theorem

LRU cache algorithm

3 letters describing which objects inside a full cache are disposed of first.


When an LRU cache (like Memcached) fills up, it will keep objects that were read or written recently, and delete the ones that haven’t been read or written in the longest amount of time (or least recently).


Coding guidelines from John Papa’s Angular Style Guide:

Locating our code is easy
Identify code at a glance
Flat structure as long as we can
Try to stay DRY (Don’t Repeat Yourself) or T-DRY

4K monitor thoughts

I stared into a Samsung U28D590D (who names these products?). It was nice, but not mind blowing. Is it for me? Well, none of my current computers can run at 3840×2160…

4K images look great on it, although I found the experience less shocking than my first time seeing a Retina display. I had to get close the screen or have my glasses on to make out the fine detail the resolution allows for. Text in some applications (like Google Chrome) is too small to read from a comfortable distance with changing the OS DPI. Could I use it? Well, Windows 7 can’t run different DPI’s on multiple monitors, so I would be out of luck there too.

Until I upgrade my hardware, OS, and my eyeballs, 4K is not a good option for me. I can achieve a similar, good-enough-for-text-editing feeling by lowering the DPI or shrinking text on a 27″ 1440p monitor. Some irrational part of my brain still wants a 4K monitor, probably just because it is new and shiny.

Maybe I’ll hold out until 5K.

Buying fancy ergonomic chairs – new vs used

You’ve decided you want that SAYL, Embody, or Zody. Now how much do you want to spend? Lots of people blog about picking up Aerons for $200 during the dot-com bust, but supply and demand for high-end ergonomic chairs has changed dramatically since then.

Why buy new?

  • Warranties! 12 years on Herman Miller, Knoll, Haworth, and 15 years on HumanScale chairs. That’s a lot of peace of mind and sometimes on-site service when fabric tears or the recline mechanism breaks
  • You can avoid meeting with someone from Craigslist in a parking lot with $500 cash on-hand
  • More payment options with retailer or dealer – credit card, get invoiced, financing etc

Why buy used?

  • Saving ~$400 on a high-end chair is ~$400 to put towards a standing desk or sports/exercise equipment which may be a better investment for your health
  • Don’t like the chair? Sell it! You won’t lose much (if any) resale value
  • Sometimes re-using stuff just feels good

Good Deals

If you are buying a Herman Miller chair new from a retailer, keep an eye out for 15% off sales. Usually there is one in November or December (holiday shopping season), and one in April or May.

Look for companies that design and build office interiors for corporations. If they are willing to serve an individual, they may have better prices on chairs than retail stores.

Retailers and showrooms occasionally sell off their floor models with a big discount and sometimes a warranty. Keep an eye out for these!

Bad Deals

The worst deals I’ve seen are used office furniture dealers selling “like new” chairs at a very small discount on Kijiji and Craigslist. They are rarely authorized dealers for the products, so chairs you buy from will not have a warranty. Paying full price for a chair without a warranty stinks!