January 6, 2021

The Most Under-Appreciated AWS Service

I knew this was gonna be a tough ask: to identify the most under-appreciated AWS, and to expound why that is. Tough for a few reasons I think, mostly because we can’t even agree on how many services AWS has. Officially AWS sticks to their usual “over 175 services”, probably because that would mean admitting to the exact number of services with horrible names. Cloud Pegboard has it pegged (see what I did there?) at 283 services, 219 total IAM prefixes, 191 product listing counts, 182 namespaces and 159 URLs. Pick whichever you want, thats a lot. But those are the just services you can touch and feel, and not the underlying stuff. E.g. all those services are the stuff that companies and developers want (read need), like the multiple types of compute or 15 choices of databases services, and even things that developers say they need (but not really) like kubernetes.

But like I said, those all the services that are listed in the catalogue, but not the hidden, below-the-line muck that supports all those hundreds of services. And thats where I think the under-appreciated ones are. But to truly appreciate it, we need to step back and see how large corporates (which I enjoy picking on, like the fat kid in the park) have been building products and the underlying infrastructure until now prior to the cloud (and the head-in-the-sand denialists - I’m looking most you at you telcos). I could easily summarise any project I worked on a until a few years ago, which typically went like this: Business launches a project to build some fancy product, writes a bunch of requirements docs for a few months that nobody reads (or read the wrong version, oops!), which leads to a specification document from IT that asks for budget to purchase the required infrastructure. That Bill of Materials (BOM) usually includes racks, network switches, cabling, servers, load-balancers, storage, cards, software licences and a bunch of consultants that will take months to procure, install and setup. Somewhere near the end (after the project is already a month late because they needed to re-install due to the driver not being compatible with the OS), the consultants declare that the application is installed and working, ready for UAT. When you look at the connection details for the URL, you realise its something like http://10.1.34.58 — WTF! Which usually means the project manager forget to include the request to the network team for a public IP and DNS name, which in-turn needed the security team to first approve the firewall request to open the ports to the Internet. And thats the point of my long-winded rambling: on-prem networking is hard!. It's hard because many different teams besides the developers (and consultants who are quoting for more because the scope has increased), like the networking and security teams, have to be involved to make that application available on the network. And that’s just the tip of the iceberg: I recall multiple times where it took weeks just to request new IP address space, then routing those IPs out on the MPLS network, then trying to convince the security team to open a bunch of ports so the front-end can speak to the back-end. Or where return-path traffic is routed is completely different, causing latency and dropped packets. And thats even before you wanted to install kubernetes on-prem!

Yet in the cloud with AWS, you can spin up instances that come with public IPs and DNS names that are immediately routable on on the Internet. Or making your RDS DB instance open on the internet with a single config (don't do it, even though you can) You don't need to even think about it - it just works. And thats why I think networking is the most under-appreciated AWS service. Sure, networking is a broad term, and AWS does have networking services but they don't typically look like the ones that you would bought on-prem. With AWS, you don't get to choose the top-of-rack and end-of-row switches (ha, I knew that CCNA 10 years ago would come in handy), and decide which SFP you need (and inevitably choose the wrong one). The networking team dont get to log onto the router and break MPLS. And talking of security, where is the all-mighty firewall - the most important device stopping you from getting hacked? Until 2 months ago (November 2020 when they released AWS Network Firewall) AWS did not offer a service with with words “Firewall” in the name. For that, you use security groups as a virtual firewall, and thats why old-school security companies were banging on about the lack of a dedicated firewall service on AWS. So how does AWS do all of this networking magic: it uses custom-made routers with software-defined networking:

We run our own custom-made routers, made to our specifications, and we have our own protocol-development team,” Hamilton said. “It was cost that caused us to head down our own path, and though there’s a big cost (improvement) . . . the biggest gain is in reliability.” This custom-made gear “has one requirement, from us, and we show judgment and keep it simple. As fun as it would be to have a lot of tricky features, we just don’t do it, because we want it to be reliable.

and thats what allows you to create VPCs, subnets, security groups, attach elastic IPs, and still log all traffic - literally with just a few commands or clicks. Theres no way this could be done on-prem with a lot of effort.

And thats the muck that AWS builds, so you don't have to. And until now, you didn't even know it.