Remember when the Internet was supposed to be decentralised for resilience?
Back during the Rant Radio days…
No, sorry, I’m not that old :P
Remember: you’re never too young to have a Vietnam flashback!
Yes. Then these assholes came along…
I remember SLAs including ‘five nines’ ensurances. That meant 99.999% uptime or an allowance of 26 seconds of downtime a month. That would be unheard of nowadays because no cloud provider can ensure that they will have that uptime.
Amazon has so much redundancy built into EC2 that I genuinely thought they’d be able to avoid this.
I may be mistaken, but I really could’ve sworn that a lot of the really strict SLA guarantees Amazon gives assume you are doing things across availability zones and/or regions. Like they’re saying “we guarantee 99.999% of uptime across regions” sort of thing. Take this with a grain of salt, it’s something I only half remember from a long time ago.
The problem comes in so many directions in real life though. Say your company has a very large database. Replicating it across regions means you’re paying for data ingress/egress and more than one region’s copy of that already sharded and/or duplicated database. It even applies when transferring data across AZs in a given region. Backing it up to S3 is expensive, backing it up to Glacier is cheaper, until you ever have to do a restore, and then you have to lay off half the staff to pay for it.
Other issues can arise, possibly through the fault of yourself, sometimes at the fault of Amazon, if data traffic routing has a glitch and data is routing to the wrong place. The onus either way is on your company to show Amazon the receipts if you expect to get credits for the overage. At larger scale, this could be hundreds of thousands of dollars in overage. Easy to torpedo smaller companies with one mistake.
They didn’t used to nickel and dime as hard as they do now, which doesn’t help, but outside of history, they set up AWS to be the biggest slippery slope of wallet-deletion, as almost every move you make costs money. Entire companies exist to manage your AWS costs (for more money, of course) and other companies’ products you may use that are hosted in your infra may accidentally delete your wallet if you don’t constantly monitor them.
Using AWS cost-efficiently is only accomplished by ostensibly day-trading your cloud resources like a high frequency stock trader, capitalizing on unpopular/weird system types, and keeping your code as portable as possible.
…but if one didn’t care about cost, one would probably get pretty good reliability out of them, sure.
Hardware? Yes
Network misconfiguration? Welll…You should plan for your app to be multi regional if it needs to be up that much.
That’s what I’m saying. I took an amazon class last summer and that seemed to be the base.
I blame the customers being cheap or app teams being dumb not Amazon if apps are still down after a few hours of regional downtime.
The cloud is just someone else’s computer. And that computer is busy printing AI videos of the President pooping out of a fighter jet, so now your files are inaccessible
Can you imagine this sentence 1 year ago much less 5 years ago?
“Oh, the deep dream stuff? Yeah, those look so trippy. What do you mean poop though? Usually it’s just dogs.”
The President of course being a convicted felon and rapist, Donald J Trump.
That’s convicted felon, rapist and pedophile, Donald J Trump, to you, mr. Twopi.
One year ago? Easily.
Five years ago? Depends on whether I was visiting 4chan at the moment.
6 months ago, I would be surprised to hear this was done by the president’s administration.
If you properly divide your instances between providers and regions and use load balancing which uses a corum of 3 availability model then it can be zero downtime pretty fairly guaranteed.
People be cheap and easy tho, so 🤷♂️
Yup. And I think I’ll add:
What do you mean we’ve blown our yearly budget in the first month.
Screw the compute budget, the tripled team size without shipping any more features is a bigger problem here.
I’ve seen the opposite. “Oh, you moved your app to the cloud and rebuilt it to be full cicd and self healing? Cool. Your team of 15 is now 3.”
I’m not sure if you are referring to the same thread.
I’m talking about the effort to build multi region and multi cloud applications, which is incredibly difficult to pull off well. And presents seemingly endless challenges.
Not the effort to move to the cloud.
Dividing between providers is not what people would be doing if the resilience of cloud services were as is being memed about.
Doing so is phenomenally expensive.
Doing so is phenomenally expensive.
It’s demonstrably little more expensive than running more instances on the same provider. I only say -little- because there is a marginal administrative overhead.
It’s phenomenally expensive from a practical standpoint, it takes an immense amount of engineering and devops effort to make this work for non trivial production applications.
It’s egregiously expensive from an engineering standpoint. And most definitely more expensive from a cloud bill standpoint as well.
We’re doing this right now with a non trivial production application built for this, and it’s incredibly difficult to do right. It affects EVERYTHING, from the ground up. The level of standardization and governance that goes into just making things stable across many teams takes an entire team to make possible.
In my experience using containers has removed requirements for additional engineering cost to deploy between providers because a container is the same wherever it’s running, and all the providers will offer container hosting, and most offer cluster private networking.
Deployment is simplified using something like octopus which can deploy to many destinations in a blue-green fashion with easy rollback.
Yes, containers make your application logic work.
That’s the lowest hanging fruit on the tree.
Let’s talk about persistence logic, fail forwards, data synchronization, and write queues next.
Let’s also talk about cloud provider network egress costs.
Let’s also talk about specific service dependencies that may not be replicatable across clouds, or even regions.
Oh, also provider specific deployment nuances, I AM differences, networking differences…etc
Containers are nice, but don’t really cover things like firewalls, network configuration, identity management, and a whole host of other things, the configuration of which varies between providers.
The administrative overhead and the overhead of engineering everything to with multiple vendors is what is massive
Only if you engineered your stack using vendor neutral tools, which is not what each cloud provider encourages you to do.
Then the adminstrative overhead of multi-cloud gets phenomenally painful.
This is why OpenTofu exists.
Yeah, Terraform or it’s FOSS fork would be ideal, but many of these infrastructures are setup by devs, using the “immediately in front of them” tools that each cloud presents. Decoupling everything back to neutral is the same nightmare as migrating any stack to any other stack.
Definitely. I go through that same nightmare every time I have to onboard some new acquisition whose devops was the startup cfo’s nephew.
Infrastructure is there to be used by apps/services. It doesn’t matter how it’s created if infrastructure across providers does not provide same API. You can’t use GCP storage SDK to call AWS s3. Even if API would be same, nothing guarantees consistent behavior. Just like JPA provides API but implementations and DBs behavior are inconsistent
You can use the S3 API to interop with basically every major provider. For most core components there are either interop APIs or libraries that translate into provider-native APIs.
It’s 100% doable to build a provider-agnostic stack from the iac all the way up to the application itself.
“But we have our load balacing with 3 different AWS buckets!!!”
Also requires AWS to do the same thing which they sometimes don’t …
