this post was submitted on 23 Feb 2024
442 points (98.5% liked)

Technology

59740 readers
3958 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 2 years ago
MODERATORS
top 50 comments
sorted by: hot top controversial new old
[–] OhmsLawn 86 points 9 months ago (5 children)

Hanlon's razor in action

"Never attribute to malice that which is adequately explained by stupidity"

[–] [email protected] 59 points 9 months ago

If you cut costs enough, including in personnel and testing, stupidity is caused by malice. Not saying that's exactly what happened here but we've all definitely seen the effects when corpos decide to lay off thousands and their capabilities suffer.

[–] Godthrilla 12 points 9 months ago

I think it was the same intern that accidentally told Hawaii it was about to be hit by an icbm

[–] [email protected] 8 points 9 months ago

While I like that saying in the context of not knowing whether it is malice or stupidity, it does need the context of 'Unless you know there is malice..."

Malicious people do stupid things too.

[–] billwashere 3 points 9 months ago

I always included obliviousness in this as well. But one could argue that’s just another form of stupidity… 🤷

[–] pdxfed 2 points 9 months ago

With At&T it's a big fucking razor

[–] AreaKode 84 points 9 months ago

Looks like I'm not the only one who tests in production...

[–] [email protected] 47 points 9 months ago (5 children)

the nationwide public safety network that was built by AT&T. Some FirstNet users reported frustrations related to the outage.

Eminent domain that fucking shit and seize the infrastructure. A public utility should not be able to be taken out by a private company's fuckups.

[–] nucleative 11 points 9 months ago (6 children)

Do you think a public IT staffer would be immune to fuckups?

[–] JDPoZ 22 points 9 months ago (1 children)

You frame the issue incorrectly.

You see - it’s not some poor IT guy fucking things up (I mean ultimately the IT guy is the one who probably pressed the button, but no IT department acts independently from the system it exists within).

It’s AT&T not having the adequate amount of funding set aside to cover for redundancies + probably adequate staffing.

See… AT&T wants to make the biggest fucking profit margin possible… everything else be damned.

Say what you will about the ineptitude of government, but given funding, the government doesn’t have an incentive to make things shittier specifically just to get some sort of larger profit margin.

Yeah the DMV sucks, but Medicare works well… mostly because Republicans slice and diced budgets as much as they can get away with everywhere they can… and it’s much harder for them to sneak cuts to Medicare - which would clearly and directly affect senior citizens, who would then be less likely to vote for Republicans again no matter what culture war bullshit they spew from the billionaire owned cable TV they stay glued to.

[–] [email protected] 3 points 9 months ago* (last edited 9 months ago)

They framed it incorrectly but they're still right, mistakes happen, and no matter what you plan for really bad things can happen.

This wasn't a catastrophe it was some downtown (and it wasn't even all their customers in all their service areas -- my uncle had this problem his wife did not, they're both on AT&T in literally the same house). It's happened with Google, it's happened with Amazon AWS, it's happened with various other major players. Nobody and no department is immune to them, making AT&T a nationalized company is very unlikely to have helped here.

In fact, because we're so bad at raising taxes to fund our federal agencies and things ... it might actually be worse in terms of reliability.

load more comments (5 replies)
load more comments (4 replies)
[–] CaptainSpaceman 39 points 9 months ago (1 children)
[–] [email protected] 5 points 9 months ago (1 children)

Yeah I definitely saw the solar flares article.

This living person trusts the living flares we're the cause, as they never rejected my coupons.

[–] humorlessrepost 3 points 9 months ago

By posting as CraigeryTheKid you’ve accepted responsibility for the actions of CRAGERYTHEKID, and now you’re fucked. Rookie mistake.

[–] NegativeLookBehind 23 points 9 months ago

I’ve worked with ATT before. Their general incompetence is breathtaking.

[–] mlg 20 points 9 months ago

I saw this happen live because my fiber endpoint (not router because I have my own ONT) went offline for exactly 3 minutes at 4am EST so I realized they were pushing updates lol.

Fiber and internet network went fine but I guess cellular kicked the bucket.

[–] [email protected] 19 points 9 months ago (1 children)

Just remember junior devs, there's always a worse junior dev than you

[–] agent_flounder 5 points 9 months ago (1 children)

Except for the guy that did this.

[–] ArtVandelay 5 points 9 months ago (4 children)

To be fair, whoever did this is probably not with their title was yesterday any longer

[–] [email protected] 3 points 9 months ago

It depends on how the place runs... If it runs well and this was an honest mistake in the established process ... there's probably a team that's getting a very stern talking to but it's unlikely heads actually roll.

It's more important to fix the issue for next time than fire somebody.

load more comments (3 replies)
[–] [email protected] 18 points 9 months ago

I thought it was solar winds

/s

[–] overzeetop 15 points 9 months ago (7 children)

Not to defend them or minimize the corporate stupidity, but it sounded like there were less than 100k people affected out of tens of millions (100m?) accounts. I get that it was a big deal for those affected, but a 0.1% outage doesn’t seem “major”.

[–] dirtySourdough 20 points 9 months ago (22 children)

I think the reported numbers are coming from downdetector.com, which relies on self reporting and people being aware that the website exists. I imagine many more customers were affected. Also, anything the prevents emergency services communication, which occurred during this outage, should be considered a major outage imo

load more comments (22 replies)
[–] EncryptKeeper 17 points 9 months ago (1 children)

I do know that FirstNet was impacted. The tablets in our fire apparatus couldn’t connect which is kind of a pain in the neck because we use that to navigate, locate fire hydrants and view their flow capabilities and whether they’re out of service, store maintenance phone numbers, view building blueprints and material safety data sheets, view responding apparatus and locations, identify helicopter landing sites, etc.

Like the job will still get done but it does throw a wrench in our ability to coordinate larger responses.

load more comments (1 replies)
[–] cultsuperstar 9 points 9 months ago

I was impacted and it sucked. Having no cell service for 8-9 hours is not fun. Can't make or receive calls or texts, every app that requires or uses an internet connection (like Waze) was impacted. Whole Waze worked with directions using offline maps and GPS, you don't fet stuff like traffic conditions and rerouting.

But when you only have a cell phone and limited wifi resources at the office, it's a major pain in the butt. And I didn't report so that 70k could've been a conservative number of people that reported.

[–] [email protected] 8 points 9 months ago* (last edited 9 months ago)

Our company phones were affected (both cellular and our ability to phone out or take external calls on our traditional phones). For us, the outage started at 4am (edit: this is just when my small department noticed, we're not IT), could be that not everyone noticed.

[–] [email protected] 6 points 9 months ago (1 children)

Yeah, but this is lemmy so.. the outrage is very real. Even for most of the people complaining about it that don’t even have AT&T.

load more comments (1 replies)
[–] surewhynotlem 4 points 9 months ago

Some of the affected users were other systems, like Duo, which then caused downstream outages of even more thousands. That's why it's being reported that way.

[–] [email protected] 3 points 9 months ago

I was affected. Haven't bothered to report since I wasn't seriously bothered. Might be different if I'd lost business or couldn't contact family

[–] jordanlund 8 points 9 months ago

tl;dr - "Somebody's getting fired!"

[–] ArtVandelay 8 points 9 months ago (2 children)

If it's that big of an upgrade, and your primary customers are North American based, why the fuck do you decide first thing in the morning of a weekday is the time to roll that out? Grab a fresh pot of coffee and start that shit at 10:00 p.m.

[–] HeyJoe 3 points 9 months ago

Normally, the odd times are because someone can't work the weekend, which is crazy to me since I am someone who accepts all the worst go live times...

Although the more I think about it, would there really have been a fantastic day or time? Even weekends suck because it affects everything so even then it would have sucked plus if it did bring down companies those people are now doing work on the weekends as well. Idk if I disagree with a late time smack in the middle of the week date for this the more I think about it.

load more comments (1 replies)
[–] [email protected] 4 points 9 months ago (1 children)

This is the best summary I could come up with:


AT&T said a botched update related to a network expansion caused the wireless outage that disrupted service for many mobile customers yesterday.

"Based on our initial review, we believe that today's outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack," AT&T said on its website last night.

While "incorrect process" is a bit vague, an ABC News report that cited anonymous sources said it was a software update that went wrong.

The outage was big enough that the Federal Communications Commission said its Public Safety and Homeland Security Bureau was actively investigating.

The San Francisco Fire Department said it was monitoring the outage because it appeared to be preventing "AT&T wireless customers from making and receiving any phone calls (including to 911)."

The US Cybersecurity and Infrastructure Security Agency reportedly said it was looking into the outage, and a White House spokesperson said the FBI was checking on it, too.


The original article contains 323 words, the summary contains 164 words. Saved 49%. I'm a bot and I'm open source!

[–] db2 14 points 9 months ago* (last edited 9 months ago) (1 children)

The outage was big enough that the Federal Communications Commission said its Public Safety and Homeland Security Bureau was actively investigating.

No single company should be big enough to cause that kind of problem.

[–] Donjuanme 4 points 9 months ago

No single publicly traded company*. Anything that interferes with government services had better have competition for the availability of contracts to service public good/government entities

load more comments
view more: next ›