this post was submitted on 18 Aug 2024
209 points (97.3% liked)

Linux

48008 readers
1007 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
 

I'm writing a program that wraps around dd to try and warn you if you are doing anything stupid. I have thus been giving the man page a good read. While doing this, I noticed that dd supported all the way up to Quettabytes, a unit orders of magnitude larger than all the data on the entire internet.

This has caused me to wonder what the largest storage operation you guys have done. I've taken a couple images of hard drives that were a single terabyte large, but I was wondering if the sysadmins among you have had to do something with e.g a giant RAID 10 array.

(page 2) 50 comments
sorted by: hot top controversial new old
[–] [email protected] 5 points 2 months ago

4 TB over my home network. 800GB download from a external server.

[–] [email protected] 5 points 2 months ago

Largest one I ever did was around 4.something TB. New off-site backup server at a friends place. Took me 4 months due to data limits and an upload speed that maxed out at 3MB/s.

[–] Presi300 5 points 2 months ago

I've imaged an entire 128GB SSD to my NAS...

[–] [email protected] 4 points 2 months ago (1 children)

Manually transferred about 7TBs to my new Rpi4 powered NAS. It took a couple of days because I was lazy and transferred 15 GBs at a time which slowed down the speed for some reason. It could handle small sub 1 GB files in half a minute otherwise.

[–] [email protected] 4 points 2 months ago

Could the slowdown be down to HDDs that cache on a section of - I think it's single layer? - and slowly rewrite that cache onto the denser (compound layer?) storage?

[–] [email protected] 4 points 2 months ago

Around 15 TB migrating to a new NAS.

[–] ShittyBeatlesFCPres 4 points 2 months ago

You should ping CERN or Fermilab about this. Or maybe the Event Horizon Telescope team but I think they used sneakernet to image the M87 black hole.

Anyway, my answer is probably just a SQL backup like everyone else.

[–] [email protected] 4 points 2 months ago

I did 100TB, 100 streams of 1TB, all simultaneous with rsync

[–] iopq 4 points 2 months ago

I synced to the BSV shitcoin which is 11+ terabytes. So large I had to turn on throwing away the rest of what I downloaded because it wouldn't fit on all of the storage media I own. I feel sorry for the people running an archive node.

[–] [email protected] 4 points 2 months ago (4 children)

When I was in highschool we toured the local EPA office. They had the most data I've ever seen accessible in person. Im going to guess how much.

It was a dome with a robot arm that spun around and grabbed tapes. It was 2000 so I'm guessing 100gb per tape. But my memory on the shape of the tapes isn't good.

Looks like tapes were four inches tall. Let's found up to six inches for housing and easier math. The dome was taller than me. Let's go with 14 shelves.

Let's guess a six foot shelf diameter. So, like 20 feet circumference. Tapes were maybe .8 inches a pop. With space between for robot fingers and stuff, let's guess 240 tapes per shelf.

That comes out to about 300 terabytes. Oh. That isn't that much these days. I mean, it's a lot. But these days you could easily get that in spinning disks. No robot arm seek time. But with modern hardware it'd be 60 petabytes.

I'm not sure how you'd transfer it these days. A truck, presumably. But you'd probably want to transfer a copy rather than disassemble it. That sounds slow too.

load more comments (4 replies)
[–] [email protected] 4 points 2 months ago

Multiple TB when setting up a new server to mirror an existing one. (Did an initial copy with both together in the same room, before moving the clone to a physically separate location. Doing that initial copy would saturate the network connection for a week or more otherwise)

[–] [email protected] 3 points 2 months ago* (last edited 2 months ago)

My cousin once stuffed an ISO through my mail server in '98. His connection up in Bella Bella restricted non-batched comms back then, so he jammed it through the server as email to get on the batched quota.

It took the data and passed it along without error, albeit with some constipation!

[–] [email protected] 3 points 2 months ago

Today I've migrated my data from my old zfs pool to a new bigger one, the rsync of 13.5TiB took roughly 18 hours. It's slow spinning disks storage so that's fine.

The second and third runs of the same rsync took like 5 seconds, blazing fast.

[–] seaQueue 3 points 2 months ago

I routinely do 1-4TB images of SSDs before making major changes to the disk. Run fstrim on all partitions and pipe dd output through zstd before writing to disk and they shrink to actually used size or a bit smaller. Largest ever backup was probably ~20T cloned from one array to another over 40/56GbE, the deltas after that were tiny by comparison.

[–] bulwark 3 points 2 months ago (2 children)

I mean dd claims they can handle a quettabyte but how can we but sure.

[–] [email protected] 3 points 2 months ago

dd can’t really handle quettabytes! GNU has taken us all for fools! Alert the masses! Wake up sheeple!

load more comments (1 replies)
[–] [email protected] 3 points 2 months ago

80GB, it was 8 hours of (supposedly) 4k content in the MP4 format. https://www.youtube.com/watch?v=VF5JWdaJlvc Here's the link (hoping for the piped bot to appear).

[–] slazer2au 3 points 2 months ago

As a single file? Likely 20GB iso.
As a collective job, 3TB of videos between hard drives for Jellyfin.

[–] Nibodhika 3 points 2 months ago (4 children)

Why would dd have a limit on the amount of data it can copy, afaik dd doesn't check not does anything fancy, if it can copy one bit it can copy infinite.

Even if it did any sort of validation, if it can do anything larger than RAM it needs to be able to do it in chunks.

[–] FooBarrington 3 points 2 months ago

No, it can't copy infinite bits, because it has to store the current address somewhere. If they implement unbounded integers for this, they are still limited by your RAM, as that number can't infinitely grow without infinite memory.

load more comments (3 replies)
[–] [email protected] 3 points 2 months ago

~340GB, more than a million small files (~10KB or less each one). It took like one week to move because the files were stored in a hard drive and it was struggling to read that many files.

[–] pete_the_cat 3 points 2 months ago* (last edited 2 months ago)

I'm currently in the process of transferring about 50 TB from one zpool to another (locally), so I can destroy and recreate it.

I've downloaded a few torrents that were around 5 TB each, they're PS4 and Xbox 360 game collections.

[–] [email protected] 3 points 2 months ago* (last edited 2 months ago)

Local file transfer?

I cloned a 1TB+ system a couple of times.

As the Anaconda installer of Fedora Atomic is broken (yes, ironic) I have one system originally meant for tweaking as my "zygote" and just clone, resize, balance and rebase that for new systems.

Remote? 10GB MicroWin 11 LTSC IOT ISO, the least garbage that OS can get.

Also, some leaked stuff 50GB over Bittorrent

load more comments
view more: ‹ prev next ›