this post was submitted on 09 Jun 2023
6 points (87.5% liked)

Python

3228 readers
1 users here now

News and discussions about the programming language Python

founded 5 years ago
MODERATORS
[email protected]
6
pickling large objects (lemmy.ml)
submitted 1 year ago by [email protected] to c/[email protected]
8 comments fedilink hide all child comments
 

I have a large object that I want to save to the disk because it takes a minute to generate. The OOM reaper kills the process while pickle.dump ing the object.

It's a tuple of dicts of tuple of array.array.

Can pickle dump in chunks? If not, is there another technique I can use?

you are viewing a single comment's thread
view the rest of the comments
[โ€“] [email protected] 3 points 1 year ago (1 children)

If you're pickling that much data you should definitely consider using a more appropriate data format. Maybe a database or HDF5?

[โ€“] [email protected] 3 points 1 year ago

Agreed. When I started, things were much simpler. Trying not to revise too much code but I can if there's no other option.