this post was submitted on 29 May 2024
28 points (93.8% liked)

Selfhosted

40341 readers
888 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
ruud
Loki
CannaVet
devve
HybridSarcasm
[email protected]
28
[SOLVED] Immich keeps restarting the backup (lemm.ee)
submitted 5 months ago* (last edited 5 months ago) by [email protected] to c/selfhosted
25 comments fedilink hide all child comments
 

Hi guys! I'm having my first attempt at Immich (...and docker, since I'm at it). So I have successfully set it up (I think), and connected the phone and it started uploading. I have enabled foreground and background backup, and I have only chosen the camera album from my Pixel/GrapheneOS phone. Thing is, after a while (when the screen turns off for a while, even though the app is unrestricted in Android/GrapheneOS, or whenever changing apps...or whenever it feels like), the backup seems to start again from scratch, uploading again and again the first videos from the album (the latest ones, from a couple of days ago), and going its way until somewhere in December 2023...which is where at some point decides to go back and re-do May 2024. It's been doing this a bunch of times. I've seen mentioned a bunch of times that I should set client_max_body_size on nginx to something large like 5000MB. However in my case it's set to 0, which should read as unrestricted. It doesn't skip large videos of several hundreds megs, it does seem to go through the upload process...but then it keeps redoing them after a while.

Any idea what might be failing? Why does it keep restarting the backup? By the way, I took a screenshot of the backup a couple days ago, and both the backed up asset number and the remainder has kept the same since (total 2658, backup 179, remainder 2479). This is a couple of days now going through what I'd think is the same files over and over?

SOLVED: So it was about adding the client_max_body_size value to my nginx server. I thought I did, so I was ignoring this even though I saw it mentioned multiple times. Mine is set to value 0, not 50000M as suggested on other threads, but I thought it should work. But then again, it was in the wrong section, applying to a different service/container, not Immich. Adding it to Immich too (with 0, in my case, which should set it to "unlimited") worked immediately after restarting nginx service. Thanks everyone for all the follow ups and suggestions!

top 25 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 5 months ago (1 children)

What do the logs say? I'd check

  • proxy log for response code
  • l assume media is storing, if not check it
  • microservice not checking for duplicates due to some error

the more I think there are more unknowns sice there are a few ways it could be running.

[–] [email protected] 2 points 5 months ago (1 children)

Good point about media storing. I seem to have some loop folders issue. My Immich folder has encoded-video (empty), profile (empty), thumbs (with stuff), and immich-files...and immich-files has encoded-video (again, empty), library (with all the stuff), profile (empty), thumbs (empty) and upload (empty). This is my .env:

UPLOAD_LOCATION=/media/MyNAS/Immich/immich_files
THUMB_LOCATION=/media/MyNAS/Immich/thumbs
ENCODED_VIDEO_LOCATION=/media/MyNAS/Immich/encoded-video
PROFILE_LOCATION=/media/NASdata/MyNAS/Immich/profile

And...this is my docker-compose.yml:

    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${THUMB_LOCATION}:/usr/src/app/upload/thumbs
      - ${ENCODED_VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - ${PROFILE_LOCATION}:/usr/src/app/upload/profile
      - /etc/localtime:/etc/localtime:ro

Is there anything obviously wrong that would make Immich place stuff in weird folders? Should I remove all the additional paths from .env, leaving only UPLOAD_LOCATION?

[–] nickhammes 2 points 5 months ago (1 children)

I'd watch those folders, especially the UPLOAD_LOCATION, when it's uploading. Are they being written? Do they persist, or are they being deleted? See if you can upload a single image through the web client, and observe that behavior too.

[–] [email protected] 2 points 5 months ago* (last edited 5 months ago) (2 children)

I mean, my concern is...it would seem in docker-compose.yml that these paths would match:

${THUMB_LOCATION} = ${UPLOAD_LOCATION}/thumbs
${ENCODED_VIDEO_LOCATION} = ${UPLOAD_LOCATION}/encoded-video
${PROFILE_LOCATION} = ${UPLOAD_LOCATION}profile

, which might explain why I'm getting a thumbs folder inside my /media/MyNAS/Immich/immich-files...same with profile and encoded-video. The thumbs clearly is being used (the one inside Immich/immich-files/thumbs), while the root-located one (Immich/thumbs) is not. The root-based folders might be created due to the .env file, but then not used...and maybe it's confusing Immich? Can I remove the entries from the .env, leaving just the UPLOAD_LOCATION one? Or am I making myself a mess this way? Maybe I should make them point equally in the .env to the same sub-paths inside immich-files so they match the structure in the docker-compose.yml? Sorry...kinda new to docker compose.

[–] blotz 3 points 5 months ago (1 children)

I think the issue might be that the config changes haven't been properly committed. Docker container won't just update based on docker compose config.

docker container stop
docker container rm

You might want to delete and reset any settings which have been set

docker volume ls
docker volume rm (IDs from docker volume ls)

(This will also wipe out any backups/accounts made on immich already tho)

But once you have deleted the old containers, running docker compose up -d will start the containers with the new config. You can use docker compose logs -f to see the server logs and check if everything is working.

[–] [email protected] 2 points 5 months ago* (last edited 5 months ago) (1 children)

Thanks for all the help. I changed the paths as I was mentioning a bit on the .env, so they matched the ones on the docker-compose.yml. But no dice. I think it gets stuck at the same picture, although I'm not 100% sure which one. After I rebuilt the container, the number of assets increased by two, but I also realized that I took a couple pics earlier. So it added those two and crashed a while later at the same spot as before...Is a picture/video capable of corrupting the whole backup?? Also, I'm not sure how to properly track which one is messing it, because the backup seems to have skipped a lot of pictures in what it copied.

[–] [email protected] 1 points 5 months ago (2 children)

If it is due to a sigle asset I imagine an error would log to the console.

[–] [email protected] 1 points 5 months ago* (last edited 5 months ago) (1 children)

O.M.G....so many hours wasted. One of my first searches already returned "you should increase client_max_body_size to something like 50000M", and I was like I aLreAdy iNCreAseD mY client_max_body_size tO zERo sO its uNLimIteD DuH <spongebob.jpg> Well turns out my client_max_body_size 0 parameter was in a section defining parameters for a different container/server. So of course it wasn't applying to Immich. Just added the same line to Immich section too, restarted nginx...and the backed up asset count is already wayy ahead of the ceiling it would always hit at 180ish assets. I think I might have found my issue.

Thanks for all the help and following up!

[–] [email protected] 1 points 5 months ago

Glad you found the issue! I fell asleep hard last night sorry I couldn't be your rubber duck haha

[–] [email protected] 1 points 5 months ago* (last edited 5 months ago)

You mean to docker logs immich_server ? I think this would be about the only error kinda it's outputting once in a bluemoon.

[Nest] 6 - 05/29/2024, 2:40:17 PM WARN [ImmichServer] [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses

Everything else is Websocket Connect, Websocket Disconnect.

[–] [email protected] 1 points 5 months ago (1 children)

Were you able to fix it? Mounting like that should work but it looks like docker isn't mounting subpaths right.

[–] [email protected] 1 points 5 months ago (1 children)

Yeah, also my thought. Seems it's still not working. I've seen it repeat the uploads multiple times, and still have quite a limited amount of pictures on the server.

[–] [email protected] 1 points 5 months ago (1 children)

My advice is start over at least temporarily. Use immich base compose with one mount for the uploads and test it before deviating from the basic setup.

[–] [email protected] 1 points 5 months ago

You mean, without defining additional paths for thumbs, profile etc? Will it work without declaring them?

Thanks!

[–] [email protected] 3 points 5 months ago (1 children)

Is the container running out of disk space for its DB? Is the container running out of memory during the backup process and crashing?

[–] [email protected] 2 points 5 months ago* (last edited 5 months ago)

The docker container where the DB lives has still another 2GB free to go (I'll increase this). The pictures storage goes to a NAS drive with nearly 10TB of storage left. The docker container has 4GB RAM to run, only immich is running on this at this moment...I didn't seen running too high just yet.

[–] [email protected] 2 points 5 months ago* (last edited 5 months ago) (2 children)

It happened to me as well. To resolve it, go to Administration > Jobs and apply them all available options. I had multiple issues, including this one, but resetting it from the Jobs page fixed it for me.

[–] [email protected] 2 points 5 months ago* (last edited 5 months ago) (1 children)

Thanks! I never walked through that screen yet...I'll try that!

EDIT: Seems...not to have done much. Backup keeps tracking and uploading a ton of files. Many more than what the server actually displays, as in, I've seen the backup go through pictures with dates the server doesn't seem to show. I've seen it slowly go through at least 15-20 videos for example, but the server only acknowledges a grand total of 2 video assets.

EDIT2: Oops, seems the screen eventually turned off and backup stalled...Let's see what it does now... ....aaaand back to May 25th. Yup, still the same problem here :(

[–] [email protected] 2 points 5 months ago

Another thing I was doing was to keep spamming the cancel button.

[–] [email protected] 1 points 5 months ago* (last edited 5 months ago)

I...Think I found the issue. A classic case of "increase max upload size on your reverse proxy". Which I thought I did... https://lemm.ee/comment/12234368

Thanks for all the help through the process!

[–] blotz 2 points 5 months ago (1 children)

What method are you using to mount the nas for immich. I ran into issues trying to mount my Nas using docker volume+sshfs but ran into zero problems using docker volume/cifs. With sshfs, immich would run until it suddenly stopped working and backups would constantly fail. Restarting the server would cause it to run for a little time longer.

[–] [email protected] 2 points 5 months ago (1 children)

I mean, the NAS is already mounted. It's an NFS share, it gets mounted at boot, and it should be just another regular folder, transparent to docker (or anything else).

[–] blotz 2 points 5 months ago

Ah yeah that should be good. I've never had issues with NFS

[–] [email protected] 1 points 5 months ago

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
HTTP Hypertext Transfer Protocol, the Web
NAS Network-Attached Storage
NFS Network File System, a Unix-based file-sharing protocol known for performance and efficiency
nginx Popular HTTP server

[Thread #772 for this sub, first seen 30th May 2024, 07:15] [FAQ] [Full list] [Contact] [Source code]

[–] [email protected] -2 points 5 months ago

New Lemmy Post: Immich keeps restarting the backup (https://lemmyverse.link/lemmy.world/post/15927332)
Tagging: #SelfHosted

(Replying in the OP of this thread (NOT THIS BOT!) will appear as a comment in the lemmy discussion.)

I am a FOSS bot. Check my README: https://github.com/db0/lemmy-tagginator/blob/main/README.md