this post was submitted on 15 Apr 2024
12 points (83.3% liked)

Selfhosted

40152 readers
545 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
ruud
Loki
CannaVet
devve
HybridSarcasm
[email protected]
12
Running DNS server in Docker (jlai.lu)
submitted 7 months ago by [email protected] to c/selfhosted
26 comments fedilink hide all child comments
 

Hi everyone,

I own a homeserver running multiple services set up in individual docker compose projects. One of them is dnsmasq + pi-hole. Pi-hole is exposed on the host's 53 port and uses dnsmasq as upstream. The host's /etc/resolv.conf points to 127.0.0.1 to use pi-hole. Other containers from other compose projects on the same machine indirectly use pi-hole as well, as they use docker's embedded DNS which forwards to the host.

Everything had worked fine for weeks before I ran into an issue after unattended-upgrades updated docker. After the upgrade, systemd attempted to restart the docker service. From the logs, it seems that the containers had a hard time honoring the SIGTERM. They started to complain about failing to reach the DNS server. At least that's my understanding from journalctl -u docker. Here's a (long) excerpt that shows:

  • containers failing to stop gracefully
  • stale processes after docker forcibly shut them down
  • repeated errors "stream copy error: reading from a closed fifo" afterwards that I am wondering what they mean
avril 12 06:46:24 prosper systemd[1]: Stopping docker.service - Docker Application Container Engine...
avril 12 06:46:24 prosper dockerd[1580]: time="2024-04-12T06:46:24.161850246+02:00" level=info msg="Processing signal 'terminated'"
avril 12 06:46:33 prosper dockerd[1580]: time="2024-04-12T06:46:33.276053218+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:37197" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.509762706+02:00" level=info msg="ignoring event" container=d518bb77113596e96db39a6770fd9955ec29efacd907cdb02148708e92156cd5 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989217237+02:00" level=info msg="ignoring event" container=81b3413f7d2409fe1a8fd8e1cf7dce3a23a6f7a4dc7cc23c13e9f1ba84b24eb7 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989312976+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=642a3c9a2c821727f30c5cf84aac52778c027083f61d571831a408a2a8c>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989126980+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=781a3eac3f08d78ef285bc0231c00bd52005fe2257fd358f7c2b51af392>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989532834+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=7406c47622ecf7fb3f9e9ff6b7a8be2a3920342b4921b33a32b3df45123>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989693081+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=e5311ad000d63615c2336db1d3630cde233495108029648289f3660ad2d>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989723995+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=e6194d95ed1b95dddeaacab68f64a8e869c7a2489144198a84865d57f89>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.350637865+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=249922bad7603a6e00bbcb0528cb7b7774a321f23e9f81b9784cc5968b5>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.506956200+02:00" level=info msg="ignoring event" container=a5ac3703480cbfd4455d4d0b373023f730da63a8792863e19bfda469ee44370b module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.602466692+02:00" level=info msg="ignoring event" container=1bd974e7ccf8f77ed93b51aa9eb0bed07cd0e26db7201fecb33c5c6f82712cb2 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.661037905+02:00" level=info msg="ignoring event" container=881fbf5034c7c0016b0715a8028e2c69788ddff218bc70fef98b30019d2b48bd module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.714184730+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=0d9862444b4a7fce35df527f034e4f673aecf44a6551858056f9d8c7ce4>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.775230427+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:42577" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.881449888+02:00" level=info msg="Container failed to exit within 10s of signal 2 - using the force" container=c0147e337cee407e7ab9b133bd20667d287d6e2e121259c4794e4e090b12>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.893446715+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=0eecbecfcaa2639946f0ed4a61a36f6ef004453af793cab91a52a97724d>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.987621257+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=3493720ae2fc565fdc052a1d137590818d418d290f4418ec6911cac4704>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.186176914+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=5cee8cfa0e068dbda35e89bcaec19450b50a0903b13bd677f6a6cb61686>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.186467614+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.186476595+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.214354986+02:00" level=info msg="ignoring event" container=e6194d95ed1b95dddeaacab68f64a8e869c7a2489144198a84865d57f8970a3f module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.339111902+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=8f42101141cf634f3495ae850b368a4b00ce065fb9a401ab7e1450d532b>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.484546249+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=881fbf5034c7c0016b0715a8028e2c69788ddff218bc70fef98b30019d2>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.728059697+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=1bd974e7ccf8f77ed93b51aa9eb0bed07cd0e26db7201fecb33c5c6f827>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.753403840+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=d518bb77113596e96db39a6770fd9955ec29efacd907cdb02148708e921>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.787737744+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=c39c900cb5ccd9ae3d48be4f953389edcab7bf051b9d505e23a816a8627>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.235834609+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=13a3101e56e66f045318ad572a19b259a8cd3fbe7772b82f9262c15b727>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.331772840+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=a5ac3703480cbfd4455d4d0b373023f730da63a8792863e19bfda469ee4>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.411877163+02:00" level=warning msg="Health check for container e6194d95ed1b95dddeaacab68f64a8e869c7a2489144198a84865d57f8970a3f error: context canceled"
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.705699519+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:58293" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.748247132+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=81b3413f7d2409fe1a8fd8e1cf7dce3a23a6f7a4dc7cc23c13e9f1ba84b>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.790571926+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=e6e82f5a61e3b412c93abdd7fcc1cf25560e7d8ae957313fd36429f3ed7>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.811976559+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=48d78ba0c55912bd57f72c6934940ad173261364d13483e630769b670d0>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.832092046+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=0c7717003531fe815060e60da13065abe8399782c750d526e7fba872f0e>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.965233556+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=73b5051331f81477c6358d74e889fedf50bf0c57696fdd8a735e5a20f3b>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.965447255+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=c73fda2593bbbeb66589373add0609f8a165eab6d3ef4f0fe9bc871b1ca>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.094152959+02:00" level=info msg="ignoring event" container=0eecbecfcaa2639946f0ed4a61a36f6ef004453af793cab91a52a97724d5225f module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.113626463+02:00" level=info msg="ignoring event" container=642a3c9a2c821727f30c5cf84aac52778c027083f61d571831a408a2a8c2c71f module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.113684911+02:00" level=info msg="ignoring event" container=e5311ad000d63615c2336db1d3630cde233495108029648289f3660ad2d6c33a module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.192923515+02:00" level=info msg="ignoring event" container=8f42101141cf634f3495ae850b368a4b00ce065fb9a401ab7e1450d532b917ba module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.212691218+02:00" level=info msg="ignoring event" container=7406c47622ecf7fb3f9e9ff6b7a8be2a3920342b4921b33a32b3df45123878f2 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.282823538+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=a8be522903b221bd8af2b6bf0533b56bc828863852d17fa604ed932d8cd>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.364351179+02:00" level=info msg="ignoring event" container=249922bad7603a6e00bbcb0528cb7b7774a321f23e9f81b9784cc5968b54ca54 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.433220060+02:00" level=info msg="ignoring event" container=0d9862444b4a7fce35df527f034e4f673aecf44a6551858056f9d8c7ce44dbd4 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.435651309+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:59049" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.503015026+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=6c0fcacc03cef94239fadb71e55ded7b01d07ed075023966427072d480a>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.531879355+02:00" level=info msg="Container failed to exit within 10s of signal 28 - using the force" container=6290112ab909221f1d3eee48a09e49403c1f61b7e1675ab3e581649ea82>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.789853647+02:00" level=info msg="Container failed to exit within 10s of signal 28 - using the force" container=89d8e915dc9ab961502830d0f8086d1a4b709d726d69e6d26570463219a>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.928854916+02:00" level=info msg="ignoring event" container=5cee8cfa0e068dbda35e89bcaec19450b50a0903b13bd677f6a6cb6168675d21 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.089539561+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=b52727b63c13631c15b7b22d400f2c913ae73a2f5b36b0f566bef0ba645>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.202791198+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=deda565519df8cd8705fff76564af4a084514fea41fe03ca1ab7e56f14b>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.290584165+02:00" level=error msg="Force shutdown daemon"
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.567930258+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=c813f03c24c960653e28a9857a5d7f1fde677bcd3e226ed1585d41626a9>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.725804932+02:00" level=info msg="Daemon shutdown complete"
avril 12 06:46:42 prosper systemd[1]: docker.service: Deactivated successfully.
avril 12 06:46:42 prosper systemd[1]: docker.service: Unit process 15700 (docker-proxy) remains running after unit stopped.
avril 12 06:46:42 prosper systemd[1]: docker.service: Unit process 15979 (docker-proxy) remains running after unit stopped.
avril 12 06:46:42 prosper systemd[1]: docker.service: Unit process 18294 (docker-proxy) remains running after unit stopped.
avril 12 06:46:42 prosper systemd[1]: Stopped docker.service - Docker Application Container Engine.
avril 12 06:46:42 prosper systemd[1]: docker.service: Consumed 3h 16min 43.639s CPU time.
avril 12 06:46:42 prosper systemd[1]: docker.service: Found left-over process 15700 (docker-proxy) in control group while starting unit. Ignoring.
avril 12 06:46:42 prosper systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
avril 12 06:46:42 prosper systemd[1]: docker.service: Found left-over process 15979 (docker-proxy) in control group while starting unit. Ignoring.
avril 12 06:46:42 prosper systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
avril 12 06:46:42 prosper systemd[1]: docker.service: Found left-over process 18294 (docker-proxy) in control group while starting unit. Ignoring.
avril 12 06:46:42 prosper systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
avril 12 06:46:42 prosper systemd[1]: Starting docker.service - Docker Application Container Engine...
avril 12 06:46:44 prosper dockerd[276530]: time="2024-04-12T06:46:44.366430751+02:00" level=info msg="Starting up"
avril 12 06:46:45 prosper dockerd[276530]: time="2024-04-12T06:46:45.443488954+02:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
avril 12 06:46:57 prosper dockerd[276530]: time="2024-04-12T06:46:57.117503878+02:00" level=info msg="Loading containers: start."
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.071827546+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.074049075+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.077609642+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.077684622+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.082780007+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.084236865+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.088120145+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.088249509+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.168036186+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.168083531+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.169898516+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.171988424+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.172240688+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.172348881+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.173855919+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.176585486+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.178804059+02:00" level=info msg="ignoring event" container=781a3eac3f08d78ef285bc0231c00bd52005fe2257fd358f7c2b51af39266c42 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.180166037+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.185876045+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.270156961+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.270186688+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.281083861+02:00" level=info msg="ignoring event" container=89d8e915dc9ab961502830d0f8086d1a4b709d726d69e6d26570463219a620f1 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.282779586+02:00" level=info msg="ignoring event" container=6290112ab909221f1d3eee48a09e49403c1f61b7e1675ab3e581649ea829ad0f module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.298497572+02:00" level=info msg="ignoring event" container=b52727b63c13631c15b7b22d400f2c913ae73a2f5b36b0f566bef0ba645e7c0f module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.316205371+02:00" level=info msg="ignoring event" container=c73fda2593bbbeb66589373add0609f8a165eab6d3ef4f0fe9bc871b1ca315a7 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.316278998+02:00" level=info msg="ignoring event" container=3493720ae2fc565fdc052a1d137590818d418d290f4418ec6911cac4704eb7e8 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.341300109+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.341872295+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.370180971+02:00" level=info msg="ignoring event" container=6c0fcacc03cef94239fadb71e55ded7b01d07ed075023966427072d480ab42f7 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.382180381+02:00" level=info msg="ignoring event" container=c0147e337cee407e7ab9b133bd20667d287d6e2e121259c4794e4e090b123e11 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.436201339+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.436352997+02:00" level=error msg="stream copy error: reading from a closed fifo"

I ssh'ed into the server and noticed that the pi-hole & dnsmasq containers were not running. I ran docker compose up -d in all projects, starting with the DNS one and it went back to normal.

My question is: how bad of an idea is this to run a DNS in docker and use it for the host and other containers? Maybe I just had back luck but I can't help thinking this setup creates some sort of circular dependency between the host, docker and the containers.

If anyone has experience with a similar setup, I'd be curious to hear their opinion on the issue I ran into.

Thanks!

top 26 comments
sorted by: hot top controversial new old
[–] vegetaaaaaaa 9 points 7 months ago (1 children)

Not an answer but still relevant: I actively avoid enabling unattended-upgrades for third-party repositories like Docker (or anything that is not an official Debian repository) because they don't have the same stability guarantees, and rely on other upgrade notification methods instead.

how bad of an idea is this to run a DNS in docker and use it for the host and other containers?

Personally I would simply install dnsmasq directly on the host because it is one apt install and a configuration file away. Keep it simple.

[–] [email protected] 1 points 7 months ago (1 children)

Yeah, that was my plan B. To be honest, I was not super confident that it would work when I put this setup together, because of the "host uses a container as DNS and docker uses the host as DNS" kind of circular dependency.

But people do use docker for DNS servers so it has to work, right? That's where I'd like to understand where I'm wrong. I'm fine with running pi hole and dnsmasq on the host as long as I get why this is not doable in docker.

Thanks for your input, though. That's helpful.

[–] vegetaaaaaaa 3 points 7 months ago (1 children)

Usually you would have a second DNS resolver configured in /etc/resolv.conf (or whatever name resolution config system you are using, resolvconf, systemd-networkd, etc). The system will fall back to this resolver if the first resolver fails to respond (and/or replies NXDOMAIN, I'm not sure. The exact order and fallback conditions may vary depending on which system you use). This can be another dnsmasq instance, a public DNS resolver, your ISP's resolver, etc. This allows at least basic DNS resolution to work before your dnsmasq instance comes back up.

I would also add automatic monitoring for dnsmasq (either check that the service/container is running, or check the TCP connection to port 53, or check that DNS resolution is working for a known domain, etc)

[–] [email protected] 2 points 7 months ago

I see. I kind of thought about it earlier today while mulling over the problem. I can definitely do that first because it's easy and makes total sense.

I already have prometheus monitoring the DNS resolution, I think. I'll check!

Thanks for taking the time to answer!

[–] [email protected] 3 points 7 months ago (1 children)

circular dependency

What's in/etc/resolv.conf inside the dns container?

What's the upstream dns for the dns servers in the dns container?

[–] [email protected] 1 points 7 months ago (1 children)

In both the pi-hole (exposed on the host) and dnsmasq (used as upstream by pi-hole) containers:

# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make
 further changes once it
# has been modified.

nameserver 127.0.0.11
options ndots:0

# Based on host file: '/etc/resolv.conf' (internal res
olver)
# ExtServers: [host(127.0.0.1)]
# Overrides: []
# Option ndots from: internal

So they are pointing to docker's embedded DNS, itself forwarding to the host.

[–] [email protected] 2 points 7 months ago (1 children)

OK yeah, that might be a problem. You want to set the dns in the dnsmasq so it can resolve stuff upstream (probably with DoT or DoH), and the dns in pihole to the IP of dnsmasq. Look into the "dns", "dns_search" and "dns_opt" compose directives.

You may also want to consider simplifing your stack by using only dnsmasq or only pihole. Both of them can do pretty much the same things if you add some plugins.

I'm using dnsmasq as lan dns server with DoH upstream as well as dhcp server. The dnsmasq is advertising itself as dns to all the dhcp clients. Also it runs on the router which allows it to hijack port 53 so any client attempting to do clear dns elsewhere ends up going through dnsmasq anyway. It also has an adblock plugin.

[–] [email protected] 2 points 7 months ago

Your suggestion looks similar to this other comment and makes sense. I'll try that!

I have never managed to wrap my head around DoH and DoT but this is on my todo list ^^

I didn't know dnsmasq has an adblock plugin, I'll have a look. Originally, I was using dnsmasq alone (running on bare metal). Then I migrated to docker and added pi-hole for ad blocking. I tried to get rid of dnsmasq but pi-hole's embedded DHCP is not as configurable as dnsmasq's and I could not address my use case.

Thanks a lot for your time!

[–] [email protected] 2 points 7 months ago

I have two docker hosts which both run Pihole and get synced via gravity. I set those instances as first and second DNS servers in DHCP and on each docker host

[–] Willdrick 1 points 7 months ago

Maybe a bit of a low tech solution, but I have an older RPI 3B running a second instance of PiHole.

[–] daddy32 1 points 7 months ago (2 children)

Shouldn't you just have a fallback DNS for cases like these? During the outage, it would be the one used and after the things settle, you would be back at primary.

Maybe some kind of monitoring/notifications on top of thatm

[–] AtariDump 2 points 6 months ago (1 children)

When it comes to a “secondary”DNS... [there is nothing like a primary and secondary DNS server. These indications are quite misleading but many systems adopted it this way. Pihole only list the DNS servers as primary and secondary, because this is what the providers write on their pages. The bad phrasing is supported especially by how Windows handles it.](https://discourse.pi-hole.net/t/primary-vs-secondary-dns/1536/2)

[Most operating systems implement DNS servers as alternatives, not as fallbacks, i.e. they will query any of both servers from time to time, so it is quite likely that you will loose your Pi-hole filtering capabilities (at least partially) [if you specify a secondary DNS server on your network].](https://discourse.pi-hole.net/t/secondary-dns-server-for-dhcp/1874)

The **ONLY** DNS server you should have set on your network is a/the PiHole(s).

[–] daddy32 3 points 6 months ago (1 children)

Oh my, TIL. But what then in cases like these, when the Pihole itself is down? You need fallback..

[–] AtariDump 3 points 6 months ago (1 children)

You need another DNS server. Fallback implies that it’s only there if the first stops responding, which isn’t how DNS works.

[–] [email protected] 1 points 6 months ago (1 children)

For some reason, I am only seeing this comment thread now, so sorry for the late response.

Thanks for those valuable details. But I am still a bit confused. I understand why you are saying that pi hole should be the only DNS server handling requests sent by LAN devices (including the machine hosting the DNS). That's because it is the only one which can resolve local names (well, that's actually its upstream dnsmasq running as a sibling container that does that but that's a minor detail).

But then you say there should be another DNS server to solve my problem. If I put two server entries in /etc/resolv.conf, one being pi hole and the other my ISP's DNS, the two of them will be randomly picked by DNS clients. When the ISP's is used, it will fail to translate local names. I guess there is a way to let the client try the other server after a failure but it will add some undesirable latency.

Sorry if I misunderstood your point but after reading the first comments I was quite convinced by the idea of adding a second nameserver entry in /etc/resolv.conf. Your explanations convinced me otherwise and now I have the impression that I can't really solve my initial problem in a reliable way.

[–] AtariDump 1 points 6 months ago (1 children)

You should have two PiHoles.

[–] [email protected] 2 points 6 months ago (1 children)

My goodness, that's some impressive responsiveness ^^

I guess see your point. But then the problem shifts to the upstream dnsmasq instance which acts as DHCP + DNS for the local devices. This is the server ultimately able to translate local names.

I don't think it's doable to have two instances of dnsmasq that are able to translate local names interchangeably. That would require two DHCPs to have authority on the network. But I'm no expert so I may be missing something obvious.

[–] AtariDump 1 points 6 months ago* (last edited 6 months ago) (1 children)

You only set one DHCP server. DHCP hands out both PiHoles as DNS servers on the network.

Name resolution is handled by DNS and not DHCP.

[–] [email protected] 1 points 6 months ago* (last edited 6 months ago) (2 children)

Sorry, I was unclear: I use dnsmasq as single source of truth. In its DHCP config, I set machine names, routes and all. And this is because this dnsmasq is the DHCP that it knows how to translate the names of the devices it configured. Pi-hole forwards all DNS requests to dnsmasq. Now if I use two instances of dnsmasq, only one can be a DHCP and the other won't know how to resolve local names, unless it uses the first dnsmasq as upstream. But in scenarios where this first dnsmasq instance is down, we are back to square one.

[–] [email protected] 1 points 6 months ago (1 children)

I had a quick look at resolv.conf's manpage on Debian and I think @[email protected]'s suggestion of adding a second nameserver would actually work:

nameserver Name server IP address
    Internet  address  of a name server that the resolver should query, either an IPv4 address (in dot notation), or an IPv6 address in colon (and possibly dot) notation as per RFC 2373.  Up to MAXNS (currently 3, see <re‐
    solv.h>) name servers may be listed, one per keyword.  If there are multiple servers, the resolver library queries them in the order listed.  If no nameserver entries are present, the default is to use the name  server
    on  the  local  machine.   (The algorithm used is to try a name server, and if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are
    made.)

According to the doc, the resolver will try each name server in order until one is successful.

[–] AtariDump 1 points 6 months ago

…if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are made.)

Queries are going to time out and fall over to the other DNS server.

[–] AtariDump 1 points 6 months ago

Why?

[–] [email protected] 2 points 7 months ago* (last edited 7 months ago)

Yes, others have suggested something similar. I'll do that first because it is easy. Monitoring-wise, I should already be covered but since prometheus is running on the same server, it was down during the outage. There is room for improvement, for sure! I have a couple of RPis on my network that I can leverage for better monitoring.

[–] [email protected] 1 points 7 months ago (1 children)

circular dependency seems to be the case. I guess adding second external resolver to /etc/resolve.conf will help. Second entry will not be used unless first one ( pi-hole) is responding. But it need to be tested.
BTW, why do you want to send host's DNS via pihole?

[–] [email protected] 1 points 6 months ago

Well, I have not really thought about why. I guess that's partly due to old habits of running services on the host with systemd (my migration to docker is recent and still a work in progress). But I guess I'd like to continue to be able to resolve names of local devices on my network when connected through ssh on the host. Is that inherently wrong, still? I will implement the secondary DNS as a fallback. I am hoping to get rid of the issue that way.

[–] [email protected] 1 points 7 months ago* (last edited 6 months ago)

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
DHCP Dynamic Host Configuration Protocol, automates assignment of IPs when connecting to a network
DNS Domain Name Service/System
IP Internet Protocol
PiHole Network-wide ad-blocker (DNS sinkhole)
RPi Raspberry Pi brand of SBC
SBC Single-Board Computer
TCP Transmission Control Protocol, most often over IP

[Thread #683 for this sub, first seen 16th Apr 2024, 11:25] [FAQ] [Full list] [Contact] [Source code]