I mainly use my phone with a cheap DAC dongle along with a Walkman DAP that I got a great deal on used. There is not a lot of differences from the sound between them, but I use the walkman DAP when I 1) Don't want to get distracted by anything other than music, 2) Want to use a bluetooth earbud/headphone that supports aptX/LDAC codec, 3) Want a built in EQ that applies to all music apps (the Sony Clearaudio+ feature is also nice in some situations), 4) I admit I love seeing the tape player screen saver when using the Sony music player, 5) the expanded audio options when using my Sony Linkbud S. 6) When I want to save battery life on my phone.
So having a DAP is not a must have, but I do sure enjoy having it available as an option.
When I first tried the LinkBud S I had a similar reaction and was like "why is this so muddy and ultra bassy?" But as I spent a bit of time deactivating any random audio presets on the phone and then trying various settings with the sony app, they actually sound amazing. Very clear details and balanced bass. Mine is an IEM and not a headphone, but hopefully the experience to find a good audio balance for you is similar.