I would give some thought to taking Eloquence over Whispers: while the bonus damage is nice, the big thing is making sure you can reliably get Fear running, and Unsettling Words allows that reliably.
Your payoff here is permanently locking down one target for a whole combat as long as you can consistently land hits, and in that situation the damage rolls won't really matter so much.
They don't get another save, the rest of your party cleans up the others while you just plink away at your target (or just shoot at other targets, honestly, you don't actually need to kill this one fast).
The primary drawback is a lack of reliable advantage,
For reliable advantage, the easiest approach would be Blindness/Deafness, but unfortunately I think that qualifies as the target not having line of sight to you anymore for Fear.
Shield Master won't work either since the target will likely just stand from prone on their turn, before taking the AoO.
The Help action would give you a consistent source of advantage though (for only one attack, but that's all you need), and it's not too hard to get Find Familiar on a bard: using a Magical Secrets on it feels bad, but Ritual Caster or Magic Initiate would work. The normal risk of your owl getting swatted is much less since you can just move him into danger after your target is locked down from Fear.