Without reading anything else for better context, I'd say you're exactly right. RAW the ruffian shouldn't get sneak attack on a crit if the damage dice is upped to a D10. That said I'd allow it at my table.
pathfinder
Paizo uses words very specifically, much like lawyers. Note that in your highlighted text they use the word “abilities.” A critical hit is not an ability, it is a game mechanic. It also isn’t modifying the weapon’s inherent damage dice, just the dice rolled for one attack.
IMO, sneak attack damage should and does apply, even on a crit with this weapon.
I would buy that if it was capital-A Abilities in the text, but it's not. We also see this with "attack rolls".
This is poor communication on their part. If they want to be lawyers about it, a halfway decent one would get these thrown out.
It would be silly to have sneak attack not apply to a certain weapon because of a crit.
Flavor wise, the small damage die restriction even make sense since small weapons are better for making well-placed sneaky attacks. If you crit with that attack it makes even more sense for sneak attack to apply.
The restriction is supposed to apply when choosing your weapon, everything else would be unnecessarily complex.
I think this largely depends on whether you consider traits abilities or not. My interpretation would be that abilities are things like class features/feats, but traits are not, and would therefore not impact the ruffian’s sneak attack specifically.
abilities are things like class features/feats, but traits are not
Why do you interpret that way? That's the only way I can make this make sense, but that seems totally arbitrary. The text could say "character feat" if that's what they meant.
The Fatal feat grants the weapon the ability to use a higher damage die, so the word is poorly chosen if it's meant to be viewed as narrowly defined.
I think they meant for it to cover more than just feats. For instance, maybe you have an item you activate that increases your damage die size or you’re under the effects of a spell. It would be too verbose for them to enumerate everything. They could have instead stated “effect” or “anything” if they intended it to cover traits. There is also this from the definition of trait (emphasis mine):
A trait is a keyword that conveys additional information about a rules element, such as a school of magic or rarity. Often, a trait indicates how other rules interact with an ability, creature, item, or another rules element that has that trait.
Trait specifically calls out ability as something separate from a trait.
But here, trait is calling itself out as a word. In this case, it's a word that signals an ability that originates from the item.