So there’s a few issues here:

• Unarmed Strikes do not require an open hand. Punches, kicks, and slams all count as the same Unarmed Strike
• If you were to allow this, there would be no reason to allow someone with two Shortswords or a Greataxe to do a BA strike
• …which would then render the BA attack from Polearm Master moot since they no longer need a feat to do that
• I’ll also note that the fighter with a sword in one hand and nothing in the other is likely using the Duelist fighting style, so that sword attack is effectively two die sizes larger. A Duelist Longsword is roughly equivalent to a Greatsword to put it in perspective

At the end of the day, allowing martials to perform a BA Unarmed Strike wouldn’t be game breaking, but it needs to be applied universally which has secondary implications