I agree with your sentiment. Unfortunately I've never used one before BUT, I will soon have the opportunity to use one and I fully intend to check the functionality w.r.t. recovery of private keys. I will report back on this thread if nobody else get's there first (unless you particularly trust my word, then I'll chime in too! Ha ha).
@drehb - from my research I discovered that the Ledger Nano S is actually very flexible. Surprisingly, it overcomes your concerns mentioned in your post above! I don't have time right now, but I intend to come back to this thread with an explanation. IIRC it involves BIP39, password salts and using different seed phrases for each device PIN. There's even the ability to employ a plausible deniability use case.
Other hardware wallets are available!