No, we don't listen to bits. Do we? Not if we listen to PCM we don't. PCM is a type of code that goes through a secondary operation before getting to a form where we can hear it. DSD not so much. We actually do listen to the direct bits in DSD.
PCM is kind of like words on a printed page. These words you're reading have meaning but not directly. They 'represent' meaning through interpreted triggers in our heads. Alone they are nothing but collections of letters in proper sequence. When you read them they generate a voice that 'speaks' them in your head. You're listening to that voice right now as you read my words. But that voice is one step removed from the actual written symbols on the page.
DSD, on the other hand, is more like the spoken word without the secondary step PCM relies on. Utterances have a much more direct path to the brain than do the written symbols we call writing.
What's all this mean to you with respect to your stereo system? Probably not much, but it is an interesting thought.
Sometimes I like to just share those interesting thought with you.