I can not have ear buds in my ears at all. It makes me throw up. That being said, I could not imagine wearing them on the trail. I would be afraid that they would rip out of my ear getting caught on branches etc. along with what everyone else has mentioned above.
I also don’t want to be one of the people I hate to encounter on the paved trails that ride/run/walk along oblivious to the world with their buds/phones on. The ones that give you a dirty look as you pass them too close b/c they were all over the lane moving erratically etc. I like to keep my pedal cadence consistent, but I do it with certain songs just going through my head