Most horses I've asked don't seem to have an opinion one way or another. They just say 'pfpfpfpfpffp'.
In all seriousness, I don't think they seem to mind being ridden much. A very close friend of mine lives on a horse farm about 3 minutes walking distance, so I've had some very close interaction with them
Btw, Bob is awesome. He's a black pony with a big white stripe on his back. He's old, doesn't see or hear very well, and didn't care when I used him as a pillow while he was laying on the ground.