Wolf by night here, draft horse by day. I wear my harness for a day at a time but only now and then, usually when I am pulling my handler in her cart. A well designed one will fit you well, especially if it's leather since it will break in and conform to your body. A poorly designed or ill fitting harness will feel like shoes that don't fit.
I made my own harness out of Biothane (you can catch cheap factory outlet rolls on ebay sometimes), and I could feel every mistake that I made until I fixed it. But I learned a lot, and now I know what I did right and what I want to do differently for my next harness, which I plan on making from leather.
On a related note, I also wear a halter or bridle when I'm pony-ing out, since horses normally don't wear collars.