Yes, it might be a bad idea. But it depends on where you are playing the speakers, what's being played on the speakers, the volume of the content, and maybe a few other factors. If you do intend to go through with this, then whatever system you have, it'll need to be kept lightweight as you'll have speakers, power system, controls, framework for carrying everything, etc..
The main issue I see is the content being played and whatever volume you have. Depending on where you go, some people may or may not like hearing something that is "drowning out" whatever it is that someone else is doing. A prime example might be at conventions where panels, meetings, and other events are going on and it would be rude and disrespectful to disturb them.