Either be dressed like you are going to or from a hunting trip, or be dressed well enough that someone wouldn't mistake you for a ne'er-do-well. Someone I know was in an Albertson's supermarket in Nampa the other night about 10:45 PM, and a guy walked in dressed rather shabbily--but with a large holstered handgun. The manager of the store approached him and told him to leave, which he did.
I know that some people really think that everyone else should just "get over it" and not react negatively to openly displayed guns. Yes, just like people need to just "get over it" when they react poorly to guys French kissing in public, or people urinating in public on the sidewalk. Think about the consequences of what you do--and ask yourself if you are doing more good or harm.
There are situations where it may make perfect sense to carry openly in a city. Perhaps you didn't have the foresight to get a concealed handgun permit, and you have good reason to believe that you are at risk. Perhaps you are between 18 and 21, and can't get a concealed handgun permit. Perhaps you live in one of those benighted states that (for a few months longer) either don't issue permits, or make it so hard to get a permit that they are effectively not available. But if so, dress in a manner such that no one is going to wonder if you are a meth freak suffering a paranoia outbreak. You might still be asked to leave the store--but there is at least a chance that you will leave a more positive impression of gun owners than if you look like you are about to clear out the entire pseudoephedrine HCl section of the store.