The problem is, again, other people worrying about what other people are wearing. Also, just as a note, not everyone is buying LV. Just because you see celebrities and their posse geared up in LV doesn't mean everyone buys it. I live in SoCal and have rarely seen LV clothes, accessories and bags around, on men at least.
Also, just because you think everyone wears them doesn't lose it's special qualities, it just means you are shallow enough to let it bother you. So if you love an Audi S5, but everyone in your neighborhood has one, it'll lose it's qualities? Does it become slower? Does it depreciate faster? Does it breakdown easier/faster? No, they don't, at least not because everyone has one.
This may differ by cities but I can almost guarantee you that if you wore a graphite hoody in LA for a day, the chances of running into someone wearing the exact same thing will be slim to none. You have more chance of running into someone wearing one of your favorite Air Jordans than any of these LV clothes.