Physics might be the most difficult major for undergrads so if you actually can stand it, go for it. I have a good friend, who got about a "B+" average at UCLA and when he graduated he got a great job as a financial engineer, in Down Town Los Angeles, making 75k base salary to start and a bonus to match. The reason is because on Wall Street and in other financial centers in the US and the rest of the world for that matter, multidimensional econometric models were very popular for trying to guess patterns of behavior and then investing accordingly.
Best of all is that two years later, while the world of high finance and exotic derivatives were shaken, in the year 2008, my friend simply left and got into electrical engineering for a bio medical company and more recently he is a going to be a PhD student this fall and instead racking up massive amounts of student loans in graduate school (so many smart people out there take a major that ends with "-studies" and they take no math and their only even remotely lucrative prospect, within the context of quaternary education, is law school) my friend is being paid to get his PhD at Princeton.
This is especially not worthy because Ivy League Grad Schools tend to have a huge time zone bias, they do not consider even Stanford or Cal or UCLA important (the plus side is that the it strangely helps students from the lower ranked UC's and from CSU schools. In the eyes of the Ivy League's admission boards, every school "out in California" is seen as too sunny and too far from I-95 to be important so an outstanding student from CSU Long Beach or Humboldt has an equally slim chance as a student with similar numbers who thrived at Cal or UCLA). Unlike Law, Medicine, Business, Fine Arts and most PhD programs, Princeton or MIT's admissions boards have far fewer great potential physics students relative to the number of spots in their graduate level program so if you had a great GPA (and in physics a 3.7, like my friend had, is great. Physics classes do not hand out "A" grades in the way that the soft social sciences and other majors are known for grading that is even more liberal than their political views. The natural result is that an "A" grade confers excellence and even a "C" shows solid competency).
So go ahead and take physics. Just graduating with it will make you look attractive to employers. If you got a school that is not in the USA Today top 100, do not worry just out perform most of your classmates and have a good shot at getting a PhD at any institution, no matter how ancient, Eastern and prestigious it may be. Wit ha PhD in Physics you can do so much in academia, engineering, finance and even military, an ROTC physics grad is a rare thing today and having an advanced degree in that field can vault you from the starting rank of Lieutenant up to Captain, quickly onto Major and maybe even Lt. Colonel within a few years out of grad school and Colonel after that and by age 45 you could be climbing the General's promotion ladder (or if not the airforce or army, you can climb the equivalent ranks and be an admiral by 40 or 45) and then retire and make very good money as a defense consultant (a former Air Force General or Navy Admiral not only has great insight but great friends at the Pentagon and that is known to sway a contract towards the employer of an honorably retired Grandee of the Air and/or Sea). Keep your options open my keeping your spare time and social life somewhat closed and major in physics.
BTW, I used California of one example of bias against anyone from a an institution that is not very North Eastern and private and familiar to the Ivy League's admission officers. If anything, California probably gets the most respect of Western States, the bias is probably worse against the South and the South West and Texas. So no matter where you are physics and its brother mathematics are so universal and they transcend the biases against your home State and/or non Ivy League undergrad programs.
