If you do Maths and Physics any Maths you do in Physics is so simple that it's a piece of piss.
Physics will involve logarithms and perhaps integration and differentiation (all quite tricky to get the hang of), which are all covered in Maths in much more detail than you'll ever need in Physics.
I know lots of people who did Physics without doing Maths but they compliment each other quite well. Basically Maths makes physics easier but isn't essential.