I think almost everyone does this. If you just stop you will tend to pull the rear brake side of the bars slightly harder and may set yourself off balance. If you feel comfortable doing it then carry on. As long as it doesn't hinder your riding then there is no problem.