Is he in private practice or part of an HMO? If part of an HMO, you can speak to the head of the department and/or to member services to express your concerns. A doctor who cannot hear his patients should be a major concern to any practice and should be something that they are made aware of.
I always say that, if you are not comfortable with your doctor, then-by all means- find one that you are comfortable with. You cannot participate in the management of your disease and/or the treatment of your symptoms if you are dealing with a doctor whom you are not comfortable with. And, not being heard (literally) is an issue that would certainly make me uncomfortable (lol).
Best of luck and let us know how it goes.
Peace and Blessings
Look For The Good and Praise It!