It is better late than never, I was in search of AUX with USB as a replacement for just ordinary AUX which is fitted on my Camry and my quest led me here.
To answer your question, the AUX on your Camry is working fine BUT toggling between other functions in your control consoles (CD, FM AM) and AUX requires a firm connection of the audio jack into the AUX, if not there would be no switching.
It could be a problem with the cable or the AUX Port, as a test of functionality, use one hand to hold the 3.5 mm audio jack cable into the AUX Port firmly whilst toggling into AUX mode on the control…voila! it works