Ok I think I managed to fix it. I don't know how long this will last for as it still seems to have a finicky behavior.
The problem I have now is that the left earbud takes a few tries of inserting it into the ear to detect that the mic works with ambient audio. I believe its due to the mesh ingress right next to the two golden colored contact pins (which I think is responsible for detecting if the earbud is in your ear, therefore capable of replaying ambient audio to you)
If anyone wants a fix, I'd recommend using a toothbrush or some vertical thin brush bristles able to fit in said ingress, along with IPA if you have that. (it also might be worth doing the same procedure to the metal grille and the microphone holes if they seem to be clogged up too, but be gentle)
Brush up and down ***''and''*** in a circular pattern to get the buildup stopping the in ear detection from working, and then finally ***''dry''*** up the isopropyl alcohol ***before powering them on***.
Sometimes I need to temporarily heat the left earbud up for it to detect that it is in my ear and re-insert the brush. After that if it still doesn't work wiggle the earbud while in your ear for the detection to do its thing, hopefully allowing you to hear the ambient audio.
Hope this helps anyone with a similar problem.