I just want to say that I have significantly less words studied in Anki than OP but I also live in Japan. I listen to Japanese on a daily basis and use it (poorly) but it really takes a wide variety of situations to get used to listening. Luckily I get exposure to Japanese from a lot of age groups. Maybe it would be best to focus on listening to a wide variety of sources because it really does depend on who is talking! Everyone has a different way of talking. I've even had to struggle with a weird dialect where I live. That being said your Japanese is probably much better than mine with all those words under your belt so take what I say with a grain of salt lol.
