I think that almost everything in N3/N4/N5 of the JLPT and lots of what's on N2 is common enough in general conversation that you should definitely learn it. That doesn't mean you need to study it formally outside of native material; if you have a good solid understanding of sentence structure, then you could probably get by just by looking things up as they come up.

I think the limiting factor when watching TV as a beginner is how quickly people talk. (And also all the things that can make speech harder to understand -- baby talk, regional accents, drunken slurring, angry shouting, music in the background, etc.) There's a difference between knowing enough grammar that you can understand a sentence if it's written down and you have time to figure it out, and internalizing enough grammar that you can understand a sentence at the speed it's spoken, and without having to think too hard about it. (Reading easy stuff will help a lot with that.)

It's easier now that you can get dramas with Japanese subs pretty easily, but I was at a high-intermediate level before I really felt like I could watch TV and get much out of it.