I hope I can do JLPT N4 without any classes because I decided not to go for them this time.  

So, I have got Nihongo Challenge for self-study. It seems fine for kanji but I am not sure about grammar part.

Which is the best book to do grammar? Tae Kim bugs me because I get overwhelmed at the kanjis I don't know while I study from it.

Don't worry, I passed N1 and N2 without classes, so it should be doable for N4 too. I can't advise you on N4 books directly, but Kanzen Master was great for my JLPT studies, and if they make N4 books, you might want to look into them.

Not knowing your current level, but I would recommend the Genki series for a start. They have quite thorough explanations of basic grammar. If I recall correctly the Kanzen N4`s explnations is more sparse (I didn't`t use the N4 book though, just flicked through it)

I can't really say if it's "the best one", but we used Genki at univeristy when I studied. I was enrolled and took the exam, but I never really followed the classes, so I essentially used it for self-study. Worked out great, but I wish I had known about SRS and Anki back then. I think it would have done the whole process much easier.

If I were you I'd use Genki as the main study tool, and supplement with a N4 grammar book such as Kanzen or Sô matome and expose myself to sentences using the grammar points through Anki.