First off, thanks for being so open, because this is exactly the sort of thing that can eat away at a person, especially language-learners. Personally, I think things like this need to be discussed when it comes to learning the language, so even though you've got some pretty negative feelings driving this post, I think you'll find that you're certainly not alone in feeling the way you do.
I'm no stranger to failure myself, and considering that Japanese is sort of "my thing," when I feel I'm making stupid mistakes in the language or get it in my head that I'm not making enough progress, I have a tendency to beat myself up over all that. The usual line of thinking that pops into my head? "You've been studying for all these years, and you STILL got that wrong? You're out of your league, and you've got no business studying Japanese in the first place!"
Writing it out like that, it's easy to see just how many ridiculous holes there are in that sort of thinking, but you know what? In the moment, it sounds like the truth, like you've finally come around to being honest with yourself. But if you take just one step back from that mindset, the entire picture changes. The fact of the matter is, failure (or rather, the feelings associated with it), will try its best to bring distortion to anything it touches.
Couldn't remember that Kanji? You forgot the passive conjugation for that verb? Don't tell me you forgot to add the honorific "o!" While small on their own, we tend to let miniature failures like that pile up in our minds simply because we remember our mistakes far better than our successes. After all, why sweat what you got right? On the other hand, with language learning especially, we have to learn from our mistakes, so naturally, we pay attention to them. While learning from your mistakes is pretty much the cornerstone of the entire process, the problem comes whenever we shift our focus away from the growth that such learning can bring, and onto the mistakes themselves. In other words, if you focus more on your mistakes than what those mistakes can teach you, you've got yourself a one-way ticket to self-depreciating hell.
While it might seem like you're being out of touch with reality or trying to fool yourself, when it gets down to it, the best thing you can do in this sort of situation is to keep your head up and keep plowing ahead. Instead of telling yourself that you'll never learn Kanji, focus on all those areas that you DO know, and remind yourself that you've got access to a whole new world with the language-- a world that not many English speakers have access to at that. Japanese looks like scribbles to my relatives and friends, and even though I still have moments where I'm left scratching my head or blanking on something simple (it took about five tries for me to recognize the word 拾う(ひろう)when my tutor used it the other day, for example), there's a sort of secretive thrill in learning a new language that can help quell the sort of doubt and misgivings that you brought up in your post. The best part, though? If you keep going, you're bound to grow more and more. It'll take time, but if there's anything over five years with the language have taught me, it's that patience and hard work pay off. Big time. Naturally, you have to put out as much effort as you can, and your own progress is bound to be different from that of others, but at the end of the day, you'll find yourself achieving far more than you ever thought possible.
So, I guess what I'm trying to say is... savor your victories. Find a study method that works for you, stick with it, and when you start seeing results, celebrate them, no matter how small. Whether it's the first time you get through a whole sentence unaided, guessing the pronunciation of a word you've never seen before correctly, or even finishing your first novel in Japanese, savor that victory, and stash it away in your mind for all those いざという時 when your failures try and convince you that you'll never make it. More than anything, remember that being honest with yourself doesn't mean gut-punching yourself over everything you ever got wrong.