Language learning has been mostly fun for me (except for certain tasks that you just HAVE TO get done, fun or not), but it also takes me five to ten years to get good at a language (in the case of Japanese, it's gonna probably be closer to ten than five). So there's a tradeoff between hard work and time spent. You can't get good fast, without working really hard at it.
As for what makes language learning fun, that's tricky. Early on, not a lot. I can't imagine ever enjoying playing a game with content in it that I don't understand, for instance. In fact, there is ONLY ONE thing I can think of that is truly fun: watching stuff (stuff I enjoy, not stuff I watch out of obligation), with English subtitles. Everything else is a drag to some extent, and all you can do is manage the pain (making sure you don't burn out...and perhaps more importantly, making sure you don't let this one activity take over your whole life...because, frankly, doing nothing but studying languages makes for a pretty empty, meaningless life).
My secret is loooooong breaks between relatively short periods of intensive studying. For instance, with English, I spent a couple of months, one summer break, going through a big fat textbook I found on my uncle's book shelf. I don't even think I finished it all. It didn't teach me English, but that wasn't my goal for doing it...my goal was to allow me to breeze through English tests in high school without having to rely on the teacher. Then, a year went by where all I did was a few classes in school (not particularly useful classes, since no one was paying attention to whatever the incompetent teacher was mumbling about), and watching the Discovery Channel (with subs) and Cartoon Network (it didn't have subs, but it was really fun...this was back when Cartoon Network was good...they had stuff like Ed, Edd & Eddy, Cow & Chicken, The Powerpuff Girls, Dexter's Lab, Courage the Cowardly Dog, etc....you didn't need to understand every word to enjoy it...plus, my friends were watching the same shows, so they were a topic of conversation in school the next day). Then, I spent a few weeks next summer (intensive) reading Alice in Wonderland, and went right back to doing nothing for the next school year. Afterwards, the only other thing that involved effort, that I did, was go through a collection of philosophy essays, and a couple of Joyce novels. Since then, everything I've done in English has been purely for entertainment and relaxation (or for work, once I got fluent). I doubt I've spent more than 300 hours actually studying English.
Same with Japanese (except Japanese is about twice as hard, so double all the numbers from the previous paragraph): I did RtK, and a few hundred sentences (much like you, it seems), for about three months, and then took a long break of just watching stuff with subs. I would occasionally do small things, like go through some song lyrics, or rip audio from my favorite variety/comedy shows and listen to it repeatedly, half paying attention, but it was mostly just time spent with subtitled stuff, or Japanese in the background, while doing something else. Then, after a year or so, I SRSd RtK again (this time, the Light version, and really fast), and a couple thousand sentences, for a few months, and then went right back to not really studying for almost two years...but, in those two years, materials that weren't subtitled have also started becoming accessible to some extent. So it was a useful two years of not studying. More recently, I've set aside another three months, this time I SRSd through about 7-8 thousand audio sentences, for listening comprehension. I've spent about 120 hours/month either SRS-ing, or doing some other kind of actual studying (be it watching vids on Nihongomori, reading manga, song lyrics, blogs, radio show transcripts...I consider all that studying, because, even though it's not very difficult to do, it involves conscious mental effort and the use of willpower to keep your ass in the seat for a few hours/day).
But now I'm pretty much done with studying, and I'm at a point where I don't foresee having to put any more effort into Japanese...I expect that I'll just get good at it naturally. I can watch stuff on TV, I can even listen to easy radio, I understand most songs, so I'm all set to just rely on immersion. But, like I said: my method is mostly fun (with short intermissions of studying), but it takes a long time. It took me over five years each to get really good at a couple of European languages, and it's been four or five years since I learned my first Japanese word (it's going to be another 4-5 before I will be able to use Japanese the way I can use English for instance). If you want to get it done faster, you're going to have to work harder for it. And, if you intend to put even less than the give or take 750 hours of total study time I put into it, you should be prepared to accept that it's going to take you even longer before your Japanese is fully functional.