The hardest thing about language
is probably vocabulary, like yogert, said but its not really the memorizing part.
The really hard part about language is basically knowing when to use words and how to phrase things. You can know all the words and know how to build grammatically correct sentences which people will understand. But if you only ever learned from a book, you'll probably discover that, if people are being frank with you, how you said something "sounded strange" or "wasn't quite how I would say it." Every language has this and its only something that's gained through years of experience.
One example for Japanese is knowing the right "tone" of words to use in a conversation. Kind of like the difference between saying something is cool or nice vs splendid. This word usage changes how stiff something sounds. There's also sentence layout which you have to think about with Japanese since you can
technically move most parts of a sentence around so long as you keep the verb at the end [most of the time]. If you do that though, don't be surprised if some Japanese don't just go "WTF?" Pragmatics (in the linguistic sense) play a huge part in Japanese, which means you can only really learn many of these things through experience/exposure.
The DBJG has a good section about general word order rules though and has a nice chart.
The lower the number, the more toward the beginning of the sentence that noun phrase(NP) should go.