This can happen often in the first part of RtK, where the kanji are not too complex visually. Also when dealing with very frequent characters. I wouldn't worry about it.

If you are being halfway into the book however, I'd recommend to always try and recall at least an idea of what the story was about. Doing this recall makes the link between the kanji and its primitives. It can be counter productive in the long run, if you rely too much on visual memory.