I'd like to discuss how we measure the level of things. Over in the Book Club thread it came up, and some people were talking about making a wiki page with books ranked by their reading level, so I think it's one of those useful conversations to have.

Two common systems of categorising come to mind - JLPT and Beginner-Intermediate-Advanced.

But if you say a book is N3, I'm not really sure what that means. Obviously I guess it means someone who could pass N3 could read the book - but what level of comprehension are we talking about? 98%+ (the 'comfortable' comprehension amount, as I understand)? 90% (the can-read-enjoyably-but-missing-stuff threshold). Or <90% with a dictionary/getting only the gist?*

Is there some kind of objective standard that the book could be compared against to declare it 'N3'? The official JLPT level summary page doesn't help much.

Beginner-Intermediate-Advanced is convenient, but very subjective, and as you get better at Japanese I think the goalposts shift. 'If I can read it easily, it's beginner. If I can't read it yet, it's advanced'. Plus there's a lot of ground covered in a label like 'beginner'. You can make it better by splitting it i.e. lower, middle, or upper beginner though. Nonetheless, it's pretty vague.

Or, we could stop looking at everything through the lens of outsiders and consider it how Japanese people would rate our reading level/book difficulty level.
But from my admittedly not very thorough search ... I'm not even sure if they do that. The English Wikipedia article for readability is massive and full of different methods and formulas for measuring a book's readability in an objective way. The Japanese version of the same article has pretty much nothing. I would have assumed it would be an important topic in Japan given how complex they reckon their language is.
I guess you could always tie ranking to the school system. Children's books often have difficulty rankings based on school level (小学中級から, for example). But given no one here's actually gone through the Japanese school system or has any way to measure what their school-relative level is, that comes rather close to just making things up.

From recent discussions here I think there are lots of things being put together in categories when they are plainly different levels (for instance: this and this might both be labeled 'beginner') - I'd guess that's a quick way to discourage reading. Being able to come up with a good, accurate list of which books match what ability level would make it much easier for people to find stuff to read.
So, is there any good way to properly match reading level with a book? What criteria can be used to judge a book's readability in Japanese, and how do we fit it into neat labels like 'beginner' in a consistent, useful way?


*Note that these percentages might be rubbish, but I saw them in a study on extensive reading somewhere.

One thing which has a great effect on reading level is the amount of furigana. I know of a few children's and YA books which were republished under a general (meant for adults) publishing label. The only difference between the versions was the republished versions dramatically reduced the amount of furigana to only rare words or words with uncommon kanji.

I'm not sure if there are any books written for native speakers that I'd be comfortable classing at a "beginner" level, outside maybe of board books for very young children that are basically just "Eggplant! Carrot! Spinach!"

The research on English books is that you need around 8,000-9,000 word families to read native texts at a 98% comprehension level (http://www.victoria.ac.nz/lals/about/sta...-vocab.pdf).

(There's research that 95% comprehension is not adequate for most students -- http://nflrc.hawaii.edu/rfl/PastIssues/r...ehchao.pdf -- I think that isn't always the case, but it's the best research I know of on the issue.)

If the same applies to Japanese, then you really have to be N1 level or very close to it to comfortably read most texts -- and I don't *think* that's true, I was certainly reading a lot of light novels when I was at about N2 level, despite breezing through a lot of words I didn't understand -- but I think it's true enough that it can easily create a plateau at the intermediate level where you need to read a lot to get better, but it's hard to find anything to read that isn't a total slog. (Pop-up dictionaries can mitigate that, but I think we REALLY need better learning materials around the N3 level.)

I suspect that the level and amount of vocabulary is the main factor in reading level, because virtually *all* texts for elementary school level and up are written for people who have native-level grammar skills, and (except for some fantasy novels that have a lot of made-up katakana words) the kanji difficulty of any given text probably relates pretty closely to the vocabulary difficulty.

So, if I had a word frequency list and text analysis software and could run all kinds of books through it, I would want to know

* For any given text, how many words would you need to know to achieve 98% comprehension?
* For any given text, out of the less common words (outside the most common 5000 words?), how many have kanji but no furigana?
* How many unique words are there in the text that are both uncommon *and* used only once or twice in the whole text? (My assumption is that it's easier to deal with a rare word that the author uses over and over, rather than fifty different rare words that the author only uses once each.)

The problem with doing this in a mechanical way is that it would tend to catch a lot of dialect stuff and casual speech that isn't actually rare or difficult but would show up as different words in text (すげー、　しちゃった, etc.) -- you could get around that to some degree by only counting words written in kanji, which are going to pose the biggest problems for second language learners anyway.

Outside of vocabulary difficulty, I suspect that the biggest obstacle to readability is how concrete or how abstract the text is, in style as well as content -- but that's hard to quantify.

I don't think English and Japanese are comparable here though. I'm not sure why it is, but I think if you take two similar children's books aimed at similar age group on a similar topic, the English one will be vastly more difficult and use a vastly larger number of distinct word families. Mind you that this is totally anecdotal and from my own experiences, but I can read most books aimed at nine year old Japanese girls just fine but English learners at a higher level than me can't easily read the books aimed at nine year old English girls that I've shown them (the first line of one of them: 'Innumerable swallows nested in the battlements of the old house and swooped and dived all day from May to October, skimming off the green lawn surface' ... Japanese children's books just don't pile on the words like that).

So, I don't think comparisons to English are applicable. I think you need much less, and I think that there are books of reasonable complexity aimed at native speakers that an actual beginner can read in Japanese (upper beginner, mind you - not that the line is clear, but with a vocabulary between 3-4000 words there are readable novels including the one that I scanned/linked the first page of in the OP, which I read at about that level reasonably comfortably)

I agree that vocabulary is the main limiting factor, but complexity of grammar varies massively too.
Looking at the readability Wikipedia page, a lot of the formulas used seem to adjust for this by assuming that long sentences are difficult. I wonder if this could just carry over to Japanese?

... actually, I guess I should have googled in English before posting. While I couldn't find anything in Japanese, it looks like this problem has been looked at before.

http://kotoba.nuee.nagoya-u.ac.jp/sc/obi...c-sato.pdf
http://www.ideosity.com/ourblog/post/ide...as#Hayashi

And, more directly useful:
http://sourceforge.net/projects/japanese...Tool_v5.0/
Which yes, I fully realise was developed by a member of this forum and I'm pretty sure I've used it before, and just forgot about it.

So I guess we have a perfectly good tool to measure readability if we're inclined to use it and think those measuring methods are trustworthy.
I think I've only read two of the innocent books (きまぐれロボット and 時をかける少女), but given that the obi score is meant to correlate with school grades, I think they're inflated a bit. きまぐれロボット is given as 6th grade when the publisher gives it as 小学中級, and 時をかける少女 is 8th grade when the publisher gives it as 小学上級 (furigana-full version). I thought both were easy and I don't think my reading level is that of a native 8th grader.

Still, if it inflates things consistently we could still get a useful grade-score from it. Would make it easy to classify things.
The data from the innocent books is here. Has anyone else read anything in the list? How does it match-up with your experience in reading the book?

And yeah, any rating should mention the amount of furigana coverage or factor it in somehow (which automatic tools can't do).

And another thing: How much of a book would you need to put into the tool to get a reasonable estimate of difficulty? Would a handful of random pages be sufficient to get an accurate result?

I think an algorithmic solution like JapaneseTextAnalysisTool_v5.0 would be preferred because the score is concrete and isn't open to interpretation. Otherwise you'd get arguments about what is considered lower intermediate or whatever.

The text analysis tool outputs hayashi and obi-2 readablity scores and also outputs a rating based on a user vocabulary list. Since most people on this forum studied the core vocabulary, why not list several different readablity scores: hayashi, obi-2, and total number of words outside core 6k. That combined with reader comments should give you a pretty good idea about readability and it would be easy to compare one book to another.

Since this topic seems to be based at least in part on the difficulty ratings I gave on the 'Book Club' topic, I'll weigh in with my thoughts.

When I rated those books by JLPT level, it was purely a subjective opinion based on the level I was at when I read them, and how difficult they seemed to me at the time. There was no scientific or quantitative basis to it.

I guess I am somewhat well qualified to make the statement, as I have taken all of the levels of the JLPT and I have also read a lot of books in Japanese (about 30). Unlike some of the people on this forum who race to N1 in a couple of years or less, skipping a lot of the levels, I progressed slowly and smoothly from nothing to 4級, 3級, 2級, N2 and then N1 with a few years between each level, so I remember what it was like to read (or attempt to read) books at each level.

In terms of percentage comprehension I have no idea, but all I meant by it was that when I was at that level, I was able to read the book and enjoy it. In a few cases I changed the level, for example I read ＺＯＯ 1 just recently but found it very quick and easy so I did not place it at N1 level. Conversely, I read 涼宮ハルヒの憂鬱 a long time ago, I think around N3 level, but I found it pretty tough at the time so I bumped it up to N2.

Regarding the whole beginner/intermediate/advanced, I agree that these labels are pretty meaningless. Learning Japanese is one of those things where the more you learn, the more you realise how much you don't know, so your perception of it changes a lot over time.

Also, bear in mind that saying ゼロの使い魔 is a "beginners book" (at least in my mind) means that it is a book for someone who is just beginning to read novels in Japanese, not a true beginner just starting to learn the language. Honestly, I think it is impossible to read any kind of Japanese novel prior to about N3, which for most people requires several years of study.

To put it into perspective, the first novel I ever read in Japanese was ハリーポッターと賢者の石, based on a recommendation on this forum. It took me about a year, and in retrospect it was not a good choice. I read ゼロの使い魔 several years later, but imho it was MUCH easier (I finished it in a couple of weeks), and I really wish that I had picked that as my first book. Or better yet, 君にしか聞こえない, which I thought was slightly easier still.

Regarding your comparisons to children's books, I have no experience of these. I did try a few times, but I could just never get into them. My personal recommendation is that until you are at a level to be able to ready "beginners" novels, you should read simple manga like 『よつばと!』, and maybe also some simple anime with exact subs, like 無人惑星サヴァイヴ (http://en.wikipedia.org/wiki/Uninhabited_Planet_Survive!). It all depends on what floats your boat, though.

Aikynaro Wrote:Really wondering about some of these other 'beginner' book suggestions though. I suspect people have either forgotten what beginner's level looks like or they read with a whole lot of automatic dictionary lookups.

This seems like the more appropriate place to reply to this comment, which I wanted to do because I'm one of the people who's pushed ゼロの使い魔 as a beginner book. Basically I agree with what NickT said: beginner in this context means beginning to read Japanese novels. Arguably there are children's books that would be more appropriate but I think there's a blurred line between those and learner material because those are books that are written to be easy, so imho it's better to recommend the easier adult/teenage books and leave it up to individuals to decide when they're ready.* For me that was quite early and I did use a popup dictionary to look up basically every word, but the obvious improvement was enough to make that feel worthwhile.

*Obviously I don't mean that people shouldn't be given recommendations for children's books, but when talking about native material I think it's appropriate to distinguish between simple language and simplified language.

Well, yeah, the thread is inspired by your ratings (which I don't have any real problem with, though I think stuff by 星 新一 and キノの旅 or ゼロの使い魔 shouldn't be put in the same level of difficulty). But for a long time before the Book Club thread I've been seeing really strange recommendations to beginner readers - 'you should read Chii's Sweet Home, Yotsuba, and Norwegian Wood'. It's like going to an English learner 'You should read Mr. Men, Clifford the Big Red Dog, and War and Peace'.

So mostly I'm just unhappy about how we talk about difficulty ratings in general, because it's making it much harder for people to get into reading than it needs to be. If we could just have one accepted way of rating things we could create a proper list that would let people start reading sooner.
I'd like a system that can stop dud recommendations like the one you got for Harry Potter, so we can definitively say 'this is easier than that'. With a proper standard of comparison you could have saved a year of effort.

ゼロの使い魔 and Harry Potter 1 have the same obi-2 rating though (6). Does that sound right to you?

Quote:Also, bear in mind that saying ゼロの使い魔 is a "beginners book" (at least in my mind) means that it is a book for someone who is just beginning to read novels in Japanese, not a true beginner just starting to learn the language. Honestly, I think it is impossible to read any kind of Japanese novel prior to about N3, which for most people requires several years of study.

Fair enough. I'm not sure where to draw the line between 'beginner' and 'intermediate' honestly, but at the time I self-identified as 'upper beginner', I found novels that I could read. Despite having taken N3, I still only have the vaguest idea of what 'N3 level' is though - I don't think I was at N3 level when I was reading those, but I might be wrong.



Also, though I'm not sure this is a good thread to debate the matter...
I feel that children's novels are unfairly targeted.
A novel is a novel, and native material is native material. There is no blurred line between a novel written to entertain children and a book written to educate adults. It's not as if they were written then simplified - it is just simple language.

After all, よつばと is a children's manga. 無人惑星サバイブ is a children's anime. I'm not sure why those seem to be treated on an even level to other things of their medium but children's novels are not.
(well, I'd suspect it's because hardly anyone actually tries reading them and don't realise how varied and interesting they can be - but it seems like such a strange double standard)

I thought about this a little bit after seeing the difference between 君にしか聞こえない and All You Need is Kill.

Seems that 'vocab density' might give a reasonable measure of relative difficulty, i.e. something like

number of different words / total number of words

where what counts as a 'word' doesn't matter too much provided it's consistent across the comparison. Easy to do with Mecab for someone who can be bothered

Aikynaro Wrote:Well, yeah, the thread is inspired by your ratings (which I don't have any real problem with, though I think stuff by 星 新一 and キノの旅 or ゼロの使い魔 shouldn't be put in the same level of difficulty).

You might be right. But, I still maintain that ゼロの使い魔 is one of the easiest novels I have read. I've read easier things, but they probably didn't count as novels.... キノの旅 is somewhere between a light novel and short stories, and 星 新一 barely even counts as short stories, as many of his stories are only a few pages long. The 総ルビ versions of his books in particular are more akin to children's books, but if you get the adult versions with hundreds of stories crammed in a longer book with no furigana, they are actually not so easy.

Aikynaro Wrote:I'd like a system that can stop dud recommendations like the one you got for Harry Potter, so we can definitively say 'this is easier than that'. With a proper standard of comparison you could have saved a year of effort.

Actually it wasn't such a bad recommendation, it is just quite a long book. Also this was pre-Kindle days, so trawled through a paper copy of the book and looked up literally every word I didn't know in a dictionary. Nowadays I am more comfortable skipping over things I don't understand - I prefer 90% comprehension with a high reading speed than 100% comprehension painfully slowly. Just tapping a word and getting the reading and definition also speeds things up no end.

Another factor is that people recommended it on the basis of "you already know the story, so it is easier." I've never read it in English (or watched it), so I didn't know the story at all, which made it harder I guess.

Aikynaro Wrote:ゼロの使い魔 and Harry Potter 1 have the same obi-2 rating though (6). Does that sound right to you?

It doesn't sound right to me, but its hard to be sure. I think HP is about twice as long as ZnT which makes a big difference. Also, ZnT was written originally in Japanese which helps, as it doesn't have to think of translations of made-up words that don't even exist in English like in HP.

But then again, it may just be because I read ZnT on a Kindle, and inbetween the two books I read about 15 others, and passed N2, so I was in a different place I guess.

Aikynaro Wrote:After all, よつばと is a children's manga. 無人惑星サバイブ is a children's anime. I'm not sure why those seem to be treated on an even level to other things of their medium but children's novels are not.

Yeah I thought that as I wrote it. I'm not sure why, but there is just something charming about よつばと that appeals to adults. I'm not even sure if it would appeal to kids in the same way, although it is ostensibly a children's manga. I showed it to my niece once and she didn't seem that interested.

Not sure if anyone else liked 無人惑星サバイブ, or if its just me, but I thought I would throw it out there as a recommendation. Generally, I do steer clear of kids stuff including kids manga and kids anime, but this was one that I didn't find too offensive.

NOTE: The following is mostly a tirade, because someone implied books for 'children' aren't as useful as books for 'adults'. Otherwise, I think the idea to assign levels based on the results of a text analysis would be good (I also like the idea to say how many words there'll be that aren't in Core).

I agree that dismissing materials based on demographic's age is strange. Especially when talking about children's books (what do you consider a 'child'? 5-10? Younger than 16? Younger than 18?); as I've mentioned before, I've read numerous books aimed at younger demographics (usually at a level that reminds me of what my favorite bookstore considered 9-12: which also housed the Harry Potter books and Inheritance books (Eragon and it's sequels)) and while they might be 'easier' I think it's more that the writers use less redundant noun and adjective phrases; other than that, the writing (and even the stories) are quite mature.

ゼロの使い魔 is aimed at a similar demographic (around 12, best I can tell; so it's mainly for elementary or middle school students), so we can conclude that, unless they are 'new reader' books, you must consider them acceptable. However, even 'new reader' books (which is a poor description, considering they cover a large range of reading ability), depending on the writer, can be enjoyable (if you like what's there, of course), as demographic is based on the publishing house (thus, books with psychotic cats murdering each other go with books about girls who like to ride horses).

If a wiki is to be created, all books should be treated equally (until you get to books containing a picture per page and one or two lines of text). I think that anything that's too simplified would be easily noticeable to people looking at the page; not to mention, these books would only be put up there if someone thought it worthy of recommendation.

Quote:Not sure if anyone else liked 無人惑星サバイブ, or if its just me, but I thought I would throw it out there as a recommendation. Generally, I do steer clear of kids stuff including kids manga and kids anime, but this was one that I didn't find too offensive.

I actually really like 無人惑星サバイブ. I watched the whole thing in a four day marathon and have subs2srsed about twenty episodes of it.
If you steer clear of children's anime, manga, novels, etc. - it only stands to reason that you won't find much of the good stuff. The bad stuff is much more visible than the good stuff - I think because a lot of the bad stuff is created by soulless corporate types with an excellent marketing budget, while the good stuff largely gets buried. And of course 90% of everything is crap.
A lot of NHK-funded anime is both for kids and really good though. Almost everything else is extended toy commercials so it's no surprise that it's almost entirely horrible.

anotherjohn:
Sounds like a reasonable metric.　I could probably write a script to calculate that from the output of the text analysis tool (or edit the text analysis tool itself, I guess, would be more sensible ... but anyway).
If I can be bothered. We'll see.