mezbup wrote:

That article was a great read, basically confirmed a lot of what we've been talking about.  A lot of the points I was making are in there. The thing I was most interested to read about was at which point a learner can tackle a novel and not have the level of difficulty be overwhelming.

Yes. It also points out that there's little point trying to look up or pre-learn every word b/c (a) many words appear only 1-2 times, and (b) you'd have to read 3-5 more novels to see half of those words again. Also, 2000 words (plus a few proper nouns) covers 95%  - which they equate to adequate comprehension and ability to guess meaning from context. For them, "pleasurable" is at 97-98% (5000 words). I can see some learners having a higher tolerance for unknowns (pleasure not necessarily equating with ease and complete comprehension.)

Isoron: wow!  That article recommends pre-learning only vocab which occurs frequently in a novel. Unfortunately, creating such a glossary requires computer analysis. Which you have done for several of them! Can we include your link in rasera's My study method thread?

Laroche: Interesting range of #. Textbooks seem easier to me than Japanese literature. I can't seem to shake this feeling that I'm only understanding stories at a most basic literal level and missing both the Big Ideas and the subtleties. I keep giving up ... 

Thora wrote:

Isoron: wow!  That article recommends pre-learning only vocab which occurs frequently in a novel. Unfortunately, creating such a glossary requires computer analysis. Which you have done for several of them! Can we include your link in rasera's My study method thread?

Sure.

BTW, I just updated the site with a more detailed breakdown of the vocab. For the curious, I'm using Mecab plus a mess of bash, python and ruby scripts.

Last edited by iSoron (2010 February 14, 5:55 pm)

Finally frequency lists based off of (light) novels instead of newspapers. Not sure what to do with them, though.

LaLoche wrote:

Reminds me of this article about the vocab needed to read a selection of first year general texts at university (in the Netherlands, but applicable, I think, to English) which came up with over 11,000 words!!
http://applij.oxfordjournals.org/cgi/gc … act%28s%29

There weren't enough grammar Nazi's derailing threads

http://forum.koohii.com/viewtopic.php?pid=93256#p93256

so ...

An Oxford Journal wrote:

This study aimed to answer the question of how many words of the Dutch language, and which words, and adult non-native speaker needs to know receptively in order to be able to understand first-year university reading materials In the first part of this study,

Have fun

Jarvik7 wrote:

yukamina wrote:

Finally frequency lists based off of (light) novels instead of newspapers. Not sure what to do with them, though.

Each list is only an individual novel. It'll be truly useful once applied to a corpus of them. As is I think it's only useful if you're reading one of the applicable novels. I've been planning on reading NHKにようこそ！ so I'm happy though. Too bad I only own the book and not a txt file though

Working on it. My imaginary friend already has a .txt of NHKetc.

@iSoron: Could you list which novels make up your corpus in the INFO?

Last edited by Jarvik7 (2010 February 15, 7:12 am)

Thanks.

It also might be useful to indicate how any times the word occurred across the entire corpus, instead of just counting how many novels it appeared in. It can give a more detailed picture of how common a word is (appeared 2 times in 11 novels vs appeared 500 times in 10 novels).

Last edited by Jarvik7 (2010 February 15, 7:21 am)

I've never done much statistical analysis using shell scripts, but you should be able to snip outliers like that. In any case, it could be in addition to the novel count.

iSoron wrote:

Jarvik7 wrote:

It also might be useful to indicate how any times the word occurred across the entire corpus, instead of just counting how many novels it appeared in. It can give a more detailed picture of how common a word is (appeared 2 times in 11 novels vs appeared 500 times in 10 novels).

Yep. I refrained from doing this at first because a single light novel could very easily skew the numbers. For example, 俺 appears almost 8000 times in the Suzumiya books, which is not typical at all, even for a light novel of its size.

Something raseru suggested was to avoid using more than 1 or (I think) a few volumes per author, that might prevent too much of a stylistic/authorial skew. From what's available in .txt form, even just 3 volumes per work/author would I think be at least 200 files? (For now... mwa ha ha...)

Last edited by nest0r (2010 February 15, 6:16 pm)

Really? Which ones are bad? I've been going through and comparing scans vs. OCR haphazardly but haven't found any errors. It's such a pain, having to check them. ;p

Last edited by nest0r (2010 February 16, 5:06 am)

Hmm, still can't find any errors, but I have noticed something that can be confusing, the way furigana is either bracketed or even given its own lines. Takes some getting used to I suppose.

Jarvik7 wrote:

The bracketed ones are aozora-bunko format, so I assume if you open it in an appropriate reader it will look pretty.

I never could get any of those readers and suchlike to work properly. Is there a program/addon thingy that turns those brackets back into ruby without a special reader?

And iSoron, dude, I don't know what you're talking about! If anything, the OCRs seem disturbingly good, as if some robot went through and hand-corrected everything page by page.

Last edited by nest0r (2010 February 16, 5:54 am)

nest0r wrote:

I never could get any of those readers and suchlike to work properly. Is there a program/addon thingy that turns those brackets back into ruby without a special reader?

This is what I use:

    http://isoron.org/stuff/japanese/aozora/aozora_to_html
    http://isoron.org/stuff/japanese/aozora/aozora_ruby.py

More scripts, sorry.

And iSoron, dude, I don't know what you're talking about! If anything, the OCRs seem disturbingly good, as if some robot went through and hand-corrected everything page by page.

I've seem some pretty bad ones, with page headers/numbering still there, for example. But well, nevermind. Will recheck later.