nest0r wrote:

a similar experiment.

Along the same lines as that experiment, apparently if you try to install it with Wine in Linux, it runs perfectly fine.  In theory, I mean.

Weird, both work for me. Good thing the txt was still there then I guess
The code <pre style="white-space:pre-wrap"> is what allows you to view it in a browser without it being wacky. Before I found out how to do that code I had to just suck it up, and even used a add-on that does word-wrap and used <pre> lol

Last edited by raseru (2010 February 14, 5:03 pm)

Just wanted to include here something from another thread. The topic was “essential”  kanji and vocab and what might be required for various types of reading.  iSoron kindly analyzed some of raseru’s texts and provided a detailed breakdown of the kanji and the vocab in each work.

iSoron wrote:

Maybe you'll find this useful?  http://isoron.org/stuff/japanese/analysis/

For the curious, I'm using Mecab plus a mess of bash, python and ruby scripts.

An article (“What Vocabulary Size is Needed to Read Unsimplified Texts for Pleasure?) recommends pre-learning words which occur frequently in a text as a good way to improve reading fluency. (As opposed to intensive reading to learn every word or pre-learning hundreds of words which occur only once or twice.)  Seems like common sense, but to actually create such vocab and frequency lists is beyond the ability of most readers. Thanks to iSoron, you have it!

Haha, just thought of a way and it works, but it's pretty ghetto. (if there's a better way, feel free to say it)

open the .html file in txt
press ctrl + h
Find : 《
replace : <!--
Find : 》
replace : -->

Would be even cooler if you could just make rikai-chan skip it though because occasionally readings are supposed to be something different

Edit: Doesn't work Rikai-chan won't skip past the invisible html
Edit: You can copy all of the text in the HTML and replace it in the txt then rikai-chan will work. Kind of a pain though

Last edited by raseru (2010 February 14, 7:55 pm)

Now someone needs to take 100 .txt files of light novels, run an analysis on the vocabulary, and sort the 95-98% most common vocabulary items and their kanji. Isn't that a good idea? I bet between us all, we could get a corpus of far more than 100.

Last edited by nest0r (2010 February 14, 7:58 pm)

《 doesn't seem to work in ctrl + h in that software

nest0r wrote:

Now someone needs to take 100 .txt files of light novels, run an analysis on the vocabulary, and sort the 95-98% most common vocabulary items and their kanji. Isn't that a good idea? I bet between us all, we could get a corpus of far more than 100.

One thing I noticed is authors often use the same vocabulary, so if you aim for maybe different authors as well, it might be more accurate

wccrawford wrote:

nest0r wrote:

Now someone needs to take 100 .txt files of light novels, run an analysis on the vocabulary, and sort the 95-98% most common vocabulary items and their kanji. Isn't that a good idea? I bet between us all, we could get a corpus of far more than 100.

Actually, I'd rather pick the 'easiest' ones and do that with them...  Getting into novels is the hard part.  Once you're in, the rest is cake.

I have to admit that analysis would be a lot better than the newspaper frequency list.  Who really reads newspapers any more?

I'm not sure what you mean. Aren't all light novels in the same basic range? I'm a n00b.

nest0r wrote:

wccrawford wrote:

nest0r wrote:

Now someone needs to take 100 .txt files of light novels, run an analysis on the vocabulary, and sort the 95-98% most common vocabulary items and their kanji. Isn't that a good idea? I bet between us all, we could get a corpus of far more than 100.

Actually, I'd rather pick the 'easiest' ones and do that with them...  Getting into novels is the hard part.  Once you're in, the rest is cake.

I have to admit that analysis would be a lot better than the newspaper frequency list.  Who really reads newspapers any more?

I'm not sure what you mean. Aren't all light novels in the same basic range? I'm a n00b.

Newspapers have really really stupid words that shouldn't exist like 日露. I saw one kanji in a newspaper before and asked 2 Japanese highschoolers to read it and they couldn't (鬨). Even with context and me telling them how to read it, they still didn't know the word

Last edited by raseru (2010 February 14, 8:10 pm)

raseru wrote:

nest0r wrote:

wccrawford wrote:

Actually, I'd rather pick the 'easiest' ones and do that with them...  Getting into novels is the hard part.  Once you're in, the rest is cake.

I have to admit that analysis would be a lot better than the newspaper frequency list.  Who really reads newspapers any more?

I'm not sure what you mean. Aren't all light novels in the same basic range? I'm a n00b.

Newspapers have really really stupid words that shouldn't exist like 日露. I saw one kanji in a newspaper before and asked 2 Japanese highschoolers to read it and they couldn't (鬨)

Oh, I was referring to the 'easiest' part of his comment. Well, I think I might already have like 50 .txt files to contribute<---That's what I would say if I were a pirate, which I'm not. So I definitely won't be posting any Google search suggestions to avoid later.

wccrawford wrote:

That analysis posted above shows that 1 uses 1500 kanji, 1 uses 2000, and the rest use 2500.  That's fairly different, with the lowest using only 3/5 the kanji of the highest.

But just like comics, there's low-end (Yotsuba&), middle (Death Note), and high-end (Read or Die).

Someone posted a nice link to a site that had a lot of easy light novels, and even had previews of them...  But I can't find it now and I lost my bookmark in a freak accident.

I see. Well as long as we preserve the breakdown of each work before doing a 'meta' analysis, it shouldn't be too much of a problem.

raseru wrote:

I wonder if that OCR program is even all that accurate though. Like even if it's correct 95% of the time, 5% of the time you'll just be super confused. I think some of the txt files were possibly marketed in that format and not OCR'd to begin with

I shall... experiment with it myself, when it "arrives" on my computer

Seems pretty accurate to me, but for my purposes it's more trouble than it's worth to correct the errors and tweak the formatting, etc. Actually I don't know, trying more samples, seems like it makes some really dumb errors. Like converts furigana to random words or something.

Last edited by nest0r (2010 February 14, 10:37 pm)

Holy crap that is a lot of stories, where ever did you find all of these? -- is what I would say if I was some kind of a pirate

... I.. Don't think I'll ever run out of txt books to read. Ever. o.o

Last edited by raseru (2010 February 15, 5:25 am)