Is this different from our 3-page thread here? Kanji Frequency in Wikipedia

With additions from Katsuo: http://forum.koohii.com/viewtopic.php?pid=57575#p57575

Not for my sake but for others, I hope someone soon runs an analysis on all those .txt files of novels I didn't post here: http://forum.koohii.com/viewtopic.php?id=5252

Nice jisho.org links. I'll leave all this fancy analysis stuff to you eggheads. ;p

Last edited by nest0r (2010 February 20, 5:40 pm)

I never actually looked at the previously posted lists, but now that I look at yours and theirs, a layperson like me really enjoys your presentation, so there's my expert opinion. ^_^

Also, I made this tool a while back to figure out your "Kanji coverage Percent" - http://answercowapp.appspot.com/app/kanjicoverage/index

It is based off the Kanji frequencies in Wikipedia.

I actually strip out most of the boilerplate code, Wikipedia conveniently puts tags around "content" and that's what I grab. There might be a couple of minor things like that left to filter though, but the vast majority of stuff you see in navigation bars and the like is gone.

I have also just finished running my scripts on the novels that nest0r did not post. Just needed to make a little tweak to work with shift-jis encoding. I believe it was a couple of gigs, so sample size is good. Here are the results:

Report table: http://foosoft.net/japanese/kanji_frequ … eport.html
Raw frequency CSV: http://foosoft.net/japanese/kanji_frequ … /novel/raw

Last edited by FooSoft (2010 February 20, 8:16 pm)

By the way, we discussed this one before, but here's a couple novel word frequency lists: http://pomax.nihongoresources.com/index … 1222520260 - I never used them because I didn't know what texts they came from.

And also here's a list taken from Goo blogs:

"Another frequency list was created in 2008 by Hiroshi Utsumi using the blog pages on the Goo site in Japan. His explanation is here.

    *edict-freq-20081002.tar.gz (3952168 bytes) archive containing the marked-up EDICT file, plus the Ruby code used to generate the markup. "

Last edited by nest0r (2010 April 12, 4:33 pm)

Would there be anyway to run a search of, say, the German wiki to find the most common words, for SRS? Or say contrasting the entire wiki with a word list of 5,000 common words, and determine which wiki entries use most of them?

killeralgae wrote:

Would there be anyway to run a search of, say, the German wiki to find the most common words, for SRS? Or say contrasting the entire wiki with a word list of 5,000 common words, and determine which wiki entries use most of them?

Of course. It would take a little bit more work, since with 漢字 you can take some shortcuts (like you know they won't show up in HTML tags, so you don't have to handle that), but yeah it would be cool to do such analysis on different versions of wikipedia.

Words are harder because you need to be able to figure out where one ends and another begins. Having no spaces can be annoying like that

Edit: I'm sure it's possible, I'll just have to think about how to do it first though

Last edited by FooSoft (2011 January 22, 10:05 pm)

With running a kanji frequency on wikipedia, wouldn't the thousands of articles on all things Chinese kind of throw off the results with a whole lot of kanji not really recognized in Japanese?

Womacks23 wrote:

With running a kanji frequency on wikipedia, wouldn't the thousands of articles on all things Chinese kind of throw off the results with a whole lot of kanji not really recognized in Japanese?

They would be greatly outnumbered by all of the kanji used normally in article text and end up as just noise. Remember that this is an exercise in statistics.

How about if you can add list of (all) 常用2009 kanji. Would be nice to see those missing ones.