lauri_ranta wrote:

A few weeks ago I downloaded Japanese subs for about 10000 anime episodes and used mecab to make a frequency list of words with at least one kanji:

for f in *.srt *.ass; do mecab -F'%t %f[6]\n' "$f" | sed -En 's/^2 (.+)/\1/p'; done | sort | uniq -c | sort -r

I wrote a Ruby script that removed words that didn't contain kanji (932/6000), and then words that weren't on the first 10000 lines of the frequency list (1436/5068).

A random sample of the 1436 words removed after the second step:

有害 harmful, hazardous
願書 application form
停留所 (bus) stop
物差し ruler
重役 executive
鰻 eel
お辞儀 bow
細長い long and thin, long and narrow
観賞 ornamental
時給 hourly wage

It's not perfect, because mecab doesn't recognize all words, and the corpus is too small and not diverse enough.

Anyway, you can download the list as TSV from http://lri.me/upload/core3632.txt. Most Anki decks are based on an older version of the data from iKnow's website. Audio files that match this version can be downloaded from this thread.

I won't make an Anki deck for it, but I'm thinking about making a huge vocabulary deck based on different word frequency lists later. Most of the top 20000 words have sound files from JapanesePod101 or iKnow.

By the way, do you happen to still have the unparsed list, with the Hiragana words and everything? I'm going through this list still and it's working a lot better then the core decks order, however I'm not getting any kana words, which is a pretty big chunk of the language.

Last edited by egoplant (2013 February 05, 7:47 am)

lauri_ranta wrote:

...

I recently made a list of hiragana-only words to review. (It's in rōmaji because I used it as input in a type training application.) There is also a version of EDICT with word frequencies indicated by number of Yahoo search hits. Here are all words tagged with on-mim (onomatopoeia / mimetic) sorted by frequency: http://lri.me/upload/edict-freq-on-mim.txt.

Thank you! ^___^

tashippy wrote:

how do i filter out the cards i have already learned/have reviews of in the first 8 decks. it should be about 1600 cards i think?

I'm not sure of the best way, but I can immediately think of two suggestions:

1) Use Anki duplication detection:

Import one deck into the other and have Anki flag duplicates. Then suspend the duplicates and remove the other deck.

2) Use MorphMan to suspend notes you've already learnt:

Have both decks in collection, run Morph Man. Suspend any i+0 facts (as they were already covered by your other decks).

overture2112 wrote:

tashippy wrote:

how do i filter out the cards i have already learned/have reviews of in the first 8 decks. it should be about 1600 cards i think?

I'm not sure of the best way, but I can immediately think of two suggestions:

1) Use Anki duplication detection:

Import one deck into the other and have Anki flag duplicates. Then suspend the duplicates and remove the other deck.

2) Use MorphMan to suspend notes you've already learnt:

Have both decks in collection, run Morph Man. Suspend any i+0 facts (as they were already covered by your other decks).

thanks. i might try using morphman. if i try your first suggestion, how do i do that? i see that finding duplicates is explained in the anki manual, but how do i combine the decks like you say? i guess what i was looking for is a list of sorts for each deck that i can just highlight multiple items and suspend or delete them. thanks

tashippy wrote:

if i try your first suggestion, how do i do that? i see that finding duplicates is explained in the anki manual, but how do i combine the decks like you say?

In Anki 2 I believe* you would do:
1) modify note type so the appropriate fields are marked as not being allowed to have duplicates
2) change the note type of the notes in one deck to the note type of the other deck, mapping fields as appropriate

* untested. always backup the collection file first before doing large changes like this, just in case

tashippy wrote:

i guess what i was looking for is a list of sorts for each deck that i can just highlight multiple items and suspend or delete them

Not sure what you mean by this. Based on what you wrote, it sounds like you're describing the card browser, but I'm guessing you intended some sort of 2-pane version or something?

Last edited by overture2112 (2013 February 12, 12:04 pm)

toshiromiballza wrote:

Are the sentences from Core 6000 licensed under any licence that needs to be mentioned when displaying them on your website? Or, worse yet, are they the "property" of smart.fm/iknow.jp and you are not allowed to reproduce them at all on your website, and thus all the Anki decks, etc. are "illegal"?

You can find out the details if you search on this forum, but I think it basically went like this (someone correct me if I'm wrong).

Core10k was produced by some company (I forget the name).  The first 6000 of those sentences were licensed by SmartFM.  SmartFM used to have a nice API that allowed those sentences to be farmed out.  They were put into the Anki deck we know as Core6k.

Then later some other company licensed all of Core10k and put it into some kind of computer software.  The software was reversed engineered (by someone on this forum) and the remaining 4k sentences were extracted and combined with Core6k to produce the the Anki deck we know as Core10k.

I think the Anki decks are technically infringing on the original company's copyright.

If you wanted to put them on your website you would probably need to license them from the original company that created them.

Last edited by partner55083777 (2013 February 13, 6:08 am)

toshiromiballza wrote:

Common Content (CC) - All of the material on iKnow is listed as common content.

I assumed this only applied for user created content on the site and not smart.fm's published content, but could be wrong.

partner55083777 wrote:

The software was reversed engineered (by someone on this forum) and the remaining 4k sentences were extracted and combined with Core6k to produce the the Anki deck we know as Core10k.

I think the Anki decks are technically infringing on the original company's copyright.

If you wanted to put them on your website you would probably need to license them from the original company that created them.

Correct. If you're using Core10k you should have purchased the jsensei app (only $14) and only share anki decks and the other materials with people who also own it (or at least use the honor system).

Core6K (the original 2000 + 4000) sentences was licensed by iKnow (smart.fm) under the Creative Commons. I'm not sure about 10k deck. The only thing that wasn't licensed like that was the images which they used a stock library for and couldn't sub license.

I remember checking up on this because the only thing of value for them was the sentences, which they gave away. Once you had the sentences what was the point of using their site, anyone could compete with them with their own data. Which is exactly what happened when everyone just used Anki.

I'm guessing they changed the license at some point, so any new additions, updates etc are not CC.