Cranks Wrote:As you know by now, I like to tag stuff using Japanese's tag function. I find it really handy as I can email myself word lists from my Phone and add it to Anki.

Splatted Wrote:I've been thinking about something similar to this for a while now. I dream of a program that can go through a text and create a list of every word it contains, then compare this list to another list, such as an anki deck, and delete any repeated words. This would leave you with a list of the words you'll need to know to read the text. I think I might try doing it manually like you.

tokyostyle Wrote:

Splatted Wrote:I dream of a program that can go through a text and create a list of every word it contains, then compare this list to another list, such as an anki deck, and delete any repeated words. This would leave you with a list of the words you'll need to know to read the text.

Are there any libraries that can already split up Japanese sentences? That seems by far the hardest part and yet intriguing at the same time.

overture2112 Wrote:

Splatted Wrote:I've been thinking about something similar to this for a while now. I dream of a program that can go through a text and create a list of every word it contains, then compare this list to another list, such as an anki deck, and delete any repeated words. This would leave you with a list of the words you'll need to know to read the text. I think I might try doing it manually like you.

I started writing something like this using mecab so I could see what vocab from my Fate/Stay Night subs2srs deck wasn't already in core6k (and thus I'd need to add as supplemental vocab). It's actually quite easy to do this once you solve the issue of interfacing with mecab (which uses euc instead of unicode).

I have some unpolished code available on github here. There's a Haskell library for interfacing with mecab as well as some python code with an anki plugin.

The plugin (extractMorphemes.py) lets you select some cards in the browser and extract all the vocab in the Expression fields to a file (hardcoded to anki.morphdb) which is basically a compressed python dictionary mapping morphemes to their number of occurrences. The morphemes.py, when used as a standalone app and given 2 file paths will find morphemes in the first and save the db to the 2nd. There's also a few functions for comparing DBs (like a set difference and symmetric difference).

I'll try to get around to cleaning this up and releasing it properly at some point, but I welcome anyone to use/read/get inspiration from the code and beat me to it.

overture2112 Wrote:

Splatted Wrote:I　have a dream........

I started writing something like this etc...

I'll try to get around to cleaning this up and releasing it properly at some point, but I welcome anyone to use/read/get inspiration from the code and beat me to it.

Snarp Wrote:Here is a very, very rough Python script I wrote last year to extract vocabulary from a .txt file. You name your text file "nihongo.txt" and run the script from the same directory, and it prints the word list to "nihongovocab.txt".

The problem with it is that it requires the MeCab Python bindings. I was able to install them on Ubuntu without much trouble, but I get this error on Windows:

Code:

>python setup.py build
'mecab-config' is not recognized as an internal or external command,
operable program or batch file.
Traceback (most recent call last):
  File "setup.py", line 13, in <module>
    version = cmd1("mecab-config --version"),
  File "setup.py", line 7, in cmd1
    return os.popen(str).readlines()[0][:-1]
IndexError: list index out of range

Anyone with any idea how to fix this?

Cranks Wrote:As you know by now, I like to tag stuff using Japanese's tag function. I find it really handy as I can email myself word lists from my Phone and add it to Anki. This ensures that the words I use and learn from natives in my corrections on Lang-8 don't go forgotten and stuff I see in text is retained. What I didn't think about was: Why not do a once over of the text (say 20 pages at a time) just adding things to the word list function in Japanese without really checking what the words mean in any depth and then make a cram deck out of it. After that I can read the 20 pages again with less look ups and each week I can sort the lists and filter our anything that I would consider "generally usable in real conversation".