Thora Wrote:A kanji order based on how they function in the language might have some advantages in the long run over an order based entirely on consistency of pace or difficulty. There are other ways to manage that. Stats and graphs sometimes miss how we actually absorb language. Fuzzy non-linear plots aren't always a bad thing.

At some other phases of the learning process I'd agree but where I'm right now consistency is the best approach for me. Learning is fun but when I get a difficult sentence introducing a simple and common kanji it really ticks me off. I get distracted this way, thinking about how stupid you have to be to think its a good example etc., doesn't help with my motivation either.

Thora Wrote:KinC is designed for advanced beg/low inter, I think. Even if the sentences are too difficult for you now, you might find the order of the kanji useful.

Thats exactly what I'm trying to avoid, getting too hard sentences before I get good enough for them. I don't subscribe to the whole sink-or-swim attitude, at least not if its about learning languages. I'd even dare to call AJATT sub-optimal with its "immersion" since day one advice. I think time is better spent working on the basics rather than bombarding your brain with useless information (for now, since your brain can't process it). If it had worked that way we'd all see thousands of anime freaks speaking perfect Japanese

nrg1701 Wrote:Wow, that would be extremely useful for Kanji in Context as well if the same method could be applied.

you can use it on any collection of sentences or words. something like iknow core which lacks a meaningful order would benefit most I think

Nukemarine Wrote:

nrg1701 Wrote:When doing the conversion, can you save reviewed card data by doing this somehow?

Probably not, but it's no stretch to export the unseen cards as .tsv file, sort just those, delete the old order then import back the new ones into Anki.

yeah you'd have to do it like that for now...

Nukemarine Wrote:The one failing of Anki has been the inability to bulk add new information to cards and reviewing cards in a user dictated order (currently limited to tags or the order of import). Kind of odd seeing that a "fact id" field could be utilized which let's anki know the imported data are either additions or replacement to fields of already existing facts. However, I'm not a programmer, so this could be more difficult than I'm making it sound.

...the overwrite-fields plugin can add new info to cards but not change the order. I was thinking of replacing it with something else that uses fact ids and could add and delete cards and change the order as well as modifying fields, but haven't gotten around to it yet

thurd Wrote:

cangy Wrote:you can't redirect into a file under windows? I guess you could use cygwin...

It might be possible but I don't know how to do this Also after seeing the results printed under command line console (random kanji instead of a proper one) I decided to use another approach.

see the example command line comment in the code. if you were seeing strange output it's probably just a lack of unicode support in the terminal

I've been mucking around for a while and can't get it to work. I've never really used any scripts before that weren't self-executable and I'm a bit lost.

I was wondering if someone could provide either STEP-BY-STEP instuctions... or upload a sorted anki deck.

Sorry to be a pain, just can't justify wasting any more time trying to work this out myself.

^Me too. I'm getting errors with the script.

EDIT: Fixed it. I think. Just to confirm, for those of you that managed to get it working, were the first 5 sentences the following?

今日の新聞はあのテーブルの上にある。
子供が線路で遊んでいて、電車にひかれた。
スーザンさんは子供が２人いる。上は男の子で５歳、下は女の子で２歳だ。
毎日庭の木に水をやる。
家の近所にテニスコートがある。時々近所の人とテニスをする。

EDIT2: Nevermind, I didn't get it working. The error message is:
>perl kanji-sort-1.4.pl kanji-freq.txt full.txt
Use of uninitialized value #sentence_field in list slice at kanji-sort-1.4.pl line 53, <> line 3107.

Nii87 Wrote:^Me too. I'm getting errors with the script.

If you use the original script by Fugounashi and want to sort the file "full.txt" by the first entry in the line in order of "kanji-freq.txt" and store the output in "output.txt", you would use:

Code:

perl kanji-sort-1.4.pl --kanji kanji-freq.txt --sentence-field 0 full.txt > output.txt

You can also just put all the files (full.txt, kanji-freq.txt, ...) in a single zip file, upload them somewhere, send me the link and I'll sort them for you.

SammyB Wrote:or upload a sorted anki deck.

I can upload a sorted Anki deck, if you give me the unsorted one and tell me which file to use to order it (e.g. 2001KO order, kanji freq, RTK order, ...)

If you are using my version of the script it should be :

perl kanji-sort-1.4.pl --kanji kanji-freq.txt --sentence-field 0 mydeck.txt

where :
kanji-sort-1.4.pl - name of this script
--kanji - argument indicator for the file with sorted (by previous script so frequency based) kanji
kanji-freq.txt - name of the file with kanji frequency
--sentence-field - argument indicator for the field in which the script should look for a sentence
0 -value for sentence-field argument, this means that in the deck export file, sentence field (fields are separated by tabs) is the first field, if its second then it should be 1 etc.
mydeck.txt - name of the deck export file you made, be sure to check out in which field your sentences are (if you have problems with the file looking "weird" open it in a real text editor like notepad++)

Hope this helps.

SammyB at what point are you stuck? Did you export your deck correctly? Do you have kanji frequency file?

cangy Wrote:...the overwrite-fields plugin can add new info to cards but not change the order. I was thinking of replacing it with something else that uses fact ids and could add and delete cards and change the order as well as modifying fields, but haven't gotten around to it yet

You can however use the command line tool "sqlite3" to change the creation time of the facts directly and e.g. update it based on a specific field of that fact. They'll just let Anki recreate all cards and they should be in the new order.

thurd Wrote:If you are using my version of the script it should be :

Rapidshare deleted it. So most people won't be able to use it. You need a premium or collector's account to store files permanently on Rapidshare. The collector's one is free, so just create one and reupload the script.

Pretty sure I exported deck correctly... It is called "unsorted" and the first lines look like:

<img src="569ca883fc9637469de306aafc9060cb.gif" /> This bag costs 15,000 yen. <img src="f5c500a3428496fb172dea7e0921c66b.gif" /> <img src="1b4592c6ee35005c4fa31870b1d51f21.gif" /> 16 13 12 16 [sound:1a066758bad8547b5882f013888fd69e.mp3] このかばんは１万５千円です。 このかばんは１まん５せんえんです。
<img src="8a65717d8b6dda27e29cba96237bb6a8.gif" /> $1 is now 105 yen. It is 1 yen higher than yesterday. <img src="749e536bfb4e6a0b009eb63e716fa0b9.gif" /> <img src="f32abd33f3ce1cad4dc0a3064f321391.gif" /> 16 16 32 90 19 101 [sound:a9a35de2838ff9a1987d6e84d63ac64d.mp3] 円は今１ドル１０５円。昨日より１円円高だ。 えんはいま１ドル１０５えん。きのうより１えんえんだかだ。

etc.

So if I want script to sort this file by the KanjiText field, which appears to be the 8th field I would do the following:

perl kanji-sort-1.4.pl -- kanji kanji-freq.txt -- sentence-field 7 unsorted.txt

is this correct so far?

I've tried that and keep getting this error:

Use of uninitialized value $sentence_field in list slice at [location of script] line 33


EDIT: I'm using windows...

SammyB Wrote:So if I want script to sort this file by the KanjiText field, which appears to be the 8th field I would do the following:
perl kanji-sort-1.4.pl -- kanji kanji-freq.txt -- sentence-field 7 unsorted.txt

No space after "--".

Code:

perl kanji-sort-1.4.pl --kanji kanji-freq.txt --sentence-field 7 unsorted.txt

Tried that, still getting the same error...

SammyB Wrote:Tried that, still getting the same error...

Due to a lot of questions and problems about the process I decided to add a link to sorted deck export on RevTK Wiki. Just use import in Anki.

Original article was already pointing to a copy of KO2001 deck, so this can't infringe any more than that. From my point of view you can't patent a sentence or even a set of sentences, KO2001 however is more of a learning method and since sorted sentences don't use the method anymore I don't think we're infringing anything.

Of course if higher power (Koohii & RevTK admins) doesn't share my views, please remove any links you see fit (and don't ban me ).

Hooray! I think I got it working. Are the first 5 sentences meant to be:

家から会社まで１時間。いつも８時に家を出る。
中に入る時、IDカードを見せなければならない。
電車は１１時４５分に出る。あと１０分しかない。
ケーキを作る時、オーブンを使う
会社に行く時、山手通りを通る。

Normally I would download the deck and check for myself, but rapidshare isn't being very nice today.

Nii87 Wrote:Hooray! I think I got it working. Are the first 5 sentences meant to be:

家から会社まで１時間。いつも８時に家を出る。
中に入る時、IDカードを見せなければならない。
電車は１１時４５分に出る。あと１０分しかない。
ケーキを作る時、オーブンを使う
会社に行く時、山手通りを通る。

Normally I would download the deck and check for myself, but rapidshare isn't being very nice today.

Yes it should be ok, I got the same sentences.

I'm not far into the deck (around 100) but I find this order much better. Beginning is smother and less frustrating, I can focus more on vocabulary and learning new readings for kanji I already know some.

thurd, can you also upload the deck somewhere else? Rapidshare only says "Currently a lot of users are downloading files. Please try again in 2 minutes or become a Premium member". Did you use the old text based deck as the input or the Google Docs sheet?

xaarg Wrote:thurd, can you also upload the deck somewhere else? Rapidshare only says "Currently a lot of users are downloading files. Please try again in 2 minutes or become a Premium member". Did you use the old text based deck as the input or the Google Docs sheet?

Indeed, I REALLY want this deck, but I'm not allowed to download it. Can you upload to megaupload.com, or another similar site??

Cheers!

xaarg Wrote:thurd, can you also upload the deck somewhere else? Rapidshare only says "Currently a lot of users are downloading files. Please try again in 2 minutes or become a Premium member". Did you use the old text based deck as the input or the Google Docs sheet?

Additional link:
http://uploading.com/files/e3a1364b/sort...ences.zip/
this could vanish soon since I didn't register but should be enough.

Tried uploading a torrent to mininova but they deleted it very quickly.

I used the old text based deck, it has some errors but its good enough for this kind of thing. If you find a hard sentence and its transcription is incorrect then don't enable this card yet.

http://www.multiupload.com/VZED3ZN86R
http://www.2shared.com/file/9185298/ae1f...ences.html
http://www.mediafire.com/download.php?nmiwkinkz0y
http://www.sendspace.com/file/i64exd
http://www.turboupload.com/5g66adgm1ycm/...s.zip.html

If anybody is interested in using this technique on a deck that you've already started, here's a method. Step-by-step below, but the idea is simple. Create the kanji frequency using the full deck, reorder only suspended cards using the full deck's kanji frequency.

I've tried this and quickly compared KO2001's reordered cards and the second book only. All the first cards are kanji that are seen in the first book, so somebody using this after they're done with the first book only will find the first sentences very easy, until eventually it'll match fairly well towards the end.

To go a step further I think using the new frequency orders, but forcing kanji you've already seen first, would be even better. For example there's about 800 kanji seen in the first book, so the new order for the second book (suspended cards) would be the cards that contain the kanji you've already seen first (sorted using frequency). Afterwards it can continue along the frequency path.

If anybody is interested I can probably write something up pretty quickly. Just remind me when I'm procrastinating my SRS reviews, and thus sleep.

1. Backup your deck, just in case.
2. Export the whole deck as "Facts in tab-separated text file"
3. Run the kanji-freq-1.1 script to create the new kanji-freq text file

4. Filter by "Suspended"
5. Select All
6. Add tags, and name them something that's not already being used.
7. Export and limit to the new tag
8. Run the kanji-sort-1.4 script
(I haven't actually done this to a real deck, but the rest shouldn't have problems)
9. Delete all the suspended cards
10. Import the deck from the new ordered text file

Sorry if this was all common sense, or there are mistakes.

Although I'm still uncomfortable with openly putting 2001KO sentences on share sites (Coscom did create and compile the sentences after all), the overall idea has been great. I'd be much more at ease had the iKnow Core 2000 and Core 6000 Decks been used since they are common creative license.

Anyway, guess I'll try to figure out how to load up the script and try this out on Core 2000 and Core 6000. Thing is though, while iKnow are group of sample sentences that contain vocabulary words, the Core are group of vocabulary that are demonstrated in the sentences.

Goes back to my original plan: With exception of the first 400 words, sort the Core words by kanji frequency, with the kana only words evenly spaced through out in original iKnow order.

Course, it leaves the weakness that some sentences will introduce kanji early on. A graph plot can show if it's a very bad situation or not.

Nukemarine Wrote:Thing is though, while iKnow are group of sample sentences that contain vocabulary words, the Core are group of vocabulary that are demonstrated in the sentences.

That's why I don't really like learning from iKnow. The voice acting was much much better, but the sentences didn't seem to build up gradually. Hopefully this sorting thing shown in this thread will fix it though, I'll try it tonight. If that is the case, I'll use iKnow instead.

Though it sucks that I've already done about 1000 cards of KO2001 and will have restart since I've sorted them =/

Nii87 Wrote:Though it sucks that I've already done about 1000 cards of KO2001 and will have restart since I've sorted them =/

Did what I posted above not work, or is there need or additional scripts? It shouldn't be even that hard to sort the KO2001 audio cards too, because you can match the English with the text sorted cards with some scripting (or maybe some excel magic).

^Great idea! Sorry, I missed your post. Now I don't have to restart KO2001! =)

thurd Wrote:3. Create a file with kanji frequency based on KO2001 sentences. Zip file already contains kanji frequency file so this step is optional but if you modified/corrected your deck you might want to run kanji-freq-1.1.pl script again so the list is updated.

if you do this you'll need to strip the 2nd column in the frequency output as the other scripts expect the kanji list to contain kanji only

thurd Wrote:5. Import sorted-sentences.txt into Anki adding all columns (14 or so in my case) naming them in the order you've written before. You should have 2 more columns than previously, you can discard them.

one of the extra columns gives the highest kanji index used in that sentence

if you sorted on a known order such as ko2001 or rtk, that might be useful when going through the cards for the first time in conjunction with the corresponding book as it'll tell you where you need to be up to to handle that sentence

if you sorted by some more arbitrary order such as frequency, then you can use the kanji-count script instead to give a list of new kanji introduced in that sentence


btw, I've put the kanji-graph script up now, but it's kind of nasty...

here are the other 2 kinds of graphs it produces

comparison of sort orders on rate of new kanji introduction (intrko2001 is the same as none, i.e. unsorted, and the ko2001 kanji (but not sentence) order actually looks a little better than frequency order)



frequency distribution



[edit: typo]

Grinkers Wrote:1. Backup your deck, just in case.
2. Export the whole deck as "Facts in tab-separated text file"
3. Run the kanji-freq-1.1 script to create the new kanji-freq text file

4. Filter by "Suspended"
5. Select All
6. Add tags, and name them something that's not already being used.
7. Export and limit to the new tag
8. Run the kanji-sort-1.4 script
(I haven't actually done this to a real deck, but the rest shouldn't have problems)
9. Delete all the suspended cards
10. Import the deck from the new ordered text file

This is maybe simple for most people on this forum, but I'm lost here - I don't understand "run a script". Looking on the internet I get a mess of stuff. Someone describes it as -

"A script is a list of commands that can be executed without user interaction. A script language is a simple programming language with which you can write scripts."

Still clueless how to do this. Seems to me all the instructions here are aimed at people who understand computers. Obviously I don't. I just want to make my existing deck be more efficient. Sorry for the moronic post....I just get lost trying to figure out tech stuff, when all I want to do is use it for studies in the least complicated way (yes I'm lazy when it comes to this stuff). Googling just brings up a sea of peoples questions about a million different situations....uhg.

I have an anki deck that is the KO2k1 smartfm version. As I'm already into the process, I would like to try to reorder, based on Nii87's testimonial on another thread.

I'm a Mac user. I know how to turn the thing on, use it for what I need, and that's about it. I realize I've opened myself up to ridicule by those who are "in the know"....

I would appreciate any help anyone is willing to offer my newbie ass.

How to run a script depends on what kind of script it is. If it's a python script, you have to have python installed. If it's a ruby script, you need to have Ruby installed. If it's a PHP script, it is run on a server when you connect. If it's a shell script, you run it like a program from your OS X command line.