KO 2001 Kanji/Word progression

KO 2001 Kanji/Word progression - Printable Version

+- kanji koohii FORUM (http://forum.koohii.com)
+-- Forum: Learning Japanese (http://forum.koohii.com/forum-4.html)
+--- Forum: Learning resources (http://forum.koohii.com/forum-9.html)
+--- Thread: KO 2001 Kanji/Word progression (/thread-4371.html)

Pages: 1 2 3 4 5 6

KO 2001 Kanji/Word progression - thurd - 2009-11-10

I've seen some statistics about KO2001 here but I've never seen anything about how many new kanji/words gets added with each sentence.

What I want to find out is there a tipping point where you start learning considerably less new things than before on this specific set of data. Everyone is saying that KO gets easier later on and its natural but what I'm interested in is where this point is: 200 sentences, 200 kanji etc. That way we'd know when to speed up our pace and learn more optimally.

There are lots of excel files with all sentences but I fail to find any macro/script that can count words/kanji in a cell, let alone "remember" them and compare with any new word.

KO 2001 Kanji/Word progression - Rooboy - 2009-11-10

I think you're looking at it the wrong way a little. Just let your speed naturally increase - or to put it better let the number of sentences you do per hour naturally increase.

Don't try analyse it. Do you have a magic switch that makes your brain go faster? Just go with the flow. I think you'd be better off setting time goals for study rather than try do a set amount of sentences/reviews (if that is what you are doing).

Time spent looking for a macro/script would have been better spent doing some more reviews Smile

(juat as i should be doing now)

To answer the question though there is no spot you can pinpoint. Its more a curve. It just gets progressively easier as the readings/kanji you learn to begin with are the more common ones. As a result they occur frequently in later sentences.

KO 2001 Kanji/Word progression - mezbup - 2009-11-10

Agreed. No amount of analysis will get you through the first part any easier.

I found it was pretty much the same all the way through cos I knew enough vocab from the start that it wasn't all coming down on me like a tonne of bricks. Guess it's just what you're level is although of course it's going to be easier toward the end but that doesn't mean it's "easy" at the end, it's just as hard as learning a new word will always be.

KO 2001 Kanji/Word progression - cangy - 2009-11-11

red 'x' is first time a kanji is seen, green '+' are other occurrences, x-axis is sentence index number, y-axis is kanji index number, blue line is cumulative number of kanji seen (so slope of blue line is rate of introduction of new kanji).

this is default ko2001 order:

[Image: ContribFugounashi?action=AttachFile&do=g...mall-2.png]

this is with sentences sorted by ko2001 frequency order:

click for larger versions

[edit: added new titles to graphs]

KO 2001 Kanji/Word progression - thurd - 2009-11-11

Rooboy Wrote:I think you're looking at it the wrong way a little. Just let your speed naturally increase - or to put it better let the number of sentences you do per hour naturally increase.

That wouldn't satisfy my hunger for statistics and graphs Smile

Also such analysis provides me with another mini-breakthrough which helps my motivation.

mezbup Wrote:I found it was pretty much the same all the way through cos I knew enough vocab from the start that it wasn't all coming down on me like a tonne of bricks. Guess it's just what you're level is although of course it's going to be easier toward the end but that doesn't mean it's "easy" at the end, it's just as hard as learning a new word will always be.

Sure but while learning 1-2 words in a sentence (while reinforcing everything else) is ok, trying to learn 6-8 words + a couple of new kanji is just not.

Thats why I want to know for how long I should move at mediocre pace (give myself more time to digest lots of new information) and when to shift to another gear.

cangy Wrote:red 'x' is first time a kanji is seen, green '+' are other occurrences, x-axis is sentence index number, y-axis is kanji index number, blue line is cumulative number of kanji seen (so slope of blue line is rate of introduction of new kanji).

this is default ko2001 order [...]

this is with sentences sorted by ko2001 frequency order

Thank you very much Cangy, thats exactly what I was looking for!!! But can you explain why there are two sorting orders? I thought default ko2001 was already sorted by frequency.

KO 2001 Kanji/Word progression - mezbup - 2009-11-11

There are points dotted throughout KO that won't exactly be apparent on that graph where you will encounter sentences in which there are 3 or 4 new kanji and 6 new words. I encountered it a number of times.

Just roll with the punches.

KO 2001 Kanji/Word progression - Nukemarine - 2009-11-11

thurd Wrote:
cangy Wrote:red 'x' is first time a kanji is seen, green '+' are other occurrences, x-axis is sentence index number, y-axis is kanji index number, blue line is cumulative number of kanji seen (so slope of blue line is rate of introduction of new kanji).

this is default ko2001 order [...]

this is with sentences sorted by ko2001 frequency order
Thank you very much Cangy, thats exactly what I was looking for!!! But can you explain why there are two sorting orders? I thought default ko2001 was already sorted by frequency.

Good graphs. Really an eye opener.

@Thurd, KO2001 is sorted by Kanji, then introduces words that use that kanji, then sentences that use those words. I think Cangy sorted the sentences, which is an interesting take on the matter. The problem with KO2k1 and iKnow is that you may get easy words, but the example sentences might use complicated words in it. Cangy's sorted sentences removes that problem.

@Cangy, perhaps you could send your sorted list to CosCom (or at least tell them about it). It'd be an interesting addition to the CD. Great job to say the least.

KO 2001 Kanji/Word progression - thurd - 2009-11-11

Nukemarine Wrote:@Thurd, KO2001 is sorted by Kanji, then introduces words that use that kanji, then sentences that use those words. I think Cangy sorted the sentences, which is an interesting take on the matter. The problem with KO2k1 and iKnow is that you may get easy words, but the example sentences might use complicated words in it. Cangy's sorted sentences removes that problem.

Nice, so this new order gives up on teaching specific kanji in isolation (which doesn't make sense anyway since they introduce so many new kanji in almost every sentence that it ruins the whole purpose of isolation) in favor of minimizing amount of new kanji that appears with the next sentence (true n+1 approach).

So where can I find an Anki deck that can be sorted like that? Smile

Update: I think I might be able to create such deck myself. I'll keep you posted.

KO 2001 Kanji/Word progression - elanlan - 2009-11-11

What did you use to generate those graphs?

KO 2001 Kanji/Word progression - angerman - 2009-11-11

looks alot like GNUPlot to me.

KO 2001 Kanji/Word progression - thurd - 2009-11-11

I managed to create an Anki deck based on the same principle as shown on graph 2.

It has the new order but still keeps the original order stored in one field, that way if Anki ever supports sorting new cards this deck could be used for both order methods.

Since I wasn't that far into KO2001 I think I'll switch to this new one. It should provide me with a smoother and more consistent learning pace. I will post my initial impressions as soon as I will have some experience with it.

KO 2001 Kanji/Word progression - Nukemarine - 2009-11-11

Definitely keep us updated.

Out of interest, what sentence pack are you thinking of using?

KO 2001 Kanji/Word progression - thurd - 2009-11-11

Nukemarine Wrote:Definitely keep us updated.

Out of interest, what sentence pack are you thinking of using?

It's just a standard KO2001 deck but sorted differently.

KO 2001 Kanji/Word progression - ninetimes - 2009-11-11

Would you mind sharing the process? I'm at an early point in KO2001 as well, and that seems like the most logical way to go about it.

KO 2001 Kanji/Word progression - RobotsAnger - 2009-11-12

So where can I get a list of the second graphs order? I don't mind starting again. (Gave up when I reached book two)

KO 2001 Kanji/Word progression - thurd - 2009-11-12

ninetimes Wrote:Would you mind sharing the process? I'm at an early point in KO2001 as well, and that seems like the most logical way to go about it.

RobotsAnger Wrote:So where can I get a list of the second graphs order? I don't mind starting again. (Gave up when I reached book two)

I don't know what's the policy about KO2001 and if we can freely share its contents, I've seen links to pre-made decks though.

But I'm sure I can share how I did it:
Here are two scripts from Fugounashi I modified a little so they would write into a file:
http://rapidshare.com/files/305836720/Sorting_decks.zip

Here's how it works:
1. Note the order of your fields in your decks card model, you will need them later during import.
2. Export your deck from Anki into a tab separated textfile. Good tip is to modify your card model so that the question and answer part contains all fields you want to export, separated by <tab> tag. This way all your fields will be exported and easily readable by scripts.
3. Create a file with kanji frequency based on KO2001 sentences. Zip file already contains kanji frequency file so this step is optional but if you modified/corrected your deck you might want to run kanji-freq-1.1.pl script again so the list is updated.
4. Run kanji-sort-1.4.pl using both files (kanji freqency & your exported deck), as a result you should have a file named sorted-sentences.txt.
5. Import sorted-sentences.txt into Anki adding all columns (14 or so in my case) naming them in the order you've written before. You should have 2 more columns than previously, you can discard them.

This should give you an Anki deck sorted in this new fashion. It will be hard at first (but still easier than the original order) but later (from the graph) it seems less steep than original. If I interpret the data correctly original order introduces HALF the kanji by 500 sentence (out of 3100), here it should be around 1200-1500.

I can post more detailed instructions later. I can also share my deck or the import file if it isn't against any forum laws and policies.

KO 2001 Kanji/Word progression - ninetimes - 2009-11-12

That's plenty fine for me, no sense in sharing the deck and reopening those old wounds. I can take it from there.

Much obliged! I'm pretty excited.

KO 2001 Kanji/Word progression - cangy - 2009-11-12

elanlan Wrote:What did you use to generate those graphs?

angerman Wrote:looks alot like GNUPlot to me.

there's a script to extract the kanji occurrence data from a deck and another script to plot it using gnuplot (but not GNUPlot!). it produces graphs for ko2001 and core2000 using various orders. I'll put the scripts up soon if you want to try them out

thurd Wrote:I modified a little so they would write into a file

you can't redirect into a file under windows? I guess you could use cygwin...

thurd Wrote:Good tip is to modify your card model so that the question and answer part contains all fields you want to export, separated by <tab> tag.

if you export facts to a tab-separated file then you don't have to change anything

thurd Wrote:This should give you an Anki deck sorted in this new fashion. It will be hard at first (but still easier than the original order) but later (from the graph) it seems less steep than original.

yeah the idea is to limit the introduction of new kanji, so the blue curve is less steep initially and closer to the ideal straight line overall

I was going to make a script to find an optimal sort oder, but frequency order is probably going to be pretty close anyway

thurd Wrote:If I interpret the data correctly original order introduces HALF the kanji by 500 sentence (out of 3100), here it should be around 1200-1500.

from the raw data it's 505 vs 1079 sentences (for 554/1108 kanji)

also there's a bit more info on sorting decks in another thread

KO 2001 Kanji/Word progression - SammyB - 2009-11-12

Super noob question but.. how do I run/use the script?

KO 2001 Kanji/Word progression - nrg1701 - 2009-11-12

Wow, that would be extremely useful for Kanji in Context as well if the same method could be applied. The learning curve for KiC is a bit steeper (I think) than KO2001 (I have already went through KO2001 and continue the reviews) and often, unseen kanji from future lessons are used.

When doing the conversion, can you save reviewed card data by doing this somehow?

KO 2001 Kanji/Word progression - Nukemarine - 2009-11-12

nrg1701 Wrote:Wow, that would be extremely useful for Kanji in Context as well if the same method could be applied. The learning curve for KiC is a bit steeper (I think) than KO2001 (I have already went through KO2001 and continue the reviews) and often, unseen kanji from future lessons are used.

When doing the conversion, can you save reviewed card data by doing this somehow?

Probably not, but it's no stretch to export the unseen cards as .tsv file, sort just those, delete the old order then import back the new ones into Anki.

The one failing of Anki has been the inability to bulk add new information to cards and reviewing cards in a user dictated order (currently limited to tags or the order of import). Kind of odd seeing that a "fact id" field could be utilized which let's anki know the imported data are either additions or replacement to fields of already existing facts. However, I'm not a programmer, so this could be more difficult than I'm making it sound.

KO 2001 Kanji/Word progression - thurd - 2009-11-12

cangy Wrote:you can't redirect into a file under windows? I guess you could use cygwin...

It might be possible but I don't know how to do this Smile

Also after seeing the results printed under command line console (random kanji instead of a proper one) I decided to use another approach.

cangy Wrote:if you export facts to a tab-separated file then you don't have to change anything

That wasn't the case for me. I had some HTML in my answer part as well as few fields mixed inside, when doing export I got some fields without any tabs between them and fields that weren't there at all wouldn't show on the export. I think I tried both methods card & fact and it worked the same way.

cangy Wrote:yeah the idea is to limit the introduction of new kanji, so the blue curve is less steep initially and closer to the ideal straight line overall

I was going to make a script to find an optimal sort oder, but frequency order is probably going to be pretty close anyway

Yeah, it should be good enough. Compared to the original everything is a big step forward Smile

SammyB Wrote:Super noob question but.. how do I run/use the script?

If under Windows then download&install some Perl interpreter (http://www.perl.org) and you run each script with "perl scriptname.pl arguments". Best way to find out is to look at the code.

Cangy if you are the author of these scripts maybe there should be a mention about them in Wiki? Article about KO could contain info about this new order.

KO 2001 Kanji/Word progression - vosmiura - 2009-11-12

thurd Wrote:Sure but while learning 1-2 words in a sentence (while reinforcing everything else) is ok, trying to learn 6-8 words + a couple of new kanji is just not.

Thats why I want to know for how long I should move at mediocre pace (give myself more time to digest lots of new information) and when to shift to another gear.

I don't see how some preprepared measure would be better than going with your gut. Is it hard for you? Then spend longer on it. Is it all easy? Throw it away.

KO 2001 Kanji/Word progression - thurd - 2009-11-12

vosmiura Wrote:I don't see how some preprepared measure would be better than going with your gut. Is it hard for you? Then spend longer on it. Is it all easy? Throw it away.

I don't see how having more information about a problem could hurt. I'll still have my gut feeling...

The issue with judging it all based on gut is that it all seems really easy. If I wanted I could do 100+ cards a day. Problem is what happens the next day and day after (reviews + lots of failed cards). On the other hand I want to maximize my learning performance and to do that I need as much information I can get. If I know that after a certain point my study material is more stable I can act accordingly: move slower, be patient etc. since it will all pay off later and I can make up for the lost time.
I got this information from the first graph and I didn't like it at all, its just random kanji appearing in a sentence. No breakthrough point, no nothing. It would get a little easier later on but even at the very end there are still "surprise" kanji to be had.

So I asked a question, got my answer and now I can try a different (I hope better) approach, isn't this what we call learning?

KO 2001 Kanji/Word progression - Thora - 2009-11-12

thurd Wrote:I don't see how having more information about a problem could hurt. [...]
So I asked a question, got my answer and now I can try a different (I hope better) approach, isn't this what we call learning?

You might want to consider other criteria for kanji order. I'm not familiar with 2001KO, but I find the Kanji in Context order effective. A kanji order based on how they function in the language might have some advantages in the long run over an order based entirely on consistency of pace or difficulty. There are other ways to manage that. Stats and graphs sometimes miss how we actually absorb language. Fuzzy non-linear plots aren't always a bad thing. Smile

Also, I think seeing the same stuff in a variety of forms can be more effective and interesting than SRSing the identical thing multiple times. You might consider, for example, working on 2-3 different books once each (and playing around with some kanji/vocab quizzes) rather than SRSing one book. Whatever you do - good luck.

*About Kanji in Context:*

KinC basically divides them into 4-5 groups based roughly on how essential they are. Beyond 1400 or so, they suggest learning them based on your interests. Within those groups, kanji are presented together based on stuff like:
- similar form
- easily confused
- most commonly (or only) found in compounds with each other
- related common vocab

The order is efficient and effective because we make associations among kanji and among vocab as we go.The KinC order would also be consistent with graded readers, if that interests you.

The book gives a handful of common vocab for each kanji and marks any kanji in vocab that won't appear until later in the book. Just note those, but quiz yourself on the target kanji. The phrases and sentences highlight common usage, opposites, collocations, equivalents, etc. No need to spend time on long sentences for every word when a sentence fragment showing common usage is often all that's needed. The workbooks have built in vocab repetition to some extent.

KinC is designed for advanced beg/low inter, I think. Even if the sentences are too difficult for you now, you might find the order of the kanji useful.