This is a offshoot of the "sentence mining" thread, as I didn't want to further sidetrack that discussion. ghinzdra, Asriel, and I got into a debate regarding the utility of "kanji counting" (measuring the number of kanji readings you know). My argument was that this is an inadequate and unreliable means of measuring Japanese ability, but the point was made that some method of quantitatively measuring Japanese ability would be beneficial.

So I put this question to the community: what quantitative measurement (talking numbers here!) can we make to measure our progress in Japanese, which is both a fair reflection of our own Japanese ability over time, and could also be used to compare against each other?

E.g, if I say I'm on frame 1500 of RTK, everyone here knows what that means with respect to my ability to write and recognize the meaning of kanji, and I know with certainty that I'm twice as far as someone on frame 750, and halfway towards finishing RTK1+3. Can anyone think of a number or set of numbers that can be quoted to measure Japanese ability in general?

Well my understanding of the question has now changed. I was unaware that this was a universal "how well do I know Japanese" question. I thought it was just about his personal quest to learn the kanji for the JLPT. (and my own quest to learn the kanji and readings)

I'm only on frame 1363 of RTK, but my knowledge of Japanese is way beyond that. I can listen and speak, but reading kanji is where I fall short.

As for a "how well do I know Japanese" gauge...I don't really have anything better than your idea of the materials you have read...It's a tough thing to stick a number on.

Knowing kanji is just knowing kanji. It's a subset of Japanese language ability, which would cover reading, writing, speaking, listening, etc. Your ability in the language should be measured by your ability to carry out tasks in the language. Can you watch a random TV program and understand it? Can you go to a random business, find out what they sell, and order something? Can you pick up a book, figure out what it's generally about, read the introduction, and tell the person next to you what it's about Japanese? Can you write about all of these sorts of experiences in Japanese and tell people what you saw, did, and read?

Those are the sorts of things that language ability measures. Not the number of kanji in your deck.

I don't see any problem with learning extra Kanji if you plan to become fluent anyway.

btw they are removing 6 from the joyo list and adding another 180 something later on in the year. Wiki has links to the list of new ones.

thought this would be a useful if slightly off on a tangent bit of info for those unaware.

JLPT is the metric I go by. Not because it's perfect or because one can easily tell two speakers apart, but because it's somewhat official, it's broad and it's standardized. Someone who can't pass JLPT1 isn't better at Japanese than someone who can, that's the only distinction I make. Does this mean that if two people pass JLPT2 but can't pass JLPT1, they are equally good? Far from it. I do think it means both are better than a third guy who can't pass it though.

Overall, I don't think a good metric can be made for a language. You can measure how many kanji you can recognize, how many kanji you can read (and how many readings you know for each kanji), how many words you can understand, how many grammatical structures you can use in a conversation, how many books you've read, how many movies you've seen... but it will never really let you compare yourself to anyone else when it comes to Japanese as a whole.

Yeah, I never really finished my post properly. The kinds of tasks I suggest are the sorts of things you need to be able to do, and the only way you can figure out if you've gotten to the level you need to be at is to try doing those sorts of things-- the sorts of things you do in everyday life in your own language.

I don't think there's really an administerable test you could devise that would be able to measure real language proficiency. All tests are artificial constructs that simply measure your ability to pass the test. Test with bubble sheets are doubly so.

Just because you can pass JLPT 2 doesn't mean you necessarily know how to ask a pharmacist for something for your sunburn. The odds are good that you can, but there's no guarantee.

That's why I'd recommend testing yourself rather than worrying about some test like the JLPT to determine if you 'know' Japanese. (Although I know the JLPT is useful for some things...)

Then again, if you have to ask yourself if you know the language, then you already have the answer, don't you?

rich_f Wrote:Then again, if you have to ask yourself if you know the language, then you already have the answer, don't you?

But the question isn't whether you know the language or not, it's about a metric. A way of putting your knowledge on a scale and comparing it to others.

What Tobberoth just said. As I said in the other thread, I personally gauge my Japanese ability by keeping track of all the things I've read and fully understand (having sentence mined anything I need to look up). But while this is fine for measuring my own progress, it's useless for comparing against others, unless we happened to read the same things.

I was thinking this morning about writing an Anki plugin that parses sentences in your SRS and extracts grammar patterns, vocabulary, kanji readings, expressions and idioms, etc., and normalizes against frequency lists to keep it grounded in reality. That might be as close as we can get to a universal, comparable metric. But it'd be a hell of an undertaking.

Well, I'm not much into the "metrics" thing, but this topic made me remind of an old idea of mine...

Have you heard of the "Turing test"? It's a test devised more than half a century ago by Alan Turing to assess the actual "intelligence" of an artificial intelligence. I'll c/p a description from Wikipedia.

The Turing test is:

Quote:a test of a machine's ability to demonstrate intelligence. It proceeds as follows: a human judge engages in a natural language conversation with one human and one machine, each of which tries to appear human. All participants are placed in isolated locations. If the judge cannot reliably tell the machine from the human, the machine is said to have passed the test.

Do you see the connection with language learning?

Just replace the words "human" with "native" and "machine" with "learner" and "intelligence" with "fluency" and you'll have what I think is the best assessment for language fluency on the long run.


So, let's see again:

Quote:a test of a language learner's ability to demonstrate fluency. It proceeds as follows: a native speaker judge engages in a natural language conversation with one native speaker and one language learner, each of which tries to appear native. All participants are placed in isolated locations. If the judge cannot reliably tell the native from the learner, the learner is said to have passed the test.

Of course, not all the occasions in which people can engage in interaction with native speakers are equal, nor is this kind of engagement the only thing you are supposed to do in the language, but the idea is that L2 has to become a "second L1". Whenever you do something in L2 and feel comfortable with your performance, that's a sign you're in the right path. If you don't feel comfortable, that's a sign you have to improve. Assessments are useful if they help you improve, but they have to be a means to an end, not an end by itself.

I think passing the JLPT and all that is cool, but the real assessment is doing things IN the language, not ABOUT the language. So, as long as unofficial assessments go, talking to other learners about the things you do IN the language is much better than comparing scores. Much the better if the conversation itself is IN Japanese.

Tobberoth Wrote:

rich_f Wrote:Then again, if you have to ask yourself if you know the language, then you already have the answer, don't you?

But the question isn't whether you know the language or not, it's about a metric. A way of putting your knowledge on a scale and comparing it to others.

What's the point in comparing it to others? Do you get a cookie?

"Good job, Johnny, you're much better in Esperanto than Fred! You get juice with your cookie! Fred must go back in the box! No crying, Fred! Connect the electrodes!"

I don't care how much the person sitting next to me knows. I'm more worried about how much I need to know, and how much more I need to study. What's the point in comparing me with others?

The thing is, if you want to take some sort of test to find out how good you are, you're going to spend/waste a lot of time prepping for it, which diverts you from your normal studies, and biases your language study towards the topics on the test. So whatever results you get won't be accurate anyway. As long as you can game the test, it's pointless. *cough*JLPT*cough*

So whatever metric comes out from the Generic Big Metric Group, LTD., there will be 30 different study guides for it, each with an approach to mastering it, nailing it, whatever. So the metric fails.

The only way it could work would be if examiners randomly broke into your home and tested you without you signing up for it, so you couldn't prepare for it. They would just assess your ability on the spot, as it currently is, without the bias of test preparation materials.

And even then, the metric would only be as good as the design of the test, and by the next day, after you had studied some more, it would be completely inaccurate, as any exam is only a snapshot of the contents of your brain in one instant in time.

...

If I had to come up with the perfect Japanese Language Ability Test, though, I'd stick by my first one, except that examinees would be randomly taken in the middle of the night to Japan, dropped in the middle of Shinjuku, given a random shopping list, and told to read a book, a magazine, and watch a TV show, and write a report about it. Maybe I'd give then 10,000 yen, too. >_>a

Wait, have I just come up with a new TV show?

That idea about the turing test is actually pretty clever. I was just thinking about how often I converse with people online through writing, and many of them are not native English speakers, but I often would never be able to guess if they didn't bring it to my attention. Of course, if you move to speech, than the test becomes much more difficult to pass, as many people never lose their foreign accent.

The Turing test is the best metric. I think I mentioned something similar on TJP forums back in 07. I thought it was stupid to let learners of a language define what is and is not fluent (offshoot to an asinine complaint about Khatzumoto's writing ability being sub-standard because he used too many kanji, limiting his ability to be of use in a Japanese company).

The only drawback to the Turing test is accent's. Having an accent does not mean you're not fluent. However, it's a dead give away that one is not "native". However, as Zarzax points out the best cases have been on these forums where Japanese, Swedes and Brazilians all post here and "native English speakers" are shocked by the command of the language when their origins are revealed.

Guess that means if you can take part in a BB discussion with natives and they're not thinking you're a foreigner, you probably have a darn good command of the language. I guess one can do conversation where the voices are digitized to remove hints of accents, but that's going a bit too far.

rich_f Wrote:

Tobberoth Wrote:

rich_f Wrote:Then again, if you have to ask yourself if you know the language, then you already have the answer, don't you?

But the question isn't whether you know the language or not, it's about a metric. A way of putting your knowledge on a scale and comparing it to others.

What's the point in comparing it to others? Do you get a cookie?

Actually, you do. A metric gives a clear indication of progress. How often didn't you think "Ah, just X amount of kanji left, then I'm done with RtK!" and feel relief? This is the exact same thing. It's free motivation.

Well, if I get a cookie, does it at least have chocolate chips in it?

Nukemarine Wrote:An asinine complaint about Khatzumoto's writing ability being sub-standard because he used too many kanji, limiting his ability to be of use in a Japanese company.

May I ask what you meant (or mean) by this? How or why would this be a hindrance to employment?

[I genuinely have no idea. I've always assumed that using lots of Kanji in written Japanese shows that you have a strong command of the language. Does it make you come across as a snob or an elitist? Is it frowned upon?]

harhol Wrote:

Nukemarine Wrote:An asinine complaint about Khatzumoto's writing ability being sub-standard because he used too many kanji, limiting his ability to be of use in a Japanese company.

May I ask what you meant (or mean) by this? How or why would this be a hindrance to employment?

[I genuinely have no idea. I've always assumed that using lots of Kanji in written Japanese shows that you have a strong command of the language. Does it make you come across as a snob or an elitist? Is it frowned upon?]

I would guess it's similar to an English person using thou and thee etc in writing.

Sorry, wall of text follows.

rich_f Wrote:What's the point in comparing it to others? Do you get a cookie?

Some people here are able to push themselves harder in a competitive environment. Just look at the "halfway" and "final countdown" threads. These people have noted that it is increasingly difficult to compare your progress against others as soon as you move beyond RTK. Some of them are using Anki kanji statistics as a way of measuring progress post-RTK. I called this out as a silly, stupid, and useless metric. Their response, in summary, was that whatever its flaws, kanji counting is the best we have at the moment.

Hence my creation of this thread.

I am not one of these people, but I understand their position. Of more interest to me is a way to test the efficiency of various learning methods for the benefit of those who follow us. For example, a variety of methods exist for learning just kanji readings. RTK2, kanji chains, and the movie method all come to mind. Right now the Anki Kanji Statistics tool could be used or adapted to measure the number of readings one knows (by counting the number of kanji in a sentence deck). Already there are people doing just that on this forum, and in time when enough data is collected I expect that we will have a definitive consensus as to which method is most efficient.

However, as I have said before kanji counting has problems. It is not an accurate measurement of how well someone can use Japanese in real life as it focuses on just one aspect of the language, and ignores frequency statistics (a rare kanji you may never, ever encounter counts just as much as 日 or 円). If there was a universal metric for measuring Japanese ability (doesn't have to be perfect, but at least better than kanji counting) there's a whole host of new, more interesting questions that we would be able to answer.

For example, the difference between RTK2 (onyomi only) and the movie method could be accurately determined by a kanji count of readings and the number of hours invested. With enough case studies (people who have done either method and reported on their results) we could fairly determine which method is better.

However, what about comparing RTK2 or the movie method to learning readings in context with KO2001? It'd be comparing apples and oranges, certainly. KO2001 teaches not just readings, but also grammar, vocabulary, and general reading comprehension skills. But it is question worth asking (after all, "Should I do movie method? Or should I jump straight into sentences and pick up readings in context?" is a common question on this board). If some universal metric existed for measuring all aspects of one's mastery of the Japanese language, we could safely ask the following more scientific question:

Given that the movie method takes ~150 hours to complete, and KO2001 ~600 hours (making up numbers here!), which is better: doing KO2001, or doing the movie method followed by 450 hours of sentence mining (both taking approximately the same amount of time).

Without a universal metric, I don't think we can answer this question with any confidence.

-----

To the people discussing JLPT: that's kinda missing the point of this thread. I hope it is clear by this point that I am talking about a day-by-day measurement that you could make to track your progress.

So you want an objective measure of Japanese language ability that would enable us to determine which learning methods work best for everyone?

Or do you just want a motivational tool for use as a sort of progress bar?

There are a few problems with the first one. I understand the second one. You need to keep motivated when learning the language.

But not everyone learns the same way. Some people thrive on visual learning styles, others thrive on verbal learning styles. Some prefer experiential learning, while others prefer theoretical learning. Some people like Anki, some people hate Anki. What's the saying, 100人、100色? Something like that.

There are so many variables just in that one aspect, I think it would make more sense for people to just understand that there are a lot of different roads to Japanese language proficiency, they all require a lot of work, and you need to find one you can stick to. I think it's more important that you enjoy what you're doing than it be 100% efficient.

If your method is 100% efficient and you are 100% miserable, the result is you'll quit. If it's 70% efficient, or heck, even 20% efficient, and you're only 10% miserable, you'll probably continue. Yes, you'll waste time, but odds are you'll eventually learn the language.

Granted, some people want the 100% efficient method, and can tolerate more misery than others. But then what's 100% efficient for one person is only 80% efficient for another person, due to their budget, time constraints, etc. So in the end I think it just boils down to experimenting on yourself, seeing what works for you, and going with that.

The problem with comparing yourself to others is something I learned in martial arts: It's not a race. If you get wrapped up in how you're doing compared to others, and constantly measure your ability in terms of how good you are compared to some random guy who claims he could speak like a native in 18 months, you're setting yourself up for some big disappointment.

I suppose an objective standard is useful, but the most useful way I've found to measure my progress in Japanese is to grab random Japanese source material, try to understand it, see how I did, and figure out how far I need to go. That's why I have TV-Japan. Whenever I turn on the TV, I get a Japanese language comprehension test on whatever random show pops up. Sometimes I pass, sometimes I flame out. Locally, we have a something called 古本市 (I think that's right) at one of the local schools, where you can get used Japanese books for dirt cheap. Just grab some random books and see what happens.

If beginners need to scale down the difficulty, they can do so by using middle school or grade school or even preschool material. There's no shame in that. You have to start somewhere.

edit: fixed wrong kanji.

Seeing that none of us are taking the same path, it is a difficult gauge. Here's mine

Every mature RTK kanji - I give myself 6 minutes
Every mature Grammar sentence - I give myself 10 minutes
Every mature Vocabulary sentence - I give myself 6 minutes.
Every mature Movie Method kanji - not sure how I'll count this, too connected to RTK.

Now the above is how I gauge my effective study time. With the above, I'm at 2400 RTK, 700 grammar, 1500 vocabulary so about 500 effective study hours.

During this time I know I've written about 500 pages of Japanese text (kanji and hiragana). So my writing metric is 500.

My reading metric I think is 1200 pages. On this, I assume every 5-6 pages of manga is 1 page of normal text. Currently I can read at a 10 page an hour pace with drama scripts.

My listening/watching metric is hard to gauge. It's not high though. My video is in the hundreds of hours, not thousands. But my listening is definately in the thousands of hours.

So there you go, that's how I'm gauging my progress: 500 Study hours, 500 written pages, 1200 read pages. So it's still low at the moment.