I've searched the forums and the broader web for algorithmic specifics (pseudocode, test code, source code...) for Koohii's SRS algorithm and came up empty. Is it a hand-rolled algorithm & implementation? Is it based on Anki/SuperMemo? I like it because it uses binary "yes/no" answers, and I wonder if it's able to do something smarter than Anki/SuperMemo spacing with this simplification.

I've been reading about using logistic regression (aka machine learning) to determine spacing on a per-question, per-student basis, e.g., Lindsey, Shroyer, Pashler, and Mozer's paper, "Improving students' long-term knowledge retention through personalized review" [1], which also makes me curious if Koohii was implementing something advanced like this.

[1] http://www.colorado.edu/ftep/documents/L...14MS-1.pdf

I'm pretty sure its just a bare-bones implementation of a Leitner system: http://en.wikipedia.org/wiki/Leitner_system

There are also a bit more details here: http://kanji.koohii.com/learnmore

Thanks Zarxrax! I can't believe I didn't notice the Leitner method mentioned there.

Much of these forums are about Anki et al., is there any public data on how many people use the Koohii SRS? If it's a reasonably large number of people, might there be some interest in trying out different spacing algorithms and get some empirical data on their performance? That is, running a large-scale experiment in psychological learning to see who learns best with what kind of SRS?

There's all kinds of ways to do this, and one simple example could be to just split up the community of Koohii SRS users into, say, 4 groups: a Leitner, Anki, SuperMemo 2, and DASH (logistic regression, explained in the paper I mentioned in the first post) group, whose members are quizzed using just one spacing algorithm. Then, with a few weeks of data, one could try to identify the group with the best performance. Coming up with an apples-to-apples way to compare the different groups (made of of different people at different levels) will certainly be a big part of the study, but independent of that, the raw data (quizzes and results) would be a huge value-add to researchers.

At the other end of the complexity spectrum, the website could decide to use a different spacing algorithm for each user after each flashcard quiz. In between these two extremes might be where the website uses the same algorithm for a window of time (a few flashcard reviews) and then switches, trying to fill out all the dimensions of the dataset.

The set of questions that could be asked and answered by experiments like this is really big. They could tell us that, say, "User X, a middle-aged kanji learner living in Japan, benefited more from Algorithm 2, whereas User Y, a teenager in Spain, did best using Algorithm 4, with these parameters...".

It wouldn't be at all surprising if learners who were exposed to kanji every day (by, say, living in Japan, or near Chinatown) needed a different spacing algorithm than someone whose only exposure to kanji were daily reps. Furthermore, it wouldn't be surprising that algorithms that adjust the spacing based on how common a kanji is, or how it's related to other kanji, could outperform or underperform simpler spacing algorithms, like Leitner and Anki, that treat all kanji as equal.

If the Koohii designers were up for something like this, there would be a lot of work in:

- front-end website programming (replacing the interface, making it less Leitner-ish and more generic)

- back-end software (implementing the Anki et al. spacing algorithms, as well as more mathy machine learning algorithms like DASH, detailed in the paper in the first post; these wouldn't run on the Koohii servers, a local or Amazon EC2 cluster would have to be obtained to do all the real-time processing)

- psychology literature review (nail down up-front how "performance" will be quantified from this kind of data)

- experimental design (field of statistics involved with deciding how to structure the experiment, i.e., where in the complexity spectrum the experiment should sit, in such a way as to maximize the value of the experimental data collected)

- some legalese (it's purely coincidence that I post this so soon after the Facebook emotions study outcry), possibly to get some demographic data from all participants to inform more advanced algorithms

- a ton of data processing and analysis (what does the data say about the algorithms)

- data sanitation (anonymize the data and share it with the world, so other researchers could backtest their own algorithms against it)

- writing (blog/forum posts containing results; could lead to academic peer-reviewed papers on experimental psychology or machine learning)

This could yield significant benefits to (1) the learners themselves: they'd find out what worked best for them, and in what conditions; (2) those studying second languages without SRS, who are not persuaded by small-scale classroom studies but who might be persuaded to use SRS (and be advised on specifically which SRS!) if the benefits were demonstrated through a large autodidact community like Koohii. Also, (3) researchers studying second-language acquisition or general learning, as well as machine learning.

I'm not going to speak for Fabrice who runs this site, but as he is just one person, and hasn't really made any changes to the scheduling on the site since it was started, I think its unlikely that he would implement all of the stuff you are talking about there. And site users probably may not be fond of being guinea pigs for differing experimental algorithms.
That sounds like something you would be best off creating your own brand new site or application for.

Thanks for the frank comments.

Zarxrax Wrote:I think its unlikely that he would implement all of the stuff you are talking about there.

I would anticipate the community participating in the project, and would offer to serve as the main programmer.

Zarxrax Wrote:And site users probably may not be fond of being guinea pigs for differing experimental algorithms.

This is certainly a most important point: presenting how the site users will benefit from the testing in the long-run, and thereby engendering broad participation. SRS is, after all, letting a computer program decide what to have us review today, and what this project would implement is a computer program that picks which computer program will do that selection. We'd only want high-quality algorithms to be in the basket so nobody actually suffers by participating.

We're already using SRS, a tool that most language learners are ignorant or suspicious of, so if we don't study it and try to optimize its use for each individual, nobody else will.

Zarxrax Wrote:That sounds like something you would be best off creating your own brand new site or application for.

Certainly setting up a separate SRS service can be done, but you can imagine so many reasons why it would get more traction if built-into Koohii, the all-in-one home base for so many kanji learners. Users would need yet one more tool to time-share with (I myself use Koohii, Anki, Memrise, colorful paper flash-cards). Even with fair-use of a few stories from each kanji (as Fabrice has graciously allowed in the past), there's now an extra step to sync stories between the two sites, again a disservice to users. Oodles of functionality duplication. Etc.

Building super-smart-SRS into Koohii is, to my view, the highest reward, lowest risk way to get the data that we all need to really start the SRS revolution.

[ファブリス]

There is no point in doing that on Koohii since by my estimates less than 10% of the active users appreciate the simplicity and integration of the SRS on Reviewing the Kanji.

Anki is very well established and people are more comfortable using one system rather than review Kanji with Koohii SRS and vocab with something else.

Thus it would make more sense if somehow you can develop a new algorithm as a plugin for Anki, or maybe fork its source code. From there on you should also find quite a lot of participatns through the Anki forums since it has a large userbase.

Plus, I would bet a lot of Japanese learners using Anki don't even know about this site here, or purposely chose not to use Remembering the Kanji. The latter probably would be better to balance your "sample" user base, since imho using a SRS with RTK is not the same as using SRS to replace typical rote memorization. It's always been the goal on this site that the SRS is to test yourself and reinforce memorization, not to replace the act of learning the characters.

Fabrice, thanks so much for your thoughtful and encouraging reply. I will try to find some other venue for this sinister plan

ファブリス Wrote:There is no point in doing that on Koohii since by my estimates less than 10% of the active users appreciate the simplicity and integration of the SRS on Reviewing the Kanji.

Out of curiosity, how many users is that, the 10% that use the SRS? I ask because that is the number that a study like this would need to be large-ish, and also because I'm curious about just how many users Koohii sees (sorry, very nosey).

ファブリス Wrote:Anki is very well established and people are more comfortable using one system rather than review Kanji with Koohii SRS and vocab with something else. Thus it would make more sense if somehow you can develop a new algorithm as a plugin for Anki, or maybe fork its source code. From there on you should also find quite a lot of participatns through the Anki forums since it has a large userbase.

Great point: while Anki lets you practice all aspects of Japanese (readings, combinations, sentences) in one setting, Koohii restricts itself to just the kanji. I found it difficult to read and understand libanki (Anki's internal backend) source code, but I am looking at documentation for Anki add-ons and it might just be convenient enough to overlook the GPL license restriction. It would make sense to write each algorithm as an Anki add-on so people could experiment with them without having to sign up with an experiment.

ファブリス Wrote:Plus, I would bet a lot of Japanese learners using Anki don't even know about this site here, or purposely chose not to use Remembering the Kanji. The latter probably would be better to balance your "sample" user base, since imho using a SRS with RTK is not the same as using SRS to replace typical rote memorization. It's always been the goal on this site that the SRS is to test yourself and reinforce memorization, not to replace the act of learning the characters.

The key point you make is that there are two orthogonal components here, (1) spaced repetition and (2) the Heisig approach of visual/story-based "mnemonics". In fact, there might be another axis, (3) Heisig's ordering of the jouyou+ kanji list in a "sensible" manner. SRS can be used without Heisig's stories or ordering, and Heisig doesn't make any mention of SRS in the RTK book or iOS app. (And it is important for me to note, per @Kuzunoha13, that Heisig didn't invent either the story method or the idea of reordering the kanji, and that RTK doesn't have a monopoly on these techniques. Please forgive me for continuing to lazily ascribe these to Heisig/RTK in this post.)

Personally, I would like to first restrict the research project to studying an ordered list of kanji (Heisig's RTK, KanjiDamage, etc.) in a visual/story-friendly way (where other people's stories are offered but not required). People using SRS for rote memorization will probably not be using a sequence like RTK/KanjiDamage, so will self-select themselves out of the study. Once I have a baseline for the variability of spacing algorithms for a specific kanji list, the scope could be expanded to different kanji lists, or to on/kun readings, or to vocab lists, etc. So to start out, Koohii's specific combination of SRS+RTK is what I was unconsciously looking for when I first posted, and your comments helped me recognize these goals and make them more concrete.

You also reminded me that a lot of people are using SRS to complement rote memorization without either of Heisig's techniques, i.e., traditionally-ordered kanji lists and no conscious notion of stories about primitives. Studying the impact of various spacing techniques for rote memorization is indeed the bread-and-butter of the psychology literature, which in one way or other informs much of the conventional thinking in SRS circles, i.e., retrieval/storage strength theory, although I should point out that there's a good deal of controversy in the field and many experimental results fly in the face of SuperMemo/Anki conventional wisdom (Jeffrey Karpicke has done experiments where equal-spaced repetitions outperform increasingly-spaced repetition for long-term retention while seeming to sacrifice short-term performance: see "Expanding Retrieval Practice Promotes Short-Term Retention, but Equally Spaced Retrieval Enhances Long-Term Retention" (2007)).

But this research-space (SRS for rote memorization) is not where I want this study to live, at least not initially. I think it is worth studying the Anki users memorizing by rote because such a group of autodidacts would be very different from the subjects typically volunteering for psychology experiments informing the literature (college students from WEIRD countries, WEIRD=Western, educated, industrial, rich, and democratic), and their practice/test styles would be very different from the experimental methodologies of (1) memorizing random vocabulary or face/names or nonsense syllables, (2) get distracted for a few minutes to a few days, (3) get tested on a subset of what was "learned". (Truly insane.)

My restriction of the initial scope of this research to study spacing algorithms for RTK-like memorization is partly personal---I am learning with RTK---and partly logistical, in that there's only a single RTK (or KanjiDamage) list, popular and well-established, whereas there are many ways to organize and describe rote memorization lists (school-ordering, different primitive-based orderings, with and without readings and compound words, etc.).

(And just how I don't want this research to live in the academic psychology niche, I don't want to restrict it to evaluating just the SuperMemo/Anki workflows. These SRS tools are highly informed by the experiences of a very few number of extraordinary learners/coders, viz., Piotr Wozniak and Damien Elmes, in learning a wide variety of things. There is a lot of structure in a specific list like RTK that these general-purpose SRS tools cannot exploit, things like the kanji-primitive dependency graph, the frequency of kanji that a specific user encounters "in the wild" outside his/her SRS reps, periodicities in the performance on specific kanji, etc. There are also arbitrary conventions like Anki's assumption that 1 day is fine-enough resolution to space new/missed items---and while this can be tweaked with add-ons, how much does that influence the underlying algorithm? What I want to say is that I am trying to find a middle space between the WEIRD methodologies favored in academia and the general-purpose SRS tools we all know and love for the narrow task of mastering kanji.)

This is just anecdotal, but a few of the people (including myself) I've talked to utilize the similarities between certain kanji as a memory aid (for example 車、 連ねる、運ぶ, etc...so it's not like the technique is exclusive to Heisig.

Kuzunoha13 Wrote:This is just anecdotal, but a few of the people (including myself) I've talked to utilize the similarities between certain kanji as a memory aid (for example 車、 連ねる、運ぶ, etc...so it's not like the technique is exclusive to Heisig.

Absolutely, I had a big disclaimer in my long-winded post saying that what I call "Heisig's techniques" (ordering by primitives and stories linking primitives--kanji) were not new when Heisig published RTK in the 1970s, and that I was just being lazy by not coming up with better names for them. It got lost during editing. I will update the post.

[ファブリス]

aldebrn Wrote:Out of curiosity, how many users is that, the 10% that use the SRS? I ask because that is the number that a study like this would need to be large-ish, and also because I'm curious about just how many users Koohii sees (sorry, very nosey).

For the Flashcard reviews, it's pretty easy. Just page through the "members" page and if you go 3 days back or so, there are 11 pages of 50 rows so you can see 550-ish active users (ie. users who reviewed during the past 3 days).

As for the Study pages, I can tell you there are 368000 pageviews in the last 30 days, so that gives you a rough idea of the activity there.