Mighty Morphin Morphology - Printable Version

Mighty Morphin Morphology - Printable Version

+- kanji koohii FORUM (http://forum.koohii.com)
+-- Forum: Learning Japanese (http://forum.koohii.com/forum-4.html)
+--- Forum: Learning resources (http://forum.koohii.com/forum-9.html)
+--- Thread: Mighty Morphin Morphology (/thread-7486.html)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Mighty Morphin Morphology - nest0r - 2011-08-24

overture2112 Wrote:
nest0r Wrote:like a little popup log that has a list of just-turned mature cards
How would you identify a card? There's the internal id number (useless to you) or you could have it tell you the value of some field of the fact it belongs to and the card type (slightly more useful?).

What about Expression and tags?

Or what about having MorphMan monitor intervals (I guess it can already do this, re: mature threshold), with an interval tag or field that it updates when it does its other automatic update stuff, but it has a popup reminder when it changes a field to 21 (or whatever the threshold is set to) from another interval, and then you could just search for interval:21 in the browser, filtering by tag:lesson1 etc., if you want to refine by tags? Perhaps you could set a threshold for how many cards minimum you want to be changed to interval 21 before it informs you. Perhaps you could monitor both that and by tag, and if cards have a tag you're monitoring, it doesn't monitor them by number of cards (it monitors when all cards with that tag are mature), but if it doesn't have a monitored tag, it will tell you when x number of those cards is mature?

Or a tag for mature that gets added/removed depending on whether the interval is equal to the mature threshold? (Remove if it gets higher, since it'll no longer be just-passed the threshold/just-turned mature.)?

Gah, it seems so simple in concept but I can't wrap my head around it.

I think the 2 key things are: 1) Having a reminder that can track by tags, so you don't have to just randomly/constantly check and track/remind yourself manually. 2) Being able to quickly and easily show and select the just-turned mature cards.

Mighty Morphin Morphology - nest0r - 2011-08-24

Ohhhhhhhhhhhhhhhhhhh. The new cards are sorted by difficulty! That is awesome! Okay! I get it now! The region of proximal learning for the win.

Is it sorted by iPlusN or something more complicated?

Edit: Oh wait, I think I confused new cards with simply due cards. Does it work with cards from the Reviews tab also? Perhaps cards that haven't met the learnt threshold? Okay, getting ahead of myself? My original plan was to deal with that in the future with Anki 2.0 or currently after tinkering with SelectiveStudy or whathaveyou.

I guess I should also adjust the default known/learnt thresholds since I have that initial learning phase of new cards that I don't want to factor in. Perhaps 5 will work, if that refers to the number of successful retrievals.

Mighty Morphin Morphology - overture2112 - 2011-08-24

nest0r Wrote:Ohhhhhhhhhhhhhhhhhhh. The new cards are sorted by difficulty! That is awesome! Okay! I get it now! The region of proximal learning for the win.

Is it sorted by iPlusN or something more complicated?

Currently the morphman index is:

i+N * 10,000
+ number of unknowns * 1000
+ vocab rank (using the vosmiura's new algorithm)

Very much so up for suggestions on this, but I like it thus far.

Mighty Morphin Morphology - nest0r - 2011-08-24

Is there a way to exempt decks? For decks that have cards where you're not doing the unknown words, so if a card passes the threshold, it'll be for other reasons? Also, for decks in the recent list that you don't want scanned, just because. ;p (Different topics, or I'm just keeping them there so I don't forget about them.)

Edit: For my subs2srs deck, I have two templates: One where the unknowns field is on the Front, effectively as the Expression, and when that's mature, that's when I add them to the known.db and regenerate the other cards. The Expression field is for the Japanese subtitles.

So for that .cfg, I'd want the field to be unknowns rather than Expression. But does it automatically add stuff from the Expression field, i.e. the Japanese subtitles, before I can change it? If so, does that mean I'm stuck with morphemes in .dbs that shouldn't be there?

Mighty Morphin Morphology - overture2112 - 2011-08-24

nest0r Wrote:Is there a way to exempt decks? For decks that have cards where you're not doing the unknown words, so if a card passes the threshold, it'll be for other reasons? Also, for decks in the recent list that you don't want scanned, just because. ;p (Different topics, or I'm just keeping them there so I don't forget about them.)

Edit: For my subs2srs deck, I have two templates: One where the unknowns field is on the Front, effectively as the Expression, and when that's mature, that's when I add them to the known.db and regenerate the other cards. The Expression field is for the Japanese subtitles.

So for that .cfg, I'd want the field to be unknowns rather than Expression. But does it automatically add stuff from the Expression field, i.e. the Japanese subtitles, before I can change it? If so, does that mean I'm stuck with morphemes in .dbs that shouldn't be there?

You can exempt a deck by going to its config and setting "enabled" to anything other than "yes".

It does the initial run with the default settings, so the easiest way to revert this is to just delete the deck's all.db and your known.db, which should cause them to be recalculated.

You can also change the default config in auto.py

Mighty Morphin Morphology - overture2112 - 2011-08-25

overture2112 Wrote:You can exempt a deck by going to its config and setting "enabled" to anything other than "yes".

It does the initial run with the default settings, so the easiest way to revert this is to just delete the deck's all.db and your known.db, which should cause them to be recalculated.

You can also change the default config in auto.py

On second thought, perhaps it's just less hassle to have all decks start disabled by default- I'm not sure why I thought otherwise. I'll change the default in the next version and tweak it to not do the first run until it's enabled.

Mighty Morphin Morphology - nest0r - 2011-08-25

Good idea. ^_^

Mighty Morphin Morphology - Boy.pockets - 2011-08-25

Here is the readme (mentioned above) for quick reference.

Mighty Morphin Morphology - visualsense - 2011-08-25

GREAT plug-in.
It crashed here, had to change Model names to non-unicode characters.
Some "ascii plugin" crashed on some "character u'\xe7'"(sorry didn't save the full message)...

Mighty Morphin Morphology - Boy.pockets - 2011-08-25

visualsense Wrote:GREAT plug-in.
It crashed here, had to change Model names to non-unicode characters.
Some "ascii plugin" crashed on some "character u'\xe7'"(sorry didn't save the full message)...

what OS are you running?

Mighty Morphin Morphology - nest0r - 2011-08-25

My 8555 sentence deck kept crashing Anki, but since I don't pay attention to the words I don't mind, though it won't create a config file so I'll have to manually create one to disable it from being processed (actually I'm just going to wait for a new version of MorphMan).

Ironically my most automation-desperate deck, I think, the subs2srs one I have the vocab cards and video clip card templates for, I don't think I'll enable the auto for that, because it would need to process both the unknowns field and the Expressions field since when the unknowns field (Which is effectively the Expression for the Vocabulary template) is checked to update the known.db, that's when I regenerate the Expression field (Japanese subtitles line) for the Comprehension template. Seems like if I just make one automatic I'll lose track of when to update the other, and even if I make them both automatic (like somehow if I can enter multiple fields in the .cfg), since the Comprehension template is suspended, I'll lose track of when to check for and use the new i+1 cards for new Vocabulary template cards and i+0 cards to unsuspend.

Complicated. ;p

(subs2srs deck explained in these two comments: http://forum.koohii.com/showthread.php?pid=141642#pid141642 - I was going to rework into a simpler tutorial but may wait and see with the new MorphMan stuff.)

Mighty Morphin Morphology - overture2112 - 2011-08-25

nest0r Wrote:My 8555 sentence deck kept crashing Anki, but since I don't pay attention to the words I don't mind, though it won't create a config file so I'll have to manually create one to disable it from being processed (actually I'm just going to wait for a new version of MorphMan).

Could you try having it process that again and send me the log file (in morph/tests/auto.log)? Note the log file is cleared whenever you start anki, so copy/send/post it before restarting anki again.

nest0r Wrote:Ironically my most automation-desperate deck, I think, the subs2srs one I have the vocab cards and video clip card templates for, I don't think I'll enable the auto for that, because it would need to process both the unknowns field and the Expressions field since when the unknowns field (Which is effectively the Expression for the Vocabulary template) is checked to update the known.db, that's when I regenerate the Expression field (Japanese subtitles line) for the Comprehension template. Seems like if I just make one automatic I'll lose track of when to update the other, and even if I make them both automatic (like somehow if I can enter multiple fields in the .cfg), since the Comprehension template is suspended, I'll lose track of when to check for and use the new i+1 cards for new Vocabulary template cards and i+0 cards to unsuspend.

So here's one solution that involves making Morph Man a tad smarter with respect to cards:

1) Currently Morph Man can check multiple fields, but I could update it to look at certain fields for only certain card types. Thus you'd specify a list of pairs like

[ ('SentenceExpression','Video Comprehension Card'), ('VocabExpression','Vocab Card') ]

2) Have Morph Man fill in a field like 'VocabExpression' whenever a card becomes i+1 (this is generally useful and I planned on doing it anyway, since the 'unknowns' field gets clobbered after the word becomes known but you still want to srs it).

Note, all of Morph Man's fields are optional, although 'Morph Man Index' is required for the new card sorting in the current version (restriction removed in the next).

Also, we could add this based on an email suggestion:

3) Have Morph Man automatically unsuspend certain card types (eg, comprehension cards) when they become i+0 in the mature.db (instead of merely known).

That said, I'd prefer to avoid (un)suspending cards automatically since people tend to use that feature for manual control and so it'd be much better just to work this into the morph man index calculation somehow (suggestions?).

Mighty Morphin Morphology - overture2112 - 2011-08-25

nest0r Wrote:(subs2srs deck explained in these two comments: http://forum.koohii.com/showthread.php?pid=141642#pid141642 - I was going to rework into a simpler tutorial but may wait and see with the new MorphMan stuff.)

Alternative solution / new method:

0) When a card becomes i+1 according to your known.db, Morph Man fills the VocabExpression field with the unknown word. When a card becomes i+0 according to your mature.db, Morph Man fills SentenceExpression with the full sentence. It will tag the card as either comprehension, vocab, or neither. It will also revert these if they no long meet the requirements.

1) Instead of separate vocab and comprehension cards, you combine them into one super card and utilize Anki's conditional replacement card template feature

set your card layout to something like

Code:

{{#SentenceExpression}}

stuff for sentence comprehension using {{SentenceExpression}} here

{{/SentenceExpression}}

{{^SentenceExpression}}

{{#VocabExpression}}

stuff for vocab word using {{VocabExpression}} here

{{/VocabExpression}}

{{^VocabExpression}}

Sorry, this fact is still too complicated to learn.

If you're seeing this you've run out of i+1 cards.

Please let Morph Man try to recalculate rankings

 (if you've learned some new stuff this session),

 add more cards, or try a different deck for now.

{{/VocabExpression}}

{{/SentenceExpression}}

Then you just study cards without touching anything (using Anki's selective study if you don't like switching between vocab/comprehension mode). The theory being that a sentence of all mature morphemes should be relatively easy and is more so a safety check and to get additional contextual exposure to its meaning.

This method has some neat effects like:
* If you fail a sentence it will normally continue being a sentence card, but will revert to a vocab card if that failure caused you to no longer have any mature cards (across all decks) that "cover" one or more of the card's morphemes. Which is exactly the kind of behavior you want for those very mature morphemes that you don't see often enough in your other studies.

Mighty Morphin Morphology - nest0r - 2011-08-25

You know what? The original name of the 8555 sentence deck was “8555 sentences - from the... ” and because the logs showed that the cards were processed it just didn't save properly and likewise the popup error, I just removed everything from the name but “8555 sentences” and now it works a treat. So I guess it's something to do with the dash, maybe? Or the length?

And the solution refined in your second comment sounds great. Especially the fact that there'd be a mature.db that can influence the automation.

I had no idea about the conditional field replacement. So that would remove the need for 2 templates? Ah and that tag would then allow for SelectiveStudy control.

I would probably need to mess with the conditional replacement stuff in the layout in order to set up my cards, as the Vocabulary cards' Front has just the unknowns (or in the future, VocabField), and for the Comprehension card the Front has just the video clip (the Back has the Expression [eventually SentenceExpression]).

So I guess I could have on the Front, the video, sentence, and vocab, where if the vocab is filled then the video and sentence aren't shown.

Other stuff I'm thinking about: The iPlusN thing also works for oPlusN (comprehensible output, or o+1), and can be applied there, and the Comprehension/i+0 thing connects to Paul Nation's fluency development strand in the Four Strands model, where all concepts are known and the idea is to develop fluency (often with time constraints, working at a faster than normal pace, which connects to the plugin thread suggesting a time limit).

Mighty Morphin Morphology - overture2112 - 2011-08-25

nest0r Wrote:I would probably need to mess with the conditional replacement stuff in the layout in order to set up my cards, as the Vocabulary cards' Front has just the unknowns (or in the future, VocabField)

That reminds me, is there an existing plugin for highlighting the contents of a field based on another field (so you can display the sentence with the unknown highlighted instead of the 2 on separate lines)?

Mighty Morphin Morphology - nest0r - 2011-08-25

Not that I can tell. Maybe you could have a field that's like, a duplicate of the expression created ({{HighlightedExpression}}) when unknowns are generated from i+1 cards, where MorphMan duplicates the original expression and fills a field like HighlightedUnknowns then replaces the unknown morpheme in {{HighlightedExpression}} with {{HighlightedUnknowns}}, and you make it conditional so HighlightedExpression is only displayed when the HighlightedUnknowns contains text? hehe. And you'd put HTML code or whathaveyou around the HighlightedUnknowns embedded in the HighlightedExpression (Edit: Or I guess you wouldn't need to since you could just change the colour in the field options, even change it to the same colour as the background so it's like a cloze).

Mighty Morphin Morphology - overture2112 - 2011-08-25

I've started a more updated readme at http://rtkwiki.koohii.com/wiki/Morph_Man (feel free to edit) as well as released version 2.01, which includes a quick attempt at the fact promotion idea above and changes decks to be disabled by default.

Mighty Morphin Morphology - nest0r - 2011-08-25

Cool, I just read the above about Morph Man 2 and the wiki after writing the following:

So check this out. After playing around some more with the Vocabulary Highlighter extension for Firefox: https://addons.mozilla.org/en-US/firefox/addon/vocabulary-highlighter/

I checked out the .xml format it uses for its vocabulary word database, and after some tinkering with regexp, got it to work! That is to say, I displayed my known.db as single columns in the MorphMan GUI, copied, and surrounded each morpheme in the list with the formatting used in the .xml file:

<vocabulary><word>morpheme</word>
<note/>
<htmlnote/>
<matchcase>0</matchcase>
<nothighlight>0</nothighlight>
<pen/>
</vocabulary>

(There's a bit at the top of the .xml for the very first word that's a bit different, haven't tried removing it yet.)

So then I copied the newly formatted list into the vocabulary.xml that I exported from the plugin options, and imported that vocabulary.xml file as a database, and voila! Now Firefox loads all my known morphemes. You can change the display options, right now instead of yellow highlighting (addon default), I'm just using a gray highlight. Will tinker with it more.

For a while I couldn't get it to work, said incorrect format, but I narrowed it down to the 。, the full stop was breaking it, so I just removed that. I also ended up having to remove numerals by searching for 0-9 in the extension's search area and removing them, so next time I'll be sure to strip those and some other punctuation.

Oh and I'm sure the .xml code changes slightly depending on what your options are, but you can see the bare essentials there required for display.

Edit: New MorphMan looks awesome so far, I guess I'll be tinkering with it for a while before commenting.

Edit 2: I noticed a feature request for not stripping the field before processing? Won't that result in garbage in the .db? Actually, speaking of which and per the above, is it really necessary for MorphMan to store punctuation and Arabic numerals in the .dbs?

Mighty Morphin Morphology - visualsense - 2011-08-25

Boy.pockets Wrote:
visualsense Wrote:GREAT plug-in.
It crashed here, had to change Model names to non-unicode characters.
Some "ascii plugin" crashed on some "character u'\xe7'"(sorry didn't save the full message)...
what OS are you running?

win xp (sp3, set Japanese for non-Unicode programs)

Mighty Morphin Morphology - overture2112 - 2011-08-25

nest0r Wrote:Edit 2: I noticed a feature request for not stripping the field before processing? Won't that result in garbage in the .db? Actually, speaking of which and per the above, is it really necessary for MorphMan to store punctuation and Arabic numerals in the .dbs?

Presumably any html markup won't result in a real morpheme. I _did_ forget to make it apply blacklisting for certain parts of speech (like punctuation and 'unknown', which is stuff mecab didn't recognize).

Mighty Morphin Morphology - nest0r - 2011-08-25

I'm drawing a blank at the moment, by the way: what are the possible categories for blacklisting? Is there like a list someplace?

Edit: Would the stuff here work? http://mecab.sourceforge.net/posid.html - Basically anything Mecab can recognize as a category?

Mighty Morphin Morphology - overture2112 - 2011-08-25

nest0r Wrote:I'm drawing a blank at the moment, by the way: what are the possible categories for blacklisting? Is there like a list someplace?

Edit: Would the stuff here work? http://mecab.sourceforge.net/posid.html - Basically anything Mecab can recognize as a category?

Yea, it's just applied to the part of speech entry (ie, the second column in the 4 column display).

Mighty Morphin Morphology - nest0r - 2011-08-25

Yes! I think I've got it! Now that I've said that, I'll play around some more to see if I jinxed myself. What could possibly go wrong?

Edit: Yes, so far it's all going smoothly. This is pretty badass, next level stuff. Thanks, O programmer-god.

So what I did was, after enabling the auto and leaving most of the defaults the same (just adjusted known threshold to compensate for my new card method, adding 2 so that it effectively doesn't count the first 2 retrievals from my initial learning phase), I went into the deck properties and left the Sterile model, which had only the Comprehension template, alone. That one is for vobsubs. I'm assuming it'll just be ignored since it's only a .jpg in its Expression field.

Then I went into the Seed model and disabled the Comprehension template, and renamed the Vocabulary Template to ConditionalS2SRS (very clever, yes), and edited the layout like so (after adding in some new fields... what's unmatures do?):

Front:

<body background="C:wallpaper.jpg" bgproperties="fixed">
{{#vocabExpression}}
{{vocabExpression}}
{{Audio}}
{{Image}}
{{/vocabExpression}}

{{^vocabExpression}}
{{Video}}
{{/vocabExpression}}

Back:

{{#sentenceExpression}}
{{sentenceExpression}}
{{Meaning}}
{{Gloss}}
{{/sentenceExpression}}

{{^sentenceExpression}}
{{Expression}}
{{Meaning}}
{{Video}}
{{Gloss}}
{{/sentenceExpression}}

And when I study, I just choose ip1known or ip0mature to toggle between vocabulary word cards and video comprehension cards.

I might just omit the usage of the audio plugin and use the video on the front of the vocab cards also (e.g. just replace {{Audio}} in the first condition with {{Video}}).

Mighty Morphin Morphology - overture2112 - 2011-08-25

nest0r Wrote:... what's unmatures do?

It's like the unknowns field but instead sees what's not in your mature.db.

Mighty Morphin Morphology - nest0r - 2011-08-25

Sweet. I'm sure I can think of some uses for that.

So let's see. Next I need to set up my extemporaneous vocabulary deck. I guess I'll have MorphMan monitor the kanji word, then when it's i+0 mature have it fill out a production field, then set up the conditionals so when there's something in the production field the layout changes accordingly.

So that's the function of the maturity reminder plugin for both subs2srs and vocabulary words! And I bet I can get it working for o+1 cards also, with longer expressions.

Then with the mature.db, in the GUI I can see all the mature morphemes and export them in various ways (such as crossword/wordsearch generators).

Next I just need to come up with something similar for people doing RTK so they can switch to Core 2000 in a timely fashion. (Or Japanese keywords, or something else.)

Edit: Also, A) resolve's a badass for having that conditional field stuff. B) You're a badass for coming up with such an elegant solution that utilizes it.

Edit 2: By the way, I noticed a typo in the GUI when you look at morphemes, in the right-hand panel: “By part of spech:”