Hello,

I've been trying to utilize LWT, because I find it very helpful. Lots of functions are reminding me of LingQ, which, unfortunately, is commercial.

The one main weakness of LWT is, the Japanese support is more or less nonexistant. Unless you can actually insert spaces between the words, there's no chance to properly use it as a learning tool for this specific language, and while I've tried to find a way to do this (apart from inserting them manually, which, for long texts, is タイヘン), is using Mecab.

Well, I've checked that, but I don't even have the slightest idea on how to use it. I tried with command line parameters, but nothing happened (no error message either). I set an input and an output file (UTF-8) for it, to no avail.

However. It would be optimal to have a little app where you paste text, press a button and have it "spaced". Kind of like a little GUI, useable for dummies (or people who rather spend their time learning the language instead of programs for learning).

Do you know of such a tool, or is it possible to make Mecab somehow useable without twisting my brain?

Thank you very much

P.S.: I tried to import something into LingQ, then copy and paste it into LWT, which led to cluttered text (-.-

See here

mecab -O wakati input.txt -o ouput.txt

The output.txt doesn't contain anything. May it have to do with Win7 security settings? I have to always confirm admin rights if I want to change stuff inside that directory..

Okay, now it works, but the output file always seems to be ANSI, not UTF-8, despite setting it every time (-.-

とても面白いことになってます。バズたんと、協会が仲間割れしまして、あいつが悪い嫌おまえがとのことになってます。
でもってこの協会も、あのKEKの野尻先生とか数名を訴えるらしい。日本でも名だたる先生ばかりですが。どうみても分が悪いのは協会。明細を出さないのはおかしいでしょうよ。
そろそろ放射能詐欺が色々ばれてくる段階になってきましたね。

results in this:

ï» ¿ ã ¨ ã ¦ ã‚ ‚é ¢ç™ ½ ã „ã “ ã ¨ ã « ã ª ã £ ã ¦ ã ¾ 㠙㠀‚ ム㠂º ã Ÿã ‚“ ã ¨ ã€ å ” ä¼ šã Œ ä» ² é– “å ‰² れ㠗 ã ¾ ã —ã ¦ 〠㠂 ã „ã ¤ ã Œæ ‚ª ã „å « Œã Š ã ¾ 㠈㠌 ã ¨ ã ® ã “ã ¨ ã « ã ª ã £ ã ¦ ã ¾ 㠙㠀‚
ã § ã‚ ‚ã £ ã ¦ ã “ã ® å ”ä ¼ šã ‚‚ 〠㠂 ã ® KEK ã ® é‡Žå °» å…ˆç ”Ÿ ã ¨ ã ‹æ •°å ã ‚’ è¨ ´ 㠈㠂‹ ら㠗 㠄。 æ— ¥ æœ ¬ ã § ã‚ ‚å ã ã Ÿ ã‚‹å …ˆ 生㠰 ã ‹ã ‚Š ã § 㠙㠌 〠‚ã © 㠆㠿 ã ¦ ã‚ ‚å ˆ† ã Œæ ‚ª ã „ã ® ã ¯ å ”ä ¼ šã €‚ æ˜ Žç ´° ã‚’å ‡º ã •ã ª ã „ã ® ã ¯ ã Šã ‹ ã —ã „ ã § ã —ã ‚‡ 㠆㠂ˆ 〠‚
ã ã ‚ ã ã ‚ æ” ¾ å° „能 è© æ ¬º ã Œè ‰² 〠…ã ° れ㠦 ã ã ‚‹ æ® µ 階㠫 ã ª ã £ ã ¦ ã ã ¾ ã —ã Ÿ ã ­ 〠‚

I now moved the complete mecab folder to the desktop to avoid the win7 security issues. The output file is still always in ANSI, even after i open it with Notepad++ and convert it to UTF-8, then save.

There must be something I'm missing here, please help

It's likely that mecab is outputting in EUC and not utf-8.

Do you have an idea on how to fix it? I've tried with -t UTF-8, but it doesn't help..

kiokuwarui Wrote:Do you have an idea on how to fix it? I've tried with -t UTF-8, but it doesn't help..

When you installed Mecab did you pick the UTF-8 option for the dictionary? I just tried the Windows version and it was fine. I've usually use it in Linux, so I'm not really sure.

In Notepad++ did you pick the right encoding before converting to UTF-8, so:
Encoding->Character Sets->Japanese->Shift JIS?

Setting it to ShiftJIS lets the text appear like this:

・ ｿ 縺 ｨ 縺 ｦ 繧 る 擇逋 ｽ 縺 ・ ％ 縺 ｨ 縺 ｫ 縺 ｪ 縺 ｣ 縺 ｦ 縺 ｾ 縺吶 €・繝舌 ぜ 縺溘 ｓ 縺 ｨ 縲 ∝ 鵠 莨 壹 ′ 莉 ｲ 髢 灘 牡 繧後 ＠ 縺 ｾ 縺励 ※ 縲 √≠ 縺 ・ ▽ 縺梧 が 縺 ・ ｫ 後 ♀ 縺 ｾ 縺医 ′ 縺 ｨ 縺 ｮ 縺薙 → 縺 ｫ 縺 ｪ 縺 ｣ 縺 ｦ 縺 ｾ 縺吶 €・
縺 ｧ 繧 ゅ ▲ 縺 ｦ 縺薙 ・ 蜊比 ｼ 壹 ｂ 縲 √≠ 縺 ｮ KEK 縺 ｮ 驥主 ｰｻ 蜈育 函 縺 ｨ 縺区 焚蜷 阪 ｒ 險 ｴ 縺医 ｋ 繧峨 ＠ 縺 ・€・譌 ･ 譛 ｬ 縺 ｧ 繧 ょ 錐 縺 ・◆ 繧句 ・ 逕溘 ・ 縺九 ｊ 縺 ｧ 縺吶 ′ 縲 ゅ ← 縺 ・ ∩ 縺 ｦ 繧 ょ ・ 縺梧 が 縺 ・・ 縺 ｯ 蜊比 ｼ 壹 €・譏 守 ｴｰ 繧貞 ・ 縺輔 ↑ 縺 ・・ 縺 ｯ 縺翫 ° 縺励 ＞ 縺 ｧ 縺励 ｇ 縺 ・ ｈ 縲 ・
縺昴 ｍ 縺昴 ｍ 謾 ｾ 蟆 ・・ 隧先 ｬｺ 縺瑚 牡 縲 ・・ 繧後 ※ 縺上 ｋ 谿 ｵ 髫弱 ↓ 縺 ｪ 縺 ｣ 縺 ｦ 縺阪 ∪ 縺励 ◆ 縺 ｭ 縲 ・

How would I set mecab to UTF-8?

I've been looking into this as well... and I have no idea how to use Mecab Not trying to hijack the thread or anything, but are there any decent manuals for Mecab in English, for relative computer noobs? I'm struggling to find anything either on this forum or elsewhere

Edit: Using Windows 7, forgot to mention!

during the installation it will ask you for the type of encoding you want to use with Mecab

I failed at setting up mecab; I really only know enough about the command line to get myself into trouble, so I don't know if I'm going to try to take another crack at it.

So far the most practical way for me to use LWT has been to use the option that makes every character as its own word, and then define words of more than one character as multi-word expressions. This isn't ideal because words composed of two or more kanji that I know well will appear as "known," but it's not too difficult to read through and capture them and add in the definitions. Easier than manually inserting white space, anyway.

mecab sounds too much like macabre.... :-(

It seems I decide to drop in once a year. I will attempt to make a little tutorial on setting mecab up for Windows 7 64bit, and if need be, I can virtualize 32 bit Win7 as I believe its install was a bit different.

Since my last post, I've moved on to using Linux and it has made working with MeCab (and Python) a comfortable stroll in the park. I recommend it, but I understand the obstacles.

One last note: using MeCab along side Python is a wonderful way to create little scripts that do exactly what you want, but getting them to work together on Windows can be a pain for new users. If that's something intriguing to you, I can make a rundown of how to do that as well.

One laster note: LWT is amazing and MeCab has a wrapper for PHP, but the maker of the software is pretty adamant about adding features that are relevant to his interests and that is perfectly fine. It is open source.

The default encoding of the ipadic dictionary that mecab uses is euc-jp. If you're using this euc-jp version of ipadic, convert your input file's encoding to euc-jp before feeding it into mecab. The output will also be in euc-jp, which you would want to encode into utf-8.

Alternatively, there is an ipadic dictionary that is compiled to use utf-8 encoding which you can install. But if you're using Anki, the utf-8 version of ipadic will most likely cause incompatibilities with the Anki Japanese plug-in.

Another thing to check is that your windows environment is set to display utf-8. Your output file may be in utf-8 encoding, but if your windows are not set to display utf-8, then you'll see garbage on the screen when you display the file's contents.

I finished the installation video and I hope it helps some of the people here in troubleshooting the issues they have.

Cacawate Wrote:I finished the installation video and I hope it helps some of the people here in troubleshooting the issues they have.

It sure did, thanks.

Btw., the syntax for spacing text, just so we don't have to watch the video every time we forget it, is

mecab.lnk -O wakati input.txt -o output.txt

kiokuwarui Wrote:The one main weakness of LWT is, the Japanese support is more or less nonexistant. Unless you can actually insert spaces between the words, there's no chance to properly use it as a learning tool for this specific language, and while I've tried to find a way to do this (apart from inserting them manually, which, for long texts, is タイヘン), is using Mecab.

Adding spaces to Japanese text is a pretty awful solution. Even if it's automated.

It's sad, I don't think LWT is going to be very useful unless someone actually takes the source and forks it for real Japanese support. The morphology Anki plugin just does so much more then LWT. Just without a custom dictionary for each word. Anki just isn't really made to read something chronologically.

If you built in proper Japanese support, you could treat different verb stems as the same word. That alone would make LWT more useful. Anyway, I'm going to quit ranting about this.

Daichi Wrote:Adding spaces to Japanese text is a pretty awful solution. Even if it's automated.

Why?

For one, you can't have hybrid English/asian text. Well, you can, but it's going to look ugly and space-less "―Theworldisnotbeautiful.Therefore,itis.―" It completely fails at dealing with furigana or ruby text in a nice manner. It doesn't have any idea how to deal with different verb conjugations, so it treats them all as different words.

I'll be honest, I didn't get very far in using it because it was too tedious for myself. I'm not saying it doesn't work, I'm just saying it doesn't work well enough. A lot of the programming is too European language focused, it really could use some programming specific to Asian languages to make it far, far more useful.

Daichi Wrote:For one, you can't have hybrid English/asian text. Well, you can, but it's going to look ugly and space-less "―Theworldisnotbeautiful.Therefore,itis.―" It completely fails at dealing with furigana or ruby text in a nice manner.

Good. The point is to practice reading actual Japanese, not English or kana. That would defeat the purpose.

Daichi Wrote:It doesn't have any idea how to deal with different verb conjugations, so it treats them all as different words.

Does it deal with verb conjugations in European languages?

Btw., for those who wouldn't use MeCab for anything except to separate words (I keep saying "words" even though I know the right term is morphemes, because I'm guessing not everyone knows what that means), there's no need to install it anymore.

There's a webpage that runs Mecab server-side to separate morphemes for you:

http://nihongo.dpwright.com/spaces/index.php

Haven't tried it for large texts, hope it doesn't have some silly, tiny limit.

Stansfield123 Wrote:

Daichi Wrote:For one, you can't have hybrid English/asian text. Well, you can, but it's going to look ugly and space-less "―Theworldisnotbeautiful.Therefore,itis.―" It completely fails at dealing with furigana or ruby text in a nice manner.

Good. The point is to practice reading actual Japanese, not English or kana. That would defeat the purpose.

That isn't the point, it should display all languages properly despite the one your trying to learn. And part of that is proper display of furigana. Adding spaces to Japanese is just a hack to make LWT work. It is certainly something that could be improved upon.

Stansfield123 Wrote:

Daichi Wrote:It doesn't have any idea how to deal with different verb conjugations, so it treats them all as different words.

Does it deal with verb conjugations in European languages?

Probably not, but the Anki morphology plugin can pick up the dictionary versions of these verbs just fine, so I don't see why it couldn't be done.

Also, when you spit something into Mecab (like with the site in the above post) for spacing you get lots of weird language fragments that you don't need to deal with in European languages.

思ってる becomes 思って る

「る」 is not exactly a vocab word, sure I can ignore it, but it would be far easier for the user to have the LWT parser just see it as 思う.

Again, I'm not saying LWT doesn't work, the point I'm trying to make is that it could be far better and far more useful to learn to read with. I'm just trying to point out some of the faults. It's too cumbersome for myself. If it works for you, great. Go about your way and ignore me.