Hi I've done a cute script to download the videos from TBS, along with the text and put them in the current folder.

I'm using it on Ubuntu, but it probably works well on any platform with ruby and mencoder run (including windows and mac).

You only need ruby, hpricot and mencoder installed.

Download it from:
http://www.inf.ufsc.br/~emilio/japanese/aranha.rb

Example of use:

$ ruby aranha.rb

It will download the last 15 articles from TBS that have video, convert the videos to mp3, and throw it all at the same folder together with each article's text.


###### ONLY for windows users: ########

You can download the mplayer package (which contains mencoder) from this link:
http://sourceforge.net/project/showfiles..._id=683443

Then you must set the MENCODER_PATH inside the script to wherever you install mencoder (notice the double \\ instead of only \ in pathnames). For example:

### BEGIN CONFIGURATION ###
MAX_FILES = 15
MENCODER_PATH = "c:\\my\\mencoder\\folder\\mencoder.exe"
HOST = "news.tbs.co.jp"
### END CONFIGURATION ###

I've tested it on my Ubuntu, I just had to install the necessary packages and it worked without problems!

Great script! Thanks!

mentat_kgs Wrote:Hi I've done a cute script to download the videos from TBS, along with the text and put them in the current folder.

Hey mentat, with no other intention than to educate, if you have some time, go over explaining this code in detail, I think everyone, programmers and non-programmers would appreciate your personal insight.

Oh, wow, this is brilliant! Thanks a lot.

Users of Ubuntu and Linux are lucky.

We don't have that luxury!

いただきま〜す

Good stuff. Might I ask why you put it all in a begin block though? You're not using that block at all.

jorgebucaran Wrote:

mentat_kgs Wrote:Hi I've done a cute script to download the videos from TBS, along with the text and put them in the current folder.

Hey mentat, with no other intention than to educate, if you have some time, go over explaining this code in detail, I think everyone, programmers and non-programmers would appreciate your personal insight.

If you know Ruby, his code is very clear and quite easy to understand. Instead of reading an explanation on his code, you should probably learn the basics of Ruby and read it yourself. I think you would learn a lot from it. Ruby is so easy to read that it's pretty much self-documenting.

Interesting title, though, you're not actually 'stealing' it.

Oh my 神... this is amazing! I could kiss you!! Thanks so much

On a side note, paste and save the ruby into a document in Ubuntu and it should color code it for even easier reading

After downloading, though, when opening in Ubuntu's default text app, Text Editor, I get this:

Quote:$B!!5v2D$J$/309q?M$H@\?($7$?$H$7$F5/AJ$5$l$?%_%c%s%^!<$NL1<g2=1?F0;XF3<T%"%&%s!&%5%s!&%9!<!&%A!<$5$s$,#2#2F|!"K!Dn$G=i$a$FH/8@$N5!2q$rM?$($i$l!"!V$I$N$h$&$J:a$bHH$7$F$$$J$$!W$HL5:a$r<gD%$7$^$7$?!#(B


$B!!<+Bp$K?/F~$7$?%"%a%j%+?M$NCK@-$rGq$a$?$H$7$F9q2HKI8fK!0cH?$N:a$KLd$o$l$F$$$k%9!<!&%A!<$5$s$O#2#2F|$NK!Dn$G!"!V;d$O$I$N$h$&$J:a$bHH$7$F$$$J$$!#L5<B$G$9!W$HH]G'$7$^$7$?!#(B

$B!!$^$?!"?3M}$N$"$H!"%9!<!&%A!<$5$s$OJ[8n;N$KBP$7!"!VCK@-$,?/F~$G$-$?$N$O7YHw$K<jMn$A$,$"$C$?$+$i$@!#;d$O?MF;E*N)>l$+$iCK@-$rGq$a$?$@$1!W$H=R$Y$?$H$$$&$3$H$G$9!#(B

$B!!0lJ}!"%"%a%j%+?M$NCK@-$OK!Dn$G!V;d$,K,$l$?$N$O%9!<!&%A!<$5$s$,0E;&$5$l$kL4$r8+$?$+$i$@!W$H?/F~$7$?M}M3$r>Z8@$7$^$7$?!#(B

$B!!:[H=$OMh=50J9_$bB3$/M=Dj$G!"%9!<!&%A!<$5$s$OM-:a$H$5$l$l$P:GD9$G6X8G#5G/$N7:$K=h$5$l$k2DG=@-$,$"$j$^$9!#!J(B23$BF|(B01:33$B!K(B

However, if I Open With > Emacs22 (Client), it comes out as normal Japanese... in... an ugly editor Any way I can just view it normally with Text Editor?

sethg, why not simply open them with firefox? It recognizes the character encoding automatically and that way you get to use rikai-chan.

sethg Wrote:After downloading, though, when opening in Ubuntu's default text app, Text Editor, I get this:

Quote:$B!!5v2D$J$/309q?M$H@\?($7$?$H$7$F5/AJ$5$l$?%_%c%s%^!<$NL1<g2=1?F0;XF3<T%"%&%s!&%5%s!&%9!<!&%A!<$5$s$,#2#2F|!"K!Dn$G=i$a$FH/8@$N5!2q$rM?$($i$l!"!V$I$N$h$&$J:a$bHH$7$F$$$J$$!W$HL5:a$r<gD%$7$^$7$?!#(B


$B!!<+Bp$K?/F~$7$?%"%a%j%+?M$NCK@-$rGq$a$?$H$7$F9q2HKI8fK!0cH?$N:a$KLd$o$l$F$$$k%9!<!&%A!<$5$s$O#2#2F|$NK!Dn$G!"!V;d$O$I$N$h$&$J:a$bHH$7$F$$$J$$!#L5<B$G$9!W$HH]G'$7$^$7$?!#(B

$B!!$^$?!"?3M}$N$"$H!"%9!<!&%A!<$5$s$OJ[8n;N$KBP$7!"!VCK@-$,?/F~$G$-$?$N$O7YHw$K<jMn$A$,$"$C$?$+$i$@!#;d$O?MF;E*N)>l$+$iCK@-$rGq$a$?$@$1!W$H=R$Y$?$H$$$&$3$H$G$9!#(B

$B!!0lJ}!"%"%a%j%+?M$NCK@-$OK!Dn$G!V;d$,K,$l$?$N$O%9!<!&%A!<$5$s$,0E;&$5$l$kL4$r8+$?$+$i$@!W$H?/F~$7$?M}M3$r>Z8@$7$^$7$?!#(B

$B!!:[H=$OMh=50J9_$bB3$/M=Dj$G!"%9!<!&%A!<$5$s$OM-:a$H$5$l$l$P:GD9$G6X8G#5G/$N7:$K=h$5$l$k2DG=@-$,$"$j$^$9!#!J(B23$BF|(B01:33$B!K(B

However, if I Open With > Emacs22 (Client), it comes out as normal Japanese... in... an ugly editor Any way I can just view it normally with Text Editor?

What is your locale? en_US.UTF-8?

@jorgebucaran
Hey, I don't think this would be an easy task to do, but I'll tell what the code does:

He opens the yomiuri frontpage.
Gets the first 15 links with a video icon.
Visieats each link and discovers the place where the video is and copy the text.

Then he downloads each of the videos and extracts the mp3.
wget and mencoder are external programs. wget is a download manager and mencoder is the encoding twin of mplayer.

@Toberoth
I put the begin block just to make the dirty part of the code clearer from the configuration variables.

Btw, I don't know what to do if your text file gets ugly, but try opening the file with

$ LC_ALL=ja_JP.UTF-8 gedit dirty_file.txt

I've tried with Firefox, and changed it to every Japanese encoding, but nothing works :/

Tobberoth, I would assume it is, but I honestly don't know how to check that.

mentat_kgs, I tried your code, but it still showed up as the symbols I pasted above.

Awfully frustrating problem. I despise the Emacsen interface. I *could* go through and cut all the text from emacsen to Gedit and replace the file, but that just takes away from the beautiful simplicity of the script.

Anybody have any more ideas?


Edit: Actually, I have, indeed, got it working in Firefox. I just had to set Auto-detect to Japanese. I don't know why this working... I ran through, literally, all 3 Japanese settings yesterday. :: sigh :: Well, at least it's working, and with rikaichan for quick lookups, to boot

Checking your locale is easy. Just open a terminal and run "locale".

As long as it ends in .UTF-8, simply copying into gedit should work...

If you use gedit, you can open it manually and select the encoding ISO-2022-JP, it should show it.

A script someone showed me:

Code:

for i in *.txt; do f=$(basename $i); iconv -f ISO-2022-JP -t UTF-8 $i > $f.utf8;done

can convert all those text files into Unicode, so it should display fine thereafter. Back up beforehand.

Edit: On that note, I just frigging wish Japan would adopt Unicode, the stubborn so-and-sos. >.<

Ok, I just fixed the text issue. The script was not tested on windows, but it should work fine now.

mentat_kgs Wrote:Ok, I just fixed the text issue. The script was not tested on windows, but it should work fine now.

YAY! Totally works fine now Thanks so much NOW it is incredible awesome

denus Wrote:A script someone showed me:

Code:

for i in *.txt; do f=$(basename $i); iconv -f ISO-2022-JP -t UTF-8 $i > $f.utf8;done

can convert all those text files into Unicode, so it should display fine thereafter. Back up beforehand.

Thanks for this as well Converted all the ones I'd already downloaded, which saved me a lot of copying and pasting!

Thanks so much, mentat. The script is indeed beyond awesome now. :3

Wow, I had no idea wget was preinstalled on Windows... Not that saving files down manually in Ruby is hard, but that's good to know, it's a useful tool.

It is not. I made ruby check the OS so it will only download using wget on linuxes.

mentat_kgs Wrote:It is not. I made ruby check the OS so it will only download using wget on linuxes.

Ah yes, I see it now... why did you decide to use wget on linux though? Is it faster? (I haven't tried it myself).

Not faster, but it resumes downloads and checks if the file was already downloaded.