mentat_kgs Wrote:Not faster, but it resumes downloads and checks if the file was already downloaded.

Ah, awesome.

I just updated it to use each article's title as filenames.

If you don't want this behavior, change the configuration option

USE_JAPANESE_NAMES = true

to

USE_JAPANESE_NAMES = false

Latest update broke it for me. Got this:

Quote:browsing http://news.tbs.co.jp/
spider.rb:67:in `iconv': ".BtBeI=Be9T$N8x@"... (Iconv::IllegalSequence)
from spider.rb:67:in `clean_text'
from spider.rb:21:in `initialize'
from spider.rb:82:in `new'
from spider.rb:82
from spider.rb:76:in `collect'
from spider.rb:76

Just fixed it.

Script broke with today's news again and I fixed it again.