How to use the flashcard and stories exports

Index » Feedback

 
ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

You can now export your stories from the My Stories page, as well as all your flashcards and associated review status from the Manage => Export page (soon live).

These files are in CSV format (Comma Separated Values), and UTF-8 character encoding, both widely recognised standards.

You will not be able to import these directly in Excel 2007 (with or without a BOM mark, btw). Excel 2007 users simply import into one of the solutions below, and then export to an Excel compatible format.

Importing in OpenOffice
Make sure to select "CSV" file type in the drop down. Then you should get extra options after you pick the file : choose UTF-8, and separated by comma (not tabs). Now all Japanese characters should appear correctly. You will see that every time you change one of those options the preview will show you in real time, what the import will look like.

Importing in Google Spreadsheets (requires are Google account, but no download!)
Select File > New > Spreadsheet, then Import, select the csv file. Edit in there, or File > Download As > Excel, and voila monsieur! Google is smart enough to detect both the CSV format and the encoding.

megaboyx
Member
From: japan
Registered: 2008-05-30
Posts: 47

This can be done in Office.  First load the program in word.  Then save it again as a text document.  here it will give you the option to save as UTF-8.  Then just rename the .txt to .rtf and you can load it up in excel and sort by whichever column you like.

ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

Tahnk you megaboyx. I suppose you meant "Wordpad" not Word ? I saved as text document, did not have option for UTF8 (this is not the blocker though). Renamed to .rtf and sure enough I could load it into Excel 2007, and got plenty of options, chose "Comma" for Delimiters. However there still is a problem with multi line text in the Stories data. Text after linefeeds appeared on new rows.

Advertising (register and sign in to hide this)
JapanesePod101
Sponsor
 
Irixmark
Member
From: 加奈陀
Registered: 2005-12-04
Posts: 291

I'm having a problem with commas as delimiter because many of the stories contain commas and even " quotation marks which are then wrongly interpreted as new fields when I try to import them -- for example into Anki.

I've tried to come up with a sed or awk script to substitute the commas, but that doesn't work either. Would it be possible to add a function to export them as tab-delimited, or using semicolon as delimiter, neither of which will be (commonly) used when people write up their stories?

ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

The story text is enclosed by double quotes, and double quotes within the story are doubled, as per CSV rules.

I'd rather keep the export simple and "application-agnostic", given that you can import the data in a spreadsheet program as described above, and then re-export with your preferred settings; adding, removing and sorting the data as needed.

PS:

I've tried to come up with a sed or awk script to substitute the commas

I was juggling with grep the other day, but gave up in frustration with the flavor of regexp used in Bash. If you know the regexp format well enough, then you'd need a regexp that looks for commas that are NOT enclosed by double quotes, these should be the valid column separators.

There's an example regexp to "Changes the delimiter from a comma into a tab." on this page.. but it's in php.. there's plenty of results in Google for "change delimiter csv" though.

Irixmark
Member
From: 加奈陀
Registered: 2005-12-04
Posts: 291

I see your point, and your probably right. Here's what I wanted to do. Take the following exported line where the comma in the story messes up the import.

509,染,"dye",0,2009-12-23 13:10:40,"To make this dye, I used *water* and the barks of *nine* different *tree*s."

Of course, I can go via OpenOffice, but I'm learning new kanji by finding  a story I like and copying it or adding one of my own, then use the "My Stories List" to export the csv file, and then import only the newly added kanji to Anki---which I'm using to review because it has an excellent spaced repetition algorithm. That's already a few steps to take, so I'm hoping to cut it short by not going via OpenOffice.

In any case, for future reference if anyone wants to do the same, here's an ugly bash line that works by creating a semicolon-separated file that Anki automatically recognizes:

sed -e 's/\*//g' -e 's/\#//g' -e 's/"//g' -e 's/, / /g' -e 's/,/;/g' my_stories.csv | \awk -F';' '{print $1, $2, $3, $6; OFS=";"}'  > myfiletoimport.csv

It works simply by deleting the commas in the stories because they're usually followed by a space, then substituting all others with a semicolon because somehow Anki reads that better than a comma.

Cheers,

Mark

gasto
New member
From: El Salvador
Registered: 2009-06-16
Posts: 1

I can't import it in Spread sheet of Open Office. There is no CSV option when opening a file.

alexsuraci
Member
From: Indiana
Registered: 2009-09-06
Posts: 23
Website

Here's a little Ruby script that will update your Anki deck with the stories in your .csv export. Requires sqlite3-ruby. Just update the DECK and STORIES constants to point to their respective files and you should be set.

Note: Close Anki before doing this so the script can modify the database. And be sure to back up the database before running this.

Code:

require 'csv'
require 'sqlite3'

DECK = "/home/alex/.anki/decks/Heisigs Remember the Kanji (RTK) 13.anki"
STORIES = "/home/alex/Downloads/my_stories.csv"

db = SQLite3::Database.new(DECK)

heisig_id = db.get_first_value("select id from fieldModels where name = 'Heisig number'")
story_id = db.get_first_value("select id from fieldModels where name = 'Story'")

csv = CSV.parse(File.read(STORIES))
csv.each do |row|
  puts "Inserting story for ##{row[0]}: #{row[5]}."
  fact_id = db.get_first_value("select factId from fields where fieldModelId = ? and value = ?",
                               heisig_id,
                               row[0])
  db.execute("update fields set value = ? where fieldModelId = ? and factId = ?",
             row[5],
             story_id,
             fact_id)
end

Last edited by alexsuraci (2010 February 13, 10:17 pm)

Reply #9 - 2010 March 13, 9:26 pm
Inny Jan
Member
From: Sydney
Registered: 2010-03-09
Posts: 527

In a previous post Irixmark was saying that he gets that sort of entries in the CSV file:

Irixmark wrote:

509,染,"dye",0,2009-12-23 13:10:40,"To make this dye, I used *water* and the barks of *nine* different *tree*s."

When I tried exporting CSV today, I was getting that:
17,"吾","I",2010-03-11 06:40:10,2010-03-14,2,1,1

The first thing you notice is that the format of the file is changed, which (depending on how you use(d) the file in the past) may or may not be a problem.

The second thing is that stories column has disappeared. This might have been an intentional development by Fabrice (something that I could actually understand...) but nevertheless I would miss that functionality a lot.

Reply #10 - 2010 March 14, 10:50 am
ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

I think you're looking at the Flashcards Export, not the stories export. I'll have to merge them at some point as it's a bit confusing (this is on the project roadmap wiki page...).

Reply #11 - 2010 March 14, 5:48 pm
Inny Jan
Member
From: Sydney
Registered: 2010-03-09
Posts: 527

You are right ファブリス, before I came here, to Forum, today, I opened koohii and looked around a bit. And I noticed this "Export to CSV" button on:
http://kanji.koohii.com/study/mystories

It seemed somehow different from what I remembered(?) last time but I tried it anyway and I got:
17,吾,"I",1,2010-03-11 06:35:34,"Descartes said, ""I think; therefore, I am,"" but this kanji implies, ""I sense; therefore, I am.""  5 openings symbolize the 5 senses"

Hooray! But hold on... it didn't work before and it's very unlikely that it was fixed just overnight and it didn't look like what I've seen before, so I kept looking for the button that I *did* see. And I found it on:
http://kanji.koohii.com/manage/export

Mystery solved, and great consolation for me!

Cheers!!!

Reply #12 - 2010 April 13, 9:36 pm
uberclimber
Member
From: Hokkaido
Registered: 2008-01-27
Posts: 11

alexsuraci wrote:

Here's a little Ruby script that will update your Anki deck with the stories in your .csv export. Requires sqlite3-ruby.

Alex,
Would you mind going into a little detail on how to get this sqlite-ruby application up and running, pretty please?  I'm quite the n00b with this database stuff, but I'd love a quick conversion of my Reviewing the Kanji stories into my Anki application.  The Ruby Forge site says I have to have the SQlite engine installed to build this module, and that's where my tech abilities fall short. I'm on a Mac BTW, but I gather that shouldn't matter?  Yoroshiku onegaishimasu!

Reply #13 - 2010 May 13, 11:14 pm
alexsuraci
Member
From: Indiana
Registered: 2009-09-06
Posts: 23
Website

uberclimber wrote:

alexsuraci wrote:

Here's a little Ruby script that will update your Anki deck with the stories in your .csv export. Requires sqlite3-ruby.

Alex,
Would you mind going into a little detail on how to get this sqlite-ruby application up and running, pretty please?  I'm quite the n00b with this database stuff, but I'd love a quick conversion of my Reviewing the Kanji stories into my Anki application.  The Ruby Forge site says I have to have the SQlite engine installed to build this module, and that's where my tech abilities fall short. I'm on a Mac BTW, but I gather that shouldn't matter?  Yoroshiku onegaishimasu!

Sorry for the delay, but you need the sqlite Ruby gem installed; you should be able to just do "sudo gem install sqlite-ruby" in the Terminal if you're on a Mac. I'm on FreeBSD though so I can't really say for sure. I recall OS X coming with Ruby + RubyGems though.

dalmim
New member
Registered: 2010-08-01
Posts: 4

IMPORT UTF-8 .CSV INTO EXCEL 2003
(These steps may require installation of the Analysis Toolpak in Tools>Add-Ins)

To open the .csv directly in Excel 2003:
1. Open a 'Blank Workbook'
2. Go to Data>Import External Data>Import Data
3. Set 'Files of type:' to 'All Files (*.*)'
4. Navigate to Folder and highlight File and Open
5. In the 'Text Import Wizard', Set 'File origin:' to '65001 : Unicode (UTF-8)'
6. Be sure that 'Delimited' is selected as the file type and click the 'Next >' button
7. Set 'Delimiters' to 'Comma' - remove any other selections
8. Be sure that the double quotation mark (") is selected as the 'Text qualifier:' and click the 'Next >' button
9. Set Date format if desired and click the 'Finish' button

That's it. Though I'm learning to like Google Docs, there's no need to use it, or another intermediary between the .csv file and Excel. And, though I haven't checked, a similar set of steps should work for other versions of Excel.

I just wish I didn't create so many links to other kanji using brackets {}.

Fabrice - Could you add an option to replace bracketed numbers with the text

*Key_Word_HERE* (# Story_Number_HERE)

to the 'Export Stories' dialog in a future Update?

Thanks for your work.

[Note - in my original post, I asked for #Key_Word_HERE# (bolded text markers '#', when I meant italicized text markers '*'. Edited above.]

Last edited by dalmim (2010 September 11, 12:46 am)

ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

Great suggestion.. added to my list.

ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

Just fyi it's done and will be in the next update. It also uses custom keywords if any are created. Next update may be before end of September, if the UI for editing keywords doesn't turn out too complicated.

dalmim
New member
Registered: 2010-08-01
Posts: 4

Thanks! This will definitely make exported data more readily useable.

Reply #18 - 2012 July 21, 4:01 am
Errol246
Member
From: Denmark
Registered: 2012-05-02
Posts: 32

Ok, can somebody please, please, please make a step-by-step guide as to how we import  our RevTK data into anki on a mac? I am totally and utterly lost, I've tried almost everything and nothing seems to work.

Reply #19 - 2012 October 21, 3:10 am
tguk911
Member
Registered: 2012-09-17
Posts: 18

I'm not good with spreadsheets so is there a way I can format the text so I can replace all the #'s and *'s and format the primitives and keywords like they are on this website?

ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

You should probably edit the file before importing in a spreadsheet. Use text editor like UltraEdit or Vim with support for regular expressions you can then replace these with surrounding tags like <em>primitive</em> or <strong>keyword</strong> or whatever else formatting tags your target application uses.

Reply #21 - 2012 October 21, 6:09 pm
tguk911
Member
Registered: 2012-09-17
Posts: 18

Alright it's better than nothing and I didn't make too many yet. thanks for the help.

amillerchip
Member
From: Edinburgh
Registered: 2011-05-31
Posts: 96
Website

Hi Fabrice,

I'm trying to take a backup of my stories using this feature, but it is resulting in a 0KB download. Any ideas?

Edit: Doh, Apparently 'Export Flashcards' here: http://kanji.koohii.com/manage means export the SRS data, of which I have none because I'm using Anki.

But there is another link, 'Export to CSV' on the 'All my stories' page: http://kanji.koohii.com/study/mystories which did what I wanted.

:-)

Last edited by amillerchip (2012 November 26, 7:45 am)

ファブリス
Administrator
From: Belgium
Registered: 2006-06-14
Posts: 3699
Website

Are you downloading from the button here? http://kanji.koohii.com/study/mystories

Try again in an hour or so, maybe the server was having a hiccup. If it still doesn't work let me know. Also note that you can't request that URL twice in a short amount of time. If you automated this somehow with a shell script, you have to wait I think 30 seconds or so before requesting again.

amillerchip
Member
From: Edinburgh
Registered: 2011-05-31
Posts: 96
Website

Ah you replied while I was editing my post. Turns out I was using the wrong link... Looks like I'm not the first one either.

Thanks :-)

amillerchip
Member
From: Edinburgh
Registered: 2011-05-31
Posts: 96
Website

ファブリス wrote:

You will not be able to import these directly in Excel 2007 (with or without a BOM mark, btw).

Just to point out for the stories export, I re-saved the results with BOM in Notepadd ++, and it opened in Excel 2007 fine.