Geolocation and CSV Import

I'm working with a group of archivists to build a site containing materials from multiple institutions. Most of the items are already in content management systems (DSpace and CONTENTdm), so it makes sense to do a series of batch uploads with the CSV Import plugin.

I would like to be able to give them instructions to include location information that could be picked up by the Geolocation plugin, but this appears to be something that hasn't been developed (or isn't documented). Before I dig into the code to see if it would be possible to have CSV Import write to the omeka_locations table as well as to omeka_element_texts (perhaps with a custom element like dc:coverage.spatial), I thought I'd check to see if anyone else has done this.

Thanks!

Danielle Cunniff Plumer
danielle@dcplumer.com
Thanks!

I'm not 100% sure, but Daniel Berthereau has a fork of CSV import that might add that feature. Might be worth giving that a try.

If not, an alternate route might be to write something that works with Omeka's API to put in the data.

Hi,

Yes my fork is a solution for that (just add geolocation:latitude and geolocation:longitude columns to your file), but this is not yet published on Github. Wait the end of the week.

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management

Thank you! I can certainly wait until the end of the week. I'd looked at your GitHub fork and I wasn't sure whether this feature was enabled.

Danielle

Hi,

I just publish it. Read the updated readme or try the test files test_extra_data.csv and test_extra_data_update.csv with geolocation plugin installed.

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management

Thank you! I'll test it out this week.

Danielle

Daniel,

I tried testing the plugin this morning. It works fine. One suggestion would be to add in the readme that for the "extra data" drop down (which only appears when selecting Mixed records or in Update records), the setting should be "Yes, so column names won't be checked."

Notes (not necessarily bugs, just things to know about):

1. I tested with a comma-separated csv with semicolons as delimiters. It worked fine. I tried variants (including latitude and longitude as well as address, latitude and longitude without address, and address only). Latitude and longitude are required for geolocation; geolocation:address on its own was not inserted into the omeka_locations table address field.

2. The element "geolocation:zoom_level" is mandatory for any items to be geolocated. Omitting zoom level will result in an import error.

3. The "Action" column in the CSV Import Status page shows "no action" when a mixed data import is done. Mixed data imports cannot be undone. Use this plugin with caution because of this! I'm sorry to lose this functionality ( a regular "Items" import retains this functionality and also the mapping step, below). This information should probably be added to the readme.

4. The intermediate step of mapping columns to Omeka elements is omitted for "Mixed records" and "Update records". That means that exact element names (of the form Dublin Core:Title) must be used. The other permitted column labels are 'sourceItemId', 'fileUrl', 'itemType', 'collection', 'public', 'featured', 'tags', 'file'. To attach files, use the column label "file." "fileUrl" may also work (as used in test_extra_data.csv). Labels are case-sensitive.

5. The file test_extra_data.csv has six rows but only three items (the remaining rows insert additional metadata). Item identifiers used for updating records appear to be indicated in this file by the column "sourceItemId". This reflects the fact that the creation and update function are both enabled in the "mixed records" import option.

6. The update item functionality worked correctly.

Danielle

Hi,

Thank you for the check.

For 1 & 2, the needed fields depends on the required fields of the plugin and CsvImport can't know them (it depends on the checks that are used in the hooks and in the validator of the model object).

For the Undo feature, I didn't implement it for Mixed and Update for the simplicity of the code and because anything can be updated (and in next release, I'll add the update mode "Delete"). So to undo, you'll just have to reimport the same file with this mode for the items or files you want to remove. I'll check if this will be simple to implement Undo, but note that an Undo can be made only for new imported records, not for updated ones (changes are not tracked).

For the intermediate step, I never use it, because it's a manual one, so a slow one, so I didn't implement it for Mixed and Update. It's easier and quicker to change the header of the csv file. I'll check if I'll have some time to implement it.
"file" and "fileUrl" column names will be merged, but currently the code is not the same for them. "file" is used to import one or multiple files during import of an item, "fileUrl" is used to import a File record.

For 5, the default "Item Metadata" imports files via the "file" column, but it can't import metadata of files (what I used often, specially to display true dimensions and rights on each image of an item). So they should be imported on a specific row, but as the item doesn't exist yet in Omeka, a common relation field should be used. This column could be optional, so the file will be attached to the previous imported item, but this may be difficult to be sure that the files will be attached to the right item if there are parallel imports. If you import only one file by item and no metadata for files, you can use the "file" column.

In fact, I'm going to merge Mixed and Update formats, with some new update modes, specially "Create", "Create if not exist, else Update" and "Delete". The idea is to do all the management of records via a simple table of data, that will be simpler than the items/edit interface of Omeka.

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management

Daniel,

Thank you very much for all your work on this plugin! It's a vast improvement on what I was doing before, which was hacking the MySQL database directly.

I've written up some documentation of the current version for a workshop I'm doing. Feel free to reuse any part of it that is helpful. You can download it from https://drive.google.com/file/d/0B7yRUzBulzurQTZWUXY1U29sVWM/view?usp=sharing

Danielle

This is great- I've been wishing for a way to do batch updates like this for a long time.

I am also very interested in using the geodata import. However, I haven't been able to get it to work. As far as I can tell, the field names of geolocation:latitude
geolocation:longitude
geolocation:zoom_level

are not being recognized. When I leave these out, the mixed records import works fine. Am I missing something about the syntax or configuration of the Geolocation plugin?

Haha, whenever I ask a question, I usually then figure it out!

I was only using 3 column headers for geolocation, because that was all I had data for. However, the spreadsheet needs to have all five geolocation fields in the header, even if you don't have them filled out.

So, if you want to import geolocation, you must have ALL of these:

geolocation:latitude

geolocation:longitude

geolocation:zoom_level

geolocation:map_type

geolocation:address

Thank you Danielle- you noted this in your word document, I just misinterpreted it.

Also, thanks to Daniel for making this amazing plugin!!!!

Karen,

I see where you mean. I've changed the sentence to read "Values for map_type and address are optional and may be omitted, though the columns must be present." I made the same mistake myself (more than once!), which is how I learned that the columns had to be there.

Danielle

One other quirk I learned while teaching a workshop using a fresh install of Omeka 2.2.2 is that the locations table is actually not created until you add a data point in the plugin. I've changed the document to reflect that, as well. The error was that the fields didn't exist. I just created a dummy item, added a location in the map editor, and saved it.

Danielle

Hi,

Ok, thanks for your presentation.

Note that the last commit yesterday cleaned the form, so some of your images are already outdated...

I ll finish the update on Monday. That will add a new import format that will merge (and deprecate) Mix and Update in order to simplify process.

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management

Daniel,

Thanks for the heads up. I'll look at things again next week and update my instructions based on the combined Mix/Update.

Danielle

Hi,

I just finish the update. Now, there is new format that allows to manage all records via a simple table. Try the three examples test_manage_one, test_manage_two and test_manage_script.

Old formats are always available and you can manually map them and undo imports... No change for extra data like Geolocation, because it worked.

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management