Automating mapping from WKT input on DC.coverage map and Neatline

Hi all,

I've been lurking around the forums for an answer to this problem for a little bit now, but haven't found anything directly relevant. If this has already been handled by someone and I just haven't seen it, my apologies.

I'm working with a dataset of many book and archival material citations, and am using the CSV import plugin to feed them into Omeka. Each citation (where 1 citation = 1 item in Omeka) contains multiple geospatial points, all of which I would like to have represented simply as points both in Neatline, and on the DC.coverage map that appears in Omeka as a result of NeatlineFeatures.

I've been geocoding my place names with the Ruby geocoder, and using excel to assemble blocks of concatenated WKT points. Here is my problem:

I understand that Neatline will automatically read DC.coverage and parse it for WKT, then display that geometry. I haven't had any trouble with this so far. The DC.spatial map, however, does not seem to be able to do this; the only way I can get a map is if I check "Use Map" in the Edit Item tab of Omeka –– and even then, it can't find the WKT, but wants user-drawn shapes. To get around this, I have (instead of importing the string of WKT points as DC.coverage text) been manually going to every item, hitting "Use Map" to enable the feature, and then using phpMyAdmin to look at the corresponding entry for that item in the "omeka_neatline_features" table, where I can then insert the WKT string under "geo".

This way, DC.spatial has a Neatline Features map in Omeka, and Neatline can read directly from that. This is actually better than importing the WKT string as text, since it means I don't have to display the WKT string on Omeka. But, it is not really feasible for me to do this for 1000+ citations. Does anyone have a suggestion for a better way to do this?

Thanks so much!

-stephen

Someone with more knowledge will have to comment on your problem with the DC.coverage map, but as for

This is actually better than importing the WKT string as text, since it means I don't have to display the WKT string on Omeka.

You can omit WKT values with a php conditional in common/record-metadata.php in your theme. We only use POINT, so I have this in my element-text DIV:

<?php
    if ($elementName == 'Coverage') {
       $pos = preg_match("/POINT\(/",$text);
    } else {
        $pos = false;
    }
?>
<?php if ($customText) {
       echo $customText;
    } elseif ($pos!=true) {

        echo $text;

    } else {
     //do nothing
    }
?>

Including the open paren prevents it from excluding legitimate locations that might have POINT in their value.

Hi stephen,

Getting data into NeatlineFeatures is unfortunately difficult. Features stores the WKT in a separate table, along with information about your viewport settings and background map and other things, and it mis-uses the DC.coverage field for its own purposes.

That's surely more than you're looking for, though.

If you (or a friend) can handle some programming, you can automate entering these through the Omeka admin interface fairly easily:

Wayne Graham has written up instructions for using Ruby to interact with your site and automate feeding this information to the site.

I haven't written anything up about it, but I've done something similar with Python. The repository is on Github called omeka_national_parks. You can find the relevant function, named populate_coverage.

Both of these are a few years old, but I don't know that anything's changed that would affect these. If something like this doesn't work, let me know.

Eric

Thank you both very much for your helpful replies!

Eric, I just wanted to clarify something:

I'm not sure how the Ruby "mechanize" method described in the blog post would help with Features (but hopefully I am just missing something). The method that Wayne Graham describes involves having this Ruby script fill in the admin "item" form multiple times –– but my understanding is that there is no way to get WKT data into a DC.coverage Features map unless either you are drawing the shapes manually using the OSM interface, or if you manually create lines in the SQL table for Features.

Features is unable to detect WKT text entered manually into the DC.coverage text field (unlike Neatline items), correct?

Thanks again!
-stephen

You're absolutely right: that's what both of our approaches do.

Except they don't attempt to draw anything directly on the map.

Between the two stages that you mention above (creating the lines and populating the database), there's actually another stage. Between there, whenever anything's changed on the map, Features uses JavaScript to update a set of hidden inputs on the page. One of those is the things drawn on the map, as WKT.

We use this knowledge in those scripts to directly populate those input values when we submit the admin item edit page.

Does this make it more clear?

Eric, thanks again for the speedy reply.

Yes, that makes it more clear, and I'll give it another shot!

-stephen

I was able to successfully use the mechanize gem to do this, and it worked great! I have now a challenge as a result of using this method (as opposed to CSV import). CSV import makes it very easy to map multiple columns in the table of data that I want to import to the same element, or elements separated by a delimiter. I have many items with multiple authors (separated by ;), which I would like Omeka to see as separate entries within "Creator"; but with mechanize, I'm not sure if I can do that.

One solution would be to run mechanize, and create items using the title, and the geometry fields in order to get Features maps for them. Then, I could use CSV import to update those existing records, and populate all of the rest of my information. The CSV import plugin doesn't seem to allow updates, but Daniel-KM's fork appears to (https://github.com/Daniel-KM/CsvImport). The only problem is that, for some reason I have been unsuccessful at debugging, whenever I try to import my CSV using this fork I am taken to a blank browse.php page (and the import does not occur).

Does anyone know what might be causing this error –– or, I suppose the larger question, how I might be able to merge my information with this mechanize method?

Thanks!

You should be able to handle this with Mechanize.

Once you've split up the author data the way you want it, you can simulate clicking "Add Input" on the field.

If you look at this screenshot, I've clicked on "Add Input" for the "Creator" field several times. I've circled the names of the input elements for each one. You can see that the second number in square brackets is incremented for each input.

Basically, all you need to do is add names like that for each author's name that you want to add to the field. That should let you import the whole thing using mechanize.

Hope this helps,
Eric

Eric, this works perfectly! Thank you so much for your help –– I appreciate it immensely.

-stephen

Hi,

Which issue do you have with CsvImport? It looks like a rights issue. The import process is the same than the official plugin. Is it working? Did you try the samples? Can you give me the headers of your csv file and one row?

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management

stephen,

Great! I'm glad to hear it.

Eric

Hi Daniel,

Thanks for your attention! I did try your fork with both one of my CSV files (which would import properly with the official plugin), and a couple of the test CSV files that you provide. After the "map columns" page, when I actually hit submit it takes me to a blank screen and does not perform the import. Actually, the blank screen occurs whenever I try to just click the "status" page in the plugin, regardless of whether or not I've initiated an import.

I suspect this might actually have to do with how my web server is handling php errors... I say that because I've been working on a simple plugin to generate Chicago-style citations, and whenever I test some code with a syntax error I'll see (instead of a messed up citation, or an error report in the position of the citation) a completely blank item page. And this is dissimilar to how a LAMP server I was running myself would handle the same error in Omeka (on that server, I'd see an error).

I do have error reporting enabled in Omeka, but nothing displays when I have a problem –– either with your CSVimport or with my citation plugin.

By rights issue, do you mean a problem with file permissions on the server?

best,
-stephen

Hi,

As like yours, my plugin works on my production installations and on a base installation of Omeka 2.2.2, so this type of issue is always related to config. When I have such a problem on a shared host or on my installation, I find always the issue in these points to check: Omeka config.ini, server config and logs, php.ini, htaccess, files rights, Omeka errors.log rights, php cli rights (used for long background jobs).

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management

Daniel,

Thank you for the suggestions, I'll try to dig deeper into the setup issues specific to my server and see if I can't get it to work.

best,
-stephen