API Import doesn't seem to be working

I'm trying to import items from another site (on which the API is enabled) into another site (also with the API enabled) using the Omeka API Import plugin. I've tried with and without an API key but it doesn't seem to be doing much. The status reads "in progress" but no items have been added after several attempts. I've been cancelling/undoing the import after about 10 minutes of apparent inactivity.

How would you advise troubleshooting this issue?

Additional details:

  • the source site is composed mostly of moderately-sized MP3 files (a couple MB each).
  • both the source and destination sites are hosted on Webfaction, as subdomains.
  • the source site is running Omeka version 2.2.2 and the destination site is running 2.1.4

First thing is to make sure that error logging is turned on for the destination site. It should give lots of messages there about its progress. That should tell how far through the process it gets.

If the source site is publicly available, could you give the url?

Hi Patrick,

The sites are linked above (to the API endpoints).

I'm not seeing anything in the logs other than 2 errors caused by submitting a URL without a protocol.

2015-01-07T19:07:23+00:00 ERR (3): exception 'Zend_Uri_Exception' with message 'Illegal scheme supplied, only alphanumeric characters are permitted' in /home/ebellempire/webapps/omekadev/application/libraries/Zend/Uri.php:111

Stack trace:
#0 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Http/Client.php(305): Zend_Uri::factory('voices.clevelan...')
#1 /home/ebellempire/webapps/omekadev/plugins/OmekaApiImport/controllers/IndexController.php(18): Zend_Http_Client->setUri('voices.clevelan...')
#2 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Controller/Action.php(516): OmekaApiImport_IndexController->indexAction()
#3 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('indexAction')
#4 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))
#5 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Application/Bootstrap/Bootstrap.php(97): Zend_Controller_Front->dispatch()
#6 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Application.php(366): Zend_Application_Bootstrap_Bootstrap->run()
#7 /home/ebellempire/webapps/omekadev/application/libraries/Omeka/Application.php(79): Zend_Application->run()
#8 /home/ebellempire/webapps/omekadev/admin/index.php(28): Omeka_Application->run()
#9 {main}

When you enter the URL for the endpoint when importing, are you including http:// before the rest of the domain name?

Yes, I'm including http:// –– the error above happens when I omit it. Now that I consider it, I guess the error is actually telling me not to include http://?

So, I maybe the question is why am I getting this error?

Not sure if it's related, but the source site also fails as an OAI-PMH repository.

Hmm...this is curious. The http:// should definitely be included when you enter the url.

When I tried with it, it looks like the importer can't find the list of resources available to import. It _should_ just parse what is at http://voices.clevelandhistory.org/api/resources , but for some reason what gets returned is the content at http://voices.clevelandhistory.org/api. That makes the importer think that nothing is available to import, so it just stops.

I'll have to dig some more to try to find why it seems to get redirected.

Huh. looks like dropping the trailing slash at the end of the url worked for me.

http://voices.clevelandhistory.org/api works
http://voices.clevelandhistory.org/api/ didn't work

That's a weakness of the plugin for me to address! Thanks!

Strange. It's still not working for me. So maybe there's something else going on with my destination site? Any ideas?

(By the way, I was wrong about OAI-PMH harvesting. It works fine.)

So, in the destination site's logs, is it now just giving two messages, one the the import started and one that it completed immediately after that?

I don't see anything new in application/logs/errors.log. Is there somewhere else to check? I have error and sql logging turned on in config.ini.

In config.ini, also look for the logging level and drop it to debug:

log.priority = Zend_Log::DEBUG

Ok, I now get the Beginning Import statement but no end, nor anything in between – thus far at least. This is consistent with what I've seen on the admin panel, where it just sort of hangs on "in progress."

Yeah, that's consistent with the front end. There are only a few places where something could go wrong between the "Beginning Import" message and where there should be more messages, or something going wrong enough to throw some kind of message. (in the perform method in libraries/ApiImport/ImportJob/Omeka.php)

Since I was able to pull things in, my first guess is that there's something wrong in a plugin in the destination site. Since the original site doesn't look like it has plugins exposing data to the API, my next guess would be to disable plugins on the destination side before trying the import.

Update: I got farther along, but also hit a snag with the import. You might need to turn off the filtering of elements in the API settings on the original site.

I've disabled all plugins (and reinstalled the API plugin from the omeka.org repo) on the destination site but am still having the same issue. The logs still show the Beginning Import message but nothing else and the admin panel hangs on ...in progress.

I also checked the source site and the filtering option is already un-checked.

I updated the destination site to 2.2.2 just to eliminate that possibility and it's basically behaving the same way.

The one difference was that on the first run it threw the error below. After a second try, it did not, reverting to the usual begin/processing message.

Anyway, not sure if this is is a clue or just an anomaly:

2015-01-08T20:21:48+00:00 ERR (3): exception 'Zend_Http_Client_Adapter_Exception' with message 'Read timed out after 10 seconds' in /home/ebellempire/webapps/omekadev/application/libraries/Zend/Http/Client/Adapter/Socket.php:511
Stack trace:
#0 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Http/Client/Adapter/Socket.php(329): Zend_Http_Client_Adapter_Socket->_checkSocketReadTimeout()
#1 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Http/Client.php(1077): Zend_Http_Client_Adapter_Socket->read()
#2 /home/ebellempire/webapps/omekadev/plugins/OmekaApiImport/controllers/IndexController.php(19): Zend_Http_Client->request()
#3 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Controller/Action.php(516): OmekaApiImport_IndexController->indexAction()
#4 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('indexAction')
#5 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))
#6 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Application/Bootstrap/Bootstrap.php(101): Zend_Controller_Front->dispatch()
#7 /home/ebellempire/webapps/omekadev/application/libraries/Zend/Application.php(366): Zend_Application_Bootstrap_Bootstrap->run()
#8 /home/ebellempire/webapps/omekadev/application/libraries/Omeka/Application.php(79): Zend_Application->run()
#9 /home/ebellempire/webapps/omekadev/admin/index.php(28): Omeka_Application->run()
#10 {main}

That looks like an anomaly of the internet connection just lagging and timing out. Though, there might be a chance that there is also a timeout someplace that's silently ignoring the error.

I'm getting to stabs in the dark here, at least until I can recreate the behavior. Best I can imagine is going into that same libraries/ApiImport/ImportJob/Omeka.php perfom method and adding additional debugging statements to figure out how far along it gets.