Problem with CSV import and/or Dropbox

I’m trying to upload a csv file of 402 records, each of which has a file path and name linking the record to jpgs already uploaded to the Dropbox folder. This is a file exported from Filemaker, edited in Excel, then saved in Notepad.

I can successfully upload a file containing up to 50 records, but for anything larger, the import stops somewhere between the 60th and 80th record, at a different record each time.

If I make a smaller file containing the record on which the import stopped, it will be imported with no problem, so it is not a specific record as far as I can tell.

After a stopped import, the last record created contains all of the metadata, but is not connected to its jpg. Derivative image files for the jpg are created in the archive directory (one each in files, fullsize, thumbnails and square_thumbnails) but there is a broken image link icon on the record where the image should show up.

If I delete that last record created, this shows up in the error log

2012-10-25T12:52:55-07:00 WARN (4): Omeka_Storage_Adapter_Filesystem: Tried to delete missing file 'files/39ddeb5f7dde66d1e7bf4192b92fb186.jpg'.

I can manually attach the jpg to that last record, but with hundreds of records and images, I’d rather be able to import large files.

I have done this over and over, and the import hangs at some seemingly random point between the 60th and the 80th record. I have uninstalled and reinstalled both the CSV Importer and Dropbox plug-ins several times. I have double and triple checked my csv file. I have deleted and re-uploaded the jpgs to the Dropbox folder and made permissions on all of the Dropbox files and folders as open as possible. I have used both Chrome and Firefox, and cleared the cache/deleted cookies in both browsers.

The error log only shows.

2012-10-25T09:41:52-07:00 DEBUG (7): [CsvImport] Started import at: Oct 25, 2012 9:41:52 AM
2012-10-25T09:41:52-07:00 DEBUG (7): [CsvImport] Item import loop started at: Oct 25, 2012 9:41:52 AM
2012-10-25T09:41:52-07:00 DEBUG (7): [CsvImport] Memory usage: 22499472

I have also gotten this error page
[http://kplma.org/om/admin/csv-import/index/map-columns]

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@kplma.org and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

I need some help, so thanks for anything.

Here's my system info

User
Browser: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4
Role: super

System
Omeka: 1.5
PHP: 5.2.17 (cgi-fcgi)
OS: Linux 3.2.15-grsec-grsec-2.9-rc1 x86_64
MySQL Server: 5.1.56
MySQL Client: 5.1.49

PHP Extensions
Regular: bcmath, calendar, cgi-fcgi, ctype, curl, date, dom, exif, filter, ftp, gd, gettext, hash, iconv, imap, json, libxml, mbstring, mcrypt, mhash, mysql, mysqli, openssl, pcntl, pcre, PDO, pdo_mysql, pdo_sqlite, posix, pspell, Reflection, session, SimpleXML, sockets, SPL, SQLite, standard, tokenizer, xml, xmlreader, xmlwriter, xsl, Zend Optimizer, zlib
Zend: Zend Extension Manager, Zend Optimizer

Plugins
CsvImport: 1.3.3
Dropbox: 0.6
DublinCoreExtended: 1.1
ExhibitBuilder: 1.2
Geolocation: 1.2 (inactive)
ItemRelations: 1.1 (inactive)
Lcsh: 1.1
OaiPmhRepository: 1.0
SimpleContactForm: 0.3 (inactive)
SimplePages: 1.2
SimpleVocab: 1.3 (inactive)

Themes
Minimalist Omeka Theme:1.2
Rhythm: 1.3 (current)

Thanks again!

Have you tried breaking your CSV into smaller chunks, of 40 records, then importing each smaller CSV file separately? There could be a problem with the post limit for your server that stops importing the files after it reaches the max limit per import which it may reach after 50 because of the file size, not the number of records.

My guess is that if you eliminated the file column completely, you should be able to import the entire CSV.

Thanks for the reply.

Yes, I can import smaller csv files.

Yes, I can import the full csv file without the link to the Dropbox image jpgs.

My issue is that either of these two solutions means far more manual work. I would have to make and import 10 files instead of a single file just for this small set, or link hundreds of jpgs to their metadata one record at a time, rather than automatically.

I have not seen any errors in the server log. I don't know if there's a post limit on imports, but will look into this. When you wrote "file size" did you mean the csv file or the image files? Any suggestions about what kind of error to look for?

I do think that this is a serious limitation to Omeka for me, as I have thousands of records to upload. I would like to find a better solution if there is one.

Thanks,
Susan

I was talking about the file size of the individual images or whatever types you are trying to import together from the CSV file.

Here some more information about post limits. This setting is found in the PHP installation running on your server. If you don't have access to that, you will need to ask a server admin to check that for you.
http://omeka.org/forums/topic/problem-adding-multiple-files-to-an-item

Thanks for pointing me in the right direction!

With that thread and some searching, I found that the php settings for script execution time, input time, post size and file size all affect image uploading in Omeka (and several other image-related applications).

I got these straightened out and can upload big batches of records pretty easily now.

Thanks again.

Hi Susan is there a way you can post more information on which exact settings you changed?

I'm doing 7000+ images migration from ContentDM and my csv files are around 150 items in size, my import usually errors out at around 80 items through the import.

Would love to know how can make my life easier of not converting those files into smaller chunks.

Thanks,
Alex

Hi Alex,
It is a fix on the server. I added a phprc file to my site with the contents

upload_max_filesize = 64M
post_max_size = 64M
max_execution_time = 1200
max_input_time = 500

The details specific to Dreamhost sites are here http://wiki.dreamhost.com/PHP.ini

Good luck.
Susan

I am struggling getting getting the correct URL from dropbox. For example, I am using:

https://www.dropbox.com/s/d9w5asc7idxv5nv/Humors_of_Piano_Selling%20_1894.pdf?dl=0

but CSV importer skips.

Gnosis Omeka - this post is about the Omeka Plugin Dropbox, not the service. Could you start a new post, please?

Sorry!