Number of Objects per Item

I have created many items per Collection. Am running into a problem with....

I have added 7 image files per item. When I try to add the 8th image file, it fails with the error message:

"Something went wrong with image creation. Please notify an administrator."

I tried sizing the image file down. No go. I tried deleting a prior image file and uploading the 8th image file which made it the 7th image file after deleting the prior one. It worked. I then tried to upload the now deleted and formerly successful 7th image file. It failed.

What I am seeing is there is not a size limitation, but a number of Image files per item limitation. Can anyone shed some light on this? I have instances where I have far more than 7 image files per item. Things have come to a grinding halt.

Any ideas?

Hi drhiii,

There is no restriction in Omeka on the number of images per item. I just added a dozen images to an item in Omeka version 1.1, 1.2, and 1.2.1. It doesn't sound like you're running into a POST limit on your files, but rather an issue with creating derivative images for the image file you're uploading. The fact that you're able to upload the image after deleting another from the item is curious, though.

Just for my own clarification:

* Are you failing to add more than seven images on other items, and with different images?

* Are you able to add more than seven files of any kind to an item, including the same images you're having trouble with?

Best,
Jeremy

Hello,

And thank you for the response.

After coming back to it fresh, have discovered the following... after having added approx 150 object/images, Imagemagick decided to stop working. I made no upgrades to the server.

The binary is located in /usr/bin/convert and everything I do verifies that it is there. I physically go to the directory and list it, I run it with just 'convert' and also with /usr/bin/convert. 'which convert' reports /usr/bin as the location. I have even set the perms to 777 for grins, and when I run the 'Test' under Settings in the admin screen, it 'Fails'.

This is perplexing because it worked for 150+ images. And when I ran into the '7' limit issue, I had some odd responses, where I was able to add up to 7 images, but the 8th image attempt failed. It seemed to suggest an item limit. But on returning to this today, I am seeing a different error that was not evident last week.

I suppose I can remove and readd Imagemagick via Debian 'apt-get', and I can also build it from source. I remember having an issue with this when I first tried Omeka several months ago. Oh, am using v1.2. A build from source solved that.

But what is perplexing is having it work for a few weeks and 150+ image files, and not touching the server but having it fail. Even tho I can manually invoke convert and have read everything I can get my hands on....

There has to be a logical answer for this. At the moment, I am perplexed.... convert is indeed in /usr/bin and it works.

Hmmm..

tx, david

For grins, here is what is reported for the version:

Version: ImageMagick 6.4.5 2009-06-04 Q16 OpenMP http://www.imagemagick.org
Copyright: Copyright (C) 1999-2008 ImageMagick Studio LLC

There are some other image rendering programs that use Imagemagick on the same server (like Gallery2 and Gallery3 - aka G2 and G3), and they all work.

I also disabled file extension validation on prior versions of Omeke installs, as well as this one, to overcome an upload issue, and all worked well until....

Perplexed....

Last report.... I have a second installation of Omeka 1.2 on the same server. And it also 'fails' in the Test Settings. Where it had been working for a few months.

All I can think is something has changed in the server environment... except I have made no changes, and it worked during a series of uploads until it decided to stop working during a session.

I may try to reinstall from source and see if that overcomes this. If anyone has a similar experience and a solution, am all ears....

Again, 'convert' is located in /usr/bin and it does work.

Do the php settings influence the running of Imagemagick? I recall reading something about that.

tx

Ok, one more post...

I have two installs of Omeka on the same server. Both report 'fail' when I test the Settings.

However, version 2 of Omeka allows uploads of .jpg and .jpeg files. version 1 of Omeka, which is a totally separate installation, which has the exact same settings, all the way down to the permissions at the file and directory level, iow I verified that all things were as equal as I could see them... and this version failed to allow an upload even tho it is on the same server. And all of the settings at the app layer and the filesystem layer, are identical.

I went so far as to deactivate all Plugins just in case something was conflicting. No go.

Head scratching....

Odd tho that both versions fails the Settings test for Imagemagick, but one version allows uploads, and the other doesn't. May install a fresh Omeka and see if it will allow uploads. If yes, I may just gut it out and reupload the items tho that would be a bit of a drag with 150+ objects already uploaded...

There has to be a logical answer somewhere. Oh, also... I do not see anything reported in the Apache error.log files either. I am not getting anything that gives me a hint as to why it is failing....

tx

Last post for reals... uploading a mpeg file works fine. As does creating a record by itself with no associated media file.

I lied... last post for reals. But here is where it gets interesting, and why I felt there was a limitation on images.

I deleted one of the records, the last record that I was successfully able to upload. I then was able to successfully upload images.

However, when I tried to add more the 7 images, this added to the 150+ images, it failed. I could add 6 images. But the 7th image, and this didn't matter whether I created a new record or appended images to an existing record, it failed.

This is why I kept thinking there was an image threshold. Because again it appears I can only add 157 images (that is not the precise number but close to it), and anything above that number, fails. I don't think there is a file number limitation... but, a file size limitation? Anyone have any clue as to why I reach this limit of file images and no matter what I do, it fails after a certain number. Appears to be at 157....

Odd...

Is it possible you have maxed out your server's storage space? Are you hosting this yourself or using a third-party host?

As Jeremy said above, there is no limitation within Omeka on the number of files you may upload but individual server settings may limit the total amount of space you have to store everything. It's possible that you've maxed out your storage so you cannot upload any more images.

I am confused by the way you have talked about the ImageMagick configuration--it works and its fails.

You've said in one post: "convert is indeed in /usr/bin and it works."

In the next post you write, "I have two installs of Omeka on the same server. Both report 'fail' when I test the Settings. "

Sounds like the path to convert is not correct.

What is the path that you are testing when it fails? /usr/bin ?

Try typing "which convert" (without quotation marks) into terminal when you are inside one of your Omeka installations to see its path and then type that in (minus the /convert) to the settings field.

Tx for responding. Will thread responses....

> Is it possible you have maxed out your server's storage space? Are you hosting this yourself or using a third-party host?

No and no. I installed this server (have run *nix servers since '93 so have experience, gobs, on running them). I have only used 27% of capacity. There is a little less than 1TB of space available, so am definitely not out of space. Also, there are no quotas on the file system. I have other apps that have used more space in their filesystems on this server, so quotas are not an issue.

> As Jeremy said above, there is no limitation within Omeka on the number of files you may upload but individual server settings may limit the total amount of space you have to store everything. It's possible that you've maxed out your storage so you cannot upload any more images.

That is the first thing I observe on these things, primarily since I mount and run these servers myself. I am definitely not maxed out on space. I know there is supposed to be no limitation on space, but that is exactly how it is acting. I can add for instance 150 image files. When I add the 151st image file, it fails. I can delete image file 148, 149 and 150. Then I can upload image file 151, 152 and 153, but image file 154 fails. It seems to hit a threshold. Very strange. I have tried different file extensions meaning .jpg and .jpeg I have tried different files meaning image files, thinking the particular set of images files may be damaged. Nawp. Still fails. I have tried renaming filenames. No go. When I uploaded a different media type as in an mpeg and avi files, it worked.

I looked at the .htaccess and php settings to see if there is some kind of threshold, but nothing jumped out.

> I am confused by the way you have talked about the ImageMagick configuration--it works and its fails.

I have two versions of Omeka 1.2 installed on the same server. Let's call them OmekaInstall1 and OmekaInstall2. When I ran the Settings 'Test' for imagemagick, both tests failed. However, when I uploaded image files into OmekaInstall1 they worked. When I uploaded image files into OmekaInstall2, it failed. That's what I meant.

However, as a last test, I deleted a few image files from OmekaInstall2 and was able to then upload the same number of image files just deleted into OmekaInstall2, but when I tried to add one more image file beyond what the so called limit appeared to be, the upload failed. I think I will make a little video and show what I mean. It is very perplexing.

> You've said in one post: "convert is indeed in /usr/bin and it works."

Yes, it is there and yes, it does work.

> In the next post you write, "I have two installs of Omeka on the same server. Both report 'fail' when I test the Settings. "

Yes, when running Imagemagick manually, it works fine. When I run both OmekaInstall1 and OmekeInstall2 on that I explained are on the same server, at one point OmekaInstall1 worked and OmekaInstall2 failed. But after deleting a few images files, I was able to upload the same number of image files until I hit this seemingly magic limit, and OmekaInstall2 failed.

> Sounds like the path to convert is not correct.

It is correct.

> What is the path that you are testing when it fails? /usr/bin ?

Yes.

> Try typing "which convert" (without quotation marks) into terminal when you are inside one of your Omeka installations to see its path and then type that in (minus the /convert) to the settings field.

Here is the output:

root@ns2:/home/dh# which convert
/usr/bin/convert

So, Imagemagick is working properly. I will make a short video and show you folks what I mean. All I can think is there is something causing a limitation in the number of image files I am able to upload. If it is something in the system environment, I will apologise profusely. Things is, I am receiving no error output in the apache logs, Omeka is not spitting out any error, and I cannot find a logical explanation for why I can upload for instance, 150 image files, but it failed on the 151st image file. Yet I can delete files 148, 148 and 150, then upload 151, 152 and 153, but when I try 154, it fails. It is not Imagemagick.

I will also revert to another server and see if the problem persists. I will report that result. If it doesn't, then it has to be a system environmental problem. It is does, then I would look to Omeka. Will let you know.

Thank you for responding...

Ok, this perhaps can put this to rest.

I began flooding the 'other' Omeka installation. Other meaning a second installation that so far had accepted every upload I had throw at it. The first installation is the one that simply stopped accepting image uploads.

On this second Omeka installation, I literally flooded it with uploads. I created Collections, uploaded dozens of image files into the Dropbox (the exact same ones that were failing on the other site that failed), and imported them every way I could think. I eventually ended up uploading the exact image files multiples of times into the same Collection. After everything I had tried, I began to think perhaps there was a filename collision, a naming convention collision, tho I would think there are code checks and balances to prevent that.

No matter what I did, I could not get the 'other' site to fail. What it told me was, Imagemagick was and is not the problem, the server environment was and is not the problem, and the only thing I am left to conclude is there has been a corruption of the data base. I am not gifted in sql, but that is ultimately what it feels like. Because using the exact same files successfully on an Omeka install, flooding it in fact with batch uploads, trying to get it to fail like the first 'problem' Omeka installation, I could not.

It rules out almost everything exact some kind of corruption, tho I still do wonder about some kind of naming collision or corruption. I have since tried to remove some Collections thinking perhaps I could remove whatever corruption may have occurred, but no go. My only recourse is to start over with the install and uploads.

Lessons learned of course are far more frequent backups which everyone should do. Implement a disaster/recovery plan which everyone should do.

But would also suggest that better reporting tools for errors, and some administrative advancements which I know have been talked about in other areas, like batch delete of items for instance. Managing out of a problem like this is very manual intensive and admin tools like deleting and moving items would of course be very helpful. More advanced things like sql integrity checks would be helpful. I have only isolated where I think the problem lies, but it definitely comes down to some kind of corruption that has not been solved by removing objects. Having data base tools for orphaned objects, integrity checks, and so on, I have a feeling would have helped all this. Some day perhaps?

I will have to revert to a fresh install and start over, and implement some far more frequent disaster/recovery things because the inability to recover from this current state is not there. All part of the game...

So, I know I have posted way too much info. But after having wandered down a lot of pre-discovery things, I felt it important to start reporting findings because to arrive at a data base corruption of this type ought to be articulated because if Omeka progresses, would be a good idea to avoid this kind of error, and more importantly, how to recover from this. I really wish I could point to exactly what has happened, but it has all the earmarks of file name collision and inability to recover from that (educated guess) or a sql corruption that has brought things to a grinding halt. Wish I had more preciseness to report.... but I am stuck and have to start over....

No need to respond.

tx

tx

One last boring tidbit in case any of the developers are curious about further behaviour of this issue. I decided not to make a small video of this, but will be happy to if any developer wishes to see this behavior.

In addition to having reached a limit of images, as in 150 images have been uploaded, when trying to upload the 151st image, it fails.

I can delete 5 images making for 145 images, and then add 5 different images to ANY record. When I try to add the 6th one, making it 151 again, it fails.

Also, I have noted that the original file uploads do make it into ./omeka/archive/files. So every image upload attempt, whether it is through the browser or via a Dropbox, the image files end up in ./omeka/archive/files and appear to have the hashed name appended to the original name. That works every time. But there are no derivatives created when I attempt to upload the 151st image. It seems 'stuck' at this limit. And as stated in a prior post, I have another Omeka install and I pounded it with uploads, image file types, through the browser and via the dropbox, and it never failed... ruling out the server environment or Imagemagick. This is what led me to that there has been some kind of corruption, at the sql level??? That's where I am now.

Also, I tried every image file format, from .jpg to .png, .tiff and so on.... they all fail. But I can upload an mpeg or other file formats, and they work. But on this particular Omeka installation, I am stuck on 150 image file uploads, and nothing I do allows me to upload a 151st image. I am having to start over, which of course makes me a bit leery of the integrity and recovery from errors, beyond the standard "make backups". Having created sites with hundreds of thousands of image files and derivatives, I know the challenges of handling vast amounts of media types. This is why the ability to record and reveal errors, and error recovery, is a huge deal, especially in this type of media site. My caution flag has just gone way up when incurring this problem. It is a problem that should be easier to recover from, but so far it has not been so. Starting over and/or building a strong disaster/recovery engine are of course options, but on a runtime system, one gets gunshy when one deals with a lot of objects, and then struggling to recover from what appears to be an data corruption of some type, well, makes one gunshy.

Anyway, above is a bit more information fwiw about what I have observed at the file system level.

No need to respond.

Figured I would add this post. After creating a separate Omeka install, and uploading a series of media files, I ran into another couple of upload problems. The errors were not the same, but they were similar.

Thinking this is too much, I dug in and tried to figure out what else might be causing this erratic behavior. Eventually I stumbled onto a simple tweak, and things started working again.

I reset the permissions on the ./archive directory. I applied both a chmod -R 777 and a a chown -R www-data:www-data which is over the top, but things started working.

What is perplexing, and I think is important, is that I never touched the ./archive directory. After the initial installation, I did not mess with them. But something, what I don't know because I did not look into the ./archive directories before resetting perms. I will when it happens again. But something bolixed up this directory when I attempted to upload, and use a Dropbox, to add files.

Because it happened on two separate installations, and because I did not touch the ./archive directory, the only things that could have happened is an act of god, or something in the code. I bring this up, and hey, I know this thread is as annoying as it gets, but I feel it important to articulate these issues because I am a power user, I have a good handle on servers, and when pushing the applications like this and they break in similar fashion, I figure your team would want to know. I know this is driving you folks nuts, but you need to know this stuff so your code and applications can mature. So much of this is really aiming in the right direction... but there are weaknesses like error reporting and recovery from conditions that, well, this is why I report this.

So, now I am able to upload stuff again into the main problem install. But only after I reset permissions and ownership to the ./archive directory and that to me is a bit spooky because I did not touch them, but something in them failed. I will report any findings if and when it arises again.