I have connected my Omeka site with Amazon S3 via Zend. I copied the entire "files" directory to my S3 bucket and created ACLs to allow PUT and GET object permissions. Using the account I'm using in config.ini, I can write to the S3 bucket via sftp, but I'm still unable to do so via Omeka. When adding a file (uploaded to Dropbox) to an item, I get the following. Permissions seems to be correct, and Omeka is pulling from S3, but this upload is not working. Ideas?
Omeka_Storage_Exception
Unable to store file.
exception 'Omeka_Storage_Exception' with message 'Unable to store file.' in /data/digitalpitts/exhibits/application/libraries/Omeka/Storage/Adapter/ZendS3.php:100
Stack trace:
#0 [internal function]: Omeka_Storage_Adapter_ZendS3->store('/tmp/f93f7e6916...', 'original/f93f7e...')
#1 /data/digitalpitts/exhibits/application/libraries/Omeka/Storage.php(67): call_user_func_array(Array, Array)
#2 /data/digitalpitts/exhibits/application/models/File.php(424): Omeka_Storage->__call('store', Array)
#3 /data/digitalpitts/exhibits/application/models/File.php(424): Omeka_Storage->store('/tmp/f93f7e6916...', 'original/f93f7e...')
#4 /data/digitalpitts/exhibits/application/models/Job/FileProcessUpload.php(23): File->storeFiles()
#5 /data/digitalpitts/exhibits/application/libraries/Omeka/Job/Dispatcher/Adapter/Synchronous.php(25): Job_FileProcessUpload->perform()
#6 /data/digitalpitts/exhibits/application/libraries/Omeka/Job/Dispatcher/Default.php(136): Omeka_Job_Dispatcher_Adapter_Synchronous->send('{"className":"J...', Array)
#7 /data/digitalpitts/exhibits/application/models/File.php(203): Omeka_Job_Dispatcher_Default->send('Job_FileProcess...', Array)
#8 [internal function]: File->afterSave(Array)
#9 /data/digitalpitts/exhibits/application/libraries/Omeka/Record/AbstractRecord.php(277): call_user_func(Array, Array)
#10 /data/digitalpitts/exhibits/application/libraries/Omeka/Record/AbstractRecord.php(550): Omeka_Record_AbstractRecord->runCallbacks('afterSave', Array)
#11 /data/digitalpitts/exhibits/application/models/Item.php(335): Omeka_Record_AbstractRecord->save()
#12 /data/digitalpitts/exhibits/application/models/Builder/Item.php(204): Item->saveFiles()
#13 /data/digitalpitts/exhibits/application/libraries/globals.php(567): Builder_Item->addFiles('Filesystem', Array, Array)
#14 /data/digitalpitts/exhibits/plugins/Dropbox/DropboxPlugin.php(100): insert_files_for_item(Object(Item), 'Filesystem', Array, Array)
#15 [internal function]: DropboxPlugin->hookAfterSaveItem(Array)
#16 /data/digitalpitts/exhibits/application/libraries/Omeka/Plugin/Broker.php(157): call_user_func(Array, Array)
#17 /data/digitalpitts/exhibits/application/libraries/Omeka/Record/AbstractRecord.php(298): Omeka_Plugin_Broker->callHook('after_save_item', Array)
#18 /data/digitalpitts/exhibits/application/libraries/Omeka/Record/AbstractRecord.php(550): Omeka_Record_AbstractRecord->runCallbacks('afterSave', Array)
#19 /data/digitalpitts/exhibits/application/libraries/Omeka/Controller/AbstractActionController.php(229): Omeka_Record_AbstractRecord->save(false)
#20 /data/digitalpitts/exhibits/application/controllers/ItemsController.php(91): Omeka_Controller_AbstractActionController->editAction()
#21 /data/digitalpitts/exhibits/application/libraries/Zend/Controller/Action.php(516): ItemsController->editAction()
#22 /data/digitalpitts/exhibits/application/libraries/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('editAction')
#23 /data/digitalpitts/exhibits/application/libraries/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))
#24 /data/digitalpitts/exhibits/application/libraries/Zend/Application/Bootstrap/Bootstrap.php(105): Zend_Controller_Front->dispatch()
#25 /data/digitalpitts/exhibits/application/libraries/Zend/Application.php(382): Zend_Application_Bootstrap_Bootstrap->run()
#26 /data/digitalpitts/exhibits/application/libraries/Omeka/Application.php(79): Zend_Application->run()
#27 /data/digitalpitts/exhibits/admin/index.php(28): Omeka_Application->run()
We don't get much in the way of feedback from Zend's S3 code, only whether the operation failed or succeeded.
Is the bucket located in some non-standard AWS endpoint? Some people's S3 problems have been caused by that kind of issue, and specifying the endpoint URL in the config file has solved the problem.
Thanks John. I have specified the endpoint in the Omeka config file. That took care of the access issue I was having. Now I can see everything that is in the s3 bucket, but Omeka seems unable to move anything from /tmp to the Omeka bucket (specifically the /original folder is where the application seems to throw the exception). Bucket permissions are there, but write is failing.
Can you post the S3 configuration you're using (with the keys redacted of course)?
Sure; config.ini section is below (keys redacted and bucket name changed), followed by User policy for relevant bucket:
; Sample S3 cloud storage configuration
;
; The accessKeyId, secretAccessKey, and bucket options are all required.
; If the expiration option is set, files will be uploaded with "private"
; access, and Omeka will generate URLs that are only valid for a limited
; time. If the expiration option is missing or left commented out,
; uploaded files will always be publicly readable.
;
storage.adapter = "Omeka_Storage_Adapter_ZendS3"
storage.adapterOptions.accessKeyId =******
storage.adapterOptions.secretAccessKey =*****
storage.adapterOptions.bucket =mybucket
storage.adapterOptions.expiration = 30 ; URL expiration time (in minutes)
storage.adapterOptions.endpoint = http://s3-us-west-2.amazonaws.com ; Custom S3 endpoint (optional)
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "WebOmeka",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::mybucket/*"
]
}
]
}
I think the ListBucket permission needs to be on the bucket itself, not the objects within, so it should be a separate grant on the Resource arn:aws:s3:::mybucket (without the trailing /*).
I'm not sure whether this is the cause of your issue, because I don't think ListBucket is required for putting an object, but it's the only thing that's jumping out at me.
Otherwise I'd be tempted to point the finger at other simple problems like the policy not being attached to the user Omeka's configured to use. We've had some previous problems with people using S3 buckets with underscores in their names being incompatible with Zend's code, but this doesn't look like that's what's happening.
Thanks again, John. I tried what you suggested, but still no luck. Just for a test, I granted all privileges to this user (yes, the correct user) on both the buckets and the contents within, and I'm still getting the "unable to store file." exception. I even created a new Omeka user and a new bucket, with the proper permissions in AWS, and still Omeka is unable to write. Thanks again for trying to help!
Have you checked PHP logs (not Omeka's logs) on your server? Depending on your setup, you might get better logging if you're doing something that doesn't happen in a background job, like a regular one-off upload.
Your problem could be something as simple as bad DNS settings and/or outbound connectivity from the server hosting your Omeka site, so it just can't connect to the AWS servers. Some situations like that would probably produce PHP notices or warnings pointing to the problem.
Thanks, I will check the logs. The confusion I have, though, is why GETs from S3 are working fine (and i can turn it on/off by changing permissions) but PUTs are not. Does that make any sense?
Well, for the GET side, Omeka just generates a URL, so it's the computer viewing the page that actually performs the GET request. On the PUT side, it's Omeka on the server side performing the request to put the object up to S3, so there's room there for the problem to just be something with the server's environment or settings.
Thanks again, John. I'm a bit confused, for the problem would seem to exist with the permissions on the S3 server, which are controlled only by the ACL in the S3 console, for which I have set up the proper permissions. Nothing on my server running Omeka should have a problem with its trying to write a file to another server (which is what it is trying to do). Indeed when I use the Omeka /files directory as my storage solution (so no S3), things work fine. There seems to be a permissions problem when Omeka tries to PUT to s3. I'll keep banging my head against the table on this, but I appreciate any ideas you might have!