If an admin unchecks "Public" on an item, I think they are likely assuming that all public access to both the item record and its component files is being revoked, but that's not currently the case. Although a 404 page is appropriately shown to visitors for .../item/show/[Some ID] for items whose "Public" property is set to false, the files associated with the items are nevertheless accessible.
(e.g. /files/fullsize/[some 32-digit hex number].jpg or
/files/thumbnails/[some 32-digit hex number].jpg )
Of course, the visitor would have to already know the URLs of the files themselves, so this issue really only applies to cases where items were made public, then some visitor saved the file URLs for later use (or, say, directly linked to them from their blog or whatever), and then the Omeka installation's admin decided that the items needed to be withdrawn from public view, for whatever reason.
So, the issue here is that Omeka isn't actually involved in serving the files. They're served directly by Apache from the files directory.
Now, it's possible to change this behavior. The S3 adapter has the ability to store files as "private" on S3, and only creates one-off signatures allowing temporary access to them, so when a item is made private, the files also automatically become private.
It's also possible to make a storage adapter that would work this way on a local machine. It would work like this: the files would be stored somewhere Apache can't serve from by itself, either by being located outside the DocumentRoot, or by being denied through server configuration or .htaccess rules. Then, the URLs the adapter generates would point to a simple script that just checks against the Omeka permissions before delivering the file. Omeka doesn't come with an adapter that does this, but it would be pretty simple to implement.
I would vote for your second suggestion, John!
You could implement this by writing an .htaccess and put it in the "files" folder redirecting all requests for files to a (new) script, say showFile.php. There is a tradeoff between speed and security, which means I would only do this for original and fullsize - not for thumbnails.
I agree to John, it's not that complicated to implement (in theory).
I'm not sure it can be implemented by writing a plugin - I think you have to perform the tweak the ugly way, i.e. put the script and htaccess file by hand. :-/