News:

"The phone is a remarkably complex, simple device,
and very rarely ever needs repairs, once you fix them." - Dan/Panther

Main Menu

A new but easily fixable problem discovered when uploading images to the forum

Started by TelePlay, July 23, 2017, 01:02:23 PM

Previous topic - Next topic

TelePlay

A member was having problems uploading 3 images, taken with a Nikon camera, directly to the forum.

The problem was a new one, never seen before. Upon attaching the images and hitting save, the screen would go blank. Nothing would be seen or show other than the top browser task bar. This was not a "security error" issue.

The member emailed the original images to me for analysis and testing.

First, I tried to upload them and got the same thing, a blank white screen so it was not his computer.
Second, I check them and they contained a lot of meta data so stripped that out but it did not help, still got the white screen.

Discovered I could work around the white screen, once it appeared, by clicking on the "back" browser arror (top left), clicking on the screen refresh "circle arrow" in the top center (or pressing F5) and once the screen refreshed, clicking on save. However, if 3 images were attached, only one would show up. The images had to be uploaded one at a time using the "back-refresh-save" process. But they were there, as taken in full size.

The real solution was to reduce the images in size. They were take with 4,320 horizontal pixels so I reduced each image in size to 2,160 images and saved them as a new file. I attached all three images to the topic they were intended to be posted and all three loaded at once.

So resizing was the solution here but don't know why. Was it something in the photo code or was it the size of the file or both.

Anyway, if you get a white screen, don't panic. Simply click "back," refresh the screen and click on save. Do not try to re-attach the image. It's there already so click save. If you try to attach the image again, you will get an error saying you already uploaded the image.

A screen capture of the white screen is attached.


unbeldi

This is actually not a new problem.  It is the identical problem that occurred before I asked to install a patch I had created to increase the memory allocated for thumbnail creation, which was implemented about two or three years ago.  But with images large enough the memory allocated would obviously be still too small.

TelePlay

Quote from: unbeldi on July 23, 2017, 01:47:11 PM
This is actually not a new problem.  It is the identical problem that occurred before I asked to install a patch I had created to increase the memory allocated for thumbnail creation, which was implemented about two or three years ago.  But with images large enough the memory allocated would obviously be still too small.

It might well be but wasn't it the case that a large image, over 4 mBytes would not just give a blank screen but would lock the top for everyone and the person uploading the large image would have to PM a moderator and that mod would have to tell Dennis which reply to delete from his administrative level access? The whole reply would be deleted and the member would have to start over with smaller images.

I have not seen a topic lock like that for years. And these images were but 3 mBytes in size. They did not lock the topic, merely stopped with a blank screen not being able to show the image even though it was attached.

I guess it goes to the larger point that images should always be cropped and resized to a reasonable file size before uploading, something we've encouraged for a long time now, ever since the topic locking issue occurred.

In the end, the solution is known, reduce the size of an image to 1,500 or 2,000 on the widest side before uploading. That's plenty of resolution to see the finest detail. Images can be edited (cropped and resized on smart phones) so that's not an excuse. And no reason to upload images that large anyway, other than a minor inconvenience to the member uploading the images.

unbeldi

Quote from: TelePlay on July 23, 2017, 02:08:55 PM
It might well be but wasn't it the case that a large image, over 4 mBytes would not just give a blank screen but would lock the top for everyone and the person uploading the large image would have to PM a moderator and that mod would have to tell Dennis which reply to delete from his administrative level access? The whole reply would be deleted and the member would have to start over with smaller images.

No.  It simply crashed the current server thread and presented a white screen (no HTML output), just as described; at least this was the predominant behavior. Before the patch this happened quite frequently. The page could be recovered with the back-button in the browser from the browsers cache.

Interfering with the subsequent display of the topic could very well be related though, but requires the crashing of the thread after partial page output and incomplete database transactions, so that HTML divisions aren't closed properly.  I have never been able to reproduce this, it may well be related to insufficient resources allocated to the PHP subsystem in general, rather than to the SMF code base.

The change for the memory limit for the resize operation was increased from 90 MB to 256 MB.

Quote
And these images were but 3 mBytes in size.
3 mBytes is very very small.   That means 3 millibytes.   The symbol for mega is capital M, as in 3 MB. 

The file size of the image is not what counts actually.  What counts is the size of the uncompressed, raw image.

They did not lock the topic, merely stopped with a blank screen not being able to show the image even though it was attached.

TelePlay

Quote from: unbeldi on July 23, 2017, 02:30:12 PM
The change for the memory limit for the resize operation was increased from 90 MB to 256 MB.
3 mBytes is very very small.   That means 3 millibytes.   The symbol for mega is capital M, as in 3 MB. 

The file size of the image is not what counts actually.  What counts is the size of the uncompressed, raw image.

They did not lock the topic, merely stopped with a blank screen not being able to show the image even though it was attached.

Didn't know that, or never saw it used or explained that way mB vs MB. Yes, I meant MBytes. I guess I've been a bit sloppy in my use of that term.

I did some more testing and yes, you are correct. Doesn't lock the screen but hangs it up and if the back button is not used, if the browser is stopped, there may be no way to go back to remove the large file without administrative help. I went to the home screen and used the back button to get back to the white screen. Had I ended the browser session, that option would have been gone a reply deletion by the administrator would have been the only fix.

So, same problem, different symptoms. Still reducing image size for all posting images is a small thing to ask. This image( 1,237 x 1,500 pixels) at 72 dpi and states it is 450.7 KBytes in size. And while it was 1/10th the size of the image causing problems, it's actual size on the forum is 193KBytes (13 KByte thumbnail shown below) - click this link to get the original image which can be click on to get the full sized image

     (  http://www.classicrotaryphones.com/forum/index.php?action=dlattach;topic=18509.0;attach=165201;image  )



The Jade Green AE 50 as saved has been reduced in size by SMF software modules to less than half that size. EXIF code for this image shows the image as saved creator being :  gd-jpeg v1.0 (using IJG JPEG v62), default quality -- it's posted size is 193KBytes.

It might be that the SMF module resizing the images to something smaller (and I'm not sure on exactly which size and above that starts at) still has an issue with very large files (>4 MBytes) except instead of instantly locking the topic, show a white screen so the member can back up and load a smaller image, something under 2 MBytes.

TelePlay

Quote from: unbeldi on July 23, 2017, 01:47:11 PM
This is actually not a new problem.  It is the identical problem that occurred before I asked to install a patch I had created to increase the memory allocated for thumbnail creation, which was implemented about two or three years ago.  But with images large enough the memory allocated would obviously be still too small.

Yes, your patch was implemented. And yes, it's most likely the case that the improvement in cameras and phones has resulted in all images taken on them being much larger than when your patch was implemented.

If you would like to PM me a new patch with a large space allocation, whatever you this is sufficient, we will get that implemented. Also, tell me how much space allocation is being allowed, expanded to with the patch.

Getting a new patch in may take some time to implement so in the meantime, the maximum image size allowed will be 3 MBytes. I got a white screen at 3.1 MBytes and a 2.9 MByte image uploaded fine so from now on, anyone trying to upload an image greater than 3 MBytes will get a over size error.

If that happens, the member will have to hit the back arrow, refresh the screen (F5) and attach a smaller image.

Jack Ryan

I know that one is often very restricted in what can be achieved with a patch but is it possible to determine the required buffer size from the original image size? A fixed size buffer should never have been used in the first place - particularly if the requirement is not being calculated and tested for.

Thanks to Unbeldi for taking the time to create the patch - the authors should be so responsive.

It is probably not necessary to post such a large image but with so many gadgets in use now there is not always the opportunity to crop and down sample.

Off on a tangent - now that "mega" has been addressed, we should address "seconds". Milliseconds is ms, mS is milli-Siemens.

Jack

TelePlay

Funny, Jack. Seriously, that made me smile.

I am guessing but I would say the last patch created by unbeldi was 3,000 kBytes. As part of my testing, I could upload a 2.9 MByte image but a 3.1 MByte image gave me a white screen.

There are size options, actually many option, for posting. 6 images per reply is one that has been in place for a long time. We just changed the maximum image size from "0" (unlimited size" to 3,000 kBytes (3 MBytes so as not to confuse anyone).

The question goes to what is considered too large an image and what is gained by such a large image?  My 12 MPixel camera and phone takes at most a 3 MByte image (the first image below is right off the my Nikon 12 MP camers). After rotating, cropping, resizing so the max side is 2,000 pixels and saving it as a jpg, the size drops to 1.2 MBytes. Both are under 3 MBytes so they both uploaded right now under the 3 MByte cap. Click on either image to expand to see the detail.

So how big of a file is really needed to upload and the question you raise is how much temporary space has to be allocated for the software to do an extensive security test of the image and if an issue is found, send the file to the GD module for stripping of meta data and recoding and downsizing the image to get a complete image before the original is erased and the newly created file sent to extensive security checks again to make sure the issue was removed? 6 MBytes? 8 MBytes? 10 MBytes for a 4 MByte file upload? Don't know but if unbeldi wrote a patch for 3 MBytes of internal work space, then he should be able to change that patch to 6, 8, or 10 MBytes and the maximum allowed file size increased to 4 MBytes.

Seems we got rid of the security error issue and came up with a space allocation issue. That setting, whatever the patch does, is deep in the code and not a box to be checked, unchecked or a number filled in.

That's where it stands today, except for the anticipated, the expected barrage of complaints about not being able to upload a file bigger than 3 MBytes. Well, unless you wanted to see the white screen, you couldn't do that before the maximum was put into place.

unbeldi

Quote from: Jack Ryan on July 23, 2017, 07:52:56 PM
I know that one is often very restricted in what can be achieved with a patch but is it possible to determine the required buffer size from the original image size? A fixed size buffer should never have been used in the first place - particularly if the requirement is not being calculated and tested for.

I agree that the software could be written more intelligently to test image size before allocating memory for the operation. But this is not universally reliable and may involve decompressing the image before the result can be known, and that requires allocation of memory in the first place.  But as written, the SMF software does not allocate fixed buffers.  The parameter that was changed was the maximum amount of memory that could be allocated for buffers. There has to be some limit, all computers have some limited amount of memory or CPU power to process large images. So the amount of memory to set aside for processing images is a decision that involves several criteria, it helps nobody, to allocate so much memory that the CPU can't possibly deliver a result in a reasonable amount of time to provide a satisfactory user experience of the web site. Compromises have to be decided.

My opinion of the image processing code in SMF is that it is in general poor, if the intent is to use and process images extensively. Another aspect is the poor handling of the 'security' concerns inherent in embedded exif data in images.  Most forums perhaps don't rely on pictures as extensively as what we do here, and small parameter limits may be acceptable.   Most of these aspects involve a combination of technical and administrative components.  The developers can't be expected to solve all problems.  Since the software is open-source anyone can customize it for specific purposes, features, and rules.  IMHO, this is more powerful, secure, usable, and affordable than buying commercial software.

A 4000 x 3000 pixel image has 12 million pixels. If the depth is full 24-bit color, this image requires 288 Mbits of storage in memory, uncompressed, or 36 MB.   If the image is a gray scale image, perhaps in PNG format, it only requires one third of 36 MB. Compression may reduce file size to a mere fraction of those amounts. It is therefore quite unpredictable which size of images are acceptable for upload to the Forum.  It always depends on the type of image and its compression method.

[PS: edited arithmetic errors]

unbeldi

Quote from: TelePlay on July 23, 2017, 09:45:44 PM
Funny, Jack. Seriously, that made me smile.

I am guessing but I would say the last patch created by unbeldi was 3,000 kBytes. As part of my testing, I could upload a 2.9 MByte image but a 3.1 MByte image gave me a white screen.


See my previous responses.   It is not the file size that is important.
The patch increased the memory limit from 94 MB (IIRC) to 256 MB,  not 3 MB.

Your example of a 4000x3000 file shows the enormous effect of compression, it is over 90%.

Quote
There are size options, actually many option, for posting. 6 images per reply is one that has been in place for a long time. We just changed the maximum image size from "0" (unlimited size" to 3,000 kBytes (3 MBytes so as not to confuse anyone).
In the process you probably also limited the file size for PDF files.  I don't think this is a good idea.  I don't recall that the admin pages have separate settings for different file types.

Did you really limit uploads in size ?


Quote
So how big of a file is really needed to upload and the question you raise is how much temporary space has to be allocated for the software to do an extensive security test of the image and if an issue is found, send the file to the GD module for stripping of meta data and recoding and downsizing the image to get a
The memory setting has nothing to do with security checks.  I think I said it is for the resizing operation when creating thumbnails only.

[PS: edited for unit conversion error bits/bytes]

unbeldi

It seems I was right.  I just tested posting a PDF file of about 4 MB.  It failed with a message.

You have limited the file size of uploads, this has nothing to do with image size.
Many PDF files uploaded here exceed the 3 MB limit.

By my experience, the software environment of the Forum, until now, has made file uploads of somewhat over 20 MB fail on most attempts.  This is likely a limit of processing time allowed in a PHP thread, but I can't be sure without having shell access to the server.

Please reconsider.

unbeldi

Quote from: Jack Ryan on July 23, 2017, 07:52:56 PM
Off on a tangent - now that "mega" has been addressed, we should address "seconds". Milliseconds is ms, mS is milli-Siemens.

Yes,  8)  and let it also be known that the unit is the siemens, not Siemens.   1 millisiemens (mS).

The SI specifies that when a unit name is derived from a person's name, the unit name is always written in lower case (in English), but the unit symbol is upper case (in any language).
Other examples:  hertz (Hz),  ampere (A).

TelePlay

The upload size limit has been changed from 3 MBytes back to unlimited.

This means large pdf files can once again be attached to posts or replies.

That does not mean a very large image can be uploaded. It is still the responsibility of the posting member to crop and/or reduce the size of an image so that it is under 3 MBytes. Failure to do so may result in the "white screen" due to the system running out of now more than sufficient allocated resources - the screen load stops when it fails to complete image processing due to it's large size.

If that happens (you get a while screen when uploading an image), the only solution is to  hit the browser "back" arrow, refresh the screen (F5), and attached a down sized version of the image that caused the white screen. Only the uploading member can recover from the "white screen" by going back one page and replacing the large image with a smaller version of that image. No exceptions - moderators can not help a member recover from a "white screen" other than writing his long paragraph.

It is recommended that a *.jpg image be uploaded with no more than 2,000 pixels on it largest side. 1,000 to 2,000 pixels on the largest side provide a good quality thumbnail that is enlargeable to see detail.

Anyone needing help cropping and/or resizing images can send me a PM.

TelePlay

As an update to this size/over flow issue, a member came across a unique event with a photo being uploaded. It was a large (4000+ by 3000+ pixel image) and caused the SMF image processing software to burp.

I know the cut off size is right around that with some image uploading and some not most likely due to the depth of the image greatly increasing the file size of an image this large.

This particular image got half way through processing and locked the screen, stopped loading and presented the blank screen shown above.

I've had this happen to me more than a few times when trying to upload large zipped files. When this happens, the topic is "locked" and nothing can be done until the topic is refreshed by using F5, the refresh icon on the browser or cancelling out of the topic and starting over. If you cancel out, simply highlight the text typed, copy it to the clip board and paste it back into a newly opened reply page.

This image ended up loading but with a thumbnail that showed it sideways and when clicking on it to enlarge the image, it rotated according to the rotation instruction in the EXIF data, which should not have been there since the SMF security module has been recoding all images uploaded stripping away almost all of the image EXIF data.

Viewing the EXIF data for this image that got "through" showed most of the EXIF data was stripped but 3 of the normally 70 pages still existed and the image did not show the recoding module information.

When this image was being processed, it maxed out the processor buffer must have been in the process of stripping the data but never finished the job leaving the rotation instructions and it did not recode the image.

When a large image or file is uploaded, two things can happen: 1) the text portion of the reply goes blank, the processing stops and the open reply screen is locked up requiring it to be refreshed; 2) a new pops up completely empty as if starting to post a new reply.

The solution is to not try to upload large images. The fix is to cancel out of the reply (saving any text), resize the image and create a new reply with the saved text and smaller images.

I've had this happen to me more than a dozen time but with trying to upload known very large pdf or zipped files. Images should be resized to 2500 pixel on the largest size which is plenty of resolution after recoding for close up detail.

If members continue to upload large images, one solution would be to reimpose the file size limitation which was not accepted last time we did that to reduce image uploading problems.

This one picture was unique being half processed, created a blank screen but was still attached to the reply with rotation information included and not recoded. This post is just to document this in case the event is needed for future problems and to remind members to rotate, crop and resize images before attaching them to a file for uploading.

Dan/Panther

Many problems of uploading photos to the internet would be eliminated if the one uploading those photos would compress or reduce the image size. Most if not all forums will give the specs for a maximum size photo to be uploaded. One way to check,  right click your photo, click on properties, scroll down to image size. If it says MB"s, your photo is most likely too large to upload, once resized the same readout should be in the Kilobyte region not Megabyte.
One more item, if you want to maintain the original detail of the photo, make a copy of it, then resize the copy, that way your original is exactly as you took it.

D/P

The More People I meet, The More I Love, and MISS My Dog.  Dan Robinson