Hi all, I have a new website im setting up, its a large image archive,
I have it setup to open a folder, go through and for each file in that
folder check a /tn sub-folder for a matching name, if the name doesnt
exist, it means a new image has been added and that a thumbnail should
be made, when thumbnails are made they are stored rather than deleted
(like most gallery software does) this check happens every time a
folder is opened. the problem is performance, this site will have
around 25k images in each folder, on my testing with 800 images in a
folder the scirpt times out 3 times before all the thumbs are made.
the server is fairly decent but it shoots up to 100% cpu usage when
the script is executed.. this is the script i have for making the
thumbs: (i should mentioned for the folders where all thumbs already
exist its very fast so the issue is in this script somewhere)

* i tried turning the quality down from 60 to 6, it still took 40
seconds to generate 137 images totalling 87mb for the fullsize and
165kb for 137 thumbs

<?php

function create_thumbnail($infile,$outfile,$maxw,$maxh,$str etch) {

if (!preg_match('%\\A/webdir/galleries/[\\wa-z]/[\\wa-z0-9]*/[\\wa-
z0-9]*\\.jpg\\z%i', $infile))
{

die("died");
// if we keep file names sane, then nothing should ever happen here
}

else {


clearstatcache();
if (!is_file($infile)) {

return FALSE;
}


$functions = array(
'image/png' => 'ImageCreateFromPng',
'image/jpeg' => 'ImageCreateFromJpeg',
);


if (function_exists('ImageCreateFromGif')) { $functions['image/gif']
= 'ImageCreateFromGif'; }

$size = getimagesize($infile);


if (!$function = $functions[$size['mime']]) {

return FALSE;
}


if (!$source_img = @$function($infile)) {

return FALSE;
}

$save_function = "image" .
strtolower(substr(strrchr($size['mime'],'/'),1));

// Scale dimensions
list($neww,$newh) = scale_dimensions($size[0],$size[1],$maxw,$maxh,
$stretch);

// Create new image
$new_img = imagecreatetruecolor($neww,$newh);

// Copy and resize image
imagecopyresized($new_img,$source_img,0,0,0,0,$new w,$newh,$size[0],
$size[1]);

// Save output file
if ($save_function == 'imagejpeg') {

if (!$save_function($new_img,$outfile,60)) {

return FALSE;
}
} else {
if (!$save_function($new_img,$outfile)) {
//trigger_error("Unable to save output
image",E_USER_WARNING);
return FALSE;
}
}

// Cleanup
imagedestroy($source_img);
imagedestroy($new_img);

return TRUE;
}
}

// Scales dimensions
function scale_dimensions($w,$h,$maxw,$maxh,$stretch) {
if ((!$stretch) && (($w < $maxw) || (!$maxw)) &&
(($h < $maxh) || (!$maxh))) return array($w,$h);

// Scale Height
if ((!$maxw) || (($h > $w) && ($maxh)) ) {
$newh = $maxh;
$neww = floor($w * $newh /$h);
}
// Scale width
elseif ((!$maxh) || (($w >= $h) && ($maxw))) {
$neww = $maxw;
$newh = floor($h * $neww / $w);
} else
// Scale neither
return array($w,$h);

return array($neww,$newh);
}



}





?>

flamer die.spam@hotmail.com wrote:
> Hi all, I have a new website im setting up, its a large image archive,
> I have it setup to open a folder, go through and for each file in that
> folder check a /tn sub-folder for a matching name, if the name doesnt
> exist, it means a new image has been added and that a thumbnail should
> be made, when thumbnails are made they are stored rather than deleted
> (like most gallery software does) this check happens every time a
> folder is opened. the problem is performance, this site will have
> around 25k images in each folder, on my testing with 800 images in a
> folder the scirpt times out 3 times before all the thumbs are made.
> the server is fairly decent but it shoots up to 100% cpu usage when
> the script is executed.. this is the script i have for making the
> thumbs: (i should mentioned for the folders where all thumbs already
> exist its very fast so the issue is in this script somewhere)
>
> * i tried turning the quality down from 60 to 6, it still took 40
> seconds to generate 137 images totalling 87mb for the fullsize and
> 165kb for 137 thumbs
>


<snipped function without studying it>

Hi flamer,

In my opinion you try to fix the wrong thing.
thumbnailing 800 images is a big job, even for a good server.

I do not understand WHY this thumbnailing must happen when the folder is
opened.
Why don't you make a cronjob that checks for missing images, or even
better: make a thumbnail the moment a new image is added?
Why do it in this strange way?

Regards,
Erwin Moller

..oO(Erwin Moller)

>In my opinion you try to fix the wrong thing.
>thumbnailing 800 images is a big job, even for a good server.
>
>I do not understand WHY this thumbnailing must happen when the folder is
>opened.
>Why don't you make a cronjob that checks for missing images, or even
>better: make a thumbnail the moment a new image is added?

Or on-the-fly if the thumb is requested on a web page: if it doesn't
exist, it's created and stored. Pretty easy to do with a custom 404
error handler.

Micha

Erwin Moller wrote:
> flamer die.spam@hotmail.com wrote:
>
> ...snipped...
>
> In my opinion you try to fix the wrong thing.
> thumbnailing 800 images is a big job, even for a good server.

ACK.

> I do not understand WHY this thumbnailing must happen when the folder is
> opened.
> Why don't you make a cronjob that checks for missing images, or even
> better: make a thumbnail the moment a new image is added?
> Why do it in this strange way?

Another strategy would be to create the thumbnails - on the fly - if
they are requested by a user the first time.

> Regards,
> Erwin Moller

Joe

Erwin Moller wrote:

> I do not understand WHY this thumbnailing must happen when the folder is
> opened.
> Why don't you make a cronjob that checks for missing images, or even
> better: make a thumbnail the moment a new image is added?

I've done this before, and it worked pretty well. It was part of a web
based file manager. Using a cronjob would have been plausible, I suppose,
but making a thumbnail when the image was added would have been
impossible, because there were so many routes that images could be added.
(The same directory was accessible via Samba amongst other means.)

But for that, a typical directory would have contained a few hundred
images, and not 25000, so performance wasn't a huge problem.

The OP could use a compromise solution -- run an overnight cronjob to
build missing thumbnails, but keep his current script to build any new
thumbnails for newly added images.

--
Toby A Inkster BSc (Hons) ARCS
[Geek of HTML/SQL/Perl/PHP/Python/Apache/Linux]
[OS: Linux 2.6.17.14-mm-desktop-9mdvsmp, up 23 days, 21:04.]

CSS to HTML Compiler
http://tobyinkster.co.uk/blog/2008/01/22/css-compile/

On Jan 23, 11:57 am, Toby A Inkster <usenet200...@tobyinkster.co.uk>
wrote:
> Erwin Moller wrote:
> > I do not understand WHY this thumbnailing must happen when the folder is
> > opened.
> > Why don't you make a cronjob that checks for missing images, or even
> > better: make a thumbnail the moment a new image is added?
>
> I've done this before, and it worked pretty well. It was part of a web
> based file manager. Using a cronjob would have been plausible, I suppose,
> but making a thumbnail when the image was added would have been
> impossible, because there were so many routes that images could be added.
> (The same directory was accessible via Samba amongst other means.)
>
> But for that, a typical directory would have contained a few hundred
> images, and not 25000, so performance wasn't a huge problem.
>
> The OP could use a compromise solution -- run an overnight cronjob to
> build missing thumbnails, but keep his current script to build any new
> thumbnails for newly added images.
>
> --
> Toby A Inkster BSc (Hons) ARCS
> [Geek of HTML/SQL/Perl/PHP/Python/Apache/Linux]
> [OS: Linux 2.6.17.14-mm-desktop-9mdvsmp, up 23 days, 21:04.]
>
> CSS to HTML Compiler
> http://tobyinkster.co.uk/blog/2008/01/22/css-compile/

did u tried http://phpthumb.sourceforge.net/ ?? i dont know how it
will behave for 25k images, but it uses imagemagik if possible(FAST) ,
and has a buit-in caching system. And with some small tweaks (replace
header("location:') with a 304 if i remember correctly(permanently
moved)) you can take advantage of the browser-cache, so the server
bandwidth will not be consumed sending the same images over and over
to the same browser.

in regards to the suggestions, the site doesnt use a database at all,
and im using array_split to split the images up onto separate pages so
it wasn't as simple as just generating the thumbanils to be shown on
each page, I may have a look at trying implement a way to generate
only the thumbnails for each page only when requested. (which would
mean making a max of 25 at once rather than thousands).. it may be as
simple as moving the thumbnail function

The cron job is an option, but wouldnt executing a script in php via
cron still not time itself out as a normal php script would??

flamer die.spam@hotmail.com wrote:

> The cron job is an option, but wouldnt executing a script in php via
> cron still not time itself out as a normal php script would??

IIRC, the command line version of PHP doesn't honour the
max_execution_time setting in php.ini. And even if it does, you can
override it easily using the "-d" option.

Plus with command line PHP you don't need to also concern yourself about
browser timeouts and the role of Apache in all this mess.

--
Toby A Inkster BSc (Hons) ARCS
[Geek of HTML/SQL/Perl/PHP/Python/Apache/Linux]
[OS: Linux 2.6.17.14-mm-desktop-9mdvsmp, up 24 days, 9:44.]

CSS to HTML Compiler
http://tobyinkster.co.uk/blog/2008/01/22/css-compile/