PHWinfo banniere

Titres
PORTAIL ANNUAIRE ARTICLES COMPARATEUR HÉBERGEURS DEVIS FORUMS RÉDUCTEUR D'URL
Précédent   PHWinfo > Autres forums > Forum Programmation & Conception > comp.databases.mysql > stats database structure - design suggestions
S'inscrire FAQ Membres Recherche Messages du jour Marquer les forums comme lus
stats database structure - design suggestions

Réponse
 
LinkBack Outils de la discussion
Vieux 06/01/2008, 07h04   #1
FFMG
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut stats database structure - design suggestions


Hi,

Currently on my site all my 'pages' are saved in the db.
Each page has a 'counter' field.
Every time the page is accessed I increase the page count so this is
nothing more than a page view, (or db SELECT count really).

But now I want to keep track of how many times a page was viewed per
day/week/months.

I could:
1- Have a huge table that saves the page id and the date, (and total
those numbers when I need to see the 'stats' for a given date range),
2- Have a smaller table that has a page id, day/month/year and a
counter so I would have a running count per day.

Neither option seems attractive purely because our site would have a
few million rows very quickly, (granted the actual size of the db would
not be more than a few 100mbs but still).

What would be the best definition for such a table? Given that it would
be updated every single time a page is loaded.

I know that most developers don't care much for those stats and that
there are better options out there, but the fact is that management
pays my salary , , and thirdparty tools don't offer an option at a
page level.

So, what do you suggest would be the best design?

FFMG


--

'webmaster forum' (http://www.httppoint.com) | 'Free Blogs'
(http://www.journalhome.com/) | 'webmaster Directory'
(http://www.webhostshunter.com/)
'Recreation Vehicle insurance'
(http://www.insurance-owl.com/other/car_rec.php) | 'Free URL
redirection service' (http://urlkick.com/)
------------------------------------------------------------------------
FFMG's Profile: http://www.httppoint.com/member.php?userid=580
View this thread: http://www.httppoint.com/showthread.php?t=23939

Message Posted via the webmaster forum http://www.httppoint.com, (Ad revenue sharing).

  Réponse avec citation
Vieux 06/01/2008, 12h26   #2
Paul Lautman
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

FFMG wrote:
> Hi,
>
> Currently on my site all my 'pages' are saved in the db.
> Each page has a 'counter' field.
> Every time the page is accessed I increase the page count so this is
> nothing more than a page view, (or db SELECT count really).
>
> But now I want to keep track of how many times a page was viewed per
> day/week/months.
>
> I could:
> 1- Have a huge table that saves the page id and the date, (and total
> those numbers when I need to see the 'stats' for a given date range),
> 2- Have a smaller table that has a page id, day/month/year and a
> counter so I would have a running count per day.
>
> Neither option seems attractive purely because our site would have a
> few million rows very quickly, (granted the actual size of the db
> would not be more than a few 100mbs but still).
>
> What would be the best definition for such a table? Given that it
> would be updated every single time a page is loaded.
>
> I know that most developers don't care much for those stats and that
> there are better options out there, but the fact is that management
> pays my salary , , and thirdparty tools don't offer an option at a
> page level.
>
> So, what do you suggest would be the best design?
>
> FFMG


I would go for timestamp and pageid.


  Réponse avec citation
Vieux 06/01/2008, 15h24   #3
Jerry Stuckle
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

FFMG wrote:
> Hi,
>
> Currently on my site all my 'pages' are saved in the db.
> Each page has a 'counter' field.
> Every time the page is accessed I increase the page count so this is
> nothing more than a page view, (or db SELECT count really).
>
> But now I want to keep track of how many times a page was viewed per
> day/week/months.
>
> I could:
> 1- Have a huge table that saves the page id and the date, (and total
> those numbers when I need to see the 'stats' for a given date range),
> 2- Have a smaller table that has a page id, day/month/year and a
> counter so I would have a running count per day.
>
> Neither option seems attractive purely because our site would have a
> few million rows very quickly, (granted the actual size of the db would
> not be more than a few 100mbs but still).
>
> What would be the best definition for such a table? Given that it would
> be updated every single time a page is loaded.
>
> I know that most developers don't care much for those stats and that
> there are better options out there, but the fact is that management
> pays my salary , , and thirdparty tools don't offer an option at a
> page level.
>
> So, what do you suggest would be the best design?
>
> FFMG
>
>


Why not just use your server's web logs? They already track this. And
there are numerous programs out there which will format it for you,
build nice graphs and charts, etc.

But keep in mind such statistics are not accurate. Pages may be cached,
for instance, and you'd never see the hit.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

  Réponse avec citation
Vieux 07/01/2008, 06h43   #4
FFMG
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions


Paul Lautman;112085 Wrote:
>
> >
> > So, what do you suggest would be the best design?
> >
> > FFMG

>
> I would go for timestamp and pageid.[/color]

hum... thanks, any particular reason why?

FFMG


--

'webmaster forum' (http://www.httppoint.com) | 'Free Blogs'
(http://www.journalhome.com/) | 'webmaster Directory'
(http://www.webhostshunter.com/)
'Recreation Vehicle insurance'
(http://www.insurance-owl.com/other/car_rec.php) | 'Free URL
redirection service' (http://urlkick.com/)
------------------------------------------------------------------------
FFMG's Profile: http://www.httppoint.com/member.php?userid=580
View this thread: http://www.httppoint.com/showthread.php?t=23939

Message Posted via the webmaster forum http://www.httppoint.com, (Ad revenue sharing).

  Réponse avec citation
Vieux 07/01/2008, 06h51   #5
FFMG
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions


Jerry Stuckle;112124 Wrote:
>
> > ...
> > Every time the page is accessed I increase the page count so this is
> > nothing more than a page view, (or db SELECT count really).
> >

>
> > ...
> > I know that most developers don't care much for those stats and that
> > there are better options out there, but the fact is that management
> > pays my salary , , and thirdparty tools don't offer an option at a
> > page level.
> >

>
> Why not just use your server's web logs? They already track this.
> And
> there are numerous programs out there which will format it for you,
> build nice graphs and charts, etc.
>[/color]

The answer is in my original post, "..and that there are better options
out there ..."

Jerry Stuckle;112124 Wrote:
>
> But keep in mind such statistics are not accurate. Pages may be
> cached,
> for instance, and you'd never see the hit.
>


I also mentioned that in my OP, "...this is nothing more than a page
view, (or db SELECT count really)".

On another, side, note, how do you know the pages will be cached? Are
you suggesting that browsers will ignore my headers and cache pages?
In that case yes, it is indeed possible for visitor to copy a page to
their hard drive and access it from there, I will be sure to add
0.00000001% to my page count to allow for this case.

Thanks for the .

FFMG


--

'webmaster forum' (http://www.httppoint.com) | 'Free Blogs'
(http://www.journalhome.com/) | 'webmaster Directory'
(http://www.webhostshunter.com/)
'Recreation Vehicle insurance'
(http://www.insurance-owl.com/other/car_rec.php) | 'Free URL
redirection service' (http://urlkick.com/)
------------------------------------------------------------------------
FFMG's Profile: http://www.httppoint.com/member.php?userid=580
View this thread: http://www.httppoint.com/showthread.php?t=23939

Message Posted via the webmaster forum http://www.httppoint.com, (Ad revenue sharing).

  Réponse avec citation
Vieux 07/01/2008, 10h07   #6
Paul Lautman
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

FFMG wrote:
> Paul Lautman;112085 Wrote:
>>
>> >
>> > So, what do you suggest would be the best design?
>> >
>> > FFMG

>>
>> I would go for timestamp and pageid.

>
> hum... thanks, any particular reason why?
>
> FFMG[/color]

You then have all the information you need to cut the data up in any way you
want. For instance you may think initially that you only want to know how
many hits per day. But you may want to in future know when during the day
you get most hints. Of course other nice information to have is stuff that
things like Google Analytics gives you such as where it "thinks" the hits
were from. So maybe holding the IP address of the request would be good.

The more assumptions you make up front when storing the data (i.e. the more
preprocessing you do on it), the less you can actually do with the data. One
of the things about data analysis is that untill you analyse it, you don't
know whet there is to find out. If you scupper the data at the start, you
reduce the amount of analysis you can do.


  Réponse avec citation
Vieux 07/01/2008, 10h38   #7
FFMG
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions


Paul Lautman;112538 Wrote:
>
> You then have all the information you need to cut the data up in any
> way you
> want. For instance you may think initially that you only want to know
> how
> many hits per day. But you may want to in future know when during the
> day
> you get most hints. Of course other nice information to have is stuff
> that
> things like Google Analytics gives you such as where it "thinks" the
> hits
> were from. So maybe holding the IP address of the request would be
> good.
>


This makes sense, but wont the table reach an astronomical size in no
time.
The site is not the busiest in the world but I am worried that overtime
the table will become unusable.
In our case, there would be hundreds of millions of rows within
months.

Paul Lautman;112538 Wrote:
>
> The more assumptions you make up front when storing the data
>


I totally agree, but as I said I am worried about the size of the db.
I wonder how others do it, maybe I should look for a php open source
project to see how they do it.

FFMG.


--

'webmaster forum' (http://www.httppoint.com) | 'Free Blogs'
(http://www.journalhome.com/) | 'webmaster Directory'
(http://www.webhostshunter.com/)
'Recreation Vehicle insurance'
(http://www.insurance-owl.com/other/car_rec.php) | 'Free URL
redirection service' (http://urlkick.com/)
------------------------------------------------------------------------
FFMG's Profile: http://www.httppoint.com/member.php?userid=580
View this thread: http://www.httppoint.com/showthread.php?t=23939

Message Posted via the webmaster forum http://www.httppoint.com, (Ad revenue sharing).

  Réponse avec citation
Vieux 07/01/2008, 13h06   #8
Jerry Stuckle
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

FFMG wrote:
> Jerry Stuckle;112124 Wrote:
>>> ...
>>> Every time the page is accessed I increase the page count so this is
>>> nothing more than a page view, (or db SELECT count really).
>>>
>>> ...
>>> I know that most developers don't care much for those stats and that
>>> there are better options out there, but the fact is that management
>>> pays my salary , , and thirdparty tools don't offer an option at a
>>> page level.
>>>

>> Why not just use your server's web logs? They already track this.
>> And
>> there are numerous programs out there which will format it for you,
>> build nice graphs and charts, etc.
>>

>
> The answer is in my original post, "..and that there are better options
> out there ..."
>[/color]

And you continue "...and thirdparty tools don't offer an option at a
page level..."

This is incorrect. They can offer detailed page information.

> Jerry Stuckle;112124 Wrote:
>> But keep in mind such statistics are not accurate. Pages may be
>> cached,
>> for instance, and you'd never see the hit.
>>

>
> I also mentioned that in my OP, "...this is nothing more than a page
> view, (or db SELECT count really)".
>


But it is not a page view. If the page is cached, your server will
never see the request. So a user has viewed your page, but your server
knows nothing about it.

> On another, side, note, how do you know the pages will be cached? Are
> you suggesting that browsers will ignore my headers and cache pages?
> In that case yes, it is indeed possible for visitor to copy a page to
> their hard drive and access it from there, I will be sure to add
> 0.00000001% to my page count to allow for this case.
>


You don't. And it may not be cached just by the user - it may be cached
by servers between you and the user.

Cache control is only a recommendation to the browser, not a
requirement. Browsers are free to ignore it - as are other servers.
The result is lower hit counts than you see. But there's nothing you
can do about it.

> Thanks for the .
>
> FFMG
>
>



--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

  Réponse avec citation
Vieux 07/01/2008, 13h55   #9
FFMG
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions


Jerry Stuckle;112584 Wrote:
>
> >> Why not just use your server's web logs? They already track this.
> >> And
> >> there are numerous programs out there which will format it for you,
> >> build nice graphs and charts, etc.

> >
> > The answer is in my original post, "..and that there are better

> options
> > out there ..."

>
> And you continue "...and thirdparty tools don't offer an option at a
> page level..."
>
> This is incorrect. They can offer detailed page information.
>


I doubt that very much, (well not as I understand server logs).
I don't know of any that would work at the page id level.

How would a server log handle something like :
http://www.example.com/somepage/12345, if server redirects it to
http://www.example.com/somescript.php?id=12345

Unless you can direct me to one, I don't claim to know all the
applications available.

Jerry Stuckle;112584 Wrote:
>
> >
> > I also mentioned that in my OP, "...this is nothing more than a page
> > view, (or db SELECT count really)".

>
> But it is not a page view.
>


Hence the reason I added it was more of a db SELECT really.

Jerry Stuckle;112584 Wrote:
>
> If the page is cached, your server will never see the request.
>


What can I do if servers or browsers do as they please?
But it would be a rather big security hole if servers took it upon
themselves to cache pages or not.
And I don't know of any browsers that would not obey a no-cache
request.
What would make them choose what to request from the server and what
they feel can be cached.
I also don't know of relay servers that cache data and serve whatever
page they see fit.

Are you suggesting that if I go to
http://wwww.example.com/somepage.html, I might or might not see a
cached page because the browser or the relay might, or might not, take
it upon themselves to choose what to show me.

FFMG


--

'webmaster forum' (http://www.httppoint.com) | 'Free Blogs'
(http://www.journalhome.com/) | 'webmaster Directory'
(http://www.webhostshunter.com/)
'Recreation Vehicle insurance'
(http://www.insurance-owl.com/other/car_rec.php) | 'Free URL
redirection service' (http://urlkick.com/)
------------------------------------------------------------------------
FFMG's Profile: http://www.httppoint.com/member.php?userid=580
View this thread: http://www.httppoint.com/showthread.php?t=23939

Message Posted via the webmaster forum http://www.httppoint.com, (Ad revenue sharing).

  Réponse avec citation
Vieux 07/01/2008, 14h50   #10
Peter H. Coffin
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

On Mon, 7 Jan 2008 05:38:44 -0500, FFMG wrote:

> Paul Lautman;112538 Wrote:
>
>> You then have all the information you need to cut the data up in any
>> way you want. For instance you may think initially that you only
>> want to know how many hits per day. But you may want to in future
>> know when during the day you get most hints. Of course other nice
>> information to have is stuff that things like Google Analytics gives
>> you such as where it "thinks" the hits were from. So maybe holding
>> the IP address of the request would be good.

>
>
> This makes sense, but wont the table reach an astronomical size in no
> time. The site is not the busiest in the world but I am worried that
> overtime the table will become unusable. In our case, there would be
> hundreds of millions of rows within months.


So chunk it off into months. That's a very reasonable size for grouping
trends anyway. You've got the timestamp right there to make it trivial
to copy off last month's data into tables for analysis and remove it
from your logging table. It'll make it MUCH easier to keep the whole
database backed up too, because once you have a month of data backed up,
you won't need to do that table ever again.

> Paul Lautman;112538 Wrote:
>
>> The more assumptions you make up front when storing the data

>
>
> I totally agree, but as I said I am worried about the size of the db.
> I wonder how others do it, maybe I should look for a php open source
> project to see how they do it.


You generally won't hit a MySQL limit on the size of a table, only
a filesystem limit on the size of a file. And, with any modern OS,
that's going to be $smallint terabytes, or probably tens of billions of
rows. You'll run out of time before you run out of room in a table, or
probably, storage. Disk is cheap.

The solution to the time issue is to chunk the data into the largest
managable units, run the analysis for the numbers you know you'll want
now, and archive the data someplace convenient for the numbers you'll
discover you want later.

--
56. My Legions of Terror will be trained in basic marksmanship. Any who
cannot learn to hit a man-sized target at 10 meters will be used for
target practice.
--Peter Anspach's list of things to do as an Evil Overlord
  Réponse avec citation
Vieux 07/01/2008, 19h47   #11
Jerry Stuckle
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

FFMG wrote:
> Jerry Stuckle;112584 Wrote:
>>>> Why not just use your server's web logs? They already track this.
>>>> And
>>>> there are numerous programs out there which will format it for you,
>>>> build nice graphs and charts, etc.
>>> The answer is in my original post, "..and that there are better

>> options
>>> out there ..."

>> And you continue "...and thirdparty tools don't offer an option at a
>> page level..."
>>
>> This is incorrect. They can offer detailed page information.
>>

>
> I doubt that very much, (well not as I understand server logs).
> I don't know of any that would work at the page id level.
>


Then you need to look at server logs more carefully.

> How would a server log handle something like :
> http://www.example.com/somepage/12345, if server redirects it to
> http://www.example.com/somescript.php?id=12345
>


Two entries. One for the first page, with the redirect code (i.e. 301
or 302). Then one for the second page with a normal return code (200).

> Unless you can direct me to one, I don't claim to know all the
> applications available.
>


Search for analyzers. I happen to prefer awstats, but there are dozens
out there.

> Jerry Stuckle;112584 Wrote:
>>> I also mentioned that in my OP, "...this is nothing more than a page
>>> view, (or db SELECT count really)".

>> But it is not a page view.
>>

>
> Hence the reason I added it was more of a db SELECT really.
>


You said you wanted to track page views. It's all in the server logs.

> Jerry Stuckle;112584 Wrote:
>> If the page is cached, your server will never see the request.
>>

>
> What can I do if servers or browsers do as they please?


Absolutely nothing you can do about it.

> But it would be a rather big security hole if servers took it upon
> themselves to cache pages or not.


Why? What's the security hole? They're just short cutting the path to
the server and back.

> And I don't know of any browsers that would not obey a no-cache
> request.


Firefox, IE, Opera,...

> What would make them choose what to request from the server and what
> they feel can be cached.


Many things. It's up to each browser.

> I also don't know of relay servers that cache data and serve whatever
> page they see fit.
>


Why not? It cuts down on both server load and network traffic. Very
commonly done.

> Are you suggesting that if I go to
> http://wwww.example.com/somepage.html, I might or might not see a
> cached page because the browser or the relay might, or might not, take
> it upon themselves to choose what to show me.
>


That's correct. That's how it works. But if the page hasn't changed,
what's the difference?

> FFMG
>
>



--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

  Réponse avec citation
Vieux 07/01/2008, 20h20   #12
FFMG
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions


Jerry Stuckle;112746 Wrote:
>
> Many things. It's up to each browser.
>
> > I also don't know of relay servers that cache data and serve

> whatever
> > page they see fit.
> >

>
> Why not? It cuts down on both server load and network traffic. Very
> commonly done.
>
> > Are you suggesting that if I go to
> > http://wwww.example.com/somepage.html, I might or might not see a
> > cached page because the browser or the relay might, or might not,

> take
> > it upon themselves to choose what to show me.
> >

>
> That's correct. That's how it works. But if the page hasn't changed,
> what's the difference?
>
> > FFMG
> >
> >

>


Please think about what you are saying. don't just reply for the sake
of replying... think about it.

You are suggesting that somehow, out of the blue a relay server will
cache data and choose what to display, (or not!).

And then, just for fun, my browser will pick and choose what to show
me.
"Maybe I will cache that page, maybe I wont"

please don't insult my, or your, intelligence.

Let me bring you back to reality, what will happen is the browser will
ask if a page has been changed if not, it will display it otherwise it
get a new version of the page.

either way, my server is asked for the page and I will log that.

No server/browser and most certainly no relay server picks and chooses
what to show.

FFMG


--

'webmaster forum' (http://www.httppoint.com) | 'Free Blogs'
(http://www.journalhome.com/) | 'webmaster Directory'
(http://www.webhostshunter.com/)
'Recreation Vehicle insurance'
(http://www.insurance-owl.com/other/car_rec.php) | 'Free URL
redirection service' (http://urlkick.com/)
------------------------------------------------------------------------
FFMG's Profile: http://www.httppoint.com/member.php?userid=580
View this thread: http://www.httppoint.com/showthread.php?t=23939

Message Posted via the webmaster forum http://www.httppoint.com, (Ad revenue sharing).

  Réponse avec citation
Vieux 07/01/2008, 21h52   #13
Peter H. Coffin
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

On Mon, 7 Jan 2008 15:20:21 -0500, FFMG wrote:
>
> Jerry Stuckle;112746 Wrote:
>>
>> Many things. It's up to each browser.
>>
>> > I also don't know of relay servers that cache data and serve

>> whatever
>> > page they see fit.
>> >

>>
>> Why not? It cuts down on both server load and network traffic. Very
>> commonly done.
>>
>> > Are you suggesting that if I go to
>> > http://wwww.example.com/somepage.html, I might or might not see a
>> > cached page because the browser or the relay might, or might not,

>> take
>> > it upon themselves to choose what to show me.
>> >

>>
>> That's correct. That's how it works. But if the page hasn't changed,
>> what's the difference?
>>
>> > FFMG
>> >
>> >

>>

>
> Please think about what you are saying. don't just reply for the sake
> of replying... think about it.
>
> You are suggesting that somehow, out of the blue a relay server will
> cache data and choose what to display, (or not!).


That's exactly what he's suggesting. Caches happen at the browser level,
sometimes at the LAN level (for a corporate user in the office, for
example) and increasingly on the ISP level as well. AOL is a classic
case of this. You can't even presume that all the page parts for an
single page show for a single user will all come to a webserver showing
the same requesting IP. ANY large ISP will see substantial savings on
the byte-count of their outfacing linkages by setting up what amount to
giant Squid engines that rapidly fill with commonly-requested pages.
Some will even overrule pragma no-cache requests. Slow links like those
via satallites or ocean cable may cache as well.

> And then, just for fun, my browser will pick and choose what to show
> me.
> "Maybe I will cache that page, maybe I wont"
>
> please don't insult my, or your, intelligence.


Sorry, but it's not up to him, me, you, or anyone else that'll listen.
Web's full of complaints from users about it and companies trying to
sell ISPs those exact things. And they do sell 'em.

> Let me bring you back to reality, what will happen is the browser will
> ask if a page has been changed if not, it will display it otherwise it
> get a new version of the page.
>
> either way, my server is asked for the page and I will log that.


Maybe. It depends on how draconian the settings are. And webservers
don't generally log GETs with If-Modified-Since that fail. The users can
ask their browsers to bypass that, but there's no way the server end can
enforce it, short of encrypting all it's real traffic through https or
similar expensive shenanigans.

> No server/browser and most certainly no relay server picks and chooses
> what to show.


Believe that if you like. It's not the best way to bet, though. There's
a huge amount of traffic to save fetching cnn.com's home page only once
a minute instead of for every bozo hitting refresh.

--
Whenever you look at a beautiful woman,
always remember that somewhere, someone is tired of her.
  Réponse avec citation
Vieux 08/01/2008, 02h11   #14
Gordon Burditt
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

>>> If the page is cached, your server will never see the request.
>>>

>>
>> What can I do if servers or browsers do as they please?

>
>Absolutely nothing you can do about it.


Agreed. You can't count hits by ESP.

>> But it would be a rather big security hole if servers took it upon
>> themselves to cache pages or not.

>
>Why? What's the security hole? They're just short cutting the path to
>the server and back.


There *ARE* possible security issues in a caching proxy used by
multiple users (e.g. all the customers of a particular ISP) that
caches too much. For example, it could cache the page that A viewed
(after he logged in with his password) with his private info on it,
and show it to B. Most caching proxies avoid the most serious issues
by refusing to cache (or even proxy) https: pages.

There are also other issues with caching proxies that aren't really
security issues, but more like stale data - like showing you the
same stock prices you saw two days ago, or giving the same weather
forecast or news.

>> And I don't know of any browsers that would not obey a no-cache
>> request.


I'd say any browser with a "work offline" mode would count. You
don't see hits made in that mode. Doesn't that include IE?

>Firefox, IE, Opera,...
>
>> What would make them choose what to request from the server and what
>> they feel can be cached.

>
>Many things. It's up to each browser.
>

  Réponse avec citation
Vieux 08/01/2008, 02h27   #15
Gordon Burditt
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

>You are suggesting that somehow, out of the blue a relay server will
>cache data and choose what to display, (or not!).


Yes. Many ISPs do this to save on bandwidth. I used to work for
one that did. And they can play router tricks so you have no idea
you're using the proxy. The decision may be:

IF: (a) we have a copy of the page, and (b) it's less than 2 days
old, then give them the cached copy. Otherwise fetch it, cache it,
and give them the newly fetched page.

The one I worked with didn't cache https: and it didn't cache POST.
It also had an exception list which turned out to take a lot of
work to maintain. Most of the issues related to "stale data", but
sometimes this caused more problems (like showing the same CAPTCHA
image repeatedly).

  Réponse avec citation
Vieux 08/01/2008, 02h28   #16
Jerry Stuckle
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

FFMG wrote:
> Jerry Stuckle;112746 Wrote:
>> Many things. It's up to each browser.
>>
>>> I also don't know of relay servers that cache data and serve

>> whatever
>>> page they see fit.
>>>

>> Why not? It cuts down on both server load and network traffic. Very
>> commonly done.
>>
>>> Are you suggesting that if I go to
>>> http://wwww.example.com/somepage.html, I might or might not see a
>>> cached page because the browser or the relay might, or might not,

>> take
>>> it upon themselves to choose what to show me.
>>>

>> That's correct. That's how it works. But if the page hasn't changed,
>> what's the difference?
>>
>>> FFMG
>>>
>>>

>
> Please think about what you are saying. don't just reply for the sake
> of replying... think about it.
>
> You are suggesting that somehow, out of the blue a relay server will
> cache data and choose what to display, (or not!).
>


That is correct.

> And then, just for fun, my browser will pick and choose what to show
> me.


That is correct.

> "Maybe I will cache that page, maybe I wont"
>


That is correct.

> please don't insult my, or your, intelligence.
>


Then learn how the internet works.

> Let me bring you back to reality, what will happen is the browser will
> ask if a page has been changed if not, it will display it otherwise it
> get a new version of the page.
>


Not necessarily.

> either way, my server is asked for the page and I will log that.
>


Nope.

> No server/browser and most certainly no relay server picks and chooses
> what to show.
>


Every one does.

> FFMG
>
>


I suggest you read up on how the internet works.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

  Réponse avec citation
Vieux 08/01/2008, 05h22   #17
FFMG
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions


Jerry Stuckle;112868 Wrote:
>
> I suggest you read up on how the internet works.
>


By the amount of replies it looks like I am obviously wrong.

Having said that I still find it very hard to believe.
Maybe scripts or images are cached, but I would have thought that a
request of some sort is made to the server regadless.

When I am logged in on a forum or looking at my emails online, (not on
an https link), I have never seen the page of someone else.
Or I have never heard of a Myspace.com or Facebook.com page of one
person been shown to another person because the browser was served the
cached page of someone else.
Maybe I have been amazingly lucky over the past +10 years on the net.

I do know how the internet work, but I didn't know how ISPs worked.
I'll have to double check what the policy of my ISP is, (although I
doubt they would tell me).

FFMG


--

'webmaster forum' (http://www.httppoint.com) | 'Free Blogs'
(http://www.journalhome.com/) | 'webmaster Directory'
(http://www.webhostshunter.com/)
'Recreation Vehicle insurance'
(http://www.insurance-owl.com/other/car_rec.php) | 'Free URL
redirection service' (http://urlkick.com/)
------------------------------------------------------------------------
FFMG's Profile: http://www.httppoint.com/member.php?userid=580
View this thread: http://www.httppoint.com/showthread.php?t=23939

Message Posted via the webmaster forum http://www.httppoint.com, (Ad revenue sharing).

  Réponse avec citation
Vieux 08/01/2008, 09h41   #18
Paul Lautman
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

FFMG wrote:
> Jerry Stuckle;112868 Wrote:
> I do know how the internet work, but I didn't know how ISPs worked.


ISPs are the part of the internet that everyone uses to connect through.
If you don't know how ISPs work, then you don't know (completely) how the
internet works. QED.


  Réponse avec citation
Vieux 09/01/2008, 00h02   #19
Gordon Burditt
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

>By the amount of replies it looks like I am obviously wrong.
>
>Having said that I still find it very hard to believe.
>Maybe scripts or images are cached, but I would have thought that a
>request of some sort is made to the server regadless.


That takes (a) time and (b) bandwidth. There are incentives to
avoid both. If it looks faster, the users are happier (until they
realize it's screwing up). Bandwidth costs money.

>When I am logged in on a forum or looking at my emails online, (not on
>an https link), I have never seen the page of someone else.


Just because YOUR ISP doesn't use a proxy, or uses a well-behaved
one, doesn't mean none of the others do. How many different ISPs
have you used? How many are there?

Would you really notice if you got a 3-hour-old weather forecast
instead of a current one?

>Or I have never heard of a Myspace.com or Facebook.com page of one
>person been shown to another person because the browser was served the
>cached page of someone else.
>Maybe I have been amazingly lucky over the past +10 years on the net.


Your Facebook or Myspace page is unlikely to have the same URL as
someone else's page. (You're supposed to be able to link to your
page, and that requires a different URL). The tracking page for
your package might have the same URL as anyone else's package, with
the difference determined by or POST data. (They are really
the same page with different personalization). The most likely
affected pages will be ones where how you logged in or personal
settings make the same page look different for different people.

>I do know how the internet work, but I didn't know how ISPs worked.


If you leave out the ISPs, you get a whole bunch of users and web sites
that are not connected to each other, except for some local nets.

>I'll have to double check what the policy of my ISP is, (although I
>doubt they would tell me).


Unless the ISP is advertising the proxy as a feature (some ISPs
have a caching, compressing proxy when used with IE with some kind
of plugin, which compresses the data going to the user. They
sometimes charge extra for this. And it may seem faster for the
user, especially over dialup lines.) the front-line tech support
or sales crew may not know about it.

  Réponse avec citation
Vieux 09/01/2008, 03h20   #20
Peter H. Coffin
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: stats database structure - design suggestions

On Wed, 09 Jan 2008 00:02:04 -0000, Gordon Burditt wrote:
> Unless the ISP is advertising the proxy as a feature (some ISPs
> have a caching, compressing proxy when used with IE with some kind
> of plugin, which compresses the data going to the user. They
> sometimes charge extra for this. And it may seem faster for the
> user, especially over dialup lines.) the front-line tech support
> or sales crew may not know about it.


Several dialup wholesalers (dialupusa.net, usbackbone.net, et al.)
provide this as an option, when they encourage their resellers to charge
for.

--
I didn't need to sabotage anything. Not being around to say "No that
won't work" or "you can't do it that way" is more than enough damage.
(Ego problem? It's not a problem.)
-- Graham Reed, on job endings
  Réponse avec citation
Réponse


Outils de la discussion

Règles de messages
Vous ne pouvez pas créer de nouvelles discussions
Vous ne pouvez pas envoyer des réponses
Vous ne pouvez pas envoyer des pièces jointes
Vous ne pouvez pas modifier vos messages

Les balises BB sont activées : oui
Les smileys sont activés : oui
La balise [IMG] est activée : oui
Le code HTML peut être employé : non
Trackbacks are oui
Pingbacks are oui
Refbacks are oui


Fuseau horaire GMT +1. Il est actuellement 12h06.


Édité par : vBulletin® version 3.7.3
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Friendly URLs by vBSEO 3.2.0 RC5 Tous droits réservés.
Version française #16 par l'association vBulletin francophone
PHWinfo est un site Éducation Sans Frontières ©2000-2008
Ad Management by RedTyger
©Tous droits réservés