PHWinfo banniere

Titres
PORTAIL ANNUAIRE ARTICLES COMPARATEUR HÉBERGEURS DEVIS FORUMS RÉDUCTEUR D'URL
Précédent   PHWinfo > Autres forums > Forum Programmation & Conception > comp.lang.php > Massive data and database, how to handle it efficiently?
S'inscrire FAQ Membres Recherche Messages du jour Marquer les forums comme lus
Massive data and database, how to handle it efficiently?

Réponse
 
LinkBack Outils de la discussion
Vieux 15/12/2007, 21h01   #1
Ming
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Massive data and database, how to handle it efficiently?

For example, 100,000 records, each record has 10 fields and may belong
to one or more categories. How shall I save those records in DB? I
think it is a very typical concern for most online shopping mall.

Option 1)

A table with all 100,000 records, and 10 fields for each record

Option 2)

Many tables by category, each table with all records for this category
and all fields other than category

Users typically browse by category but cross category query is
frequently used, and query may be made on one or more fields.

How shall I do in this?
  Réponse avec citation
Vieux 15/12/2007, 21h38   #2
macca
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: Massive data and database, how to handle it efficiently?

This is not a php question.

Please ask in a database newsgroup...or read up on database
normalization
  Réponse avec citation
Vieux 15/12/2007, 23h19   #3
Michael Fesser
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: Massive data and database, how to handle it efficiently?

..oO(Ming)

>For example, 100,000 records, each record has 10 fields and may belong
>to one or more categories. How shall I save those records in DB? I
>think it is a very typical concern for most online shopping mall.


100.000 records is nothing for a DB, assuming a properly built and
indexed table structure.

>Option 1)
>
>A table with all 100,000 records, and 10 fields for each record


More or less. Since each record may belong to one or more categories,
you will need two additional tables, one for the categories and one to
combine them both (m:n relation). Nothing special so far.

>Option 2)
>
>Many tables by category, each table with all records for this category
>and all fields other than category


Nope. That would be a really broken design.

>Users typically browse by category but cross category query is
>frequently used, and query may be made on one or more fields.


Peanuts. But you should get a good book or at least some readings on the
Web about proper DB design, normalization and such things.

Micha
  Réponse avec citation
Vieux 16/12/2007, 02h57   #4
Jerry Stuckle
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: Massive data and database, how to handle it efficiently?

Ming wrote:
> For example, 100,000 records, each record has 10 fields and may belong
> to one or more categories. How shall I save those records in DB? I
> think it is a very typical concern for most online shopping mall.
>
> Option 1)
>
> A table with all 100,000 records, and 10 fields for each record
>
> Option 2)
>
> Many tables by category, each table with all records for this category
> and all fields other than category
>
> Users typically browse by category but cross category query is
> frequently used, and query may be made on one or more fields.
>
> How shall I do in this?
>


Option 3: try a newsgroup related to the database you'll be using.
There you will get much better on DB design and normalization.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

  Réponse avec citation
Vieux 16/12/2007, 15h24   #5
Matthew
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: Massive data and database, how to handle it efficiently?

100,000 is not massive at all. I had a database of about 10 million
records held in approx. 150 tables and served by an old 1990s 386 PC with
64MB RAM, a bare bones Linux OS, and MySQL (nothing else of note running
on it, not even a GUI). Despite the large amount of data and relative low
power of the PC, simple search queries and inserts returned in about a
second or 2, and more complex queries took about 5-10 seconds. I did write
a few queries that took a few minutes to return but these were very
sophisticated and not things I needed to run regularly.

I'd be surprised if the most efficient way was not Option 1. A definitive
answer can't be given without a lot more info.. You need to read about
'database normalization' to learn more and then -afterwards- ask and
discuss the design on a database newsgroup/forum. If using MySQL try their
forums http://forums.mysql.com/ they are a ful bunch.

HTH.


Ming emailed this:
> For example, 100,000 records, each record has 10 fields and may belong
> to one or more categories. How shall I save those records in DB? I
> think it is a very typical concern for most online shopping mall.
>
> Option 1)
>
> A table with all 100,000 records, and 10 fields for each record
>
> Option 2)
>
> Many tables by category, each table with all records for this category
> and all fields other than category
>
> Users typically browse by category but cross category query is
> frequently used, and query may be made on one or more fields.
>
> How shall I do in this?

  Réponse avec citation
Vieux 16/12/2007, 16h47   #6
The Natural Philosopher
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: Massive data and database, how to handle it efficiently?

Matthew wrote:
> 100,000 is not massive at all. I had a database of about 10 million
> records held in approx. 150 tables and served by an old 1990s 386 PC
> with 64MB RAM, a bare bones Linux OS, and MySQL (nothing else of note
> running on it, not even a GUI). Despite the large amount of data and
> relative low power of the PC, simple search queries and inserts returned
> in about a second or 2, and more complex queries took about 5-10
> seconds. I did write a few queries that took a few minutes to return but
> these were very sophisticated and not things I needed to run regularly.
>
> I'd be surprised if the most efficient way was not Option 1. A
> definitive answer can't be given without a lot more info.. You need to
> read about 'database normalization' to learn more and then -afterwards-
> ask and discuss the design on a database newsgroup/forum. If using MySQL
> try their forums http://forums.mysql.com/ they are a ful bunch.
>


A lot depends on how you set things up.

If say you have item with item numbers as primary key, and categories in
a category table, adding an index to the category field of the database
will speed up searches by category immensely.

It is pretty much equivalent to having tables for each category, in
<SELECT> performance.

If you want to search by name a lot, add an index to that as well.


Its a lot easier to optimise a database for routine known access types
than rewrite the whole structure...
> HTH.
>
>
> Ming emailed this:
>> For example, 100,000 records, each record has 10 fields and may belong
>> to one or more categories. How shall I save those records in DB? I
>> think it is a very typical concern for most online shopping mall.
>>
>> Option 1)
>>
>> A table with all 100,000 records, and 10 fields for each record
>>
>> Option 2)
>>
>> Many tables by category, each table with all records for this category
>> and all fields other than category
>>
>> Users typically browse by category but cross category query is
>> frequently used, and query may be made on one or more fields.
>>
>> How shall I do in this?

  Réponse avec citation
Vieux 16/12/2007, 19h40   #7
jebblue
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: Massive data and database, how to handle it efficiently?

On Sat, 15 Dec 2007 12:01:29 -0800, Ming wrote:

> For example, 100,000 records, each record has 10 fields and may belong
> to one or more categories. How shall I save those records in DB? I think
> it is a very typical concern for most online shopping mall.


There is a php.db newsgroup, might be good to try there.

--
// This is my opinion.
  Réponse avec citation
Vieux 17/12/2007, 18h32   #8
Ivan Marsh
Aucun Avatar
 
Messages: n/a
Hébergeur:
Par défaut Re: Massive data and database, how to handle it efficiently?

On Sat, 15 Dec 2007 12:01:29 -0800, Ming wrote:

> For example, 100,000 records, each record has 10 fields and may belong
> to one or more categories. How shall I save those records in DB? I think
> it is a very typical concern for most online shopping mall.


Though this database is nowhere near "massive", the info you're looking
for can be found by searching for "database normalization".


--
I told you this was going to happen.

  Réponse avec citation
Réponse


Outils de la discussion

Règles de messages
Vous ne pouvez pas créer de nouvelles discussions
Vous ne pouvez pas envoyer des réponses
Vous ne pouvez pas envoyer des pièces jointes
Vous ne pouvez pas modifier vos messages

Les balises BB sont activées : oui
Les smileys sont activés : oui
La balise [IMG] est activée : oui
Le code HTML peut être employé : non
Trackbacks are oui
Pingbacks are oui
Refbacks are oui


Fuseau horaire GMT +1. Il est actuellement 12h59.


Édité par : vBulletin®
Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
Search Engine Friendly URLs by vBSEO 3.2.0 RC5 Tous droits réservés.
Version française #16 par l'association vBulletin francophone
PHWinfo est un site Éducation Sans Frontières ©2000-2008
Ad Management by RedTyger
©Tous droits réservés par les parties respectives
Page generated in 0,13314 seconds with 16 queries