Hi List,

I was wondering if there's a Debian specific tool that could facilitate
managing thousands of machines via APT. I'm aware that many people would
recommend a sync option, where 1 machine serves as the master, and the
others sync off of that. Perhaps that is the only reliable approach, but
I thought I'd just check in w/ the list and see what people recommend.

Thanks,
Ben


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Ben wrote:
> I was wondering if there's a Debian specific tool that could facilitate
> managing thousands of machines via APT. I'm aware that many people would
> recommend a sync option, where 1 machine serves as the master, and the
> others sync off of that. Perhaps that is the only reliable approach, but
> I thought I'd just check in w/ the list and see what people recommend.

There is no single standardized tool to do this. Some customized
script writing will be required. However there are lots of options
and lots of possibilities.

Unfortunately you did not say whether you were upgrading from Sarge to
Etch or if this is a routine daily installation of security upgrades
or if other conditions applied. I would suggest different things in
each of those cases. If you could say more about your environment
then better suggestions might be provided.

The majority of users have a small number of machines. The standard
solutions all center around the Debian Release Notes and upgrading
manually. That is the most flexible method but is of course the most
manual method. The numbers of administrators such as yourself with a
large number of machines is smaller. Also they usually have
customized environments making use of completely standardized tools
out of the box difficult. It is harder to make a generalized
solution. But custom solutions for any one particular environment are
almost always possible.

What I have done in the past (also with thousands of machines) to
provide security upgrades is to run a daily cron task that ran a an
upgrade script. I used a private mirror that I controlled. I staged
victim machines to get security upgrades immediately and other
machines received them after a waiting period if no problems were seen
on the "canary" machines. A standard Debian package that may be
useful in this case is fine 'cron-apt' however I found a custom script
solution to be better in my case.

However security upgrades are nice, tidy special cases. Configuration
files don't change. Package names don't change. Very little changes.
But for distribution changes from Sarge to Etch it is more complicated
to automatically upgrade machines. More is needed than tools designed
for security upgrades can provide. In those cases I think only a
custom script upgrade process can work successfully.

Are all of your machines identical? Are there small numbers of known
variations? Are there large numbers of large variations? Desktops?
Servers? A mix? Of course the more similar the pool of machines to
upgrade automatically then the easier this will be but one of the
strengths of Debian systems is the ability to handle gracefully a lot
of variations.

Assuming that you have thousands of machines running Sarge and that
some variation exists but that most are very similar then it is fairly
easy to create a script to automate the upgrade Sarge to Etch. I have
done this several stable Debian releases previously. I would be happy
to provide further information from my own experience and I am sure
that others on the list would as well. Start small and test the
script on a representative machine. Fix any issues found. Work
slowly through several more machines. Gain confidence is the process.
Increase the rollout to the large pool of machines. Finish off any
exception machines that were held off during the original deployment.
It will be done before you know it!

I did not provide details here because they would be overwhelming. If
you (or others) are interested then please keep the dialog going. It
is an interesting topic.

Bob


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Ben <mack.stout@gmail.com>:
>
> I was wondering if there's a Debian specific tool that could facilitate
> managing thousands of machines via APT. I'm aware that many people would
> recommend a sync option, where 1 machine serves as the master, and the
> others sync off of that. Perhaps that is the only reliable approach, but
> I thought I'd just check in w/ the list and see what people recommend.

I'll answer that with, why would someone in your position not want to
do that?


--
Any technology distinguishable from magic is insufficiently advanced.
(*) http://www.spots.ab.ca/~keeling Linux Counter #80292
- - http://www.faqs.org/rfcs/rfc1855.html Please, don't Cc: me.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

s. keeling wrote:
> Ben wrote:
> > I was wondering if there's a Debian specific tool that could facilitate
> > managing thousands of machines via APT. I'm aware that many people would
> > recommend a sync option, where 1 machine serves as the master, and the
> > others sync off of that. Perhaps that is the only reliable approach, but
> > I thought I'd just check in w/ the list and see what people recommend.
>
> I'll answer that with, why would someone in your position not want to
> do that?

I am not sure to which you direction you are referring. Why would
someone administering a large number of machines not want to use APT?
Why would someone administering a large number of machiens not want to
use system images?

On a server farm of identical machines using system images works
pretty well. Tools like SystemImager, FAI, Mondo, etc. are great.
But in an engineering desktop pool for a counter example there will be
machine variations. In the case that variations are expected then
images do not work as well. In which case I prefer to simply upgrade
the machine with APT. APT is great for this and works very well.

In my experience most environments have variations. Perhaps the
largest groups of machines are identical. But other machines are
different. In addition to compute servers there are always
DNS/NTP/NIS/YP servers, NFS servers, LDAP servers, web servers,
database servers, and others that don't quite ever fit a standard
image. As soon as I need to deal with these variations I find that
using APT for everything is just easier.

But as always every environment is unique. An informed and reasoned
decision is always best even if it might not make sense to people not
involved in the process.

Bob


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

All of our machines are identical... er at least the variety of machines
that we're planning to manage (thin-clients, servers). We do use tools
like system imager to image the boxes before they are deployed in the
field. I find apt appealing because it encourages our programmers to
compile their code and create debian packages, which promotes the idea
that we should keep our source code organized (something that has not
occurred in the past). I like debs more than rpms (no standardization)
and portage (emerge requires bulky rsync of makefiles). We can keep
track of changes, effectively rollout packages/releases with version and
revision control, downgrade, upgrade... etc...

I plan to use apt as a tool to do software rollouts and periodic
upgrades and security releases rather than dist-upgrades. If this works,
I would manage several private mirrors (stable, testing, unstable) of
our own in-house software and gnu software. Now the
apt-frontend-network-tool that I envision creating would act as sort of
a mass-network admin gui that facilitates running apt-get upgrade. I'm
ultimately thinking of a gui-tree-like-display of all the debian
machines on our network. There would be a way to change machines'
sources.list by the handful (highlight a bunch of machines and set them
for stable, testing, unstable, whatever). I would also like to employ
some sort of sync/imaging utility that could be used in case a machine
were to become corrupted and needed to be re-imaged via pxeboot or some
other method.

Fancy extra features may include links to startup VNC/SSH sessions to
any given machine on the network, having machines call home via a small
script or C app, report their mac address to an sql database, report
diskspace, mac address, assett tags, deliver alerts, etc...

But yeah, basically, there is no gnu utility that I have found that does
these sorts of things, and certainly not one that focuses on linux or
debian's packaging tools. If I can get my company to get excited about
this tool, I'd love to start up a project and see what we can get going.

Ben


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On 06/17/07 22:08, Ben wrote:
[snip]
>
> Fancy extra features may include links to startup VNC/SSH sessions to
> any given machine on the network, having machines call home via a small
> script or C app, report their mac address to an sql database, report
> diskspace, mac address, assett tags, deliver alerts, etc...
>
> But yeah, basically, there is no gnu utility that I have found that does
> these sorts of things, and certainly not one that focuses on linux or
> debian's packaging tools.

wajig might be a good starting point for this kind of tool.

> If I can get my company to get excited about
> this tool, I'd love to start up a project and see what we can get going.

--
Ron Johnson, Jr.
Jefferson LA USA

Give a man a fish, and he eats for a day.
Hit him with a fish, and he goes away for good!


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Maybe, dsh -distributed shell- is a good starting point?
--
Regards,
Jörg-Volker.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Jörg-Volker Peetz wrote:
> Maybe, dsh -distributed shell- is a good starting point?

Using dsh illustrates a push-method. For a small number of machines
pushing works and for when you are monitoring the process manually
then pushing to machines work pretty well. But when setting up
hundreds or thousands of machines then invariably a number of those
machines will be down at the moment that you are trying to push to
them. Although dsh is a fine tool using the push-method runs into
process problems pretty quickly.

A better way is a pull-method. Have the clients pull the updates from
a server. This way if the client is powered down, hung on a stale nfs
mount, run out of memory by a runaway process, or otherwise
incapacitated then when it becomes operational again it will pull all
pending updates and get back up to date again.

Here is a summary of the issues:

http://www.infrastructures.org/bootstrap/pushpull.shtml

I strongly recommend going with a pull process.

Bob


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Bob Proulx <bob@proulx.com>:
> s. keeling wrote:
> > Ben wrote:
> > > I was wondering if there's a Debian specific tool that could facilitate
> > > managing thousands of machines via APT. I'm aware that many people would
> > > recommend a sync option, where 1 machine serves as the master, and the
> > > others sync off of that. Perhaps that is the only reliable approach, but
> >
> > I'll answer that with, why would someone in your position not want to
> > do that?
>
> I am not sure to which you direction you are referring. Why would

i. Thousands of machines updating from repositories.

ii. Thousands of machines updating from one/some local mirror(s).

> On a server farm of identical machines using system images works
> pretty well. Tools like SystemImager, FAI, Mondo, etc. are great.

Where did system imaging come into this?


--
Any technology distinguishable from magic is insufficiently advanced.
(*) http://www.spots.ab.ca/~keeling Linux Counter #80292
- - http://www.faqs.org/rfcs/rfc1855.html Please, don't Cc: me.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

s. keeling wrote:
> Bob Proulx <bob@proulx.com>:
> > s. keeling wrote:
> > I am not sure to which you direction you are referring. Why would
> i. Thousands of machines updating from repositories.
> ii. Thousands of machines updating from one/some local mirror(s).

Ah, yes, definitely use a local depot for reliability, performance and
efficiency. Agreed.

> > On a server farm of identical machines using system images works
> > pretty well. Tools like SystemImager, FAI, Mondo, etc. are great.
>
> Where did system imaging come into this?

The wording in the original posting is a little ambiguous here.
> Ben wrote:
> > I'm aware that many people would recommend a sync option, where 1
> > machine serves as the master, and the others sync off of
> > that. Perhaps that is the only reliable approach, but

Depending what comes to mind with "sync" I think a lot of people would
think binary imaging. I was not sure which was meant by it.

For distros that don't handle upgrades binary imaging is a typical
thought pattern. All upgrades on those systems are effectively
reinstalls from scratch. But since Debian is all about being able to
upgrade then doing imaging, while useful in many contexts, wastes the
potential of continuous incremental improvement available by apt
driven upgrades. I prefer apt driven upgrades.

Thanks for the clarification.
Bob


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

> I strongly recommend going with a pull process.
>
> Bob
>
I believe that I'm trying to create a push/pull method. The pull method
(cron-apt) would be for minor updates/releases, but push would be used
for special cases (critical rollouts, individual adjustments). In
addition, the tool that I want to create, would easily facilitate
adjusting how individual clients or groups of clients pull.

Ben


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org