Hi,

Since this week-end, I have big problems on my RHEL3 server, with sshd
:
- I can no more connect from clients whith ssh. I was able to connect
from 2 years,
- I have 2 new lines in my sshd_config : "DString sweetgeorgiana" and
"DVersion openssh-.3.6p2...",
- I deleted these 2 lines last Saturday evening, and they came back on
Sunday,
- between Saturday and Sunday, I restarted my sshd, and was able to
connect,
- I have a process which abnormally opens the port 417 : this process
is called "ssh/bin/initsshd -p 417", directly connected to "init". The
command line of this process does not begin by "/". I performed a find
/ -name initsshd without finding it,
- I also killed this process on Sunday morning, and is now back on
Monday morning,
- I have reboot on Sunday, after examination of /etc/init.d, but I saw
nothing special.

Does somebody knows what happens to me ? And could I reprotect my
server, if I am really hacked ?

TIA

Sylvain

sdonnet@ddo-org.com writes:

>Hi,

>Since this week-end, I have big problems on my RHEL3 server, with sshd
>:
>- I can no more connect from clients whith ssh. I was able to connect
>from 2 years,
>- I have 2 new lines in my sshd_config : "DString sweetgeorgiana" and
>"DVersion openssh-.3.6p2...",
>- I deleted these 2 lines last Saturday evening, and they came back on
>Sunday,
>- between Saturday and Sunday, I restarted my sshd, and was able to
>connect,
>- I have a process which abnormally opens the port 417 : this process
>is called "ssh/bin/initsshd -p 417", directly connected to "init". The
>command line of this process does not begin by "/". I performed a find
>/ -name initsshd without finding it,
>- I also killed this process on Sunday morning, and is now back on
>Monday morning,
>- I have reboot on Sunday, after examination of /etc/init.d, but I saw
>nothing special.

>Does somebody knows what happens to me ? And could I reprotect my
>server, if I am really hacked ?


Sounds to me like you have been cracked.

backup your crucial stuff.
Wipe the disk
reinstall the operating system
Change ALL passwords
use find to scan the reinstalled backup for suid root programs.

sdonnet@ddo-org.com writes:

> Does somebody knows what happens to me ? And could I reprotect my
> server, if I am really hacked ?

Reformat and reinstall operating system from original media, apply all
updates before opening any internet facing service.

Sounds like you may well be owned.

Best Regards,
--
Todd H.
http://www.toddh.net/

Todd H. wrote:
> Reformat and reinstall operating system from original media, apply all
> updates before opening any internet facing service.
>
> Sounds like you may well be owned.
>
> Best Regards,

consider installing an IDS to track what changes when

--
try a random act of kindness today -- you just might surprise even
yourself

Todd H. a écrit :

> sdonnet@ddo-org.com writes:
>
> > Does somebody knows what happens to me ? And could I reprotect my
> > server, if I am really hacked ?
>
> Reformat and reinstall operating system from original media, apply all
> updates before opening any internet facing service.
>
> Sounds like you may well be owned.
>
> Best Regards,
> --
> Todd H.
> http://www.toddh.net/

Thanks for your support. I was hoping that corrections could be done...
I am going to prepare a reinstallation...

Sylvain

sdonnet@ddo-org.com writes:

> Todd H. a écrit :
>
> > sdonnet@ddo-org.com writes:
> >
> > > Does somebody knows what happens to me ? And could I reprotect my
> > > server, if I am really hacked ?
> >
> > Reformat and reinstall operating system from original media, apply all
> > updates before opening any internet facing service.
> >
> > Sounds like you may well be owned.
> >
> > Best Regards,
> > --
> > Todd H.
> > http://www.toddh.net/
>
> Thanks for your support. I was hoping that corrections could be
> done...

Unfortunately it's just not feasible or advisable. The problem is
that it's impossible to gauge the depth of the penetration. Even if
you scan and find 10 problems and fix them, there could be several
others that your scanner, root kit detector, etc doesn't know about.
As such, it's always "best practice" to format and start from original
media.

> I am going to prepare a reinstallation...

Yeah, sadly as much as it is a pain in the butt, it's still the
fastest way to get to a known state again.

--
Todd H.
http://www.toddh.net/

comp@toddh.net (Todd H.) writes:

>sdonnet@ddo-org.com writes:

>> Todd H. a écrit :
>>
>> > sdonnet@ddo-org.com writes:
>> >
>> > > Does somebody knows what happens to me ? And could I reprotect my
>> > > server, if I am really hacked ?
>> >
>> > Reformat and reinstall operating system from original media, apply all
>> > updates before opening any internet facing service.
>> >
>> > Sounds like you may well be owned.
>> >
>> > Best Regards,
>> > --
>> > Todd H.
>> > http://www.toddh.net/
>>
>> Thanks for your support. I was hoping that corrections could be
>> done...

>Unfortunately it's just not feasible or advisable. The problem is
>that it's impossible to gauge the depth of the penetration. Even if
>you scan and find 10 problems and fix them, there could be several
>others that your scanner, root kit detector, etc doesn't know about.
>As such, it's always "best practice" to format and start from original
>media.

>> I am going to prepare a reinstallation...

>Yeah, sadly as much as it is a pain in the butt, it's still the
>fastest way to get to a known state again.

It is however also crucial that you scan the stuff you reinstall as well.
When I was broken into I found files scattered all over the file system--
/tmp, /dev/, /home, ....
which were suid shells-- ie anyone knowing about them if they had any entry
at all onto the machine could simply run that program and be root.

Ie, scan all of the files you restore for suid
find / -perm +6000 -ls
check each one to see if it should be suid. su is fine. /tmp/banana is not.

Unruh <unruh-spam@physics.ubc.ca> writes:

> It is however also crucial that you scan the stuff you reinstall as well.
> When I was broken into I found files scattered all over the file system--
> /tmp, /dev/, /home, ....
> which were suid shells-- ie anyone knowing about them if they had any entry
> at all onto the machine could simply run that program and be root.
>
> Ie, scan all of the files you restore for suid
> find / -perm +6000 -ls
> check each one to see if it should be suid. su is fine. /tmp/banana
> is not.

This is a good anecdote as why reformating is a good first step before
the reinstall.


--
Todd H.
http://www.toddh.net/

On 24 Oct 2006 12:44:49 -0500, comp@toddh.net (Todd H.) wrote:

>Unruh <unruh-spam@physics.ubc.ca> writes:
>
>> It is however also crucial that you scan the stuff you reinstall as well.
>> When I was broken into I found files scattered all over the file system--
>> /tmp, /dev/, /home, ....
>> which were suid shells-- ie anyone knowing about them if they had any entry
>> at all onto the machine could simply run that program and be root.
>>
>> Ie, scan all of the files you restore for suid
>> find / -perm +6000 -ls
>> check each one to see if it should be suid. su is fine. /tmp/banana
>> is not.
>
>This is a good anecdote as why reformating is a good first step before
>the reinstall.

I'd clear OS partition to zero, then reformat prior to install.

Grant.
--
http://bugsplatter.mine.nu/

Grant wrote:
> On 24 Oct 2006 12:44:49 -0500, comp@toddh.net (Todd H.) wrote:
>
>> Unruh <unruh-spam@physics.ubc.ca> writes:
>>
>>> It is however also crucial that you scan the stuff you reinstall as well.
>>> When I was broken into I found files scattered all over the file system--
>>> /tmp, /dev/, /home, ....
>>> which were suid shells-- ie anyone knowing about them if they had any entry
>>> at all onto the machine could simply run that program and be root.
>>>
>>> Ie, scan all of the files you restore for suid
>>> find / -perm +6000 -ls
>>> check each one to see if it should be suid. su is fine. /tmp/banana
>>> is not.
>> This is a good anecdote as why reformating is a good first step before
>> the reinstall.
>
> I'd clear OS partition to zero, then reformat prior to install.
>
> Grant.

Who's to say the format utility isn't compromised as well? I'd boot from
a CD-ROM (something non-writable) and do the format from there.

"Chuck" <skilover_nospam@bluebottle.com> wrote in message
news:qNt%g.5129$fA.704@trnddc05...
> Grant wrote:
>> On 24 Oct 2006 12:44:49 -0500, comp@toddh.net (Todd H.) wrote:
>>
>>> Unruh <unruh-spam@physics.ubc.ca> writes:
>>>
>>>> It is however also crucial that you scan the stuff you reinstall as well.
>>>> When I was broken into I found files scattered all over the file system--
>>>> /tmp, /dev/, /home, ....
>>>> which were suid shells-- ie anyone knowing about them if they had any entry
>>>> at all onto the machine could simply run that program and be root.
>>>>
>>>> Ie, scan all of the files you restore for suid
>>>> find / -perm +6000 -ls
>>>> check each one to see if it should be suid. su is fine. /tmp/banana
>>>> is not.
>>> This is a good anecdote as why reformating is a good first step before
>>> the reinstall.
>>
>> I'd clear OS partition to zero, then reformat prior to install.
>>
>> Grant.
>
> Who's to say the format utility isn't compromised as well? I'd boot from
> a CD-ROM (something non-writable) and do the format from there.

DBAN.
http://dban.sourceforge.net/

> > > Reformat and reinstall operating system from original media, apply all
> > > updates before opening any internet facing service.
> > >

Thanks for all your advices.

I already have desinstalled openssh-server. I sitll cannot deinstall
openssh and openssh-clients due to dependencies.

As I cannot stop the server, because it is on production, my only
solution is to install a new fresh one, transfer the data (it is a mail
server), and reinstall+format the first one, and transfer back.

I think I'll be busy on next week...

Sylvain

Mike Lowery wrote:
> DBAN.
> http://dban.sourceforge.net/

Voor opruimen na een inbraak is het onzin, maar ik zocht al een tijdje
naar zoiets. Handig voor het weggooien/verkopen van bakken waar
vertrouwelijke informatie op staat.

Steven Mocking wrote:
> Mike Lowery wrote:
>> DBAN.
>> http://dban.sourceforge.net/
>
> Voor opruimen na een inbraak is het onzin, maar ik zocht al een tijdje
> naar zoiets. Handig voor het weggooien/verkopen van bakken waar
> vertrouwelijke informatie op staat.

Oops, wrong language.

For cleaning up after a breakin, secure deletion is useless, but I've
been looking for something like this for a while in order to get rid of
some machines without wondering what confidential data was on them again.

On Fri, 27 Oct 2006 00:44:40 +0200, Steven Mocking <ufo@quicknet.youmightwanttogetridofthis.nl> wrote:

>For cleaning up after a breakin, secure deletion is useless, but I've
>been looking for something like this for a while in order to get rid of
>some machines without wondering what confidential data was on them again.

A 4" nail hammered through the HDD is a much faster method

Grant.
--
http://bugsplatter.mine.nu/

"Steven Mocking" <ufo@quicknet.youmightwanttogetridofthis.nl> wrote in message
news:12k2eip9qheppd1@corp.supernews.com...
> Steven Mocking wrote:
>> Mike Lowery wrote:
>>> DBAN.
>>> http://dban.sourceforge.net/
>>
>> Voor opruimen na een inbraak is het onzin, maar ik zocht al een tijdje
>> naar zoiets. Handig voor het weggooien/verkopen van bakken waar
>> vertrouwelijke informatie op staat.
>
> Oops, wrong language.
>
> For cleaning up after a breakin, secure deletion is useless, but I've
> been looking for something like this for a while in order to get rid of
> some machines without wondering what confidential data was on them again.

Perhaps unnecessary, but not useless. It's a handy boot disk for wiping a hard
drive, securely or otherwise (may not be the fastest however.)

Chuck wrote:

> Who's to say the format utility isn't compromised as well? I'd boot from
> a CD-ROM (something non-writable) and do the format from there.

That's how you normally reformat the / partition, where core software
lives.

Now, I'd invest in a second disk if feasible, install the new OS on the
second disk with only critical text configuration files brought over
from backup, very, very carefully, and set aside the first disk for
examining as a spare drive in a safe environment (such as booting from
a Knoppix LiveCD) to apply some analysis to it and look for traces in
the logs.

Assume also that every password and account on that system have been
sniffed and cracked: if you haven't been paying attention to how to
protect your systems from an attack from the inside, you are now very
vulnerable to any accounts that existed on that system.

Chuck wrote:

> > I'd clear OS partition to zero, then reformat prior to install.
> >
> > Grant.
>
> Who's to say the format utility isn't compromised as well? I'd boot from
> a CD-ROM (something non-writable) and do the format from there.

Doing the repartition is normally done from a boot CD or network boot.
Personally, if possible, I'd set aside the compromised disk and install
a new one, then use the compromised disk on an isolated Knoppix LiveCD
box to examine the logs with the tools from Knoppix, not local tools,
to see what traces of the attack are in place.

But the OP should consider every account that existed on that machine,
or which could be sniffed from that machine, compromised. This sort of
thing is common in environments where some admin says "we have a
firewall, we don't have to worry about internal machine updates,
they're stable, don't patch them" and one machine gets compromised.
It's now appropriate to lock down *EVERYTHING*, make sure your backups
are offsite, and probably change everyone's passwords and rebuild core
servers to make sure they haven't also been compromised.

These are harsh lessons learned way, way back in the Morris Worm
incident of 1988: the lessons are still valid.

sdonnet@ddo-org.com wrote:

> > > > Reformat and reinstall operating system from original media, apply all
> > > > updates before opening any internet facing service.
> > > >
>
> Thanks for all your advices.
>
> I already have desinstalled openssh-server. I sitll cannot deinstall
> openssh and openssh-clients due to dependencies.
>
> As I cannot stop the server, because it is on production, my only
> solution is to install a new fresh one, transfer the data (it is a mail
> server), and reinstall+format the first one, and transfer back.
>
> I think I'll be busy on next week...

This is nowhere near enough. The cracker may have every password from
that system, including sudo passwords or root or SSH keys that are
stored locally, especially those without passphrases. They've been into
the system: even if the rest of the binaries are not corrupted, the
SSHD the cracker installed was doubtless sniffing passwords, and many
old tools such as CVS or many Subversiion clients store passwords
locally in clear text.

You're due for a *LOT* of work. This is a good time to hop from RHEL
3.x to RHEL 4.x or CentOS 4.x, and pursuing it as a policy to prevent
future such cracks.

Nico wrote:
> Chuck wrote:
>
>> Who's to say the format utility isn't compromised as well? I'd boot from
>> a CD-ROM (something non-writable) and do the format from there.
>
> That's how you normally reformat the / partition, where core software
> lives.
>
> Now, I'd invest in a second disk if feasible, install the new OS on the
> second disk with only critical text configuration files brought over
> from backup, very, very carefully, and set aside the first disk for
> examining as a spare drive in a safe environment (such as booting from
> a Knoppix LiveCD) to apply some analysis to it and look for traces in
> the logs.
>
> Assume also that every password and account on that system have been
> sniffed and cracked: if you haven't been paying attention to how to
> protect your systems from an attack from the inside, you are now very
> vulnerable to any accounts that existed on that system.
>

IOW change all passwords.

If there were any unencrypted private keys stored on the box assume they
are now compromised. Remove the corresponding public key from all
servers immediately and generate new keypairs. This goes for SSH as well
as PGP and GnuPG.

Chuck wrote:

> IOW change all passwords.
>
> If there were any unencrypted private keys stored on the box assume they
> are now compromised. Remove the corresponding public key from all
> servers immediately and generate new keypairs. This goes for SSH as well
> as PGP and GnuPG.

And be prepared to change them *AGAIN* in the near future. Once a
clever cracker has access to a poorly secured password in your dial-up
or VPN systems, they can often re-install their rootkits or system
tools on other poorly secured systems behind your corporate firewall,
to lurk like hunters in a duck blind for the next set of passwords or
private keys to steal.

Backup sytems in particular are vulnerable to this kind of abuse.
Unencrypted /etc/passwd files, or backup systems that allow restore
restoration by anyone inside the company, are also fun targets of a
cracker inside your systems. This is why security is not a single tool:
it's a set of practices that prevent damage and confine it as much as
possible.