19 November 2009 - 20:13find out if your Apache is really serving the right hosts

From time to time it can happen that websites that used to be on your webservers are being moved by your clients without notifying you. Therefore, you will have to search for «zombie websites».

I have the following setup: There is an Apache webserver that has a lot of websites. Each website has at least two Virtual Hosts, a first for the main website and a second one fetching all the aliases and redirecting them to the first. Each website has its own config file.

So, if I want to know whether a website is still being served by my machine, I run the following:

for host in `grep "Server\(Alias\|Name\)" _config/apache/*
| awk '{ for(i=3 ; i<=NF ; i++)printf "%s\n", $i}'`; do echo
 -n `nslookup "$host" | grep -A1 Name | grep Address | awk
'{print $2}'`; echo " $host"; done


Okay, let’s go through it step by step:


for host in `grep “Server\(Alias\|Name\)” _config/apache/*

Here we extract all lines containing ServerAlias or ServerName in all config files.  The output will look like

_config/apache/foo.bar.conf: ServerName www.foo.bar
 _config/apache/foo.bar.conf: ServerAlias foo.bar fuh.bar www.foo.bar
_config/apache/example.com.conf: ServerName www.example.com
_config/apache/yah.conf: ServerAlias    example.com

As we can see, the results have a varying size. Therefore, we have to tell awk about it.


| awk ‘{ for(i=3 ; i<=NF ; i++)printf “%s\n”, $i}’`

Awk takes each line and starts to read it from the third column (each seperated with a space) until EOL. What we get is being printed with a CRLF so that we do not get example.com fuh.bar www.foo.bar but

example.com
fuh.bar
www.foo.bar

do echo -n

Now we print each of the results of the following command without a CRLF afterwards.


`nslookup “$host” | grep -A1 Name | grep Address | awk ‘{print $2}’`;

Here we perform a IP lookup of the current host, grep for a string called “Name” and let it print the following line as well as it contains the IP address we are looking for. As we just need the IP address, we do another grep on the result and pick the second line. But we don’t want the “Address” string here, so we awk it away. To make it a little bit easier to understand, I will quickly show you what these commands do:

nslookup example.com
Server:         213.133.100.100
Address:        213.133.100.100#53
Non-authoritative answer:
Name:   example.com
Address: 192.0.32.10
---
nslookup example.com | grep -A1 Name
Name:   example.com
Address: 192.0.32.10
---
nslookup example.com | grep -A1 Name | grep Address
Address: 192.0.32.10
---
nslookup example.com | grep -A1 Name | grep Address | awk '{print $2}'
192.0.32.10


echo ” $host”

Now we have an IP but we would like to know what the corresponding hostname is, so we quickly echo it.

Done. Afterwards we can see:

192.0.32.10 www.example.com
yetanotherhost.mil

No Comments
Categories: Linux, Webserver, howto
Tags: , , , , , , , , , , , , , , , ,

12 February 2009 - 19:25SFTP only and SSH only OpenSSH system with gentoo

# Any random options you want to pass to sshd.
# See the sshd(8) manpage for more info.
SSHD_OPTS=”"

# Pid file to use (needs to be absolute path).
SSHD_PIDFILE=”/var/run/sshd2.pid”

# Path to the sshd binary (needs to be absolute path).
SSHD_BINARY=”/usr/sbin/sshd2″

Although there are many ways workarounding the lack of security with ye olde FTP, there is a quite handy solution: use OpenSSH via SCP/SFTP to handle the file transfers.

There are many solutions which all try to restrict the access for some users. That’s not what we are trying to do. At the end, we will have two running openssh-instances. One for ssh and the other for sftp/scp.

With Gentoo, this is quite easy to do, but even for the other distros, this howto should be usable.

Here are the steps:

1. Get OpenSSH. (Should already been done at install time)

emerge openssh

2. Copy some files, make links

cp /etc/ssh /etc/ssh2 -R
cp /etc/conf.d/ssh /etc/conf.d/ssh2
ln -s /etc/init.d/sshd /etc/init.d/sshd2
ln -s /usr/sbin/sshd /usr/sbin/sshd2

3. SSH-Server

Now edit /etc/ssh/sshd_config and remove the line containing «internal subsystem». Now you can decide what to do: either bind the servers to different IP and the same ports or vice versa or both. :-) Anyway, the options for this are:

Port <portnumber>
ListenAddress <ip-address>

3. SFTP-Server

Now edit /etc/ssh2/sshd_config and keep the Port- & IP-Settings of the SSH-server in mind.

Subsystem       sftp    internal-sftp

# These lines must appear at the *end* of sshd_config
ChrootDirectory %h
ForceCommand internal-sftp

This will force every successful login to start the internal sftp server and chroot to its home directory.

Edit /etc/conf.d/sshd2

# /etc/conf.d/sshd: config file for /etc/init.d/sshd
# Where is your sshd_config file stored?
SSHD_CONFDIR=”/etc/ssh2″# Any random options you want to pass to sshd.
# See the sshd(8) manpage for more info.
SSHD_OPTS=”"
# Pid file to use (needs to be absolute path).
SSHD_PIDFILE=”/var/run/sshd2.pid”
# Path to the sshd binary (needs to be absolute path).
SSHD_BINARY=”/usr/sbin/sshd2″

4. Add SFTP to the runlevels

rc-update add ssh2 default

5. Check permissions

Make sure, that the path to each user’s home directory is being set 0755 for root:root. Otherwise, you won’t be able to log in. Let’s say, your home directory is /home/users/domains/e/example.com/t/testuser. Then, each of the path’ elements must be set to 0755 root:root. This leads to an inability of creating and removing files in the home-root. Create an incoming-files directory to get around of this.

6. EXTRA: DenyHosts just for SFTP

emerge denyhosts

edit /etc/denyhosts.conf and adapt the options to fit your needs. There is just one thing you must change:

BLOCK_SERVICE  = sshd2

If you choose to run denyhosts as daemon, I suggest to add t to the default runlevel as well. And – of course – start it.

rc-update add denyhosts default
/etc/init.d/denyhosts start

That’s all, folks! :-)

No Comments
Categories: Linux, Software, howto
Tags: , , , , , , , , ,

15 January 2009 - 16:10rebuilding Cyrus indexes

You might know the situation, something happened, and afterwards you get error messages from Cyrus telling you that your databases just have crashed. Then, your Inbox is shown empty while the filesystem does show the correct files.

What happened?

Every Cyrus folder contains three files «cyrus.cache», «cyrus.header» and «cyrus.index». These are responsible for telling the mail client how many mails the specific folder contains and which flag has been set per mail. they are your per-folder-message-database. And that’s exactly the point why Cyrus is that much faster than any other MDA. If a mail client connects, the server just looks into these databases to serve the necessary information and just if you actually read the message, it is being loaded.
So if these databases are being corrupted, you won’t see anything but big emptiness although you might have several millions of messages physically stored in the folder.

Here are some example error messages…

Jan 12 15:02:32 host cyrus/imap[23319]: DBERROR: dbenv->open ‘/var/imap/db’ failed: Permission denied
Jan 12 15:02:32 host cyrus/imap[23319]: DBERROR: init() on berkeley
Jan 12 15:02:32 host cyrus/imap[23319]: DBERROR: reading /var/imap/db/skipstamp, assuming the worst: Permission denied
Jan 12 15:02:32 host cyrus/imap[23319]: executed
Jan 12 15:02:32 host cyrus/imap[23319]: IOERROR: opening /var/imap/mailboxes.db: Permission denied
Jan 12 15:02:32 host cyrus/imap[23319]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Jan 12 15:02:32 host cyrus/imap[23319]: Fatal error: can’t read mailboxes file
Jan 12 16:00:21 host cyrus/imaps[31010]: DBERROR db4: PANIC: fatal region error detected; run recovery
Jan 12 16:00:21 host cyrus/imaps[31010]: DBERROR: critical database situation
Jan 12 16:04:32 host cyrus/lmtpunix[31504]: DBERROR: opening /var/imap/deliver.db: Permission denied
Jan 12 16:04:32 host cyrus/lmtpunix[31504]: DBERROR: opening /var/imap/deliver.db: cyrusdb error
Jan 12 16:04:32 host cyrus/lmtpunix[31504]: FATAL: lmtpd: unable to init duplicate delivery database

Yes, this is exactly my problem! What’s next?

We created a little script that runs through every user’s mailbox and rebuilds the databases. the only problem is that after a successfull rebuild every message is being marked as unread. But that’s a rather small problem, we think.  :o )

Where can I get it from? I’m in a hurry!

We know you are. But before you download, please have a look at the readme or at least at the file in order to assure that the settings are correct. And don’t forget to backup. We do not grant for anything.

Download

Gentoo users can use our siczb portage overlay. Please have a look at this article to get to know how to access the overlay.

No Comments
Categories: Linux, Mailserver, Software, howto
Tags: , , , , , , , , , , , ,

1 April 2008 - 13:09icecast and awstats

Not much has yet been written about this topic and it really is no big deal but some sentences more would have made me feel more secure. Therefore, I tell what to do.

  1. We need icecast and awstats (definitely) and a webserver.
  2. Create your awstats-file and edit it as you like.
  3. Icecast’s way of logging is very much the same as apache, so the only thing to do is to say LogType=S

Notice:

  • Pages and Hits are the same in many statistics, so you do not need to show both in the timeline-stats (monthly/weekly/daily/hourly), hosts and origin tables.
  • As there are several tables just showing hits, I suggest to drop the pages columns

1 Comment
Categories: Linux, Software, howto
Tags: , , ,

11 February 2008 - 10:39lighttpd and awstats

There are lots of howtos for running awstats with lighthttpd. Nearly all of them define a subdomain for statistics but miss to add the only line for defaulting to awstats.pl. Here is what you have to do to do it the common way $HOST/awstats:

  1. open /etc/lighttpd/lighttpd.conf
  2. uncomment mod_cgi.conf
  3. add the following for a single host (don’t forget to change www.example.com as well as the path to awstats) or adopt it to your needs with mod_evhost:

# {{{ awstats.
$HTTP["host"] =~ “www.example.com” {
alias.url = (
“/awstats/icons” => “/usr/share/webapps/awstats/6.5-r1/htdocs/icon/”,
“/awstats/classes” => “/usr/share/webapps/awstats/6.5-r1/htdocs/classes/”,
“/awstats/css” => “/usr/share/webapps/awstats/6.5-r1/htdocs/css/”,
“/awstats.pl” => “/usr/share/webapps/awstats/6.5-r1/hostroot/cgi-bin/awstats.pl”,
“/awstats” => “/usr/share/webapps/awstats/6.5-r1/hostroot/cgi-bin/awstats.pl”)
cgi.assign = ( “.pl” => “/usr/bin/perl”, “.cgi” => “/usr/bin/perl” )
}
# }}}

No Comments
Categories: Linux, Webserver, howto
Tags: , , ,

30 January 2008 - 19:41bringing awstats and mod_vhost_alias together

<!– @page { size: 21cm 29.7cm; margin: 2cm } P { margin-bottom: 0.21cm } H1 { margin-bottom: 0.21cm } H1.western { font-family: “Albany”; font-size: 16pt } H1.cjk { font-family: “HG Mincho Light J”; font-size: 16pt } H1.ctl { font-family: “Lucidasans”; font-size: 16pt } –>

Perhaps you know the situation: You’re happy not having to add the same six lines to your httpd.conf because you got mod_vhost_alias to work. You feel fine, the sun’s shining, birds are singing, the full monty. Everything seems quite usual except one thing: your awstats are going to burst your system on each run. The more often you cronjob them, the sooner you’ll know about it. Now you realise that letting awstats_updateall parse your 300 MB daily all-in-one-access.log several dozen times per day might not be such a good idea. :-) So you’re looking for a suitable solution. Great. This is why you’re here.

You’ll find the script underneath this little howto, but I would suggest to read it before running the script.

 

The situation

We have a mixed environment consisting of mass vhosts, SSL- and Tomcat hosts. Therefore, we have two different kinds of logfiles: A huge all-in-one for the mass vhosts and several vhost-based files. As a result, we’ll have to change every awstats.$HOST.conf to fit into our new situation. Nevertheless, all awstats hosts parse the same big log file.

Solution I

Let awstats parse the All-in-one-logfile n times. Possible but somewhat brutal.

Solution II

You just could use split-logfile. Great! Just change your CommonLog’s format to vhost and adapt the awstats configs accordingly. But if you do the nasty tweaky stuff offered by mod_rewrite, you’ll miss all the host aliases (formerly noted via ServerAlias).

Solution III

Use my script. :-)

Here is what’s you’ll have to do:

  1. Download the script (I suggest to save it in one of the sbin folders)

  2. chmod +x awstats_manager.sh

  3. edit it with your preferred editor and change the variables to your needs. Keep $TEMPDIR’s value in mind

  4. backup your awstats configs (mv $awstatsdir $awstatsdir.orig)

  5. now fill in the SiteDomain,HostAliases and the LogFile

    The latter one has to have the following syntax: $TEMPDIR/merged/$SiteDomain.log as awstats will find its freshmade configs there

    If you already have a bunch of mass hosts and don’t want to change them manually, change awstats.model.conf and run the following:

      newlog=/tmp/apache2/merged/;for file in awstats.www*; do cp awstats.model.conf $file; echo “SiteDomain=\”`echo $file | sed -e ’s/^awstats.//g;s/.conf$//g’`\”" >> $file; echo “HostAliases=\”REGEX[`echo $file | sed -e 's/^awstats.www.//g;s/.conf$//g'`]\”" >> $file; echo “LogFile=\”$newlog`echo $file | sed -e ’s/awstats.//g;s/conf$/log/g’`\”" >> $file; done

    This will do the following: newlog is the name of your new logfile, SiteDomain will be the domain name included in theawstats file name and HostAliases will be just the domain.
    Here’s an example:
    awstats.www.example.com.conf will set SiteDomain to www.example.com, HostAliases to REGEX[example.com] and Logfile to /tmp/apache2/merged/www.example.com.log

  6. Done? Great. Now add awstats_manager.sh to your cron and you’re done: ln -s /path/to/ awstats_manager.sh /etc/cron.daily/

Now you know what you have to do, you should know what the script does….

  1. create the temporary directories and chmod them 0700

  2. copy the big log file and split it

  3. create a list of awstats configs which get their input from logfiles in this temporary directory

  4. fetch SiteDomain and HostAliases for each of the files

  5. check for empty entries and duplicates

  6. extract what is regularly expressed

  7. find the corresponding log files (if they exist)

  8. run logrotatemerge.pl with all of the found log files

  9. run awstats -update for the corresponding host

  10. afterwards find all the other awstats hosts and update them as well

  11. clean up

Thats’s all. Have a lot of fun! Comments are welcome.

Download: awstats_manager.sh

No Comments
Categories: Linux, Software, Webserver, howto
Tags: , ,

blogoscoop