Gentoo ebuild: Nginx With Support For Upstream Fair Proxy Load Balancer

Posted on January 10, 2012 by Direktion

Nginx is a powerful web server and therefore our choice. Unfortunately, the Gentoo ebuild is missing one essential extension: a load balancer that is querying the servers not via round robin but by their current load. Therefore, we had to extend the default ebuild to support gnosek’s upstream_fair module.

How to use this ebuild:

  1. download the ebuild (currently, we only have one for nginx 1.1.12)
  2. place it anywhere portage has access to (i.e. rough: /usr/portage/www-servers/nginx/)
  3. run ebuild nginx-1.1.12-r2.ebuild digest
  4. add upstream_fair to NGINX_MODULES_HTTP in /etc/make.conf
  5. add your keywords to /etc/portage/package.keywords
  6. emerge nginx
  7. done.
Posted in howto, Linux, Software, Webserver | Tagged , , , , , , , , | Leave a comment

dovecot: remove maildirs

Posted on November 17, 2011 by Direktion

We are running dovecot as MDA. Dovecot gets its user details from OpenLDAP and adds new users automatically.  But removing a user in LDAP does not mean it gets removed in Dovecot as well. To have this a little bit more comfortable, I created this little script here:

#!/bin/bash

MAILDIR="/mails"
LDAP_HOST="ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock"
LDAP_BIND_USER="cn=Manager,dc=example,dc=com"
LDAP_BIND_PASS="password"
LDAP_BASE_DN="ou=People,dc=example,dc=com"

  for DOMAINDIR in $MAILDIR/*;do
    if [ -d $DOMAINDIR ];then
      DOMAIN=$(echo $DOMAINDIR | sed 's/\//\ /g' | awk '{print $4}')
      for USERDIR in $DOMAINDIR/*;do
        if [ -d $USERDIR ];then
          USER=$(echo $USERDIR | sed 's/\//\ /g' | awk '{print $5}')
          MAIL="$USER@$DOMAIN"
          EXISTS=$(ldapsearch -H $LDAP_HOST -D $LDAP_BIND_USER \
                     -w $LDAP_BIND_PASS -x -b $LDAP_BASE_DN \
                     mail=$MAIL mail | grep -c dn\:)
          if [ "$EXISTS" == "0" ];then
            echo "$MAIL is obsolete."
            echo -n "Removing userdir..."
            rm -rf $USERDIR
            echo " done."
          fi
        fi
      done
    fi
done

What the script does is to crawl every subdirectory of MAILDIR. This is where we receive the domain names through a sed/awk-combination. For every domain name we crawl its userbase. A similar sed/awk-combination is being used to receive the user names. Then we create an eMail address out of the two retrieved bits of information.  Now we are ready to check this mail address against the LDAP. If we receive a negative answer (address is not found and therefore no “dn”), we can be sure the eMail account has been removed. Finally, we remove the mail directory of the non-existing user.

The script itself should be handed over to the cron, I’d say.

Here is the download for the lazy.

Posted in howto, Linux, Mailserver | Tagged , , , , , , , , , , , , | Leave a comment

Locate all Apple serial numbers in a subnet

Posted on November 11, 2011 by Direktion

The command ioreg -l | grep IOPlatformSerialNumber will show you your Mac’s serial number. With this, you can find out when your Mac has been created if you enter this number to https://selfsolve.apple.com/agreementWarrantyDynamic.do

For a whole network of Macs, this can become very boring. However, there is a nice way to solve this task more conveniently. The following script is your little helper.

It needs two arguments: an IP range (i.e. 192.168.0) and an user that is allowed to log into every Mac. Of course it would be great if every Mac would have your public key already in its authorized_keys file.

You will need nmap installed for we need it to check whether port 22 is opened.

What the script does is to check every IP from IP-range.1 to IP-range.254 if port 22 is opened. If so, we try to log in and to run the ioreg command. The output is grep’d and awk’d so that we receive nothing but the serial number. If we have the serial, we add this together with the host’s IP to a file.

#!/bin/bash
if [ "$#" == "2" ] ; then
  _IP_RANGE=$1
  _SSH_USER=$2
else
  echo "usage $0 IP-range ssh-user";
  echo "IP-range should be of format xxx.yyy.zzz (no trailing .)"
  echo "ssh-user must exist on every accessible host"
  exit 1;
fi

_I=0

while [ $_I -lt 254 ]; do
  ((_I++))
  currentip="$_IP_RANGE.$_I"
  if [ $(nmap $currentip -p22 | grep -c open ) -eq 1 ]; then
    ssh-keyscan $currentip 2>/dev/null 1>> ~/.ssh/known_hosts
    currentserial=$(ssh $_SSH_USER@$currentip ioreg -l | grep \
                     IOPlatformSerialNumber | awk '{print $4}')
    if [ $currentserial ];then
      echo "$currentip $currentserial" >> $_IP_RANGE.txt
    fi
  fi
done
Posted in howto, Software | Tagged , , , , , , , , , , , , | Leave a comment

SVN: search all repositories for file

Posted on September 29, 2011 by Direktion

Say you have shell access to the SVN server of you desire which contains like a million repos. Say you are looking for a file but don’t know where it is, you can do the following:

myfile="FILE.NAME"; for dir in /var/svn/*; do echo -n "$dir"; \
svn list -R file:///$dir | grep -i $myfile;echo "... done"; done

At first, you tell which string to look for. »svn list -R file:///$dir« gives us a listing of al files in the current repository, the file is being found via case-insensitive search by grepping the output. And first but not least, we are performing this for every directory (==repository) of »/var/svn«. The echos around are just to make it fancy. :)

(via stackoverflow)

Posted in howto, Linux, Verschiedenes | Tagged | Leave a comment

sslh: let https,ssh and openvpn share a single port

Posted on September 2, 2011 by Direktion

Introduction

Maybe you know the situation: You successfully logged in to a corporate network and somehow you can’t reach your server. You check all possibilities and realize that some of the ports you desperately need to manage your daily life have been blocked. And mostly this happens when you have to have shell access.

The first thing you think of is to set up OpenVPN. Unfortunately, the default port 1194 is blocked for UDP and TCP. So having a normal OpenVPN is not an option.

Port 80 and 443 are accessible, but you already have services running there.

Wouldn’t it be absolutely great to be able to share one of these ports, to have two or three services listening on a single port? Indeed.

This is possible: sslh is an easy-to-configure and small tool which is doing exactly what we want. (Fantastic, isn’t it?)

Configuration

sshl has been packaged for Debian, gentoo and FreeBSD. I will show you the gentoo-way.

1. install your favourite webserver and configure it to listen on localhost:443
2. install OpenVPN an let it run in TCP mode on localhost:1194
3. set up OpenSSH to listen to localhost:22
4. install sslh:

emerge sslh

edit /etc/conf.d/sslh to look like this (replace the x-es with your IP)

OPTIONS="--listen xxx.xxx.xxx.xxx:443 --ssl localhost:443 \
 --ssh localhost:22 --openvpn localhost:1194"

restart your services (if necessary), start sslh and add it to your default runlevel:

/etc/init.d/sslh start
rc-update add sslh default

That’s all. Really. :)

 


Posted in howto, Linux, Software | Tagged , , , , , , | Leave a comment

If Your Squirrel Won’t Keep The House Clean

Posted on July 25, 2011 by Marcus Pauli

There are not many usable open source webmailer. We are running two: Squirrelmail and Roundcube. Besides that this can be really annoying when it comes to server-side filtering, this offers lots of benefits to our users. Some like it simple and rough – these tend to Squirrelmail -, others like it more stylish – these usually choose Roundcube and its fancy AJAX interface.

However, the Squirrelmail – or at least our version which is 1.4.18 – doesn’t clean its data directory by itself. In this directory you can find user preferences and uploaded attachments. Obviously, we don’t want to remove any user’s settings but the already sent attachments.

Luckily, an attachment file gets a new name: a hash. Whereas user settings contain two very important characters: . and _

This makes it very easy for us to set up a cron job which removes an uploaded file that is older than 7 days (just to be sure)

And this is how we do it:

crontab -e

add this line

@daily find /YOUR/PATH/TO/SQUIRRELMAIL'S/DATA/DIRECTORY -type f \
-not \( -iname '*\.*' -o -iname '*_*' \) -mtime +7 -exec rm -f {} \;

save, exit and you’re done. :)

This will set up a daily cron job (usually running at 3.00 am) that scans a given directory recursively for files (-type f) which don’t contain a dot or an underscore (-not \( -iname ‘*\.*’ -o -iname ‘*_*’ \)) and have been last modified at least 7 days ago (-mtime +7) and finally deletes them (-exec rm -f {} \;)

Posted in howto, Linux, Mailserver | Tagged , , , , , , | Leave a comment

Howto split a SQL database dump into table-wise files

Posted on July 12, 2011 by Marcus Pauli

sql files containing a single database

Splitting a sql file containing a whole database into per-table files is quite easy:

  1. Grep the .sql for any occurence of DROP TABLE.
  2. Generate the file name from the table name that is included in the DROP TABLE statement.
  3. Echo the output to a file.

Here is a little script that expects a .sql file as input:

#!/bin/bash

file=$1 # the input file
directory="$file-splitted" # the output directory
output="$directory/header" # the first file containing the header
GREP="DROP TABLE" # what we are looking for

mkdir $directory # create the output directory

while read line
do
   # if the current line contains the wanted statement
   if [ $(echo "$line" | grep -c "$GREP") == "1" ]
   then
      # extract the file name
      myfile=$(echo $line | awk '{print $5}' | sed -e 's/`//g' -e 's/;//g')
      # set the new file name
      output="$directory/$myfile"
   fi
       echo "$line" >> $output # write to file
done < $file

Here you can download the script if you don’t want to copy & paste.

[UPDATE] sql files containing multiple databases

Stephan Schier [thanks a lot] extended the script in a way that it’s working with complete MySQL snapshots:

#!/bin/bash

file=$1 # the input file
directory="$file-splitted" # the output directory
output="$directory/__header" # the first file containing the header
GREPTABLE="DROP TABLE" # the string we are looking for to find a new table
GREPDB="Current Database: " # the string we are looking for to find a new database

mkdir $directory # create the output directory
while read line
do
  # if the current line contains the wanted DB
  if [ $(echo "$line" | grep -c "$GREPDB") == "1" ]
    then
    # extract database name
    mydb=$(echo $line | awk '{print $4}' | sed -e 's/`//g')
    # create db directory
    mkdir $directory/$mydb
    # set the new header file name
    output="$directory/$mydb/__header"
  elif [ $(echo "$line" | grep -c "$GREPTABLE") == "1" ]
  then
    # extract the file name
    myfile=$(echo $line | awk '{print $5}' | sed -e 's/`//g' -e 's/;//g')
    # set the new file name
    if [ -z "$mydb" ]
    then
      output="$directory/$myfile"
    else
      output="$directory/$mydb/$myfile"
    fi
  fi
  echo "$line" >> $output # write to file
done < $file

Here you can download the updated script if you don’t want to copy & paste.

 

 

Posted in howto, Linux | Tagged , , , , , , , , , , | 1 Comment

from Squirrel to Crystal

Posted on September 24, 2010 by Marcus Pauli

For years, we were using Squirrelmail as the default web mail client. Well, it would be a lie to say its design looks good but it offers a lot of functionality that nearly compensates the design deficits. The really basic functionalities we need are (a) to be able to authorize against a LDAP server in a way that we can define and (b) to control Sieve mail filters. All the rest is (frankly) luxury. Well, of course it has to be open source.

From time to time we are looking for alternatives to Squirrelmail. Unfortunately, this market is not the fastest. So there is the Squirrel. The Squirrel is PHP and that’s it. And though they say they will AJAX-ialize the whole thing, nobody knows when they will finish.
Then there is the DIMP from The Horde. The D stands for Dynamic. Impressing, isn’t it? Actually, the DIMP is an dynamized version of the IMP, so a conceptual hybrid. Sounds not too good, right? I assume they came to the same conclusion and decided to merge DIMP back into the original – IMP. But it’s still in beta and the look’n'feel has not improved that much.  So beside the conceptual and design deficites, it’s still a beta which i the mot obvious reasons not to chose The Horde.
The next we saw was Zimbra. It’s not just a webmailer, it’s a whole groupware with everything we already have: a mail transfer agent, a mail delivery agent, a LDAP server. Now the thing is that though it might be a nice all-in-one-product the efforts and benefits are not that balanced. Sorry Zimbra.
Finally I found a tool called Roundcube and it was pretty much what I expected. Except one thing: No mobile client support. That’s an enormous disadvantage because I have one of those incredibly useless Symbian S60/5 phones and the internal mail client is not really able to handle one or two larger IMAP mailboxes. At least, Roundcube offered skins. Crawling through the sites I found a skin that tried to be mobile and Apple-ish. Then said they had stopped their development (doh!) because they’re forking  form Roundcube to have something better (yay?) – CrystalMail. I checked their website and found a promo video. Well, I’m usually not the type of guy that waits for a new software release just because of some propaganda but this time I had to make an exception.

In the meantime, CrystalMail became 1.2 and somewhat stable. It offers everything I need – except the mobile version. On the other side it has this magnificent Meebo toolbar where you can have your FacebookChat, your ICQ, whatever opened – where ever you are. Isn’t that cool? And even more: You can add additional IMAP acounts and check your mails on one platform. It is even possible to import your google contacts.

The migration process will be explained in detail in the next article…

Posted in Linux, Mailserver, Software, Webserver | Tagged , , , , , , , , , | Leave a comment

simplify your daily ssh usage… even more…

Posted on August 5, 2010 by Marcus Pauli

On MeyerMillerSmith.com I recently posted this one:

Typing ssh commands can be very annoying but it doesn’t has to. There’s a quite nice way to simplify your daily admin life: shell scripts.

Okay, let’s say we have a dozen servers which we regularly connect to. Let’s assume that they have quite long names and we don’t want to type them every time. Surely, we could simply use Ctrl-i, but if the server name is not in the history any more, we will have to type again – and loose time.

The solution is quite easy. Create the following script connect-to-server.sh and save it i.e. under /usr/local/bin

ssh `basename $0`

Don’t forget to

chmod +x /usr/local/bin/connect-to-server.sh

Done? Great! Now we just need symlinks like this

ln -s /usr/local/bin/connect-to-server.sh \
this-is-the-name-of-the-server.my-companies-name.com

So if you want to connect to you server just enter the first letters of the server’s name and hit Tab to autocomplete, then Enter and you’re done.


Okay. This was quite cool… but not cool enough… I still had to manually symlink.

Now I wanted to extend the little script to add a new host:

#!/bin/bash
if [ $1 ]
then
 echo "adding new host: $1"
 if [ `id -u` == "0" ]
 then
 ln -s $0 `dirname $0`/$1
 else
 echo "need root privileges... using sudo..."
 sudo ln -s $0 `dirname $0`/$1
 fi
else
 echo "connecting to `basename $0`"
 ssh `basename $0`
fi

So what does it? If we don’t get a parameter, we assume that we’re in SSH mode and connect to the host represented through the filename. If we have a parameter, we assume that this is the new host to connect to and add a new host. As we are writing in /usr/local/bin we have to check whether we have root privileges. If not, we use sudo. That’s all.

Here you can download the script if you don’t want to copy & paste.

Posted in howto, Linux | Tagged , , , | Leave a comment

ProFTPD with mod_dnsbl as gentoo ebuild

Posted on December 7, 2009 by Marcus Pauli

Recently we realized that proftpd misses several modules that are available but not included in the standard distribution. Therefore, we created an ebuild. As time was short today, we only included mod_dnsbl. In the near future, we will add more modules.

The ebuild is available on our SVN repository: http://dev.informations-compagnie.de/svn/gentoo/net-ftp/proftpd/proftpd-1.3.2b-r1.ebuild. Please have a look at our repository intro page for instructions on how to use the additional overlay.

Posted in Linux, Software, Verschiedenes | Tagged , , , , , , , , , , , , , | Leave a comment