Fixing Ubuntu soft lockup – CPU error

We have some strange issue with an Ubuntu 12.04 System. It takes a lot of time for booting to the login screen. It was approximately 4 minutes, which is definitely not normal. When we read log files, we found real problem:

kernel: BUG: soft lockup - CPU#1 stuck for 23s

We have made hardware tests: CPU stability, Memory test, Hard disk check. There is no hardware problem on it, so it was software issue. Most recent solutions that we found was to make system boot with no ACPI. Way to do it is to edit /etc/default/grub and change GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash” with: GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash acpi=off”

After this change you have to make update-grub and grub-install commands to take effect. When we reboot the system it takes no more than minute to load from grub to login screen. But when we checked the system, we found that it isn’t a good solution, because when you shut down it freezes (without powering off the PC). We tried with other options in grub, but nothing helps for real, so we made it as it was : GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash”

Real solutions for us came when we installed this package “amd64-microcode”(Processor microcode firmware for AMD CPUs). Our system runs with AMD CPU, so if it happens on Intel system, the proper package is: intel-microcode.

sudo apt-get install amd64-microcode # For AMD CPUs

or

sudo apt-get install intel-microcode # For Intel CPUs

In that way system runs as it should be – fast loading at boot and normal shutdown, without problems for all used services and applications.
Hope that helps 🙂

Apache secure – part 1

Howdy m8s

In the following article we’ll make our Apache web server a little bit more secured by adding few simple directives to it, in some well know files.

Let’s assume,that we haven’t install Apache server. Now we will install it together step by step:

apt-get install apache2

after procedure is complete successfully, check if the web server is working by opening the “localhost” or “ip-address” of machine that we use for installation. In my case it’s a Virtualbox’s VM with internal IP address 125.15.253.12
so i type in my browser:

http://125.15.253.12/

and i see this:

It works!

This is the default web page for this server.

The web server is running, but no content has been added yet. So…what if I request from my apache server to show me file that does not exist ?
Type in browser:

http://125.15.253.12/no_file

and see:

Not Found

The requested URL /no_file was not found on this server.
Apache/2.2.16 (Debian) Server at 125.15.253.12 Port 80

“nothing unusual” you will say, but please look closer to this lines.
From this innocent act we found two major holes in Apache default configuration:

1. That server use Apache/2.2.16
2. Operation system is Debian
3. Port number, witch in this case is by default , but if web-server is behind NAT you should understand that too if there is port-forwarding 🙂

Why we should “hello world” that info? Really …why? In fact, no one care about our server configuration except those guys that came to hack.

Let’s hide that info !
Find Apache main configuration file and open it with my favorite editor 🙂

nano /etc/apache2/apache2.conf

locate the bottom of file and add these lines

ServerSignature Off
ServerTokens NothingToSee
TraceEnable off

“ServerSignature Off” tells my Apache not to show footer line under server-generated documents
“ServerTokens NothingToSee” make Apache to suppressing OS, major and minor version info.

“TraceEnable off” Normally you will have this enabled by default, if you want to check on your server just telnet on the port your web server is running and request for ”TRACE / HTTP/1.0” if you get a positive reply it means TRACE is enabled on your system. The output look like this:

pavlin@pavlin:/home/pavlin# telnet 125.15.253.12 80
Trying 125.15.253.12...
Connected to 125.15.253.12.
Escape character is '^]'.
TRACE / HTTP/1.0
Host: myhostname.com
--------- hit enter twice ---------
HTTP/1.1 200 OK --------- TRACE IS ENABLED !!! THAT IS BAAAAAD
Date: Tue, 12 Feb 2013 22:09:31 GMT
Server: Apache/2.2.16 (Debian)
Connection: close
Content-Type: message/http
TRACE / HTTP/1.0
Host: myhostname.com
Connection closed by foreign host.

when you disable trace, the same output will look a like that:

pavlin@pavlin:/home/pavlin# telnet 125.15.253.12 80
Trying 125.15.253.12...
Connected to 125.15.253.12.
Escape character is '^]'.
TRACE / HTTP/1.0
Host: myhostname.com
--------- hit enter twice ---------
HTTP/1.1 405 Method Not Allowed
Date: Tue, 12 Feb 2013 22:14:54 GMT
Server: Apache/2.2.16 (Debian)
Allow: 
Vary: Accept-Encoding
Content-Length: 303
Connection: close
Content-Type: text/html; charset=iso-8859-1
....

Let’s test the other two directives that we’ve changed in Apache config file.

Open this URL in your browser:

http://125.15.253.12/no_file

Now the output looks a little bit less, then it was before 🙂

Not Found

The requested URL /no_file was not found on this server.

Did you note the difference ? No version , no OS info.

Let’s test for what ‘-Indexes’ options is used.

Now lets create one directory in ours Apache’s “www” folder and few empty files:

mkdir /var/www/test_dir
touch /var/www/test_dir/blah.txt
touch /var/www/test_dir/blah02.txt
touch /var/www/test_dir/blah03.txt

and open it in browser

http://125.15.253.12/test_dir/

You will see all content of our directory and that is not good at all!

Now open your default configuration file :

nano /etc/apache2/sites-enabled/000-default

Change settings to look like this:

Options -Indexes FollowSymLinks MultiViews -Includes
AllowOverride all

save and reload the Apache

/etc/init.d/apache2 reload

and open it in browser

http://125.15.253.12/test_dir/

you should receive

Forbidden

You don't have permission to access /test_dir/ on this server.

O.K -Indexes and -Includes work

Now it’s time to add some simple re-write rules in our default host file
First enable mod_rewrite by type:

a2enmod rewrite

and then let’s add the rewrite rules BEFORE close ” directive

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.html [L]

save and reload the Apache server again.

Lets finally try Re-write rules

Those rules tell my apache to “forward” every document that is requested and can’t be found to index.html file. You can test it when you type anything you want after the backslash in address bar like this:

http://125.15.253.12/test_my_rewrite_rulez_and_i_hope_it_wor

That’s it, I hope that you got the idea 🙂

One more thing ! Here is one cool tool with you can do test to your apache server

Nikto2

How to delete metadata left from LSI 3ware controller

We have many servers with hardware controllers like LSI 3Ware. From time to time disks degrade and we have to change it with another one. However we found a little problem when trying to use unwiped disk in LSI 3ware controller, which was previously used in another 3ware controller. The following error appears in the disk status – Unsupported_DCB.

LSI 3Ware stores the RAID information on the disks. The main advantage is that we can replace controller without having to reconfigure it. That information is stored on a section called Disk Control Block (DCB). There are two versions of DCB – on both of them the information is stored at the end of the disk. DCB appears to be 1024 LBAs long (1024 * 512 bytes per sector = 512KiB). Additionally in one of the versions there is a copy of the DCB as the first 1024 LBAs of the disk.

So we have to wipe that info before replacing disk from one controller to another. Here’s a simple script that do it for you:

#!/bin/bash
DISK=sdz
LBAS=$(cat /sys/block/$DISK/size)
dd if=/dev/zero of=/dev/$DISK bs=512 count=1024
dd if=/dev/zero of=/dev/$DISK bs=512 seek=$(($LBAS-1024)) count=1024

Very Important!!! Please replace sdz with the device name. I advice you to triple check the disk before executing the script. I am not responsible if you delete another disk by mistake.