Solr issues with Drupal

Posted on Tue 24 February 2015 in SysAdmin • Tagged with drupal, search, smartos, solr, tomcat, truss

Broken search

One of our clients have been having issue with the search function on their Drupal site. This full text search was powered by Solr behind the scenes. The issue only only ever occurred when the user was logged in so it was never flagged up to us.

Recently we've tried to migrate them to a new server and lots of new changes had to be made; the issue started appearing for all users, even those that aren't logged in so I decided to take a look.

Symptoms and debugging

The issue only manifested itself with a simple error message:

The Apache Solr search engine is not available. Please contact your site administrator.

This wasn't really helpful and the Tomcat log files didn't show any issues. The Solr logs didn't show the query so I assumed it didn't even hit the Solr app; manual queries were working fine via the Solr web interface and appeared in the logs. The next step was to extract the Solr query that Drupal was running.

The easy way - Drupal

If you're running Drupal or any codebase where developers are on hand to debug the app with you, you can dig into the code and make the app log or print out the query to the browser.

I came across this very helpful link on how to debug Solr queries in Drupal. In my case, possibly a different version of Drupal, the file was different:

# grep -irH drupal_http_request . | grep solr
sites/all/modules/contrib/apachesolr/Drupal_Apache_Solr_Service.php: $result = drupal_http_request($url, $headers, $method, $content);

I added "drupal_set_message(check_plain(\$url));" and I saw the query printed in my browser. And it was a huge query...

The hard way - anything else

A lot of the time we would host custom CMS or code that doesn't isn't maintained by the same developers anymore; in those cases it's much faster to do the debugging ourselves without digging into the code. If you're running SmartOS and have access to truss, this is how to go about it (strace can probably do something similar in Linux).
This is actually how I went about it before I realised Drupal probably has an easier way for me to debug the issue.

# truss -o /tmp/truss -wall -rall -vall -p $(pgrep httpd)

It will help to have a clone of your server or a dev environment so that the truss output is less messy to read.
Run this and do a search on your website. Wait until it finishes and Ctrl+C truss.

Search for the query:

# grep -n "G E T / s o l r / s e l e c t " /tmp/truss
248606:24614: G E T / s o l r / s e l e c t ? f l = i d % 2 C n i d % 2 C t
870437:24611: G E T / s o l r / s e l e c t ? f l = i d % 2 C n i d % 2 C t

You can also search for 8080 or whichever port Tomcat is running on and the GET line should be not far below:

# grep -n AF_INET /tmp/truss.1 | grep 8080
248599:24614: AF_INET name = port = 8080
870430:24611: AF_INET name = port = 8080

Go to line 248606 in your truss output and your query should be there, probably spanning a few send() blocks, it should end with a "\r\n\r\n" string.
After that you should see a recv() block with a response from Tomcat/Solr. For me, it just showed a "400 Bad Request" error. When I ran the extracted query with wget, I saw this:

Resolving localhost (localhost)...
Connecting to localhost (localhost)||:8080... connected.
HTTP request sent, awaiting response... 400 Bad Request
2015-02-24 15:26:15 ERROR 400: Bad Request.

Once you have the truss output cut down to just the query, you can remove all the whitespaces, PIDs and system calls; all that will remain is the actual query that Drupal is running. Mine was quite big:

# wc /tmp/truss 
1 9 39733 /tmp/truss

That's 39,733 characters!

Solution was rather straightforward

So I discovered that the query that Drupal send to Tomcat was humongous. After a lot of pointless debugging,enabling log files, and looking in the wrong place, it struck me that it was probably hitting a character limit. After all the query was not hitting Solr so it must be Tomcat refusing to process it.

A quick search turned up that the default URL limit on Tomcat was 4096... The setting you need to change is maxHttpHeaderSize in Tomcat's server.xml. I set it to 65536 to accommodate some of big queries that were running and sure enough everything worked fine after a restart!

Retrospectively, I googled the issue again and came across this link which might be helpful it the above doesn't resolve your issue.

SMTP authentication in Exim

Posted on Wed 28 January 2015 in Linux, SysAdmin • Tagged with auth, authentication, exim, exim4, gmail, login, plain, sha1, smtp, smtpauth

No more Google SMTP for your aliases

Since August last year, Google have grandfathered a feature in GMail that allows you to create aliases in your account and send emails as those aliases from their SMTP servers:

For any new aliases, you will need to provide your own SMTP servers.
If you have a VPS of some sort, this is how you can set up simple SMTP authentication in Exim to pipe your emails through your server and keep using the Google web interface.

SMTP Authentication

LOGIN and PLAIN auth differs in the fact that LOGIN will prompt for the username/password explicitely whereas in PLAIN the client is expected to send both in one blob.
Practically the only difference is an additional null byte at the beginning of the LOGIN blob send to the server.


How to set up simple SMTP LOGIN auth with SHA1 (alternative is crypt or MD5..):

exim.conf file (based on a  Ubuntu template):

CONFDIR = /etc/exim4/
begin authenticators

driver = plaintext
public_name = LOGIN
server_prompts = <| Username: | Password:
server_advertise_condition = ${if def:tls_cipher }
server_condition = "${if crypteq {$auth2}{\\\{sha1\\\}${extract{1}{:}{${lookup{$auth1}lsearch{CONFDIR/passwd}{$value}{*:*}}}}}{1}{0}}"
server_set_id = $auth1

passwd file:

username:[PASSWORD HASH]

How to generate password hash:

perl -MDigest::SHA=sha1_hex -e 'print sha1_hex($ARGV[0]),"\n"' [PASSWORD]

To test this, you need to encode the username/pass in base64:

# cat
use MIME::Base64;
printf ("%s", encode_base64(eval "\"$ARGV[0]\""));
# perl 'username\0password'
# exim -bh localhost
> ehlo test
> auth login dXNlcm5hbWUAcGFzc3dvcmQ=

If you'd rather have a PLAIN auth, just change the snippet in exim.conf to:

exim.conf file

CONFDIR = /etc/exim4/
begin authenticators

driver = plaintext
public_name = PLAIN
server_advertise_condition = ${if def:tls_cipher }
server_condition = "${if crypteq {$auth3}{\\\{sha1\\\}${extract{1}{:}{${lookup{$auth2}lsearch{CONFDIR/passwd}{$value}{*:*}}}}}{1}{0}}"
server_set_id = $auth2

to test, run:

# perl '\0username\0password'
# exim -bh localhost
> ehlo test
> auth plain AHVzZXJuYW1lAHBhc3N3b3Jk

Test with SSL by using this instead of 'exim -bh':

openssl s_client -connect


Updating and resetting Raritan Dominion PX PDUs

Posted on Tue 29 May 2012 in SysAdmin • Tagged with pdu, raritan, reset

I just had the pleasure of resetting some PDUs to factory defaults today. The models in question were Raritan Dominion PX20 (DPXR20A-16/32). After a mostly fruitless search for 'raritan px reset to factory default' and 'raritan px restore to factory default', I have managed to dig out a way to complete my task.

Nothing like homemade commando plugs with kettle leads, courtesy of my colleague JT.

Commando leads

First I needed to upgrade the firmware to the latest version as the command didn't seem to run on v1.3 for me. Grabbed my binaries from the Raritan website. I had to flash v1.5.2 before 1.5.5, YMMV. You can upgrade firmware using the web interface:

  • Hook up to the console of the PDU with a DB9 to USB converter (top) and a special Raritan DB9 to Ethernet console cable (bottom), like this:
  • Console cables
  • One of these, a Cisco compatible cable, won't work:
  • Wrong console cables
  • As I use Arch Linux I just ran the following as root:
screen /dev/ttyUSB0
  • Set the IP configuration to dhcp: 'config' at the prompt and then 'dhcp'
[Old IP address] command: config
IP autoconfiguration (none/dhcp/bootp) [none]: dhcp

Press [Enter] a few times until the old IP address becomes the new one, you might have to unplug the network cable and plug it back in. Proceed with the firmware upgrade in the web browser!

Once the upgrade is completed, go into 'clp' mode in the console and type:

clp:/-> set /system1 FactoryDefaults=true

Sorted! Now your PDU is ready to go on eBay :)

For more detailed information, check out the Raritan Dominion PX Online Help site.

PHP errors not displaying in IIS

Posted on Fri 18 May 2012 in SysAdmin • Tagged with errors, iis, php

A customer called in today saying that their site is not working after re-pointing the DNS. No changes have been made in a week and it was working fine before. Typical.

So I turned on Firebug and notice it's a 500 error. Logged into the server and saw it's a WordPress instance running with PHP on IIS. Great! For some reason the pages in the browser are completely blank. I loaded up the page from a local browser and the same thing happens. I double checked that detailed errors messages have been enabled in IIS and they were. Very strange.

The IIS log was as unhelpful as ever so I turned to DuckDuckGo (nearly wrote 'Google' there but I switched my search engine recently). Now I'm not usually used to WordPress in IIS so didn't think of checking the php.ini file; later on it turned out that error messages option was not set  but it's supposedly on by default but errors just decided not to show.

Cutting to the chase, I dropped the following into the PHP file:


This uncovered  a plugin was installed twice and WordPress was having none of that. Renamed the file and all was well again. Thanks to Sadi from for pointing that out!

Modding the Filco Majestouch Tenkeyless

Posted on Wed 16 May 2012 in Uncategorized • Tagged with cherrymx, filco, keyboard, majestouch, modding, tenkeyless

I've recently made an investment and purchased a Filco Majestouch 2 Tenkeyless, blue switch edition. It was a toss-up between that and the HappyHacker and I decided I like the arrow keys too much to let go. KeyboardCo have exclusive distribution rights here in the UK so I got mine from there. Next day delivery, top notch service.

Filco Majestouch 2 Tenkeyless

Just above it is my Logitech Wave; quite a nifty design but the quality isn't amazing and the keys started to stick after about 6 months. Not bad for £20 though.

The Filco is quite a pricey beast at £114 but I figured as I'm typing away every day, it will definitely pay itself off.

Filco Majestouch 2

Shortly after I showed it off to everyone (who cared and had any interest in keyboards), my friend decided to get a full sized one from WASD keyboards. For those who don't know and haven't clicked the link yet, WASD sell mechanical keyboards, but what makes them different from the competition though is that they let you fully customise your keycaps!

As WASD are based on the other side of the pond, I decided to piggy back on his order and get some custom keys and some dampeners (at the bottom of the link there is a noise comparison between with and without o-rings).

Filco and

Now that little baby on the bottom left is a keycap puller that came free with the WASD keyboard but you can get it separately too. Very nice of them to include that and it will definitely come in handy if one day you decide to customise your keycaps or maybe just clean your keyboard.

Starting with the Esc key and the qwerty row.

Qwerty and Esc

Nice and easy. Now I heard that the bigger keys are quite different and it's quite easy to break them if you pull them out like a madman. I decided to consult the infinite wisdom of the Internet, most precisely Youtube, for some advice. Stumbled upon this video which was most helpful! You might want to mute the music though. And so off the big keys went.

Bigger keys are

There are only 4 long keys on this: the 3 visible ones and the backspace. After about 1.5 hours, all the keys were dampened. And here it is in its full glory:

Modded Filco MJ

The picture doesn't show it very well but my 'Windows' keys, or 'Meta' keys as I prefer to call them now, have a Tux on them.

I must say, the dampened keys are gonna take a while to get used to but they are much quieter than before. Needless to say the feel is amazing and it's definitely one of the best keyboards I've typed on so far and I've owned quite a few (Logitech Wave/G15, countless scissor and laptop keyboards as well as stock DELL keyboards). If your job involves a lot of tapping away, this will definitely improve your typing comforts.

For more reference, I urge you to read:

And if you just want a chat, sign up to the GeekHack forums.