Search Engines for Fun and Profit Part Four: Separating Resources for Queries and Indexing

NOTE Before you go and do any of this please read Interlude: The Bungle.

If you're indexing a wide array of URLs you will quickly find indexing operations are load-intensive and this can affect the speed of delivery in your front-end. To overcome this problem we can put the index/ices on an NFS share and run two instances of OSS, one dedicated to crawling and one that will only serve queries. Virtual machines lend themselves especially well to this setup since you will be making an exact copy of the original server, plus some minor configuration changes.

You can either set up the original OSS server as the NFS server or use a third server. I prefer using a dedicated file server however your resources may not permit. If you are using a dedicated file server first shut down OSS then move the contents of the /data/ directory down a level. Mount the NFS share onto the /data/ directory, move the indices onto the share then set the ownership for the mount point:

# chown oss: data/ -R

If you're going to use the indexing server as the NFS server your /etc/exports should look something like this:

/opt/open-search-server/data [QUERY_SERVER_ADDRESS](sync,no_subtree_check,ro,root_squash)

Note that if you are hosting the indices on a dedicated NFS server you should be using 'rw' in your exports file and the fstab of the indexing server instead of 'ro'. Add this to the /etc/fstab of the query server:        /opt/open-search-server/data    nfs     defaults,ro,noac             0 0

It is important that you mount the share with the noac option or you may end up with ?java.io.IOException: Stale NFS file handle errors resulting from the file attribute cache lagging behind changes made by the indexing server. Now restart your nfs and netmount init scripts (where available) or mount the share manually:

# mount /opt/open-search-server/data

It's now safe to start OSS on the query server.

# cd /opt/open-search-server/
# sudo -u oss ./start.sh


There are no comments for this item.