Installing & Configuring Nutch, Nutch-Gui, Sun JDK & Tomcat 6 On Centos

This howto will explain how to get Nutch, Nutch-Gui, Sun JDK & Tomcat 6.0.16 working on Centos 5.x or 6.x while maintaining a normally functioning Centos system. Currently, Centos 5.x ships with Tomcat 5.5, however, while it does run, there are problems with the default install of this version that results in errors which are undocumented and persistent at this time. If you have information or believe that these errors have been addressed and can point to a fix, please use the contact form on this website to let us know. The following instructions allow for easy removal of any software installed through following this howto by either using “rpm -e foo.rpm” or “rm -rf /opt/foo” returning your system to its original state.

Applicable to Centos Versions:

Centos 5.x
Centos 6.x

Requirements

Explanation of requirements.

Root or sudo access with appropriate privileges to the system you intend to install on.
A server preferably on a high-speed network.
Sun JDK rpm.bin.
Tomcat 6 rpm.
Nutch 1.0 tar.gz.
Nutch-Gui 0.2 tar.gz.

Doing the Work

Basic description of what will be done and what is expected.

Install a few dependencies:

sudo yum install ant xml-commons-apis ant-trax

1	sudo yum install ant xml-commons-apis ant-trax

Download & install the latest Sun JDK rpm.bin:

<strong>Go here:</strong> <a class="external-link" href="http://java.sun.com/javase/downloads/index.jsp">http://java.sun.com/javase/downloads/index.jsp</a><strong>

Get the following:
Java SE Development Kit (JDK) 32bit (approx. 73.98MB)
JDK 6 Update 16 (or the latest update, the version is important in setting your JAVA_HOME path variable)</strong>

<strong>Once downloaded install using the following:</strong>
chmod +x jdk-6u16-linux-i586-rpm.bin; ./jdk-6u16-linux-i586-rpm.bin
<strong>answer "yes" to the EULA</strong>
sudo rpm -ivh jdk* sun*

Go here: <a class="external-link" href="http://java.sun.com/javase/downloads/index.jsp">http://java.sun.com/javase/downloads/index.jsp</a>

Get the following:

Java SE Development Kit (JDK) 32bit (approx. 73.98MB)

JDK 6 Update 16 (or the latest update, the version is important in setting your JAVA_HOME path variable)

Once downloaded install using the following:

chmod +x jdk-6u16-linux-i586-rpm.bin; ./jdk-6u16-linux-i586-rpm.bin

answer "yes" to the EULA

sudo rpm -ivh jdk* sun*

Download & install Tomcat 6:

<a class="external-link" href="http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.noarch.rpm">http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.noarch.rpm</a>
<a class="external-link" href="http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.src.rpm">http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.src.rpm</a> (<strong>provided for reference</strong>)

<strong>Once downloaded install the rpm with the following command:</strong>
sudo rpm -ivh tomcat-6.0.16-0.noarch.rpm (<strong>this installs entirely into /opt/tomcat and can be removed with: rpm -e tomcat</strong>)
sudo vi /opt/tomcat/conf/tomcat-env.sh (<strong>set: JAVA_HOME="/usr/java/jre1.6.0_16"</strong>)

<a class="external-link" href="http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.noarch.rpm">http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.noarch.rpm</a>

<a class="external-link" href="http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.src.rpm">http://www.webdroid.org:8080/archives/tomcat-package/tomcat-6.0.16-0.src.rpm</a> (provided for reference)

Once downloaded install the rpm with the following command:

sudo rpm -ivh tomcat-6.0.16-0.noarch.rpm (this installs entirely into /opt/tomcat and can be removed with: rpm -e tomcat)

sudo vi /opt/tomcat/conf/tomcat-env.sh (set: JAVA_HOME="/usr/java/jre1.6.0_16")

Download & install Nutch 1.0:

<strong>Dowmload Nutch 1.0 from a mirror here:</strong> <a class="external-link" href="http://www.apache.org/dyn/closer.cgi/lucene/nutch/">http://www.apache.org/dyn/closer.cgi/lucene/nutch/</a>

sudo cp nutch-1.0.tar.gz /opt; cd /opt && tar xvfz nutch-1.0.tar.gz; cd nutch-1.0
sudo ant
sudo ant war (<strong>this creates the "build" directory</strong>)

sudo ln -s /opt/nutch-1.0/build/nutch.xml /opt/tomcat/conf/Catalina/localhost/nutch.xml
(<strong>modify the property "searcher.dir" to: /opt/nutch-1.0/crawl/ & the docBase attribute to the full path of your nutch-1.0 war file: docBase="nutch.war" path="/opt/tomcat/webapps/"</strong>)

sudo ant
sudo ant war (<strong>this compiles with the new build/nutch.xml file</strong>)

sudo cp build/nutch-1.0.war /opt/tomcat/webapps/nutch.war
(<strong>a .war file is a zip/jar file known as a "web archive" or war file, it is uncompressed when tomcat is started</strong>)

Dowmload Nutch 1.0 from a mirror here: <a class="external-link" href="http://www.apache.org/dyn/closer.cgi/lucene/nutch/">http://www.apache.org/dyn/closer.cgi/lucene/nutch/</a>

sudo cp nutch-1.0.tar.gz /opt; cd /opt && tar xvfz nutch-1.0.tar.gz; cd nutch-1.0

sudo ant

sudo ant war (this creates the "build" directory)

sudo ln -s /opt/nutch-1.0/build/nutch.xml /opt/tomcat/conf/Catalina/localhost/nutch.xml

(modify the property "searcher.dir" to: /opt/nutch-1.0/crawl/ & the docBase attribute to the full path of your nutch-1.0 war file: docBase="nutch.war" path="/opt/tomcat/webapps/")

sudo ant

sudo ant war (this compiles with the new build/nutch.xml file)

sudo cp build/nutch-1.0.war /opt/tomcat/webapps/nutch.war

(a .war file is a zip/jar file known as a "web archive" or war file, it is uncompressed when tomcat is started)

Edit /etc/profile:

<strong>Add these lines just above: # ksh workaround</strong>

sudo vi /etc/profile

##Tomcat 6 / Java##
JAVA_HOME="/usr/java/jdk1.6.0_16"
export JAVA_HOME
CATALINA_HOME="/opt/tomcat"
export CATALINA_HOME
NUTCH_JAVA_HOME="/usr/java/jdk1.6.0_16"
export NUTCH_JAVA_HOME
##End Tomcat 6 / Java##

Add these lines just above: # ksh workaround

sudo vi /etc/profile

##Tomcat 6 / Java##

JAVA_HOME="/usr/java/jdk1.6.0_16"

export JAVA_HOME

CATALINA_HOME="/opt/tomcat"

export CATALINA_HOME

NUTCH_JAVA_HOME="/usr/java/jdk1.6.0_16"

export NUTCH_JAVA_HOME

##End Tomcat 6 / Java##

Configure Nutch to fetch URLs:

cd /opt/nutch-1.0; sudo mkdir urls
(<strong>Make a flat text file in here called "seed" and create a list of urls to be crawled, with each url on a new separate line: http://www.example.com</strong>)

sudo vi conf/nutch-default.xml
<strong>Edit the following:</strong>
http.agent.name &lt;value&gt;My Spider&lt;/value&gt;
http.robots.agents &lt;value&gt;My Spider&lt;/value&gt;
http.agent.description &lt;value&gt;My Bot&lt;/value&gt;
http.agent.url &lt;value&gt;http://www.example.com&lt;/value&gt;
http.agent.email &lt;value&gt;admin@example.com&lt;/value&gt;
<strong>all other values remain as default, do not attempt to alter them unless you have a backup and/or you know what you're doing.</strong>

cd /opt/nutch-1.0; sudo mkdir urls

(Make a flat text file in here called "seed" and create a list of urls to be crawled, with each url on a new separate line: http://www.example.com)

sudo vi conf/nutch-default.xml

Edit the following:

http.agent.name <value>My Spider</value>

http.robots.agents <value>My Spider</value>

http.agent.description <value>My Bot</value>

http.agent.url <value>http://www.example.com</value>

http.agent.email <value>admin@example.com</value>

all other values remain as default, do not attempt to alter them unless you have a backup and/or you know what you're doing.

Nutch “deepcrawler” script:

Put <a title="deepcrawler" class="internal-link" href="/scripts/deepcrawler">this script</a> in /opt/nutch-1.0/bin<br />chmod +x deepcrawler
<strong>Note: This script assumes the urls you plan to inject are stored in /opt/nutch-1.0/urls/seed and will create a new dir in: /opt/nutch-1.0/crawl1 to store the new crawl.</strong>

Put <a title="deepcrawler" class="internal-link" href="/scripts/deepcrawler">this script</a> in /opt/nutch-1.0/bin chmod +x deepcrawler

Note: This script assumes the urls you plan to inject are stored in /opt/nutch-1.0/urls/seed and will create a new dir in: /opt/nutch-1.0/crawl1 to store the new crawl.

Fetch URLs with Nutch via command line:

<strong>If you do not alter the deepcrawler script it will most likely run for many days or weeks depending on the amount of urls you inject, so you'll want to run it in screen so you can detach and reattach to check progress.

</strong>screen -S nutch
sudo service tomcat start
cd /opt/nutch-1.0; su -c "bin/deepcrawler"

If you do not alter the deepcrawler script it will most likely run for many days or weeks depending on the amount of urls you inject, so you'll want to run it in screen so you can detach and reattach to check progress.

screen -S nutch

sudo service tomcat start

cd /opt/nutch-1.0; su -c "bin/deepcrawler"

Download & install Nutch-Gui 0.2:

<strong>Note: if you use the script provided above, you can skip the GUI altogether.

Download Nutch-Gui 0.2 from:</strong> <a class="external-link" href="http://github.com/101tec/nutch/downloads">http://github.com/101tec/nutch/downloads</a>

sudo cp nutch-gui-0.2.tar.gz /opt; cd /opt && tar xvfz nutch-gui-0.2.tar.gz; cd nutch-gui-0.2

sudo ant clean package
cd build/nutch-gui-0.2
sudo cp nutch-gui-0.2.war /opt/tomcat/webapps/nutch-gui.war

<strong>unsecured quick test method, to assure it's working:</strong>
su -c "bin/nutch admin /opt/nutch-1.0 50060"
http://example.com:50060/general

<strong>more secure password protection:</strong>
sudo vi conf/nutchguiUsers.properties
(<strong>edit the following information: user=password, admin, where user is the usename, password is the password you want, and admin is the role</strong>)

screen -S nutch-gui (<strong>since we'll probably run it for a while</strong>)
su -c "bin/nutch admin /opt/nutch-1.0 50060 —secure"
http://example.com:50060/general

Note: if you use the script provided above, you can skip the GUI altogether.

Download Nutch-Gui 0.2 from: <a class="external-link" href="http://github.com/101tec/nutch/downloads">http://github.com/101tec/nutch/downloads</a>

sudo cp nutch-gui-0.2.tar.gz /opt; cd /opt && tar xvfz nutch-gui-0.2.tar.gz; cd nutch-gui-0.2

sudo ant clean package

cd build/nutch-gui-0.2

sudo cp nutch-gui-0.2.war /opt/tomcat/webapps/nutch-gui.war

unsecured quick test method, to assure it's working:

su -c "bin/nutch admin /opt/nutch-1.0 50060"

http://example.com:50060/general

more secure password protection:

sudo vi conf/nutchguiUsers.properties

(edit the following information: user=password, admin, where user is the usename, password is the password you want, and admin is the role)

screen -S nutch-gui (since we'll probably run it for a while)

su -c "bin/nutch admin /opt/nutch-1.0 50060 —secure"

http://example.com:50060/general

Troubleshooting / How To Test

Explanation troubleshooting basics and expectations.

Make sure the required packages are installed and JAVA_HOME path variable is set in /etc/profile:

rpm -q tomcat jdk ant xml-commons-apis ant-trax; echo $JAVA_HOME
tomcat-6.0.16-0
jdk-1.6.0_16-fcs
ant-1.6.5-2jpp.2
xml-commons-apis-1.3.02-0.b2.7jpp.10
ant-trax-1.6.5-2jpp.2
/usr/java/jdk1.6.0_16

Replace "localhost" with your machines IP

<strong>Try accessing Tomcat here: http://localhost:8080/
Try accessing Nutch here: http://localhost:8080/nutch/
Try accessing Nutch-Gui here: http://localhost:50060/general</strong>

rpm -q tomcat jdk ant xml-commons-apis ant-trax; echo $JAVA_HOME

tomcat-6.0.16-0

jdk-1.6.0_16-fcs

ant-1.6.5-2jpp.2

xml-commons-apis-1.3.02-0.b2.7jpp.10

ant-trax-1.6.5-2jpp.2

/usr/java/jdk1.6.0_16

Replace "localhost" with your machines IP

Try accessing Tomcat here: http://localhost:8080/

Try accessing Nutch here: http://localhost:8080/nutch/

Try accessing Nutch-Gui here: http://localhost:50060/general

Set Tomcat to start on boot:

sudo chkconfig --level 2345 tomcat on; chkconfig --list | grep tomcat
tomcat          0:off   1:off   2:on    3:on    4:on    5:on    6:off

1 2	sudo chkconfig --level 2345 tomcat on; chkconfig --list \| grep tomcat tomcat 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Common problems and fixes

Describe common problems here, include links to known common problems if on another site

More Information

Any additional information or notes.

Disclaimer

We test this stuff on our own machines, really we do. But you may run into problems, if you do, come to #centoshelp on irc.freenode.net

Added Reading

http://www.redhat.com

Last Modified: 22 Apr, 2020 at 16:44:56