Useful Maven Commands

To run a unit test using maven from commandline:
mvn -Dtest=ActualTestName test

Eclipse gets stuck at ‘Refreshing Maven Model’ endlessly. How to resolve?

Problem Statement
On my Eclipse / STS IDE, when i open a (maven) project, it often gets stuck trying to resolve dependencies with ‘refreshing maven model’ message endlessly.

Solution
Not sure why this happens. But the following resolves the problem -
1. Close your Eclipse / STS IDE.
2. Go to your workspace project directory on command line (where the pom.xml exists)
3. From command-line run: mvn dependency:resolve
This will resolve all dependencies outside of Eclipse / STS IDE.
For all projects that have large number of dependencies, this may be the thing to do!

Thats It. Now when you open your IDE, there are no dependencies to resolve, and you will not see the refreshing maven model problem.

How to install a jar manually in local maven repository?

Problem Statement: Your pom.xml dependencies is not able to download a certain jar from the repository.

Solution:
You can manually download the jar and add it to your local maven repository.

Assuming this dependency (mentioned in pom.xml) fails (because the jar fails to download):

<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>3.3.2.GA</version>
</dependency>

Then download hibernate-entitymanager-3.3.2.GA manually, and run the command:
mvn install:install-file -DgroupId=org.hibernate -DartifactId=hibernate-entitymanager -Dversion=3.3.2.GA -Dpackaging=jar -Dfile=./hibernate-entitymanager-3.3.2.GA.jar

Thats it, you are all set.

How to use geo spatial with MongoDB?

Problem Statement
Find all locations that are within a certain radius of a particular location.

Solution Approach
We will demonstrate this with pseudo-code.

Let us insert lots of location specific data in MongoDB.

We will search this very data later.

static Mongo m;
static DB db;
static DBCollection coll;

m = new Mongo("localhost");
db = m.getDB("mydb");
coll = db.getCollection("people");
coll.ensureIndex((DBObject) JSON.parse("{ 'loc' : '2d' }"));

Random rand = new Random();
long startTime = Calendar.getInstance().getTimeInMillis();

for (int i = 0; i < 100000; i++)
{
    double lat = rand.nextDouble() * 180 - 90;
    BigDecimal bd1 = new BigDecimal(lat).setScale(2,RoundingMode.HALF_EVEN);
    lat = bd1.doubleValue();

    if (lat = 90)
         lat = lat - 0.5;
         double lon = rand.nextDouble() * 360 - 180;
         BigDecimal bd = new BigDecimal(lon).setScale(2, RoundingMode.HALF_EVEN);
         lon = bd.doubleValue();

    if (lon = 90)
         lon = lon - 0.5;

    // System.out.println("lon / lat = " + lon + " / " + lat);
    // lat=1; lon=1;
    coll.insert((DBObject) JSON.parse("{'loc' : [ " + lon + " , " + lat	+ " ]}"), WriteConcern.SAFE);
}

long timeTaken = Calendar.getInstance().getTimeInMillis() - startTime;
System.out.println("Total Time Taken to Insert (milliseconds): " + timeTaken);

Now we want to do a geo spatial search on the above inserted data.
i.e find all documents(rows) whose location is within a certain miles(radius) from a particular location(lat,lon):

There are several ways to do this search.
Lets look at them one by one.

//Using $nearSphere to find all locations that are within 1000 miles from location 78,10

static void searchWithNearSphere()
{
    // 1 radian = 3959 miles (6371 km).
    // So, 100 miles = 100/3959 = 0.0253 radians.
    // So, 1000 miles = 1000/3959 = 0.253 radians.
    System.out.println("searchWithNearSphere: Query to run: "
    + "{'$nearSphere' : [ 78 , 10 ] , '$maxDistance' : 0.253}");

    long startTime = Calendar.getInstance().getTimeInMillis();

    // Must use long/lat format, and not lat/long.
    // Valid bounds for latitude is -90/90.
    // Valid bounds for longitude is -180/180
    DBCursor cur = coll.find(new BasicDBObject("loc", JSON.parse("{'$nearSphere' : [ 78 , 10 ] , '$maxDistance' : 0.253}"))).limit(100000);
    long timeTaken = Calendar.getInstance().getTimeInMillis() - startTime;
    System.out.println("Total Time Taken for 'searchWithNearSphere(milliseconds): " + timeTaken);

    System.out.println("Total Documents Found: " + cur.count());
    System.out.println("Total Documents Found: " + cur.length());
    System.out.println("Total Documents Found: " + cur.size());

    /*
    * while (cur.hasNext()) { System.out.println(cur.next()); }
    *
    * System.out.println("Total Documents Found: " + cur.size());
    */
}

// Using geoNear command to find all locations that are within 1000 miles from location 78,10
static void searchWithGeoNear()
{
    BasicDBObject cmd = new BasicDBObject();
    cmd.put("geoNear", "people");
    int coord[] = { 78, 10 };
    cmd.put("near", coord);
    cmd.put("num", 1000);
    cmd.put("maxDistance", 0.253);
    cmd.put("spherical", true);

    long startTime = Calendar.getInstance().getTimeInMillis();
    CommandResult r = db.command(cmd);
    long timeTaken = Calendar.getInstance().getTimeInMillis() - startTime;
    System.out.println("Total Time Taken for 'searchWithGeoNear(milliseconds): " + timeTaken);
    System.out.println(r.ok());
    System.out.println(r.getErrorMessage());
    // System.out.println(r.toString());

    // print(r);//Print all the returned data.

    com.mongodb.BasicDBList resultsBasicDBList = (BasicDBList) r.get("results");
    System.out.println("Total Results: " + resultsBasicDBList.size());

    // System.out.println("Results: " + resultsBasicDBList); //This will
    // print the value of 'results' key.

    // print(resultsBasicDBList);//This will print individual values.
}

//The print function to print a DBObject:
public static void print(DBObject doc)
{
    Set allKeys = doc.keySet();
    Iterator it = allKeys.iterator();
    while (it.hasNext())
    {
        Object o = it.next();
        String temp = (String) o; // it.next();
        // System.out.println("K CLASS: " + o);
        // System.out.println("V CLASS: " + doc.get(temp).getClass());
        System.out.print(temp + "-");

        if (doc.get(temp) instanceof BasicDBObject)
        {
            System.out.println("\n");
	    print((DBObject) doc.get(temp));
        }
        else
        {
            System.out.println(doc.get(temp));
	    if (temp.toString().equalsIgnoreCase("loc"))
	        System.out.println("-------------");
        }
    }//end of while
}

// To clean up a collection completely

static void clean()
{
    // clear records if any
    DBCursor cur = coll.find();

    while (cur.hasNext())
        coll.remove(cur.next());
}

Thats It!

How to use geo spatial with PostGres (PostGIS)?

Problem Statement
Find all locations that are within a certain radius of a particular location.

Solution Approach
We will demonstrate this with pseudo-code.

Lets create a table in MySQL that will store geo information:

CREATE TABLE geoinfo
(
  id bigserial NOT NULL,
  latitude numeric(9,4),
  longitude numeric(9,4),
  "location" geometry,
  CONSTRAINT enforce_dims_geom CHECK (st_ndims(location) = 2),
  CONSTRAINT enforce_geotype_geom CHECK (geometrytype(location) = 'POINT'::text OR location IS NULL),
  CONSTRAINT enforce_srid_geom CHECK (st_srid(location) = (-1))
)
WITH (
  OIDS=FALSE
);

NOTE the use of the geometry datatype above.

Now, what we want to do is insert lots of data in this table:
The following code inserts 100,000 random records into geoinfo table. We will do a search on this very data later.
The code is same as what we saw in: , only difference being this line:
geomFunc=”st_pointfromtext”;// we are using a PostGIS specific function to insert data in location column.

static String dbUrl = "jdbc:postgresql://localhost:5432/postgis?";
static String dbClass = "org.postgresql.Driver";
static String schemaPrefix="";
static String DB_USERNAME="whatever";
static String DB_PASSWD="whatever";

Class.forName(dbClass);
Connection con = DriverManager.getConnection(dbUrl, DB_USERNAME, DB_PASSWD);
Statement stmt = con.createStatement();

String query1;
long startTime = Calendar.getInstance().getTimeInMillis();
//Randomize the lat, long inputs
Random rand = new Random();
String geomFunc="";

geomFunc="st_pointfromtext";

for (int i = 0; i < 100000; i++)
{
    double lat = rand.nextDouble() * 180 - 90;
    BigDecimal bd1 = new BigDecimal(lat).setScale(2, RoundingMode.HALF_EVEN);
    lat = bd1.doubleValue();
    double lon = rand.nextDouble() * 360 - 180;
    BigDecimal bd = new BigDecimal(lon).setScale(2, RoundingMode.HALF_EVEN);
    lon = bd.doubleValue();
    query1 = "INSERT INTO " + schemaPrefix + "geoinfo(latitude, longitude, location) VALUES (" + lat + "," + lon +             ", " + geomFunc + "('POINT(" + lat + " " + lon + ")') )";
    System.out.println(query1);
    int res = stmt.executeUpdate(query1);
    // System.out.println("res: " + res);
}

long timeTaken = Calendar.getInstance().getTimeInMillis() - startTime;
System.out.println("Total Time Taken to insert data : " + timeTaken + " milliseconds");


Now we want to do a geo spatial search on geoinfo table
i.e find all rows from the geoinfo table whose location is within a certain miles(radius) from a particular location(lat,lon):

We will write a findGeoLocations function that will print all records that meet this criteria:


public void findGeoLocations(double lat, double lon, double radiusInMiles)
{

Connection con = null;
try {
    //get DB Connection from somewhere.
    con = getConnection();

    //Statement stmt = con.createStatement();
    String query="";
    double radius=radiusInMiles/69;

    query="SELECT id, latitude, longitude, ST_DISTANCE(location, 'POINT(" + lat + " " + lon + ")') as distance FROM geoinfo WHERE ST_DWithin(location, 'POINT(" + lat + " " + lon + ")', " + radius + ")";
    //IMPORTANT:
    //If : geography suffix is used against POINT in SQL - then the order should be - lon lat and not - lat lon.
    //And the data too should have been inserted as lon lat for geocode location column. Geography suffix uses distances in meters.
    //If geography suffix is not used - then we can simply use - lat lon format.

    System.out.println("Query to use: " + query);

    PreparedStatement pStmt = con.prepareStatement(query);
    ResultSet rs;
    long startTime = Calendar.getInstance().getTimeInMillis();
    rs = pStmt.executeQuery();
    long timeTaken = Calendar.getInstance().getTimeInMillis() - startTime;

    int rows = 0;
    while (rs.next())
    {
        rows++;
        System.out.println ("RS: Id: " + rs.getInt("id") + " Lat: " + rs.getDouble("latitude") + " Lon: " + 		rs.getDouble("longitude") + " Dist: " +					rs.getDouble("distance"));
    }
    System.out.println("Total Rows: " + rows);
} // end try
catch (SQLException e)
{
    e.printStackTrace();
}
finally {
    con=null;
}
}

Thats It!

How to use Haversine Formula for location (distance) searches with MySQL?

Problem Statement
Find all locations that are within a certain radius from a particular location(point).

Solution Approach
We will demonstrate that with pseudo-code.
Refer the post titled: How to use Geo Spatial with MySQL
Everything else remains same we will modify the function findGeoLocations(…) and call it findGeoLocationsWithHaversine(…).
NOTE that it does not use geo spatial capabilities of MySQL at all.

This is what the function looks like:

public void findGeoLocationsWithHaversine(double lat, double lon,
double radiusInMiles)
{

Connection con = null;

try {
    con = Constants.getConnection();

    //Statement stmt = con.createStatement();
    String query="";
    double radius=radiusInMiles/69;

    double x1 = lat - radius; double y1 = lon - radius;
    double x2 = lat + radius; double y2 = lon - radius;
    double x3 = lat + radius; double y3 = lon + radius;
    double x4 = lat - radius; double y4 = lon + radius;
    String polygonStr = x1 + " " + y1 + "," + x2 + " " + y2 + "," +
    x3 + " " + y3 + "," + x4 + " " + y4 + "," + x1 + " " + y1;
    String pointStr = lat + " " + lon;

    query = "select g.id, g.latitude, g.longitude, (3956 * 2 *
    ASIN ( SQRT ( POWER(SIN(("+lat+" - g.latitude)*pi()/180 / 2),";
    query = query + " 2) + COS(" + lat + "* pi()/180) *
    COS(g.latitude * pi()/180) * POWER(SIN((" + lon +
    "- g.longitude)     *";
    query = query + " pi()/180 / 2), 2) ) ))/69 as distance" ;
    query = query + " from geoinfo g";
    query = query + " where g.longitude between (" + lon +
    " - " + radius + "/abs(cos(radians(" + lat + "))))";
    query = query + " and (" + lon + " + " + radius +
    "/abs(cos(radians(" + lat + ")))";
    query = query + " ) and g.latitude between (" + lat +
    " - " + radius + ") and (" + lat + "+" + radius + ")";
    query = query + " having distance < " + radius;
    query = query + " order by distance";

    System.out.println("Query to use: " + query);
    PreparedStatement pStmt = con.prepareStatement(query);
    ResultSet rs;

    long startTime = Calendar.getInstance().getTimeInMillis();
    rs = pStmt.executeQuery();
    long timeTaken = Calendar.getInstance().getTimeInMillis()
    - startTime;

    System.out.println("Time Taken: " + timeTaken + " milliseconds");

    int rows = 0;
    while (rs.next())
    {
        rows++;
        /*
        System.out.println ("RS: Id: " + rs.getInt("id") +
        " Lat: " + rs.getDouble("latitude") + " Lon: " +
        rs.getDouble("longitude") + " Dist: " +
        rs.getDouble("distance"));
        */
    }
    System.out.println("Total Rows: " + rows);

} // end try

catch (SQLException e)
{
    e.printStackTrace();
}
finally
{
    con=null;
}
}

Thats It!

How to use Geo Spatial with MySQL?

Problem Statement
Find all locations that are within a certain radius of a particular location.

Solution Approach
We will demonstrate this with pseudo-code.

Lets create a table in MySQL that will store geo information:

create table geoinfo (
id bigint(20) NOT NULL AUTO_INCREMENT,
latitude DECIMAL(9,4) NULL ,
longitude DECIMAL(9,4) NULL ,
location POINT NOT NULL ,
PRIMARY KEY (ID) )
ENGINE = MyISAM DEFAULT CHARSET=utf8

-- Note the use of POINT data type against the location column above.
-- Spatial Indexes cannot be created on InnoDB tables.
create spatial index sp_index ON geocode(location);

Now, what we want to do is insert lots of data in this table:

static String dbUrl = "jdbc:mysql://localhost:3306/test?";
static String dbClass = "com.mysql.jdbc.Driver";
static String schemaPrefix="test."
static String DB_USERNAME="whatever";
static String DB_PASSWD="whatever";

Class.forName(dbClass);
Connection con = DriverManager.getConnection(dbUrl, DB_USERNAME, DB_PASSWD);
Statement stmt = con.createStatement();

String query1;
long startTime = Calendar.getInstance().getTimeInMillis();
//Randomize the lat, long inputs
Random rand = new Random();
String geomFunc="";

geomFunc="GeomFromText";

for (int i = 0; i < 100000; i++) {
double lat = rand.nextDouble() * 180 - 90;
BigDecimal bd1 = new BigDecimal(lat).setScale(2, RoundingMode.HALF_EVEN);
lat = bd1.doubleValue();
double lon = rand.nextDouble() * 360 - 180;
BigDecimal bd = new BigDecimal(lon).setScale(2, RoundingMode.HALF_EVEN);
lon = bd.doubleValue();
query1 = "INSERT INTO " + schemaPrefix + "geoinfo(latitude, longitude, location) VALUES (" + lat + "," + lon + ", " + geomFunc + "('POINT(" + lat + " " + lon + ")') )";
System.out.println(query1);
int res = stmt.executeUpdate(query1);
// System.out.println("res: " + res);
}

long timeTaken = Calendar.getInstance().getTimeInMillis() - startTime;
System.out.println("Total Time Taken to insert data : " + timeTaken + " milliseconds");

Now we want to do a geo spatial search on geoinfo table
i.e find all rows from the geoinfo table whose location is within a certain miles(radius) from a particular location(lat,lon):

We will write a findGeoLocations function that will print all records that meet this criteria.

public void findGeoLocations(double lat, double lon, double radiusInMiles) {

Connection con = null;

try {
con = Constants.getConnection();

//Statement stmt = con.createStatement();
String query="";
double radius=radiusInMiles/69;

double x1 = lat - radius; double y1 = lon - radius;
double x2 = lat + radius; double y2 = lon - radius;
double x3 = lat + radius; double y3 = lon + radius;
double x4 = lat - radius; double y4 = lon + radius;
String polygonStr = x1 + " " + y1 + "," + x2 + " " + y2 + "," + x3 + " " + y3 + "," + x4 + " " + y4 + "," + x1 + " " + y1;
String pointStr = lat + " " + lon;

query = "SELECT id, latitude, longitude, AsText(location),";
query = query + " SQRT(POW( ABS( X(location) - " + lat + "), 2) +";
query = query + " POW( ABS(Y(location) - " + lon + "), 2 )) AS distance";
query = query + " FROM geoinfo ";
query = query + " WHERE Intersects( location, GeomFromText('POLYGON((" + polygonStr + "))'))";
query = query + " AND (SQRT(POW( ABS( X(location) - X(GeomFromText('POINT(" + pointStr + ")'))), 2) +";
query = query + " POW( ABS(Y(location) - Y(GeomFromText('POINT(" + pointStr + ")'))), 2 ))) < " + radius ;
query = query + " order by distance";

System.out.println("Query to use: " + query);

PreparedStatement pStmt = con.prepareStatement(query);
ResultSet rs;

long startTime = Calendar.getInstance().getTimeInMillis();
rs = pStmt.executeQuery();
long timeTaken = Calendar.getInstance().getTimeInMillis() - startTime;

System.out.println("Time Taken: " + timeTaken + " milliseconds");

int rows = 0;
while (rs.next()) {
rows++;
/*
System.out.println ("RS: Id: " + rs.getInt("id") +
" Lat: " + rs.getDouble("latitude") + " Lon: " +
rs.getDouble("longitude") + " Dist: " +
rs.getDouble("distance"));
*/
}
System.out.println("Total Rows: " + rows);

} // end try

catch (SQLException e) {
e.printStackTrace();
} finally {
con=null;
}
}

So Thats It!

How do i set Proxy for Maven (in Eclipse)?

Problem Statement:
(1) I am behind a firewall, therefore Maven is unable to connect to repositories to download jars etc.

Solution:
In MAVEN_HOME\apache-maven-3.0.3\conf dir you have settings.xml. Open it and update the following:

<proxies>
<!-- proxy
| Specification for one proxy, to be used in connecting to the network. -->
<proxy>
<id>optional</id>
<active>true</active>
<protocol>http</protocol>
<host></host>
<port></port>
<username></username>
<password></password>
<nonProxyHosts></nonProxyHosts>
</proxy>
</proxies>

Also check the following dir on windows:
C:\Documents and Settings\useName\.m2
If this has settings.xml – update that too.

Remember that username may require to be prefixed with domain name eg: mycorp/myuser

Problem Statement:
(2) I am behind a firewall, using Eclipse IDE and have a Maven Plugin. But it is unable to connect to repositories.

Solution:
Menu> Windows > Preferences > General > Network Connections.

Update the Proxy entries there.
In case this does not work, try doing the Step 1 solution as well.

NOTE:
1. Many Corporate Proxies will block you (temporarily?) on repeated authentication failures. Therefore be aware of this possibility.
2. Try to authenticate yourself against your network say by logging off and logging into your system to ensure that your login credentials are still valid.

How can i measure CPU and Memory utilization on a Linux?

CPU Utilization Commands:

CPU utilization; measured every 2 seconds; 5 times:
$ sar -u 2 5

Processor related stats:
$ mpstat
$ mpstat -P ALL

Top 10 CPU users/processes on the Linux system:
$ ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

Memory Utilization Commands

ps aux
ps aux | awk ‘{print $4″\t”$11}’ | sort | uniq -c | awk ‘{print $2″ “$1″ “$3}’ | sort -nr

Overall

To see lots of stats with a single command in real-time
$ top

How to do a Proximity Search using Haversine Formula?

Let us consider a table:

create table `geoaddress` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`latitude` DECIMAL(9,4) NULL ,
`longitude` DECIMAL(9,4) NULL,
‘address’ VARCHAR(1000) NULL,
PRIMARY KEY (`ID`) )
ENGINE = InnoDB DEFAULT CHARSET=utf8

The above table must ofcourse be populated with appropriate data.
Here is a simple java code snippet to insert some random data:

...
Statement stmt = con.createStatement();
Random rand = new Random();
for (int i = 0; i < 100000; i++) {
double lat = rand.nextDouble() * 180 - 90;
BigDecimal bd1 = new BigDecimal(lat).setScale(2, RoundingMode.HALF_EVEN);
lat = bd1.doubleValue();

double lon = rand.nextDouble() * 360 - 180;
BigDecimal bd = new BigDecimal(lon).setScale(2, RoundingMode.HALF_EVEN);
lon = bd.doubleValue();

String query = "INSERT INTO test.geocode(latitude, longitude, address)
VALUES (" + lat + "," + lon + ", 'some addr')";
System.out.println(query);
int res = stmt.executeUpdate(query1);
// System.out.println("res: " + res);
}
....


Given a latitude/longitude, we want to find all records from geoaddress table that are within a certain radius from a given latitude/longitude.

Here is java code snippet for that:

//Update lat, lon, radius variables with whatever values you want! The lat/lon together form the point from where search is performed.
double lat=20;
double lon=55;
double radius=1; //Note: 1 unit (or degree) is equivalent to 69 miles. So for a 100 mile search, this should be set to 1.45

String query;

//The query below is implementation of Haversine Formula!
query = "select g.id, g.latitude, g.longitude, (3956 * 2 * ASIN ( SQRT ( POWER(SIN(("+lat+" - g.latitude)*pi()/180 / 2),";
query = query + " 2) + COS(" + lat + "* pi()/180) * COS(g.latitude * pi()/180) * POWER(SIN((" + lon + "- g.longitude) *";
query = query + " pi()/180 / 2), 2) ) ))/69 as distance" ;
query = query + " from geocode g";
query = query + " where g.longitude between (" + lon + " - " + radius + "/abs(cos(radians(" + lat + "))))";
query = query + " and (" + lon + " + " + radius + "/abs(cos(radians(" + lat + ")))";
query = query + " ) and g.latitude between (" + lat + " - " + radius + ") and (" + lat + "+" + radius + ")";
query = query + " having distance < " + radius;
query = query + " order by distance";

....
PreparedStatement pStmt = con.prepareStatement(query);
ResultSet rs = pStmt.executeQuery();
...

Thats it! The query will return all records from geoaddress table that are within 69 mile radius (1 unit) from lat/long of 20/55!

Reference: http://www.scribd.com/doc/2569355/Geo-Distance-Search-with-MySQL