Spark Solr Integration on Windows
Prerequisite :
1) Download the spark-solr sources from following link and unzip to a location e.g. d:\spark-solr-master. Lets call this location SS_HOME
https://github.com/LucidWorks/spark-solr
set SS_HOME=d:\spark-solr-master
2) Download maven and set following env variables
set M2_HOME=D:\apps\apache-maven-3.0.5
set path=%M2_HOME%\bin;%PATH%;
3) open a command prompt, make sure m2_home is set and and added to path. Run command
cd %SS_HOME%
mvn install -DskipTests
This will build the spark-solr-1.0.SNAPSHOT.jar in %SS_HOME%\target location.
4) Download the spark solr java sample from here ; (you will need Eclipse with maven 'm2eclipse' plugin)
https://github.com/freissmann/SolrWithSparks
Import it in eclipse using Import .. > Existing Maven Project > Browse to the downloaded and unzipped example directory e.g. D:\SolrWithSparks-master
The SparkSolrJobApp.java sample needs JDK1.8 for lambda functions
Open the pom.xml and add following if not already specified :
- Install Spark 1.5.X (see here for details http://prasi82.blogspot.com/2015/11/setup-spark-on-windows.html )
- Install zookeeper and Solr 5.3.X
- If you have solr 4.X you will need to upgrade your solr 4.X indexed data to Solr 5.X.
1) Download the spark-solr sources from following link and unzip to a location e.g. d:\spark-solr-master. Lets call this location SS_HOME
https://github.com/LucidWorks/spark-solr
set SS_HOME=d:\spark-solr-master
2) Download maven and set following env variables
set M2_HOME=D:\apps\apache-maven-3.0.5
set path=%M2_HOME%\bin;%PATH%;
3) open a command prompt, make sure m2_home is set and and added to path. Run command
cd %SS_HOME%
mvn install -DskipTests
This will build the spark-solr-1.0.SNAPSHOT.jar in %SS_HOME%\target location.
4) Download the spark solr java sample from here ; (you will need Eclipse with maven 'm2eclipse' plugin)
https://github.com/freissmann/SolrWithSparks
Import it in eclipse using Import .. > Existing Maven Project > Browse to the downloaded and unzipped example directory e.g. D:\SolrWithSparks-master
The SparkSolrJobApp.java sample needs JDK1.8 for lambda functions
Open the pom.xml and add following if not already specified :
If you are running eclipse with JAVA_HOME pointing to a JRE path instead of JDK path, you will get 1.6 tools.jar missing error. To resolve this error, restart eclipse with following changes in your eclipse_home\eclipse.ini file :
See link below for additional info :
http://stackoverflow.com/questions/11118070/buiding-hadoop-with-eclipse-maven-missing-artifact-jdk-toolsjdk-toolsjar1
package de.blogspot.qaware.spark;
import com.lucidworks.spark.SolrRDD;
import de.blogspot.qaware.spark.common.ContextMaker;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.common.SolrDocument;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import java.util.Arrays;
public class SparkSolrJobApp {
private static final String ZOOKEEPER_HOST_AND_PORT = "zkhost:zkport";
private static final String SOLR_COLLECTION = "collection1";
private static final String QUERY_ALL = "*:*";
public static void main(String[] args) throws Exception {
String zkHost = ZOOKEEPER_HOST_AND_PORT;
String collection = SOLR_COLLECTION;
String queryStr = QUERY_ALL;
JavaSparkContext javaSparkContext = ContextMaker.makeJavaSparkContext("Querying Solr");
SolrRDD solrRDD = new SolrRDD(zkHost, collection);
final SolrQuery solrQuery = SolrRDD.toQuery(queryStr);
JavaRDD
JavaRDD
Object possibleTitle = doc.get("title");
String title = possibleTitle != null ? possibleTitle.toString() : "";
return Arrays.asList(title);
}).filter(s -> !s.isEmpty());
System.out.println("\n# of found titles: " + titleNumbers.count());
// Now use schema information in Solr to build a queryable SchemaRDD
SQLContext sqlContext = new SQLContext(javaSparkContext);
// Pro Tip: SolrRDD will figure out the schema if you don't supply a list of field names in your query
DataFrame tweets = solrRDD.asTempTable(sqlContext, queryStr, "documents");
// SQL can be run over RDDs that have been registered as tables.
DataFrame results = sqlContext.sql("SELECT * FROM documents where id LIKE 'one%'");
// The results of SQL queries are SchemaRDDs and support all the normal RDD operations.
// The columns of a row in the result can be accessed by ordinal.
JavaRDD
System.out.println("\n\n# of documents where 'id' starts with 'one': " + resultsRDD.count());
javaSparkContext.stop();
}
}
5) Build Assembly Jar of your code :
Modify pom.xml to build the assembly jar of your example code .. i.e. jar with all the dependencies included :
Open a command prompt , make sure m2_home is set as indicated step2) and "%m2_home%\bin"included in the path.
cd D:\SolrWithSparks-master
mvn clean
mvn package -DskipTests
This will build the jar of your code with all its dependency classes included in it in the "target" folder.
6) Run solr spark example :
Before you run this program, you will need to start your zookeeper and solrcloud shards.
Go to command prompt and run following :
set M2_HOME=D:\apps\apache-maven-3.0.5
set path=%M2_HOME%\bin;%PATH%;
set java_home=d:\apps\Java\jdk1.8.0_65
set jre_home=%java_home%\jre
set jdk_home=%JAVA_HOME%
set path=%java_home%\bin;%path%
set SPARK_HOME=D:\udu\hk\spark-1.5.1
set SPARK_CONF_DIR=%SPARK_HOME%\conf
call %SPARK_HOME%\bin\load-spark-env.cmd
%spark_home%\bin\spark-submit.cmd --class org.sparkexample.SparkSolrJobApp file://d:/spark-solr-master/target/spark-test-id-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Here note that you will need to specify the path to your spark job assembly jar starting with "file://" and use '/' as path separator instead of '\'
After the jar path you may specify any additional parameters required by your spark job.
7) Troubleshooting :
If you get Connection time out error, make sure that in your SPARK_HOME\conf\spark-env.cmd :
SPARK_LOCAL_IP is set.
set SPARK_LOCAL_IP=127.0.0.1
REM set ip address of spark master
REM set SPARK_MASTER_IP=127.0.0.1
If you get error of following type :
Exception Details:
Location:
org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/http/conn/ClientConnectionManager;)Lorg/apache/http/impl/client/CloseableHttpClient; @62: areturn
Reason:
Type 'org/apache/http/impl/client/DefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
Current Frame:
bci: @62
flags: { }
locals:
{ 'org/apache/solr/common/params/SolrParams', 'org/apache/http/conn/ClientConnectionManager', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/DefaultHttpClient' }Location:
org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/http/conn/ClientConnectionManager;)Lorg/apache/http/impl/client/CloseableHttpClient; @62: areturn
Reason:
Type 'org/apache/http/impl/client/DefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
Current Frame:
bci: @62
flags: { }
locals:
stack:
{ 'org/apache/http/impl/client/DefaultHttpClient' }
Bytecode:
0000000: bb00 0359 2ab7 0004 4db2 0005 b900 0601
0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
0000020: b600 0a2c b600 0bb6 000c b900 0d02 00bb
0000030: 0011 592b b700 124e 2d2c b800 102d b0
Stackmap Table:
append_frame(@47,Object127)
0000000: bb00 0359 2ab7 0004 4db2 0005 b900 0601
0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
0000020: b600 0a2c b600 0bb6 000c b900 0d02 00bb
0000030: 0011 592b b700 124e 2d2c b800 102d b0
Stackmap Table:
append_frame(@47,Object127)
Download the following jars (just google for it) and copy them to your %hadoop_home%\share\hadoop\common\lib folder.
httpclient-4.4.1.jar
httpcore-4.4.1.jar
See here for details : https://issues.apache.org/jira/browse/SOLR-7948
8) FAQ
How to build solr Queries in Spark Job :
https://svn.apache.org/repos/asf/lucene/solr/tags/release-1.3.0/client/java/solrj/test/org/apache/solr/client/solrj/SolrExampleTests.java
How to sort a JavaRDD
http://stackoverflow.com/questions/27151943/sortby-in-javardd
How to invoke a spark job from Java Program :
https://github.com/spark-jobserver/spark-jobserver
http://apache-spark-user-list.1001560.n3.nabble.com/Programatically-running-of-the-Spark-Jobs-td13426.html
Migrating Solr 4.X Index data to Solr 5.X index
To upgrade your solr 4.X indexed data to Solr 5.X , run the following command
java -cp D:\solr-5.3.1\server\solr-webapp\webapp\WEB-INF\lib\* org.apache.lucene.index.IndexUpgrader D:\solr-4.4.0\example\solr\collection1\data\index
Here assuming "D:\solr-4.4.0\awc\solr\collection1\data\index" is the directory that contains indexed data that you want to upgrade to Solr 5.3.X.
After this you can copy your Solr 4.4.X colleciton / core directory (e.g. D:\solr-4.4.0\example\solr\collection1 in above command) to Solr 5.3.X home directory.
After copying to Solr5.3.X home directory, you will need to make few changes in the schema.xml and solrConfig.xml of your collection :
In your collection\conf\solrConfig.xml, comment out following :
In your collection\conf\schema.xml, change following, field values to include "Trie" Prefix
java -cp D:\solr-5.3.1\server\solr-webapp\webapp\WEB-INF\lib\* org.apache.lucene.index.IndexUpgrader D:\solr-4.4.0\example\solr\collection1\data\index
Here assuming "D:\solr-4.4.0\awc\solr\collection1\data\index" is the directory that contains indexed data that you want to upgrade to Solr 5.3.X.
After this you can copy your Solr 4.4.X colleciton / core directory (e.g. D:\solr-4.4.0\example\solr\collection1 in above command) to Solr 5.3.X home directory.
After copying to Solr5.3.X home directory, you will need to make few changes in the schema.xml and solrConfig.xml of your collection :
In your collection\conf\solrConfig.xml, comment out following :
In your collection\conf\schema.xml, change following, field values to include "Trie" Prefix
Setup Apache Spark On Windows
download spark and unzip E.g
spark_Home=D:\udu\hk\spark-1.5.1
spark needs hadoop jars. download hadoop binaries for windows (hadoop 2.6.0) from
http://www.barik.net/archive/2015/01/19/172716/
unzip hadoop at some locaiton e.g.
hadoop_home=D:\udu\hk\hadoop-2.6.0
If your java_home or hadoop_home path contains space charcters in it ' ', you will need to convert to path to short paths:
- Create a batch script with following contents
@ECHO OFF
echo %~s1
- Run the above batch script file from java_home directory to get the short path for java-home
- Run the above batch script file from hadoop_home directory to get the short path for hadoop_home
set java_home=short path obtained from above command
set hadoop_home=short path obatained from above command.
Run following command and copy the classpath generated by the command for next step
%HADOOP_HOME%\bin\hadoop classpath
under spark_home\conf, create a file named "spark-env.cmd" like below
@echo off
set HADOOP_HOME=D:\Utils\hadoop-2.7.1
set PATH=%HADOOP_HOME%\bin;%PATH%
set SPARK_DIST_CLASSPATH=
on Command prompt
cd %spark_home%\bin
set SPARK_CONF_DIR=%SPARK_HOME%\conf
load-spark-env.cmd
spark-shell.cmd //To start spark shell
spark-submit.cmd
Refer below to create a spark word count example
http://www.robertomarchetto.com/spark_java_maven_example
To run a spark job (written using Java) word count example from above URL
spark-submit --class org.sparkexample.WordCount --master local[2] your_spark_job_jar Any_additional_parameters_needed_by_your_job_jar
References :
http://stackoverflow.com/questions/30906412/noclassdeffounderror-com-apache-hadoop-fs-fsdatainputstream-when-execute-spark-s
https://blogs.perficient.com/multi-shoring/blog/2015/05/07/setup-local-standalone-spark-node/
http://nishutayaltech.blogspot.com/2015/04/how-to-run-apache-spark-on-windows7-in.html
How to split a solr core into mutiple shards on different machine
download zookeeper
make a copy of conf\zk_sample.cfg as zoo.cfg
creaeta zkdata folder.
in zoo.cfg, uncomment datadir and provide path to zkdata folder (replace '\' in the path with '/' )
Modify zk_home/bin/zk_env.cmd to replace ~dp0.. with actual paths like below
REM assuming Zookeeper is installed at : D:\zk1
set ZOOCFGDIR=D:\zk1\conf
set ZOO_LOG_DIR=D:\zk1
set ZOO_LOG4J_PROP=INFO,CONSOLE
set CLASSPATH=%ZOOCFGDIR%
SET CLASSPATH=D:\zk1\*;D:\zk1\lib\*;%CLASSPATH%
SET CLASSPATH=D:\zk1\build\classes;D:\zk1\build\lib\*;%CLASSPATH%
set ZOOCFG=%ZOOCFGDIR%\zoo.cfg
Lets assume you have a stand alone solr with "collection1" core and we want to convert this into soldCloud cluster and split this core into multiple shards on diff machine.
1) First start zookeepr, run following command from zk_home/bin path:
zkServer.cmd
2) Next upload & link your standalone core config to Zookeeper (Assuming Solr is installed at : D:\solr-4.4.0\mySolr1)
make sure to use '/' as path seprate instead of '\' for the confdir path below
cd D:\solr-4.4.0\mySolr1\cloud-scripts
zkcli.bat -zkhost localhost:2181 -cmd upconfig -confdir D:/solr-4.4.0/mySolr1/solr/collection1/conf -confname collection1cfg
zkcli.bat -zkhost localhost:2181 -cmd linkconfig -collection collection1 -confname collection1cfg
3) Next start your solr with following command (assuming zookeeper is running on localhost:2181) and solr is installed at D:\solr-4.4.0\mySolr1
java -Xmx2g -Djetty.port=8983 -DzkHost=localhost:2181 -Dsolr.solr.home=D:\solr-4.4.0\mySolr1\solr -jar start.jar
4) Go to localhost:8989/solr and verify the # of documents in your core with following URL:
http://localhost:8983/solr/collection1/select?q=*:*&rows=0
make a note of numFound value in the response returned.
Also verify that following URL shows the "collection1" core we have as "shard1"
http://localhost:8983/solr/#/~cloud
5) Next in the browser run the following command to split your core "collection1" into multiple shards (assuming shard1 is the name your shard and collection1 is the name of your core)
http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1&async=5
This should generated 2 shards / cores
collection1_shard1_1_replica1
collection1_shard1_0_replica1
6) Next on another machine (e.g. machine2) , start solr with following command (assuming zookeeper is running on <>:2181 (replace "machine1" with name or ip address of machine1)
and solr on machine 2 is installed at : D:/solr-4.4.0/mySolr2
java -Xmx2g -Djetty.port=7574 -DzkHost=machine1:2181 -Dsolr.solr.home=D:/solr-4.4.0/mySolr2/solr -jar start.jar
This will start solr on port 7574 on machine 2
7) On machine2, Run following command to create a new Core (collection1_shard1_1_core2) specifying the collection=collection1 (same as original core name on machine1 before splitting into shards)
and shard=shard1_1 (same as what shown on http://localhost:8983/solr/#/~cloud page for the shard that you want to move to machine2)
http://localhost:7574/solr/admin/cores?action=CREATE&collection=collection1&shard=shard1_1&name=collection1_shard1_1_core2
8) now shutdown solr (not zookeeper) running on machine1 by pressing ctrl + C in the running solr window on machine 1.
Move the shard1_1_replica1 folder from solr installation, to some other location outside SOLR_HOME\solr directory.
And re-run the solr on machine 1
java -Xmx2g -Djetty.port=8983 -DzkHost=localhost:2181 -Dsolr.solr.home=D:\solr-4.4.0\mySolr1\solr -jar start.jar
Verify now shard on machine 2 is shown as shard leader for shard1_1.
We have now sucessfully converted a single core named 'collection1' into a distributed collection 'collection1' with shard 'shard1_0' on original machine1 and moved shard1_1 to another machine2.
make a copy of conf\zk_sample.cfg as zoo.cfg
creaeta zkdata folder.
in zoo.cfg, uncomment datadir and provide path to zkdata folder (replace '\' in the path with '/' )
Modify zk_home/bin/zk_env.cmd to replace ~dp0.. with actual paths like below
REM assuming Zookeeper is installed at : D:\zk1
set ZOOCFGDIR=D:\zk1\conf
set ZOO_LOG_DIR=D:\zk1
set ZOO_LOG4J_PROP=INFO,CONSOLE
set CLASSPATH=%ZOOCFGDIR%
SET CLASSPATH=D:\zk1\*;D:\zk1\lib\*;%CLASSPATH%
SET CLASSPATH=D:\zk1\build\classes;D:\zk1\build\lib\*;%CLASSPATH%
set ZOOCFG=%ZOOCFGDIR%\zoo.cfg
Lets assume you have a stand alone solr with "collection1" core and we want to convert this into soldCloud cluster and split this core into multiple shards on diff machine.
1) First start zookeepr, run following command from zk_home/bin path:
zkServer.cmd
2) Next upload & link your standalone core config to Zookeeper (Assuming Solr is installed at : D:\solr-4.4.0\mySolr1)
make sure to use '/' as path seprate instead of '\' for the confdir path below
cd D:\solr-4.4.0\mySolr1\cloud-scripts
zkcli.bat -zkhost localhost:2181 -cmd upconfig -confdir D:/solr-4.4.0/mySolr1/solr/collection1/conf -confname collection1cfg
zkcli.bat -zkhost localhost:2181 -cmd linkconfig -collection collection1 -confname collection1cfg
3) Next start your solr with following command (assuming zookeeper is running on localhost:2181) and solr is installed at D:\solr-4.4.0\mySolr1
java -Xmx2g -Djetty.port=8983 -DzkHost=localhost:2181 -Dsolr.solr.home=D:\solr-4.4.0\mySolr1\solr -jar start.jar
4) Go to localhost:8989/solr and verify the # of documents in your core with following URL:
http://localhost:8983/solr/collection1/select?q=*:*&rows=0
make a note of numFound value in the response returned.
Also verify that following URL shows the "collection1" core we have as "shard1"
http://localhost:8983/solr/#/~cloud
5) Next in the browser run the following command to split your core "collection1" into multiple shards (assuming shard1 is the name your shard and collection1 is the name of your core)
http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1&async=5
This should generated 2 shards / cores
collection1_shard1_1_replica1
collection1_shard1_0_replica1
6) Next on another machine (e.g. machine2) , start solr with following command (assuming zookeeper is running on <
and solr on machine 2 is installed at : D:/solr-4.4.0/mySolr2
java -Xmx2g -Djetty.port=7574 -DzkHost=machine1:2181 -Dsolr.solr.home=D:/solr-4.4.0/mySolr2/solr -jar start.jar
This will start solr on port 7574 on machine 2
7) On machine2, Run following command to create a new Core (collection1_shard1_1_core2) specifying the collection=collection1 (same as original core name on machine1 before splitting into shards)
and shard=shard1_1 (same as what shown on http://localhost:8983/solr/#/~cloud page for the shard that you want to move to machine2)
http://localhost:7574/solr/admin/cores?action=CREATE&collection=collection1&shard=shard1_1&name=collection1_shard1_1_core2
8) now shutdown solr (not zookeeper) running on machine1 by pressing ctrl + C in the running solr window on machine 1.
Move the shard1_1_replica1 folder from solr installation, to some other location outside SOLR_HOME\solr directory.
And re-run the solr on machine 1
java -Xmx2g -Djetty.port=8983 -DzkHost=localhost:2181 -Dsolr.solr.home=D:\solr-4.4.0\mySolr1\solr -jar start.jar
Verify now shard on machine 2 is shown as shard leader for shard1_1.
We have now sucessfully converted a single core named 'collection1' into a distributed collection 'collection1' with shard 'shard1_0' on original machine1 and moved shard1_1 to another machine2.
Java Regx to parse RGBA Color Values
String keywords_color_regex = "^[a-z]*$";
String hex_color_regex = "^#[0-9a-f]{3}([0-9a-f]{3})?$";
String rgb_color_regex = "^rgb\\(\\s*(0|[1-9]\\d?|1\\d\\d?|2[0-4]\\d|25[0-5])\\s*,\\s*(0|[1-9]\\d?|1\\d\\d?|2[0-4]\\d|25[0-5])\\s*,\\s*(0|[1-9]\\d?|1\\d\\d?|2[0-4]\\d|25[0-5])\\s*\\)$";
String rgba_color_regex = "^rgba\\(\\s*(0|[1-9]\\d?|1\\d\\d?|2[0-4]\\d|25[0-5])\\s*,\\s*(0|[1-9]\\d?|1\\d\\d?|2[0-4]\\d|25[0-5])\\s*,\\s*(0|[1-9]\\d?|1\\d\\d?|2[0-4]\\d|25[0-5])\\s*,\\s*((0.[1-9])|[01])\\s*\\)$";
String hsl_color_regex = "^hsl\\(\\s*(0|[1-9]\\d?|[12]\\d\\d|3[0-5]\\d)\\s*,\\s*((0|[1-9]\\d?|100)%)\\s*,\\s*((0|[1-9]\\d?|100)%)\\s*\\)$";Source : http://stackoverflow.com/questions/12385500/regex-pattern-for-rgb-rgba-hsl-hsla-color-coding
Mock it..
If you try to mock a method that returns primitive types, eclipse gives following compiler Error :
The expression of type boolean is boxed into Boolean
to get rid of this error :
Globally: Window-> Preferences: Java-> Compiler-> Errors / Warnings-> Potential programming problems-> Boxing and unboxing conversions.
For the project: Properties: Java Compiler-> Errors / Warnings-> further as well.
Source :
http://translate.google.co.in/translate?hl=en&sl=ru&u=http://www.sql.ru/forum/1059132/mockito-strannaya-konstrukciya-vnutri-when&prev=search
The expression of type boolean is boxed into Boolean
to get rid of this error :
Globally: Window-> Preferences: Java-> Compiler-> Errors / Warnings-> Potential programming problems-> Boxing and unboxing conversions.
For the project: Properties: Java Compiler-> Errors / Warnings-> further as well.
Source :
http://translate.google.co.in/translate?hl=en&sl=ru&u=http://www.sql.ru/forum/1059132/mockito-strannaya-konstrukciya-vnutri-when&prev=search
Closest Matching NameToken Algo
Recently worked on a GWT app. This app has a components laid out in a hierarchy.
A Primary Page has a set of Tabs. Each Tab has a primary Section listing primary
objects and a secondary section listing contents for the selected primary object.
Also each tab has a command bar to which commands can be contributed. Since this is a GWT app, every section here is a GWT Presenter widget.
The requirement here was to be able to contribute commands in one or more sections mentioned above using wildcards in a GIN module. And in the code for the given section, we should be able to retrieve commands contributed for that section.
Here was my solution to this problem :
Every Presenter is assigned a unique ID (let say PresenterContext) based on where its contributed. E.g.
"Inbox:_:SignOff:_:SecondaryWorkArea" signifies a presenter in Inbox (Primary Page) > SignOff (Tab) > SecondaryWorkArea.
Commands are contributed in GIN module using such presenter context IDs. And same command can be contributed to multiple Presenter Contexts. Here are different examples of Contributions for a command :
The requirement here was to be able to contribute commands in one or more sections mentioned above using wildcards in a GIN module. And in the code for the given section, we should be able to retrieve commands contributed for that section.
Here was my solution to this problem :
Every Presenter is assigned a unique ID (let say PresenterContext) based on where its contributed. E.g.
"Inbox:_:SignOff:_:SecondaryWorkArea" signifies a presenter in Inbox (Primary Page) > SignOff (Tab) > SecondaryWorkArea.
Commands are contributed in GIN module using such presenter context IDs. And same command can be contributed to multiple Presenter Contexts. Here are different examples of Contributions for a command :
private String m_contributions[] = new String[] { "",
"Inbox",
"Inbox:_:PrimaryWorkArea",
"Inbox:_:SecondaryWorkArea",
"Inbox:_:SecondaryWorkArea:_:SignOff",
"Inbox:_:SignOff:_:PrimaryWorkArea",
"Inbox:_:SignOff:_:SecondaryWorkArea",
"Inbox:_:SignOff",
"Inbox:_:MyTasks",
"Inbox:_:MyTasks:_:SignOff",
"Inbox:_:MyTasks:_:SecondaryWorkArea",
"Inbox:_:MyTasks:_:SecondaryWorkArea:_:SignOff",
"PrimaryWorkArea",
"SecondaryWorkArea",
"SecondaryWorkArea:_:SignOff",
"SignOff",
"ShowObject:_:SecondaryWorkArea:_:Attachments"
};
And here is a test method with desired output :
@Test
public void testGetClosestMatchingContext()
{
String input = "Inbox:_:AllTasks:_:SecondaryWorkArea:_:Attachments";
String matchingContribution = .getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "Inbox:_:SecondaryWorkArea", matchingContribution );
input = "Inbox:_:AllTasks:_:SignOff";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "Inbox:_:SignOff", matchingContribution );
input = "MyTasks:_:SecondaryWorkArea:_:SignOff";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "SecondaryWorkArea:_:SignOff", matchingContribution );
input = "AllTasks:_:SecondaryWorkArea:_:SignOff";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "SecondaryWorkArea:_:SignOff", matchingContribution );
input = "Inbox:_:AllTasks:_:SecondaryWorkArea:_:SignOff";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "Inbox:_:SecondaryWorkArea:_:SignOff", matchingContribution );
input = "ShowObject:_:Attachements";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertNull( matchingContribution );
input = "ShowObject:_:PrimaryWorkArea:_:Attachements";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "PrimaryWorkArea", matchingContribution );
input = "ShowObject:_:SignOff";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "SignOff", matchingContribution );
input = "ShowObject:_:SecondaryWorkArea:_:SignOff";
matchingContribution = getClosestMatchingContext( input, m_contributions );
Assert.assertEquals( "SecondaryWorkArea:_:SignOff", matchingContribution );
}
Here is the Algo to get closest matching context for an input presenter context:
public String getClosestMatchingContext( String targetContext, String[] contributedContextIDs )
{
String tokens[] = split( targetContext );
String[] lastMatch = contributedContextIDs;
String result = null;
int index = 0;
for( String token : tokens )
{
String[] currentMatch = startingWithToken( token, lastMatch, index );
if( currentMatch != null && currentMatch.length > 0 )
{
String match = endingWithToken( token, currentMatch );
if( match != null )
{
result = match;
}
if( result != null )
{
lastMatch = currentMatch;
index++;
}
}
}
return result;
}
/**
* Splits the context id into separate tokens
*
* @param contextID presenter context ID
* @return array of tokens
*/
private String[] split( String contextID )
{
String[] tokens = contextID.split( IPresenterContext.ID_SEPARATOR );
if( tokens == null )
{
tokens = new String[0];
}
return tokens;
}
/**
* Finds the contributions that end with given token
*
* @param token context token
* @param contributions contributed context IDs
* @return closest matching context contribution for the specified token, else null is returned.
*/
private String endingWithToken( String token, String[] contributions )
{
assert token != null;
assert contributions != null;
String tokenWithSperator = IPresenterContext.ID_SEPARATOR + token;
for( String contribution : contributions )
{
if( contribution.equals( token ) || contribution.endsWith( tokenWithSperator ) )
{
return contribution;
}
}
return null;
}
/**
* Finds the contributions that start with specified token.
*
* @param iToken input token
* @param contributions context contributions
* @param index index in the specified contributions array to search for in the contributions array
* @return contributions starting with specified token, else an empty Array is returned.
*/
private String[] startingWithToken( String iToken, String[] contributions, int index )
{
assert iToken != null;
assert contributions != null;
assert index >= 0;
List<String> matches = new ArrayList<>();
for( String contribution : contributions )
{
String[] tokens = split( contribution );
if( tokens.length > index && tokens[index].equals( iToken ) )
{
matches.add( contribution );
}
}
return matches.toArray( new String[0] );
}
JNA : Get window handle at mouse cusor position
public static final int POINT_Y(long i)
{
return (int) (i >> 32);
}
public static final int POINT_X(long i)
{
return (int) (i & 0xFFFF);
}
public interface User32Ex extends W32APIOptions
{
User32Ex instance = (User32Ex) Native.loadLibrary("user32", User32Ex.class, DEFAULT_OPTIONS);
boolean GetCursorPos(long[] lpPoint); //use macros POINT_X() and POINT_Y() on long lpPoint[0]
HWND WindowFromPoint(long point);
}
long[] getPos = new long [1];
User32Ex.instance.GetCursorPos(getPos);
HWND hwnd = User32Ex.instance.WindowFromPoint(getPos[0]);
https://groups.google.com/forum/#!topic/jna-users/jVSfNCHMQ-M
Scrum.. make sure you get it right
http://scrumtrainingseries.com/DailyScrumMeeting/DailyScrumMeeting.htm
http://scrumtrainingseries.com/
http://scrumtrainingseries.com/
export/ copy maven project with its dependencies
Command to export a maven plugin with its
dependencies
cd to maven project / module with pom.xml
and simple run
mvn
org.apache.maven.plugins:maven-dependency-plugin:2.8:copy-dependencies
under target/dependency copies all the
dependency jars
Character Encoding : JSON, Java
http://stackoverflow.com/questions/17487058/how-to-solve-json-parse-bad-control-character-in-string-literal-in-this-code
GWT's JSONParser.parseStrict() doesn;'t allow control and private for use characters. So if you have input fields in your form where you take input form the user and parse it using JSON parser, make sure that you check for invalid chars as below :
if(Character.isISOControl( c ) || c >= 0xE000 && c <= 0xF8FF ))
//flag error
For java apps, you can use java.nio.charset.CharsetEncoder class to check if input stinrg is valid for your encoding.. use CharsetEncoder.canEncode() method.
GWT's JSONParser.parseStrict() doesn;'t allow control and private for use characters. So if you have input fields in your form where you take input form the user and parse it using JSON parser, make sure that you check for invalid chars as below :
if(Character.isISOControl( c ) || c >= 0xE000 && c <= 0xF8FF ))
//flag error
For java apps, you can use java.nio.charset.CharsetEncoder class to check if input stinrg is valid for your encoding.. use CharsetEncoder.canEncode() method.
Wanna Test Natives from Java ?
For quite a few years I have been working on an application which is native UI (written in C++) embedded in Eclipse SWT / Swing panels / composites. While I wrote some JUnit Tests for the GUI automation using SWTBot, I could not write test cases for native UI dialogs since SWTBot doesn't support for Windows Native UI components.
While looking for some open source library / framework for the same.. I came across a nice white paper by some VMWare guy which solved my problem :
http://istabg.org/adding-capabilities-for-windows-native-gui-to-your-test-automation-framework-based-on-java/
All you need is couple of JNA libraries and a handful of classes in the white paper.
While looking for some open source library / framework for the same.. I came across a nice white paper by some VMWare guy which solved my problem :
http://istabg.org/adding-capabilities-for-windows-native-gui-to-your-test-automation-framework-based-on-java/
All you need is couple of JNA libraries and a handful of classes in the white paper.
CanIUse.com
To check for an API / html tag support on various browsers and mobile devices. E.g.
http://caniuse.com/filesystem
http://caniuse.com/filesystem
Css3 : Units Conversion
Using PTs :
PT units should only be used for printable Form CSS as they represent the DOts on the printer / papper.
Using PX:
PX unit repsent a pixel on screen and should only be used for displayable Forms only. Drawback is if user changes the base font size of the browser, your fonts will be still of same size. they won't grow along with your base font size.
Use PX for Icons, Images width / height etc as they are pixel perfect.
Using EMs :
EM units are relative and scalable along with your base font size and W3C best practices guidelines recommend using EMs whenever possible.
If you are planning to use Ems in your CSS, be aware that EMs are compounding and relative to parent element Font size.
useful Regx for conversion :
Converting PTs to PX
12 PT = 16 PX
so to convert pt to PX just multiply the PTs by 1.333. However for accuracy instead divide the PTs by 0.75
Converting for a specific style e.g. Font-Size:
To convert Pixel to EMs (16 PX = 1 EM). So divide the PXs by 16.
Here is the conversion table : http://reeddesign.co.uk/test/points-pixels.html
Centimeters to PX conversion : http://www.unitconversion.org/typography/centimeters-to-pixels-y-conversion.html
PT units should only be used for printable Form CSS as they represent the DOts on the printer / papper.
Using PX:
PX unit repsent a pixel on screen and should only be used for displayable Forms only. Drawback is if user changes the base font size of the browser, your fonts will be still of same size. they won't grow along with your base font size.
Use PX for Icons, Images width / height etc as they are pixel perfect.
Using EMs :
EM units are relative and scalable along with your base font size and W3C best practices guidelines recommend using EMs whenever possible.
If you are planning to use Ems in your CSS, be aware that EMs are compounding and relative to parent element Font size.
useful Regx for conversion :
String regexDecimal = "(\\d+(\\.\\d+)?)";
String regexPT = regexDecimal + "pt";
String regexPX = regexDecimal + "px";
String regexCSS = "\\s*:\\s*";
String convert( String line, String regX, float convertFactor, String suffix )
{
Pattern pt = Pattern.compile( regX, Pattern.CASE_INSENSITIVE );
Matcher ptMatcher = pt.matcher( line );
StringBuffer s = new StringBuffer();
while( ptMatcher.find() )
{
float i = Float.parseFloat( ptMatcher.group( 1 ) );
float f = i / convertFactor;
String em = String.format( "%.3f", Float.valueOf( f ) ) + suffix;
ptMatcher.appendReplacement( s, em );
}
ptMatcher.appendTail( s );
return s.toString();
}
Converting PTs to PX
12 PT = 16 PX
so to convert pt to PX just multiply the PTs by 1.333. However for accuracy instead divide the PTs by 0.75
String input = "font-size: 12pt; hello 12pt; 12.0pt convert another14pt16px8px:"; convert( input, regexPT, (float) 0.75, "px" );
Converting for a specific style e.g. Font-Size:
To convert Pixel to EMs (16 PX = 1 EM). So divide the PXs by 16.
String line = "font-size: 12pt;";
convertCSS( line, "font-size:" + regexCSS,
regexPX, (float) 16.0, "em" );
String convertCSS( String line, String style, String regX, float convertFactor, String suffix )
{
Pattern pt1 = Pattern.compile( "(" + style + ")", Pattern.CASE_INSENSITIVE );
Matcher pt1Matcher = pt1.matcher( line );
boolean regXMatches = false;
StringBuffer s = new StringBuffer();
while( pt1Matcher.find() )
{
Pattern pt = Pattern.compile( regX, Pattern.CASE_INSENSITIVE );
Matcher ptMatcher = pt.matcher( line );
while( ptMatcher.find() )
{
float i = Float.parseFloat( ptMatcher.group( 1 ) );
float f = i / convertFactor;
String em = String.format( "%.3f", Float.valueOf( f ) ) + suffix;
ptMatcher.appendReplacement( s, em );
}
ptMatcher.appendTail( s );
regXMatches = true;
}
if( !regXMatches )
{
pt1Matcher.appendTail( s );
}
return s.toString();
}
Here is the conversion table : http://reeddesign.co.uk/test/points-pixels.html
Centimeters to PX conversion : http://www.unitconversion.org/typography/centimeters-to-pixels-y-conversion.html
safely-casting-long-to-int in Java
The window 'handle' parameter is normally a 'long' but it was seen as an 'int' on at least some Linux RedHat boxes. So if you have a native API to set window handle, you can use the following code to convert long to int so that the API signature does not vary between platforms:
public int longToInt(Long longVariable)
{
try{
return Integer.valueOf(longVariable.toString());
}
catch(IllegalArgumentException e) {
Log.e(e.printstackstrace());
}
}
http://stackoverflow.com/questions/1590831/safely-casting-long-to-int-in-java
Its Joel On Software
Make better software series : http://training.fogcreek.com/
All the Articles by Joel : http://www.joelonsoftware.com/backIssues.html
Guerrilla Interview Guide ( One of the Classics of Joel ): http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html
Few Articles from Top 10 by Joel :
Mistakes you should avoid when writing software http://www.joelonsoftware.com/articles/fog0000000069.html
Building your own Company.. read this first : http://www.joelonsoftware.com/articles/fog0000000056.html
All the Articles by Joel : http://www.joelonsoftware.com/backIssues.html
Guerrilla Interview Guide ( One of the Classics of Joel ): http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html
Few Articles from Top 10 by Joel :
Mistakes you should avoid when writing software http://www.joelonsoftware.com/articles/fog0000000069.html
Building your own Company.. read this first : http://www.joelonsoftware.com/articles/fog0000000056.html
Nice Article on What is granularity?
Here is a good article that describes how to determine right level of granularity in your code / design
Granularity is the extent to which a system is broken down into it’s behavioral entities. Coarse-grained entities tend to be richer in behavior than fine-grained entities. Because coarse-grained entities do more, they tend to be larger than fine-grained entities, and are easier to use. In general, a fine-grained entity is more reusable than a coarse-grained entity because it does a little bit less. This is pretty natural. If it does less, then it’s more likely to apply across a wider variety of usage scenarios. I spent some time discussing this tension in Reuse: Is the Dream Dead?
Unfortunately, while there are a lot of patterns, principles, and idioms that offer really good guidance surrounding many aspects of architecture and design, there isn’t a lot of guidance surrounding granularity. Let’s start with a really simple example that illustrates the challenge.
A Really Simple Example
Consider the following save method:
As can be seen, the save method invokes the validate method before saving the information to the database. Generally speaking, this makes saving the information easier (and possibly safer) since whatever invokes save doesn’t have to worry about validating. And this seems to be a good thing.
Yet, what happens when a new consumer of this method wants to save the information but apply a slightly different set of rules than the current validation method provides? Here’s where the existing save method is simply too coarse-grained. It does just a little bit too much. We’ve made it easier to use, but also less reusable.
There are quite a few variations I could go with at this point to improve the design. One would be to make the validate method public, and have clients invoke the validate method before invoking save. Then, if validate didn’t do what it was supposed to, clients could always opt out of invoking the validate method and apply their own validation before invoking save. Certainly less safe though because validation is optional.
Yet another alternative might be the following:
This seems to offer the best of both worlds. I now require a Validator be passed in before I can save, and if the Validator is an interface, I can certainly allow clients to pass in the Validator they need. Of course, I could easily pass in a NOP validator.
Whatever…there are lots of different options, and each has their own set of tradeoffs. I’d like to avoid extensive debate surrounding which approach to save and validation is best and instead focus on the main point here, which is that achieving the right level of granularity for an entity can be very challenging. It requires more than just looking at the problem from a code level viewpoint and demands that we possess contextual information that will help us answer the question, “What’s the right level of granularity?”
For further reading :
http://techdistrict.kirkk.com/2010/04/22/granularity-architectures-nemesis/
Granularity is the extent to which a system is broken down into it’s behavioral entities. Coarse-grained entities tend to be richer in behavior than fine-grained entities. Because coarse-grained entities do more, they tend to be larger than fine-grained entities, and are easier to use. In general, a fine-grained entity is more reusable than a coarse-grained entity because it does a little bit less. This is pretty natural. If it does less, then it’s more likely to apply across a wider variety of usage scenarios. I spent some time discussing this tension in Reuse: Is the Dream Dead?
Unfortunately, while there are a lot of patterns, principles, and idioms that offer really good guidance surrounding many aspects of architecture and design, there isn’t a lot of guidance surrounding granularity. Let’s start with a really simple example that illustrates the challenge.
A Really Simple Example
Consider the following save method:
public class Person {
private String firstName;
private String lastName;
private String ssNumber;
public void save() {
ValidationErrors errors = this.validate();
if (errors != null) {
//handle the errors somehow
} else {
//save to the database.
}
}
private ValidationErrors validate() {
//perform validation
}
}
As can be seen, the save method invokes the validate method before saving the information to the database. Generally speaking, this makes saving the information easier (and possibly safer) since whatever invokes save doesn’t have to worry about validating. And this seems to be a good thing.
Yet, what happens when a new consumer of this method wants to save the information but apply a slightly different set of rules than the current validation method provides? Here’s where the existing save method is simply too coarse-grained. It does just a little bit too much. We’ve made it easier to use, but also less reusable.
There are quite a few variations I could go with at this point to improve the design. One would be to make the validate method public, and have clients invoke the validate method before invoking save. Then, if validate didn’t do what it was supposed to, clients could always opt out of invoking the validate method and apply their own validation before invoking save. Certainly less safe though because validation is optional.
Yet another alternative might be the following:
public class Person {
private String firstName;
private String lastName;
private String ssNumber;
public void save(Validator validator) {
ValidationErrors errors = validator.validate();
if (errors != null) {
//handle the errors somehow
} else {
//save to the database.
}
}
}
This seems to offer the best of both worlds. I now require a Validator be passed in before I can save, and if the Validator is an interface, I can certainly allow clients to pass in the Validator they need. Of course, I could easily pass in a NOP validator.
Whatever…there are lots of different options, and each has their own set of tradeoffs. I’d like to avoid extensive debate surrounding which approach to save and validation is best and instead focus on the main point here, which is that achieving the right level of granularity for an entity can be very challenging. It requires more than just looking at the problem from a code level viewpoint and demands that we possess contextual information that will help us answer the question, “What’s the right level of granularity?”
For further reading :
http://techdistrict.kirkk.com/2010/04/22/granularity-architectures-nemesis/
Unix Tips
To find out if processor is 32 bit or 64 bit on Unix :
$ getconf -a | grep KERN
KERNEL_BITMODE: 64
http://www.stata.com/support/faqs/win/64bit.html
$ getconf -a | grep KERN
KERNEL_BITMODE: 64
http://www.stata.com/support/faqs/win/64bit.html
eclipse ui activity to hide everything except for a view in a plugin
If you want to hide a view (or any other UI component in a plugin) from user, you can write a ui.activities extension as below :
This activity is by default disabled, so it hides the view by default. Now if you want to show the view programatically, you can do so by enabling this activity as below :
.
So far so good. if you want to hide everything except a particular UI component, you can do so by specifying negative lookbehind ("?<!") regular expression in your activity pattern.
Lets say you want to hide all the ui components in your plugin execpt for 2 views in your plugin :
Plugin ID : com.myapp.myappid
View1 ID : com.myapp.myviewid1
View2 ID : com.myapp.myviewid2
Here is the snippet to Hide everything in plugin com.myapp.myappid except for view1 (com.myapp.myviewid1) & view2 (com.myapp.myviewid2).
Note that you have to use "?<!" instead of "?<!" as in xthml '<' character is not allowed.
<extension point="org.eclipse.ui.activities">
<activity description="Turn off My View"
id="com.myapp.myactivity" name="MyViewActivity">
</activity>
<activityPatternBinding activityId="com.myapp.myactivity"
pattern="com\.myapp\.mypluginid/com\.myapp\.myviewid">
</activityPatternBinding>
</extension>
This activity is by default disabled, so it hides the view by default. Now if you want to show the view programatically, you can do so by enabling this activity as below :
IWorkbenchActivitySupport workbenchActivitySupport =
PlatformUI.getWorkbench().getActivitySupport();
IActivityManager activityManager =
workbenchActivitySupport.getActivityManager();
Set<String> enabledActivityIds =
new HashSet<String>( activityManager.getEnabledActivityIds() );
enabledActivityIds.add( "MyViewActivity" ) );
.
So far so good. if you want to hide everything except a particular UI component, you can do so by specifying negative lookbehind ("?<!") regular expression in your activity pattern.
Lets say you want to hide all the ui components in your plugin execpt for 2 views in your plugin :
Plugin ID : com.myapp.myappid
View1 ID : com.myapp.myviewid1
View2 ID : com.myapp.myviewid2
Here is the snippet to Hide everything in plugin com.myapp.myappid except for view1 (com.myapp.myviewid1) & view2 (com.myapp.myviewid2).
Note that you have to use "?<!" instead of "?<!" as in xthml '<' character is not allowed.
<extension point="org.eclipse.ui.activities">
<activity description="Turn off My View"
id="com.myapp.myactivity" name="MyViewActivity">
</activity>
<activityPatternBinding activityId="com.myapp.myactivity"
pattern="com\.myapp\.mypluginid/.*(?<!myviewid1|myviewid2)">
</activityPatternBinding>
</extension>
Subscribe to:
Posts (Atom)