Setting proxy for SBT environment

I was required to setup 'Apache Spark Scala' environment with SBT in a network that was protected with firewall settings.

As those aware about SBT know, it is a powerful build tool that does most of the activities in background with very little user interventions. It downloads various files listed as dependency over internet and saves them in below local repository (in Windows 7):

C:\Users\<uname>\.ivy2

As my network was behind proxy, I need to instruct SBT to download the files over proxy network.

To let SBT download via proxy, proxy server details need to be updated in below location:

C:\Program Files (x86)\sbt\conf\sbtconfig.txt

Commands to be update in above file are:

-Dhttp.proxyHost:<proxy_ip>

-Dhttp.proxyPort:<proxy_port>

-Dhttps.proxyHost:<proxy_ip>

-Dhttps.proxyPort:<proxy_port>

Please let me know if you face any challenges over here.

How to build Kibana from Source!!

Dear All,

In this blog post, I am going to summarize how I built Kibana - open source, browser-based analytics dashboard, from source code.

Environment: Linux RHEL from AWS

Version: 4.4 (At the time of writing this article, master version of Kibana is 5.0. But my requirement is to build it in 4.4 version.)

Visit the github page of Kibana here and select the branch version of 4.4. You will be landing here.

This page provides a detailed information about building Kibana, but I had to do few more tweaks - I will be explaining them in this blog.

Steps:

1. Clone the version 4.4 from Kibana source.
git clone -b 4.4 https://github.com/elastic/kibana.git
cd kibana

2. Check the version of npm from the file .node-version and install the respective npm using below command.
nvm install "$(cat .node-version)"

3. Install the dependencies and start the Kibana and Elasticsearch using below commands:

(Please note that below commands would require lot of dependencies to be installed in linux machine. Please use 'Yum' installer for this purpose.)

npm install
npm run elasticsearch
npm start

Hope everything is fine till this point. In the coming steps, I will be explaining about the build process.

4. Navigate to the OS packages section of the page and execute below commands:
gem install pleaserun
gem install fpm
npm run build:ospackages

5. When I ran the above command, system displayed an error as below related to some path issue:

Invalid package configuration: Cannot package the path '/home/ec2-user/kibana_src1/kibana_0711/build/services/systemd/lib', does it exist? {:level=>:error}

Upon googling, I found that this issue is related to pleaserun package update as described here:

https://github.com/elastic/kibana/issues/7265

Initial suggestion was to downgrade pleaserun package, but later it also did not work - as fpm package internally uses the latest version of pleaserun.

So, I had to manually update the 'tasks/build/osPackages.js' file as per the fix provided here:

https://github.com/elastic/kibana/pull/7276/files

6. Open the file 'tasks/build/osPackages.js' using vi editor and replace the below code from:

      const systemd = servicesByName.systemd.outputDir + '/lib/=/lib/';

      const systemd = servicesByName.systemd.outputDir + '/etc/=/etc/';

7. Above change would resolve the error associated with file path.

8. But upon rebuilding, I was getting another error related to 'shasum' package.

What I did is, I copied the content of 'shasum.js' file from master branch and pasted it in 'tasks/build/shasum.js'. Also, have modified below line from

var cmd = /^win/.test(platform) ? 'sha1sum ' : 'shasum ';

to

var cmd = /^win/.test(platform) ? 'sha1sum ' : 'sha256sum ';

This is how updated shasum.js file was looking like:

var { promisify } = require('bluebird');
var readdir = promisify(require('fs').readdir);
var exec = promisify(require('child_process').exec);
var platform = require('os').platform();
var cmd = /^win/.test(platform) ? 'sha1sum ' : 'sha256sum ';

module.exports = function (grunt) {
  grunt.registerTask('_build:shasums', function () {
    var targetDir = grunt.config.get('target');

    // for when shasums is run but archives and ospackages was not
    grunt.file.mkdir(targetDir);

    readdir(targetDir)
    .map(function (archive) {
      // only sha the archives and packages
      if (!archive.match(/\.zip$|\.tar.gz$|\.deb$|\.rpm$/)) return;

      return exec(cmd + archive + ' > ' + archive + '.sha1.txt', {
        cwd: targetDir
      });
    })
    .nodeify(this.async());
  });

};

9. After making above modification, build ran like a charm and I have a working copy of Kibana available in various os under 'build' folder.

Please let me know if you need any additional information.

Authentication in Rest API using Apache HttpClient

In this post, I am going to share my experience related to Basic authorization of password protected rest api using Apache HttpClient.

Let me post the code that does the above activity:

                String username = "<uname>";
                String password = "<pwd>";
String domainURL = "com.test.domain";

String restPwdProtectedURL = "com.test.domain/password/protected";

String enc = username + ":" + password;

DefaultHttpClient client = new DefaultHttpClient();
HttpResponse httpResponse;
HttpGet request = new HttpGet(restPwdProtectedURL);

request.addHeader("Authorization", "Basic " + new BASE64Encoder().encode(enc.getBytes()));
   httpResponse = client.execute(request);

Jars required:
1. apache-httpcomponents-httpclient.jar
2. apache-httpcomponent-httpcore.jar
3.commons-logging-1.2.jar
4. sun.misc.BASE64Decoder.jar

This is a simple program, so it does not require much explanation :-) Please let me know if there are any queries.

Weblogic - Deployment of Java EE based web application

Hello Everyone,

In this post, I am going to explain about my learning related to deployment of Java EE based web application in Oracle Weblogic server.

There are so many tutorials out there, but when I was trying to deploy - I was having tough time in finding the basic tutorial. Hence, in this tutorial - I am going to explain the process of web app deployment in weblogic using straight forward, simplistic approach.

First download Oracle Weblogic eval version from here and perform the installation.

After the installation, check whether the installation is proper by launching 'Weblogic Configuration Wizard' from here: All Programs -> Oracle -> OracleHome -> Weblogic Server 12c -> Tools
-> Configuration Wizard

If the above operation launches Configuration Wizard, then your installation is successful. If it throws any error related to JVM, then there is an issue between Weblogic installation and your Java installation.

Unfortunately, my installed Java had an issue and I had to resolve it :-( Resolving Java issue is a separate activity, I will write another blog on this separately.

Assuming all your installation issues are resolved and your Weblogic installation is successful, you need to create a domain for your weblogic server in order to carry on the application deployment.

To create a domain, use 'Domain Template Builder' wizard available under '..Weblogic Server 12c -> Tools' menu, Its a simple and straighforward process.

Now web application can be deployed in 3 ways as mentioned here:
http://www.techpaste.com/2011/12/30/web-applications-deployment-methods-weblogic-server-environment/

We will use the approach of deployment using 'Weblogic Deployer' utility.

Navigate to the location that you have specified in your domain creation. There you would find no of directories created along with your domain. Perform below steps in order to create your web app to be deployed in weblogic:

1. Create new folder for your web app under your domain directory.
2. Under the web app directory, create folders named 'WEB-INF'.
3. Under 'WEB-INF', create the folders 'lib' and 'classes'.
4. Place all your java library reference files under 'lib' folder.
5. Place all your compiled class files of the web app under 'classes' folder.
6. Place your 'web.xml' file under 'WEB-INF' folder.
7. If your web app contains any static files such as jsp, html or xhtml - place them under web app directory.

Now you have placed your web app files in the directory structure that Weblogic would understand.

Next we will be initiating the deployment of web app in Weblogic as below:

1. Under your domain directory, there will be a file 'startWebLogic.cmd'. Double click on it.
This step would start the server.
2. Next step is to initiate the deployment by invoking 'weblogic.Deployer' file present under 'weblogic.jar'.
3. In command prompt, navigate to the directory of your domain and execute the below command:

java -cp .;C:\Oracle\Middleware\Oracle_Home\wlserver\server\lib\weblogic.jar weblogic.Deployer -username weblogic -password welcome1 -adminurl t3://localhost:7001 -name <Your web app name> -source <Your web app folder name> -deploy -debug

As shown above, you are adding weblogic.jar into your java classpath and initiating the deployment.

If everything else is fine, your application would have been successfully deployed and you would be able to access it from below url:

http://localhost:7001/<Your web app folder name>

For the application containing JSF/JSP, it requires some additional steps. I will cover them separately. Please let me know if there is any clarifications.

Thanks!!

Fire javascript events manually!!!

Hi All,

In this post, I am going to explain how I fired javascript events of my webpage manually. I came across number of posts, when I googled about it.

But the solution, I am going to explain below is the most simplest one and I am unable to find further reference related to how this feature works.

Anyway, I am documenting my findings here. Once, I come across more info - I will update this post.

Lets get into the scenario:

I want to fire 'onchange' event in one of my field. But I need to change the value of the corresponding field using javascript - hence 'onchange' event will not get triggered.

So, I was looking for javascript that would trigger the 'onchange' event and I found the simple solution as below:

var sel1 = document.getElementById('testID');
sel1.value='testdata';
sel1['onchange'](sel1);

As you can observe, I am just getting the reference of the my element and updating the value of it.

Then with a single line code, I am triggering the 'onchange' event. It is as simple as that and we are done. Event will get triggered!!!

Below post elaborates this concept in bit detail. You can refer this for your reference:

http://www.guahanweb.com/2010/03/02/how-to-manually-trigger-events-in-javascript/

I used above mentioned code in Firefox and Chrome. It worked. I am not sure about older version of IE. But, I hope it should work in IE10+ version.

wkhtmltopdf - Generate pdf of password protected pages

Hi All,

In this post, I am going to explain about how to generate pdf images of password protected pages using wkhtmltopdf solution.

Wkhtmltopdf is a simple but powerful opensource solution for generating pdfs out of web page images. You can find more information about this solution here.

wkhtmltopdf uses headless browsers to launch the webpages and generate the pdf. As they are headless browsers, we cannot enter data in to their UI fields directly. Hence using wkhtmltopdf on password protected pages is a challenge.

Wkhtmltopdf asks us to handle this feature by using the command --cookie.jar (as stated in help command):

--cookie-jar <path>             Read and write cookies from and to the
                                      supplied cookie jar file

It asks the user to use above mentioned command for login operation, store the cookie information of login process in one of the jar file and use that jar file while accessing subsequent password protected pages.

Above task is not straight forward and it requires basi c understanding of how http request and response works. Whenever we perform login operation, browser sends the login information that we entered as a 'http post' parameter to the server. Server, in turn, validates the credentials and authenticates the user (if the credentials are valid) by sending the confirmation in cookie.

So, lets see how this process works with a demo site 'http://demo.testfire.net'. We will login to this website using wkhtmltopdf and we will generate the pdf of password protected pages with it.

At first, we need to understand what are all the post parameters concerned with the login operation of our website. We can use 'Tamper Data' plugin of Firefox to identify the post parameters.

Start the 'Tamper Data' and perform the login operation. You can find the parameters that are posted when performing login operation. Below screenshot shows the 'post' parameters of 'demo.testfire.net' captured using Tamper Data:

As you can observe, username and password are not the only fields, and we have additional post parameters as well (In another website, I had 4 parameters corresponding to login operation). Please note the values 'Post Parameter Name' and 'Post Parameter Value' of all the parameters.

Now we need to make a request to login page of our application with all the post parameters and should save the generated 'cookie' information in one jar. Then we need to make request for password protected pages with the cookie jar to generate pdf.

Lets see the commands of wkhtml to perform above operations:

wkhtmltopdf --cookie-jar my.jar --post uid jsmith --post passw demo1234 --post btnSubmit Login http://demo.testfire.net/bank/login.aspx demo.pdf

Above command, makes a request to login page and pass the post parameters with the '--post' command as mentioned. Now wkhtmltopdf will make a post request from login page and would save the cookie details in 'my.jar'.

Now we are done!!!

Hereafter, make every other request to password protected pages (as below) with the cookie jar and wkhtmltopdf would awesomely generate the pdf of the requested pages.

wkhtmltopdf --cookie-jar my.jar http://demo.testfire.net/bank/main.aspx demo1.pdf

Thank you!!!

Java Reflections on inner classes

Java Reflections is another powerful feature of Java that enables you to inspect the contents of java classes in run time without even knowing the name of the class.

Consider this scenario that you have an access to java object and you want to read the following properties of the object:

1. Object's class name
2. Details of the fields in the class
3. Details of the methods present in classes and etc.

Java reflections allows you to perform the above mentioned activities with its powerful feature and you can find more details about it here.

Consider another scenario, that you have an access to the object that is an instance of inner class of another class. Now you want to perform the same activities that are listed above on the inner class.

In this blog, I am going to explain how to inspect the fields and properties of inner classes using Java reflections.

Say, you are having access to the object (innerObj), which is an instance of the inner class 'innerClass' (say, its enclosing class is com.test.outerClass').

Whenever you fetch the name of the object innerObj with below statement

                   innerObj.getClass().getName();

it will return the string value as 'com.test.outerClass$innerClass'

With this value as a reference - we can obtain the name of the outer class, as below:

                String parent = innerObj.getClass().getName().split("\\$")[0];

From then on, we can get the access to inner class details as below:

Class<?> enclosingClass = Class.forName(parent);
Object enclosingInstance = enclosingClass.newInstance();

Class<?> innerClass = Class.forName(myObj.getClass().getName());
Constructor<?> ctor = innerClass.getDeclaredConstructor(enclosingClass);
Object innerInstance = ctor.newInstance(enclosingInstance);

Field[] fields = innerClass.getDeclaredFields();

Now, we can perform various operations as detailed in the Java reflections on 'innerClass' and inspect the various attributes of inner classes.