Jetty 9.3 HTTP2 dependency issue with Ubuntu 14.04


The other day I was toying around with HTTP2 support and Jetty 9.3 on my Ubuntu Server 14.04 VM install. I had mostly followed the helpful instructions in the Jetty: The Definitive Reference on setting up the application as a web server with support for http, https, deploy modules. But whenever I tried to implement HTTP2 support Jetty would fail with an unhelpful Java error dump.

java -jar /opt/jetty/start.jar --add-to-startd=http2

org.elecipse.jetty.start.graph.GraphException: Missing referenced dependency: alpn-impl/alpn-1.8.0_45-internal
org.eclipse.jetty.start.graph.GraphException: Missing referenced dependency: alpn-impl/alpn-1.8.0_45-internal
at org.eclipse.jetty.start.graph.Graph.selectNodes(Graph.java:470)
at org.eclipse.jetty.start.graph.Graph.selectNode(Graph.java:447)
at org.eclipse.jetty.start.graph.Graph.selectNode(Graph.java:415)
at org.eclipse.jetty.start.Main.processCommandLine(Main.java:302)
at org.eclipse.jetty.start.Main.main(Main.java:74)

Usage: java -jar start.jar [options] [properties] [configs]
java -jar start.jar --help # for more information

The Jetty log files and Google were equally unhelpful in finding a solution. But after some painful trial and error I worked out a fix.

Jetty’s implementation of HTTP2 requires ALPN support so it can communicate with TLS handshake. Java 8 currently does not support ALPN and so Jetty implements a workaround extension for various versions of OpenJDK.

The Java error is actually telling you that Jetty module alpn-impl, a requirement for http2 is looking for a file dependency that doesn’t exist. The module calling for the dependency can be found at modules/alpn.mod in your jetty home. A look into the file reveals the following.

cat /opt/jetty/modules/alpn.mod

[name]
alpn

[depend]
ssl
alpn-impl/alpn-${java.version}

So alpn.mod is using a constant value based on the Java version to generate the dependency name. As it turns out my install of  OpenJDK (openjdk-8-jre-headless) returns an interesting version and this is what is causing the problem.

java -version

openjdk version "1.8.0_45-internal"
OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)

When we take a look in modules/alpn-impl/ there is no such file as 1.8.0_45-internal.mod which matches the Java dependency error message we were receiving.

ls /opt/jetty/modules/alpn-impl

alpn-1.8.0_05.mod alpn-1.8.0_11.mod alpn-1.8.0_20.mod alpn-1.8.0_25.mod alpn-1.8.0_31.mod alpn-1.8.0_40.mod alpn-1.8.0_45.mod alpn-1.8.0.mod

Fortunately the fix is easy.

cd /opt/jetty/modules/alpn-impl/

sudo ln -s alpn-1.8.0_45.mod alpn-1.8.0_45-internal.mod

Now if we try to load the http2 module it works!

cd /opt/jetty

java -jar /opt/jetty/start.jar --add-to-startd=http2

ALERT: There are enabled module(s) with licenses.
The following 1 module(s):
+ contains software not provided by the Eclipse Foundation!
+ contains software not covered by the Eclipse Public License!
+ has not been audited for compliance with its license

Module: alpn
+ ALPN is a hosted at github under the GPL v2 with ClassPath Exception.
+ ALPN replaces/modifies OpenJDK classes in the java.sun.security.ssl package.
+ http://github.com/jetty-project/jetty-alpn
+ http://openjdk.java.net/legal/gplv2+ce.html

Proceed (y/N)?

Secure and harden Apache Tomcat’s SSL/TLS


Introduction

In this guide I will walk through the process of hardening HTTPS connectors used by Apache Tomcat. As unfortunately the default configuration of Ubuntu 14.04 LTS using Tomcat 7 and OpenJDK 7 are vulnerable to a number of attacks and weak encryptions.

You can test your own site’s HTTPS implementation against these weaknesses at Qualys SSL Lab SSL Server Test. With this guide we can hopefully boost a F or even a B grade up to an A grade rating.

java 7 ssl F grade

Alarmingly most default Tomcat over Java 7 HTTPS configurations usually receive an F grade due to some well known vulnerabilities that they permit. Such as the well publicised POODLE attack and the unauthenticated Diffie-Hellman man-in-the-middle key exchange attack.

Upgrading to a recent release of OpenJDK 8 will remove these vulnerabilities. But if you are not able to update Java you can still use this guide as it will improve your site’s HTTPS security.

Users of Apache HTTP Reverse-Proxy configurations should skip to the end of this article titled Apache HTTP Reverse-Proxy users. Where you’ll learn how to add a SSLCipherSuite directive into your VirtualHost configuration to harden HTTPS.

This guide is targeted towards Ubuntu 14.04 LTS but should work for other distributions.

Install OpenJDK 8 (Java 8)

I will be using a 3rd party ppa to install OpenJDK. On some setups using 3rd party PPAs could be considered a security risk so use your own discretion. Though this process will not delete existing Java installs so you can always revert back to your original install and configuration. Are PPA’s safe to add to my system…?

Check your Java version.

java -version

java version 7

Install OpenJDK 8 headless.

sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update
sudo apt-get install openjdk-8-jre-headless

add ppa openjdk

apt-get install open-jdk-8-headless

Switch your server to use your new OpenJDK 8 install.

sudo update-alternatives --config java

Select the listing that points to Java 8 OpenJDK, in my screenshot it is option 2.

/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1069 manual mode

select a java alternative

Recheck your Java version.

java -version

java version 8

Now the final step is to configure your Apache Tomcat install to use OpenJDK 8.

sudo nano -B /etc/default/tomcat7

Search for the following comment block. In my config it was found at line #12.

#JAVA_HOME=/usr/lib/jvm/openjdk-6-jdk

Add the following variable below the comment block.

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

etc-tomcat7

Restart Tomcat to use the new configuration.

sudo service tomcat7 restart

Note if you’re using Oracle’s implementation of Java JDK then you may need to install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction. This removes Oracle’s imposed restriction that limits keys to a length of 128-bits. OpenJDK does not suffer this restriction. How to install Java Cryptography Extension (JCE) unlimited strength jurisdiction policy files.

Harden your Tomcat HTTPS implementation

A base configuration of a standard HTTPS connector, yours may look different.

<Connector port="443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               keystoreFile="/etc/ssl/rsa/example.com.pfx"
               keystoreType="PKCS12"
               keystorePass="changeit"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               compression="on" 
               compressionMinSize="2048"
               compressableMimeType="text/html,text/xml,text/csv,text/css,text/javascript" />

You can discover what these various attributes do in the Tomcat 8 Configuration reference – The HTTP Connector.

The first major attribute change we need to implement is the protocol. In Tomcat 7 and below this is set to use the Java BIO Connector with the value of "HTTP/1.1" or "org.apache.coyote.http11.Http11Protocol".

Tomcat 7 users should change this attribute to use the Java NIO Connector which offers a similar functionality to the BIO (HTTP/1.1) connector with a smaller performance footprint. Tomcat 6 users should keep to the default BIO connector attribute while Tomcat 8 and later already use the NIO connector as default.

protocol="org.apache.coyote.http11.Http11NioProtocol"

The default Java 7 BIO and NIO connectors enable SSLv2 and SSLv3 protocols which are vulnerable to the POODLE attack. Connectors using OpenJDK 8+ have these insecure protocols disabled. All users of Java 7 SHOULD add the following attribute to disable this vulnerability.

sslEnabledProtocols="TLSv1, TLSv1.1, TLSv1.2"

The next attribute we’ll use to harden our setup is to manually select the encryption ciphers the connector is permitted to use when communicating with the a browser in HTTPS. Not all ciphers are equal and so I’ve compiled a list of 40 of the most secure yet compatible ciphers available in Java. This collection uses Forward Secrecy (FS) and disables weak algorithms such as the unauthorised Diffie–Hellman (DH) key exchange.

Unfortunately they break the ability to make successful HTTPS connections for many versions of Internet Explorer on Windows XP as well as on Android 2.3.7 or earlier. But for Java 7 users implementing this list is a must as the default collection used by its BIO and NIO connectors contain some weak and insecure ciphers.

	ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
	TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,
	TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,
	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
	TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,
	TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,
	TLS_DHE_DSS_WITH_AES_256_CBC_SHA,
	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
	TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,
	TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,
	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
	TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,
	TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,
	TLS_DHE_DSS_WITH_AES_128_CBC_SHA,
	TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,
	TLS_ECDH_ECDSA_WITH_RC4_128_SHA,
	TLS_ECDH_RSA_WITH_RC4_128_SHA,
	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
	TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
	TLS_RSA_WITH_AES_256_GCM_SHA384,
	TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,
	TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,
	TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,
	TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
	TLS_RSA_WITH_AES_128_GCM_SHA256,
	TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,
	TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,
	TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
	TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,
	TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,
	TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,
	TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,
	TLS_EMPTY_RENEGOTIATION_INFO_SCSVF"

Finally Java 8+ users should add the useServerCipherSuitesOrder attribute and set it to true.

useServerCipherSuitesOrder="true"

By default Tomcat will use the first acceptable cipher presented by the client browser. But often this selection is not the strongest cipher available or supported. The previous cipher attribute lists the ciphers in a preferential order of strength. So by enabling useServerCipherSuitesOrder Tomcat will probe the client using this ordered sequence until a supported cipher is matched making sure the most secure connection available is always used.

After implementing the changes my harden connector looks like this. Again yours may look different depending on which Java edition is in use.

<Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtoco" SSLEnabled="true"
	maxThreads="150" scheme="https" secure="true"
	keystoreFile="/etc/ssl/rsa/example.com.pfx"
	keystoreType="PKCS12"
	keystorePass="changeit"
	connectionTimeout="20000"
	URIEncoding="UTF-8"
	compression="on" compressionMinSize="2048"
	compressableMimeType="text/html,text/xml,text/csv,text/css,text/javascript"
	useServerCipherSuitesOrder="true"
	ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
	TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,
	TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,
	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
	TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,
	TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,
	TLS_DHE_DSS_WITH_AES_256_CBC_SHA,
	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
	TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,
	TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,
	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
	TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,
	TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,
	TLS_DHE_DSS_WITH_AES_128_CBC_SHA,
	TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,
	TLS_ECDH_ECDSA_WITH_RC4_128_SHA,
	TLS_ECDH_RSA_WITH_RC4_128_SHA,
	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
	TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
	TLS_RSA_WITH_AES_256_GCM_SHA384,
	TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,
	TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,
	TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,
	TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
	TLS_RSA_WITH_AES_128_GCM_SHA256,
	TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,
	TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,
	TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
	TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,
	TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,
	TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,
	TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,
	TLS_EMPTY_RENEGOTIATION_INFO_SCSVF" />

Finally to implement your new configuration reboot Tomcat.

sudo service tomcat7 reboot

Congratulations you’re done. Go and retest your configuration with the Qualys SSL Lab SSL Server Test and hopefully your hardened configuration will receive an A grade.

java 8 ssl A grade

XML dumps of the modified connectors examples can be found on GitHub.

Apache HTTP Reverse-Proxy users

Users of Apache HTTPD in a Reverse-Proxy configuration do not need to reconfigure Tomcat. As Apache HTTP 2.4+ lets you add a simple environment variable directive into a site’s configuration.

sudo nano -B /etc/apache2/sites-available/[your site].conf

Add the following into the configuration block.

SSLCipherSuite HIGH:!aNULL:!MD5
Listen 443

NameVirtualHost *:443
<VirtualHost *:443>
    SSLEngine On
    SSLCipherSuite HIGH:!aNULL:!MD5
    SSLCertificateFile /etc/apache2/ssl/file.pem
    ProxyPass / http://0.0.0.0:8080/
    ProxyPassReverse / http://0.0.0.0:8080/
</VirtualHost>

Learn more in the Apache HTTP documentation or how to setup a reverse-proxy.

References and sources used for this guide:

  1. How to Install OpenJDK 8 in Ubuntu 14.04 & 12.04 LTS
  2. Java Cryptography Architecture Oracle Providers Documentation for JDK 8
  3. Apache Tomcat 8 SSL/TLS Configuration HOW-TO
  4. Apache Tomcat 8 Configuration Reference The HTTP Connector

Create self-signed certificates for HTTPS with Apache Tomcat


This entry will guide through the process of creating a self-signed certificate to use on an Apache Tomcat 7 or 8 HTTPS connector. Self-signed certificates allow secure, encrypted HTTPS connections but are not certified by any trusted certificate authority. So first time client connections will receive all kinds of warnings from their web browser. Because of this they are not recommended for use in production environments but are useful for secure LAN traffic or testing HTTPS configurations.

I am using Ubuntu 14.04 LTS and Tomcat 7 but this setup should be similar for other distributions.

Create a keystore and certificates

Firstly create and secure a directory to hold our certificates for Tomcat.

sudo mkdir /etc/ssl/tomcat
sudo chown :ssl-cert /etc/ssl/tomcat
sudo chmod 755 /etc/ssl/tomcat

Next we are going to create a 2048-bit RSA key and use it to generate an X509 self-signed certificate. Note the -days argument hard codes a usable duration value into the certificate. A argument of of -days 365 means the certificate is valid for a year.

In the code examples you can replace "example" with a domain or site name of your choosing such as localhost or mydomain.com.

sudo openssl req -newkey tomcat:2048 -nodes -keyout /etc/ssl/tomcat/example.key -x509 -days 365 -out /etc/ssl/tomcat/example.crt

You’ll be prompted for some optional organisation information which you can skip by pressing [Enter] at each prompt.

Next we bundle the certificate into a PKS12 keystore so we can use it in a Tomcat BIO or NIO HTTPS connector.

sudo openssl pkcs12 -inkey /etc/ssl/tomcat/example.key -in /etc/ssl/tomcat/example.crt -export -out /etc/ssl/tomcat/example.pfx

When prompted set an Export Password and then confirm it. For my examples I will use the password changeit.

Create a Connector

Now we will create a HTTPS connector that will use your self-signed certificate.

sudo nano -B /etc/tomcat7/server.xml

Find and uncomment a connector similar to the following…

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
	maxThreads="150" scheme="https" secure="true"
	clientAuth="false" sslProtocol="TLS" />

Add to it the following attributes.

keystoreFile="/etc/ssl/tomcat/example.pfx"
keystoreType="PKCS12"
keystorePass="changeit"

Your updated connector should look similar to the following.

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
	maxThreads="150" scheme="https" secure="true"
	clientAuth="false" sslProtocol="TLS"
	keystoreFile="/etc/ssl/tomcat/example.pfx"
	keystoreType="PKCS12"
	keystorePass="changeit" />

Save your changes and exit nano. Now reboot Tomcat to apply your changes.

sudo service tomcat7 restart

Point a web browser to your Tomcat webserver but append port 8443 to the URL like so. https://192.168.1.221:8443 or https://www.example.com:8443 etc.

You should receive a warning from your browser but you can ignore these and proceed with the connection.

Back to safety
Back to safety

Congratulations you’re viewing your Tomcat served website over an encrypted HTTPS connection. If the connection over HTTPS didn’t work but you can still connect using standard HTTP. Then you best check your Tomcat logs for any configuration errors.

tail -n 100 catalina.out | less

Configure Tomcat to use port 443 (HTTPS default)

If you want to use HTTPS without the need to append :8443 to the end of the URL then we have to bind port 443 (the default HTTPS port) to Tomcat.

sudo touch /etc/authbind/byport/443
sudo chmod 500 /etc/authbind/byport/443
sudo chown tomcat7 /etc/authbind/byport/443

sudo nano -B /etc/default/tomcat7

Scroll to the end of the file and add or change …

AUTHBIND=yes

Save your changes and exit.

Edit Tomcat’s HTTPS connector to use port 443.

sudo nano /etc/tomcat7/server.xml

Change <Connector port="8443" to <Connector port="8443"

Save, exit and restart Tomcat.

sudo service tomcat7 restart

Congratulations now point a web browser to your Tomcat web server using only a https:// prefix such as. https://192.168.1.221 or https://www.example.com

Connection information
Connection information

Helpers and common usage for tar


tar is a feature rich but often confusing archiving tool most commonly used on Linux and BSD systems. One of the great benefits of tar over other more modern tools such as 7z, RAR or ZIP is that it is open source, free and platform agnostic. Even more importantly for Linux users tar archives preserve user and file permissions allowing for easy directory backups and restoration.

Rsync – Helpers and common usage


rsync is a widely used tool for synchronisation so to keep copies of a file on multiple computers the same. Because of its flexibility it has become the defacto standard on Linux and other similar systems. While newer protocols and tools such as Dropbox and BitTorrent Sync overlap with and improve on some of rsync’s capability. Rsync is still relevant today both for syncing and for other function such as local directory duplication and large file copying.

Backup your MySQL/MariaDB database easily using Percona XtraBackup


Over the years I have looked around for easy to use MySQL database backup tools that I can quickly and reliably automate. I’ve tried many unsatisfactory solutions but finally I have found something I am happy with, Perscona’s XtraBackup.

Unfortunately for Windows users this is a Linux only solution.

Percona is an enterprise focused database company with the aim of delivering their clients faster and more reliable MySQL solutions. Fortunately for the rest of us they also believe in open source and offer a number of free tools suitable for all MySQL and MariaDB users in addition to their own MySQL server platform.

Below is a Markdown formatted cheat sheet that I wrote which enables Linux users to easily back-up their MySQL compatible databases. With simple instructions on how to create, restore and automate backups. While the instructions were written with Ubuntu/Debian in mind they should be useful for all other distributions with the usual adjustments.

Percona XtraDB Software page.
Percona XtraDB downloads (not needed for Ubuntu/Debian users).

Gist bengarrett / percona-xtrabackup.md

Make your Windows Command Prompt, Linux-like


Updated on 2015, March 25. Added information on rsync, OpenSSH clients and updated some links plus io.js.

In this entry I will explain my method of improving the Windows Command Prompt and its command-line. To turn it from a crude shell prompt into an ANSI coloured interface with a useful set of shell commands that can commonly be found on Linux and Mac terminals. We will also integrate a number of optional popular programming languages and features. This will require some existing knowledge of Windows and the command-line. But before we get into that let’s go on a trip down memory lane for some context.

Historical Context

(you can skip this if you wish)

In the ancient days of personal computing during the mid to late 1970s and the early days of Microsoft. There was a popular text only operating system called CP/M. By today’s standards it was rather crude and rudimental but at the time it was popular in business due to its relative ease of use.

Years later when Microsoft delivered to IBM its first PC operating system. It became apparent that the Microsoft operating system MS-DOS was a derivative of this earlier CP/M system.

I bring this up because to this day the default and most common Windows command-line the Command Prompt cmd.exe. Can trace its lineage back to this 1970s operating system from the defunct Digital Research.

After Windows became popular Microsoft merged both its MS-DOS shell and Windows overlay together into a operating system known as Windows 95. So for the longest time until the release of Windows 2000 and XP. Most consumers used a version of Windows that had a MS-DOS Command Prompt as a critical component of the operating system.

With Windows 2000, XP and later operating systems the MS-DOS functionality was pushed to the wayside and has been neglected ever since. Since the release of PowerShell in 2006 this neglect deepened. In fact the current iteration of the Command Prompt in Windows 8.1 is not much different to that which was found in Windows 95 nearly twenty years ago.

Windows 95 MS-DOS prompt
Windows 95 MS-DOS prompt
Windows 8.1 Command Prompt
Windows 8.1 Command Prompt

Windows Command Prompt

So let us start with the default Command Prompt.

In Windows use the Run feature (usually this can be done with the Windows key + R) and type in CMD. This should launch the default Command Prompt application which should be similar to the prompt above.

The Command Prompt CMD.EXE is the application that we use to input and receive feedback from the command-line shell. The shell is a collection of programs that allows you to interact with the Windows filesystem and operating system.

This is an important distinction because we are going to replace the Command Prompt but not the underlying command-line shell.

Replace the Command Prompt CMD.EXE with ConsoleZ

First create a directory on the root of your hard drive to contain all your command-line tools. I will use c:\terminal but some other directory name suggestions could be c:\prompt or c:\cp or c:\shell. I do recommend that you keep the directory name short and use a complete word without any spaces.

Created c:\terminal directory
Created c:\terminal directory

I am going to use an open sourced Command Prompt replacement application called ConsoleZ by Christophe Bucher.  Which itself is a fork of a more popular replacement application known as Console2 by Marko Bozikovic.

Go to the ConsoleZ download page and select the x86 for the 32-bit or amd64 for the 64-bit edition. Save the download to your c:\terminal directory and unzip it.

Download ConsoleZ
Download ConsoleZ

For all the rest of the article I am going to use and reference the 64-bit edition.

ConsoleZ is a portable application which means it does not need to be installed to work. This means you can easily archive and backup the whole c:\terminal directory into a zip file or copy it to a USB stick or a secondary computer without losing any of your customisations or settings.

ConesoleZ
ConesoleZ

Customize ConsoleZ

With ConsoleZ you have an alternative Command Prompt. I do the following customisations to give it a less cluttered look but you can pick and choose which ever you want.

Menu > View > Toolbar off
Menu > View > Status Bar off

ConsoleZ View adjustments
ConsoleZ View adjustments

Menu > Edit > Settings > Console

Change the Windows size Rows value if you want a longer command-line interface by default.

Change the Buffer size Row value if you want to keep a larger backlog of command-line output. By default 500 lines of text are stored to memory but increasing this value is useful if you want to display and scroll through large logs or text files. I usually increase it to its maximum value of 32766.

Windows and Buffer sizes
Windows and Buffer sizes

Menu > Edit > Settings > Appearance > Styles
Window Transparency > Alpha > Active Window

I set the application transparency as a low value of 20 but Windows 8+ users may not wish to use this aesthetic effect.

Customize the ConsoleZ font

Next up is I adjust the font selection which can be an important personal choice depending on your requirements and screen size. The default font is Courier New at a size of 10.

Menu > Edit > Settings > Appearance > Font

I change Name: to use Lucida Console in regular with a Size: of 10. But play around until you find a font and size combination that you like.

Font Selection
Font Selection

Command and Batch Scripts

In c:\terminal create a new subdirectory named cmd-scripts ie c:\terminal\cmd-scripts. This will contain my custom text based scripts that use the file extension .bat or .cmd. Modern Windows scripts use the .cmd (Command) extension while the .bat (Batch) is a MS-DOS legacy convention that functions exactly the same.

Colourise Your Command-line with ANSI Colour

In the 1980s and 1990s Windows and MS-DOS supported a widely used feature called ANSI escape sequences that allowed additional functionality such as colour text. But this support was dropped in more recent Windows editions. So I will show you how to re-implement ANSI escape sequence support using Jason Hood’s excellent ANSICon shell overlay.

Visit the ANSICon website at https://github.com/adoxa/ansicon/releases. Download the file ansi166.zip and save it to c:\terminal.

Now if you run the command type "ANSI Prompt Colours.txt" you should see a whole lot of garbled text dumped to the ConsoleZ window. These are ANSI escape sequences combined with plain text that Windows does not know how to interpret.

ANSI Esc garbled
ANSI Esc garbled

Download then save ANSICon to your c:\terminal directory and unzip it into its own ansicon directory. I keep it in c:\terminal\ansicon.

Use notepad or notepad++ to create a new file called shell-colour.cmd. Copy and save the following code hosted on my Github account to the shell-colour.cmd file.

If you are using 32-bit Windows or 32-bit ConsoleZ you should replace the following code c:\terminal\ansiconx64\ansicon.exe with c:\terminal\ansiconx86\ansicon.exe.

Back in ConsoleZ we will set the shell-colour.cmd as our default shell. This is a hack that loads ANSICon in addition to the default cmd.exe shell to give us ANSI escape sequence support without replacing the underlying shell.

Menu > Edit > Settings > Console

Under Shell: add c:\terminal\cmd-scripts\shell-colour.cmd

Console Settings shell-colour
Console Settings shell-colour

Now load a new tab.

Menu > File > New Tab > Console 2

Or relaunch ConsoleZ. In c:\terminal display the ANSI Prompt Colours.txt using the following commands.

cd terminal
type "ANSI Prompt Colours.txt"

If you see coloured text congratulations you now have ANSI escape sequence support.

ANSI Esc working
ANSI Esc working

Customise and Colourise the Prompt

The default Windows text input prompt only lists the active drive and current directory. As a frequent Linux Bash shell user I like a bit more flare and information to my prompt.

Use notepad or notepad++ to open your existing shell-colour.cmd file in c:\terminal\cmd-scripts. Copy and save the following code hosted on Github to the shell-colour.cmd file.

Notepad++ editing shell-colour
Notepad++ editing shell-colour

This code adds a couple of new commands. The echo command displays text to your command-line while the prompt command customises the input text prompt. The strings that are wrap within percentage symbols % are environment variables that are accessible from the shell. They allow you to display tidbits of information that are stored by Windows to the computer memory. A complete list of environment variables can be found at SS64.com.

The prompt command has some rather cryptic ANSI escape sequences that introduce colours. The $E string is a prompt argument to display escape characters which conveniently is needed by the ANSI escape sequences as a trigger. The [number;number;40m is a code sequence used to trigger an effect. A list of ANSI escape effects and colours is listed on Pueblo.

Reload ConsoleZ or open up a new tab to apply the changes. You should see a more information pack and colourful input text prompt.

The first part of the prompt in green displays the USER @ DOMAIN while the second part in blue displays the active drive and path.

Colour prompt
Colour prompt

Set a Default Directory at Launch

By default ConsoleZ sets the active directory to the location of its application which in my case is c:\terminal\ConsoleZ.x64. But you can easy change this by doing the following.

Menu > Edit > Settings > Console

Set Startup dir: to the path of your choosing such as c: or you can use a Windows environmental variable such as %userprofile%.

Startup directory setting
Startup directory setting

Create a Run as Administrator tab

A neat feature of ConsoleZ is the ability to create shell tabs with different user account permissions including those of an Administrator. As there are times when less restricted access is required to interact with parts of the Windows and its settings.

In ConsoleZ do.

Menu > Edit > Settings > Tabs > Add

In the Main tab change the Title: value to something more meaningful such as Administrator Console.

Point the Icon: to C:\Windows\System32\imageres.dll and select the yellow and blue shield icon which represents Run as administrator.

Under Shell make sure the Run as current user option is selected and the Administrator checkbox is checked.

Press OK when done. You now have a new Run as administrator tab accessible from ConsoleZ.

Admin tab
Create an Administrator Console tab

Configuring Paths

To run most tools and software from the command-line the active directory has to be the same as the location of the tool.

For example if I am in C:and try to run our script shell-colour.cmd that is located in c:\terminal\cmd-scripts. The shell returns the error “’shell-colour.cmd’ is not recognized as an internal or external command, operable program or batch file”. Which basically means the shell could not find the command I was trying to run.

Shell-colour not found
Shell-colour.cmd not found

But by using the Windows environment variable called %PATH% we can add additional directories that the shell will scan in when it searches for the existence of programs.

Configuring the %PATH% variable in Windows is a bit messy using the default Windows Control Panel option. So I prefer to use a 3rd party portable tool called Rapid Environment Editor by Oleg Danilov.

Download either the 64-bit or 32-bit editions, unzip and run the editor.

Rapid Environment Editor
Rapid Environment Editor

The pane on the left contains a list of editable Windows system variables. While the pane to the right has variables that are restricted to your current Windows user account. You may notice there is a PATH= variable for both the system and the user panes.

I generally use the system PATH for my c:\terminal path links. But to modify this you need to run Rapid Editor in administrator mode.

Restart as administrator
Restart as administrator

By default Windows stores multiple values of the PATH in a single string separated by semicolons ;. Fortunately Rapid Editor allows you to list and edit each individual PATH entry.

PATH expanded
PATH expanded

To add new directories to the PATH, right-click the PATH= string in Rapid Editor and select Add value or use the Alt+Ins keyboard combination.

Add value
Add value

Press F7 or select Insert directory path… Then point it to your c:\terminal\cmd-script directory. You should now have a new entry in your PATH list. Save the changes by pressing the Save icon or using Ctrl+S.

Insert directory path
Insert directory path

Now for changes to have an effect you have to either restart ConsoleZ or open up a new tab. To test that the PATH modifications are active, issue a cd command to return to the directory root and try running the shell-colour command again. If it works then congratulations you can now run any custom scripts placed into c:\terminal\cmd-script from anywhere in your command-line.

PATH modification works
PATH modification works (yellow highlight was added by myself)

Launch Windows Notepad++ or Notepad from the Command-line

While I love using command-line shells I generally prefer a GUI when it comes to programming or editing text files. I use the following script to to edit a file from the command-line. It launches Notepad++ and opens the file supplied but you can use any text editor including Windows Notepad located in C:\Windows\System32\notepad.exe.

Download and save edit.cmd to your c:\terminal\cmd-scripts directory.

In a new session of ConsoleZ test the edit script with the following commands.

cd\terminal

edit "ANSI Prompt Colours"

Edit command
Edit command
EDIT command launches notepad++
EDIT command launches notepad++

To regain your prompt either close Notepad++ or press Ctrl+c in ConsoleZ.

Congratulations the core of this guide is complete. The remainder paragraphs are optional so you can pick and mix which topics you wish to implement. These include adding extra Windows and Linux tools; installing and running Node.js, Perl, PHP, Python or Ruby scripts from the command-line.

Useful Windows Command-line Tools

Besides the complete collection of commands and command-line programs built into Windows that you can discover at http://technet.microsoft.com/en-us/library/bb490890.aspx and http://ss64.com/nt/. There are quite a number of useful third-party command-line programs that are available for the Windows platform. I like to keep these contained in a single directory at c:\terminal\bin. Bin is an abbreviation for binary an alternative term for a program file.

Create the directory c:\terminal\bin and then using the processes covered in ‘Configuring Paths’ add c:\terminal\bin to your PATH variable. Restart ConsoleZ and so that any programs placed into c:\terminal\bin should be accessible from anywhere within your shell.

Here are a few Windows native command-line tools I recommend.

There are a number of useful tools created by NirSoft.
I personally use nircmd for use with various command scripts as well as WirelessNetConsole, bluetoothcl, whosip and whoiscl.

Microsoft’s Systeminternals also has a few command-line utilities.
AccessChk, Coreinfo, Handle, ProcDump and PsTools Suite.

For handling compressed files.
7-Zip Command Line Version.
UnRAR for Windows

ImageMagick can be used to convert and create image files.

Git for source-code management.

MySQL Utilities.
SQLite portable database tools.

Some other potential sources for standalone utilities include.

Joeware Utilities.
NoNags.

Linux/Unix Terminal Command-line Tools for Windows

As a frequent Linux Bash shell user there are a number of tools I miss on Windows. Fortunately as much of Linux is open sourced many of those tools have been ported over. Unfortunately some of these ports are horribly out of date so they should be best avoided.

There are also a couple of popular Windows open source C compilers such as Cygwin and MinGW that contain ports of Linux terminal tools. But in my opinion their use are overkill and many of the ported tools are rather old.

My favourite collection of ported Linux tools for Windows is the still current GOW (GNU On Windows) by Brent Matzelle.
It includes a large collection of GNU terminal tools and programs including cURL, gawk, grep, ls, nano, tar, vim and many more.

Download the latest release of Gow and install it to a directory of your choosing. I prefer placing it into c:\terminal\gow instead of its default directory C:\Program Files (x86)\Gow.

Gow will automatically copy all its application and configure PATH settings so you can use it straight away after a ConsoleZ reload or a new tab.

Gow setup
Gow setup
Testing Gow
Testing Gow

One command Gow does not replicate is the Bash alias ll (double L). I have created a command script that replicates that functionality using Gow’s ls command. Save ll.cmd to your c:\terminal\cmd-scripts directory to use it.

ll In terminal
ll In terminal

OpenSSH

Finally any useful terminal needs a copy of OpenSSH for remote logins. While there are many out of date variants of OpenSSH on Windows. I personally use OpenSSH for Windows ported by mls-software.com which is the most up to date port I have found. Simply download the current ‘New’ version from their website, the download link is named setupssh-6.[version].exe and run the setup installer. The installer should automatically configure the %PATH% for you.

To test OpenSSH just output its version. If it doesn’t match the version number listed on mls-software.com site (6.7 p1 as of writing) then you probably have one or more copies of OpenSSH on your system that are taking president in your %PATH%. Tools such as Cygwin or Git can have their own out of date ports of OpenSSH.

ssh -V

ssh version
ssh -V

rsync

The popular Linux synchronisation tool rsync also has a Windows port.

ITeF!x offers a commercial package of the daemon/server with a GUI client and a free edition of the command prompt client.

The free edition comes without an installer. So I just unzip it to C:\terminal\cwrsync and use Rapid Environment Editor to add a path to point to it.

To test that rsync is working.

rsync --version

rsync --version
rsync –version

Run PHP Scripts from the Windows Command-line

Download a copy of PHP For Windows from http://windows.php.net/download/.
Unzip the PHP package to a directory of your choosing. I will use c:\terminal\php.
Without any arguments PHP does not do much in the terminal. So I created a command script to display the language version as the default behaviour. Download and place the php.cmd script into your c:\terminal\cmd-script directory and make any changes if needed.

To test your PHP configuration and the script run the command php.

php.cmd
php.cmd

Download the hi-php.php command line script and run it. It will print Hello, World! to the screen and quit.

php hi-php.php

php php-hi.php
php php-hi.php

Now to associate script files with the .php extension to run under our php.cmd script you do the following commands in a Run as Administrator session of ConsoleZ.

assoc .php=PHP.File
ftype PHP.File=c:\terminal\cmd-scripts\php.cmd "%1" %*

assoc .php
assoc .php

Test your configuration and run php-hi.php.

You can refine this one step further so you don’t even have to type the .php file extension. Open the Rapid Editor and under System variables add the value .PHP to the PATHEXT variable. In a new ConsoleZ tab you should be able to run hi-php without the file extension.

php-hi without extension
php-hi without extension

Run Perl Scripts from the Windows Command-line

For Perl I use Strawberry Perl for Windows portable edition which can be downloaded from its website.
Unzip it to a directory of your choosing. I place it in c:\terminal\perl.
Download and place the perl.cmd script into your c:\terminal\cmd-script directory and make any changes if needed.

To associate script files with the .pl extension to run under our perl.cmd script you do the following commands in a Run as Administrator session of ConsoleZ.

assoc .pl=Perl.File
ftype Perl.File=c:\terminal\cmd-scripts\perl.cmd "%1" %*

Download the hi-perl.pl command line script and run it. It will print Hello, World! to the screen and quit.

You can refine this one step further so you don’t even have to type the .pl file extension. Open the Rapid Environment Editor and under System variables add the value .PL to the PATHEXT variable. In a new ConsoleZ tab you should be able to run hi-perl without including the file extension.

hi-perl.pl
hi-perl.pl

Run Python Scripts and Programs from the Windows Command-line

The default Windows download of Python 3.4+ automatically configures itself to enable you to run Python scripts and programs from the Windows command-line.

Download and install a Python edition of your choice from https://www.python.org/downloads/.

To test that Python has installed itself correctly you can download and run hi-python.py.

hi-python.py
hi-python.py

Run Ruby Scripts and Programs from the Windows Command-line

At the time of writing Ruby is a little bit behind on the Windows platform. The current build of Ruby is at edition 2.1.x, while the recommended edition to use on Windows is 1.9.x. So with this in mind I recommend using the RubyInstaller for Windows which is a self-contained Ruby install.

Run the setup program and when prompted make sure you select both checkboxes for the options below.

Add Ruby executables to your PATH
Associate .rb and .rbw files with this Ruby installation

Install and setup Ruby
Install and setup Ruby

To test your Ruby installation download and run hi-ruby.rb.

hi-ruby.rb
hi-ruby.rb

Run io.js or Node.js and Javascript from the Windows Command-line

Download and install the latest io.js release from https://iojs.org/ or its predecessor Node.js release from http://nodejs.org/. Both io.js and Node will create the required PATH variables for you. Unfortunately most Windows editions associate files with the io.js, Node and JavaScript .js file extension to the legacy Windows Script Host application. It is probably best that you do not overwrite the Windows Script Host configurations and instead run all Node and JavaScript scripts using the node or iojs command.

iojs somescriptfile.js
node somescriptfile.js

To test your Node installation download and run hi-node.js.

node hi-node.js

node hi-node.js
node hi-node.js

Creating Interactive Scripting Shells

Another great feature of ConsoleZ is because it offers separate, tabbed environments you can use it as an interactive programming tool.

In ConsoleZ do the following.

Menu > Edit > Settings > Tabs > Add

In the Main tab change the Title: value to Python 3

Point the Shell: to the language interpreter. For a default Python 3.4 installation I have it pointed to c:\python3\python.exe.

Under Main you can set the Icon: value which usually should be the same as the Shell: value.

When done, press OK and you now have a new Interactive Python shell tab accessible from ConsoleZ.

Create a Python shell
Create a Python shell

For an interactive Ruby shell you need to set the Shell: value to c:\ruby193\bin\irb.bat (or wherever your Ruby installation is located). And you probably want to set the Icon to c:\ruby193\bin\ruby.exe.

Create a Ruby shell
Create a Ruby shell

PowerShell in ConsoleZ

While this guide mostly uses the standard CMD.EXE shell. Microsoft provides an alternative, feature rich and overall better shell known as PowerShell. PowerShell is an optional download for most consumer versions of Windows and offers a more powerful and complicated command-line. Fortunately PowerShell can also easily be incorporated into ConsoleZ.

PowerShell
PowerShell

Within ConsoleZ do the following.

Menu > Edit > Settings > Tabs > Add

Add a new tab and set the Title: to PowerShell.
Set the Shell: value to %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe
Set the Icon: to use that same file path.

Create a PowerShell profile
Create a PowerShell profile

If you want you can also create an administration version of this tab. Select your newly created PowerShell tab and click the Clone button. Change the Title: to Administrator PowerShell and the Icon: to use the PowerShell icon with Administrator shield. Finally make sure the Run as current user option also has the Administrator check-box checked.

Create a PowerShell administrator profile
Create a PowerShell administrator profile

End.

Congratulations, you’re done. The GitHub repository of the code in this article https://github.com/bengarrett/devtidbits/tree/master/post_1226.

Complete.
Complete.