Create A Better Windows Command-line Prompt

Complete!

In this entry I will explain my method of improving the Windows Command Prompt and its command-line. To turn it from a crude shell prompt into an ANSI coloured interface with a useful set of shell commands that can commonly be found on Linux and Mac terminals. We will also integrate a number of optional popular programming languages and features. This will require some existing knowledge of Windows and the command-line. But before we get into that let’s go on a trip down memory lane for some context.

Historical Context

(you can skip this if you wish)

In the ancient days of personal computing during the mid to late 1970s and the early days of Microsoft. There was a popular text only operating system called CP/M. By today’s standards it was rather crude and rudimental but at the time it was popular in business due to its relative ease of use.

Years later when Microsoft delivered to IBM its first PC operating system. It became apparent that the Microsoft operating system MS-DOS was a derivative of this earlier CP/M system.

I bring this up because to this day the default and most common Windows command-line the Command Prompt cmd.exe. Can trace its lineage back to this 1970s operating system from the defunct Digital Research.

After Windows became popular Microsoft merged both its MS-DOS shell and Windows overlay together into a operating system known as Windows 95. So for the longest time until the release of Windows 2000 and XP. Most consumers used a version of Windows that had a MS-DOS Command Prompt as a critical component of the operating system.

With Windows 2000, XP and later operating systems the MS-DOS functionality was pushed to the wayside and has been neglected ever since. Since the release of PowerShell in 2006 this neglect deepened. In fact the current iteration of the Command Prompt in Windows 8.1 is not much different to that which was found in Windows 95 nearly twenty years ago.

 

Windows 95 MS-DOS prompt

Windows 95 MS-DOS prompt

Windows 8.1 Command Prompt

Windows 8.1 Command Prompt

Windows Command Prompt

So let us start with the default Command Prompt.

In Windows use the Run feature (usually this can be done with the Windows key + R) and type in CMD. This should launch the default Command Prompt application which should be similar to the prompt above.

The Command Prompt CMD.EXE is the application that we use to input and receive feedback from the command-line shell. The shell is a collection of programs that allows you to interact with the Windows filesystem and operating system.

This is an important distinction because we are going to replace the Command Prompt but not the underlying command-line shell.

Replace the Command Prompt CMD.EXE with ConsoleZ

First create a directory on the root of your hard drive to contain all your command-line tools. I will use c:\terminal\ but some other directory name suggestions could be c:\prompt\ or c:\cp\ or c:\shell\. I do recommend that you keep the directory name short and use a complete word without any spaces.

Created c:\terminal\ directory

Created c:\terminal\ directory

I am going to use an open sourced Command Prompt replacement application called ConsoleZ by Christophe Bucher.  Which itself is a fork of a more popular replacement application known as Console2 by Marko Bozikovic.

Go to the ConsoleZ download page and select the x86 for the 32-bit or amd64 for the 64-bit edition. Save the download to your c:\terminal\ directory and unzip it.

Download ConsoleZ

Download ConsoleZ

For all the rest of the article I am going to use and reference the 64-bit edition.

ConsoleZ is a portable application which means it does not need to be installed to work. This means you can easily archive and backup the whole c:\terminal\ directory into a zip file or copy it to a USB stick or a secondary computer without losing any of your customisations or settings.

ConesoleZ

ConesoleZ

Customize ConsoleZ

With ConsoleZ you have an alternative Command Prompt. I do the following customisations to give it a less cluttered look but you can pick and choose which ever you want.

Menu > View > Toolbar off
Menu > View > Status Bar off

ConsoleZ View adjustments

ConsoleZ View adjustments

Menu > Edit > Settings > Console

Change the Windows size Rows value if you want a longer command-line interface by default.

Change the Buffer size Row value if you want to keep a larger backlog of command-line output. By default 500 lines of text are stored to memory but increasing this value is useful if you want to display and scroll through large logs or text files. I usually increase it to its maximum value of 32766.

Windows and Buffer sizes

Windows and Buffer sizes

Menu > Edit > Settings > Appearance > Styles
Window Transparency > Alpha > Active Window

I set the application transparency as a low value of 20 but Windows 8+ users may not wish to use this aesthetic effect.

Customize the ConsoleZ font

Next up is I adjust the font selection which can be an important personal choice depending on your requirements and screen size. The default font is Courier New at a size of 10.

Menu > Edit > Settings > Appearance > Font

I change Name: to use Lucida Console in regular with a Size: of 10. But play around until you find a font and size combination that you like.

Font Selection

Font Selection

Command and Batch Scripts

In c:\terminal\ create a new subdirectory named cmd-scripts ie c:\terminal\cmd-scripts\. This will contain my custom text based scripts that use the file extension .bat or .cmd. Modern Windows scripts use the .cmd (Command) extension while the .bat (Batch) is a MS-DOS legacy convention that functions exactly the same.

Colourise Your Command-line with ANSI Colour

In the 1980s and 1990s Windows and MS-DOS supported a widely used feature called ANSI escape sequences that allowed additional functionality such as colour text. But this support was dropped in more recent Windows editions. So I will show you how to re-implement ANSI escape sequence support using Jason Hood’s excellent ANSICon shell overlay.

Visit the ANSICon website at http://adoxa.altervista.org/ansicon/. Download the file ANSI Prompt Colours and save it to c:\terminal\.

Now if you run the command type "ANSI Prompt Colours.txt" you should see a whole lot of garbled text dumped to the ConsoleZ window. These are ANSI escape sequences combined with plain text that Windows does not know how to interpret.

ANSI Esc garbled

ANSI Esc garbled

Download then save ANSICon to your c:\terminal\ directory and unzip it into its own ansicon directory. I keep it in c:\terminal\ansicon\.

Use notepad or notepad++ to create a new file called shell-colour.cmd. Copy and save the following code hosted on my Github account to the shell-colour.cmd file.

If you are using 32-bit Windows or 32-bit ConsoleZ you should replace the following code c:\terminal\ansicon\x64\ansicon.exe with c:\terminal\ansicon\x86\ansicon.exe.

Back in ConsoleZ we will set the shell-colour.cmd as our default shell. This is a hack that loads ANSICon in addition to the default cmd.exe shell to give us ANSI escape sequence support without replacing the underlying shell.

Menu > Edit > Settings > Console

Under Shell: add c:\terminal\cmd-scripts\shell-colour.cmd

Console Settings shell-colour

Console Settings shell-colour

Now load a new tab.

Menu > File > New Tab > Console 2

Or relaunch ConsoleZ. In c:\terminal display the ANSI Prompt Colours.txt using the following commands.

cd terminal
type "ANSI Prompt Colours.txt"

If you see coloured text congratulations you now have ANSI escape sequence support.

ANSI Esc working

ANSI Esc working

Customise and Colourise the Prompt

The default Windows text input prompt only lists the active drive and current directory. As a frequent Linux Bash shell user I like a bit more flare and information to my prompt.

Use notepad or notepad++ to open your existing shell-colour.cmd file in c:\terminal\cmd-scripts\. Copy and save the following code hosted on Github to the shell-colour.cmd file.

Notepad++ editing shell-colour

Notepad++ editing shell-colour

This code adds a couple of new commands. The echo command displays text to your command-line while the prompt command customises the input text prompt. The strings that are wrap within percentage symbols % are environment variables that are accessible from the shell. They allow you to display tidbits of information that are stored by Windows to the computer memory. A complete list of environment variables can be found at SS64.com.

The prompt command has some rather cryptic ANSI escape sequences that introduce colours. The $E string is a prompt argument to display escape characters which conveniently is needed by the ANSI escape sequences as a trigger. The [number;number;40m is a code sequence used to trigger an effect. A list of ANSI escape effects and colours is listed on Pueblo.

Reload ConsoleZ or open up a new tab to apply the changes. You should see a more information pack and colourful input text prompt.

The first part of the prompt in green displays the USER @ DOMAIN while the second part in blue displays the active drive and path.

Colour prompt

Colour prompt

Set a Default Directory at Launch

By default ConsoleZ sets the active directory to the location of its application which in my case is c:\terminal\ConsoleZ.x64. But you can easy change this by doing the following.

Menu > Edit > Settings > Console

Set Startup dir: to the path of your choosing such as c:\ or you can use a Windows environmental variable such as %userprofile%.

Startup directory setting

Startup directory setting

Create a Run as Administrator tab

A neat feature of ConsoleZ is the ability to create shell tabs with different user account permissions including those of an Administrator. As there are times when less restricted access is required to interact with parts of the Windows and its settings.

In ConsoleZ do.

Menu > Edit > Settings > Tabs > Add

In the Main tab change the Title: value to something more meaningful such as Administrator Console.

Point the Icon: to C:\Windows\System32\image\res.dll and select the yellow and blue shield icon which represents Run as administrator.

Under Shell make sure the Run as current user option is selected and the Administrator checkbox is checked.

Press OK when done. You now have a new Run as administrator tab accessible from ConsoleZ.

Admin tab

Create an Administrator Console tab

Configuring Paths

To run most tools and software from the command-line the active directory has to be the same as the location of the tool.

For example if I am in C:\ and try to run our script shell-colour.cmd that is located in c:\terminal\cmd-scripts\. The shell returns the error “’shell-colour.cmd’ is not recognized as an internal or external command, operable program or batch file”. Which basically means the shell could not find the command I was trying to run.

Shell-colour not found

Shell-colour.cmd not found

But by using the Windows environment variable called %PATH% we can add additional directories that the shell will scan in when it searches for the existence of programs.

Configuring the %PATH% variable in Windows is a bit messy using the default Windows Control Panel option. So I prefer to use a 3rd party portable tool called Rapid Environment Editor by Oleg Danilov.

Download either the 64-bit or 32-bit editions, unzip and run the editor.

Rapid Environment Editor

Rapid Environment Editor

The pane on the left contains a list of editable Windows system variables. While the pane to the right has variables that are restricted to your current Windows user account. You may notice there is a PATH= variable for both the system and the user panes.

I generally use the system PATH for my c:\terminal\ path links. But to modify this you need to run Rapid Editor in administrator mode.

Restart as administrator

Restart as administrator

By default Windows stores multiple values of the PATH in a single string separated by semicolons ;. Fortunately Rapid Editor allows you to list and edit each individual PATH entry.

PATH expanded

PATH expanded

To add new directories to the PATH, right-click the PATH= string in Rapid Editor and select Add value or use the Alt+Ins keyboard combination.

Add value

Add value

Press F7 or select Insert directory path… Then point it to your c:\terminal\cmd-script\ directory. You should now have a new entry in your PATH list. Save the changes by pressing the Save icon or using Ctrl+S.

Insert directory path

Insert directory path

Now for changes to have an effect you have to either restart ConsoleZ or open up a new tab. To test that the PATH modifications are active, issue a cd\ command to return to the directory root and try running the shell-colour command again. If it works then congratulations you can now run any custom scripts placed into c:\terminal\cmd-script\ from anywhere in your command-line.

PATH modification works

PATH modification works (yellow highlight was added by myself)

Launch Windows Notepad++ or Notepad from the Command-line

While I love using command-line shells I generally prefer a GUI when it comes to programming or editing text files. I use the following script to to edit a file from the command-line. It launches Notepad++ and opens the file supplied but you can use any text editor including Windows Notepad located in C:\Windows\System32\notepad.exe.

Download and save edit.cmd to your c:\terminal\cmd-scripts\ directory.

In a new session of ConsoleZ test the edit script with the following commands.

cd\terminal

edit "ANSI Prompt Colours"

Edit command

Edit command

EDIT command launches notepad++

EDIT command launches notepad++

To regain your prompt either close Notepad++ or press Ctrl+c in ConsoleZ.

Congratulations the core of this guide is complete. The remainder paragraphs are optional so you can pick and mix which topics you wish to implement. These include adding extra Windows and Linux tools; installing and running Node.js, Perl, PHP, Python or Ruby scripts from the command-line.

Useful Windows Command-line Tools

Besides the complete collection of commands and command-line programs built into Windows that you can discover at http://technet.microsoft.com/en-us/library/bb490890.aspx and http://ss64.com/nt/. There are quite a number of useful third-party command-line programs that are available for the Windows platform. I like to keep these contained in a single directory at c:\terminal\bin\. Bin is an abbreviation for binary an alternative term for a program file.

Create the directory c:\terminal\bin\ and then using the processes covered in ‘Configuring Paths’ add c:\terminal\bin\ to your PATH variable. Restart ConsoleZ and so that any programs placed into c:\terminal\bin\ should be accessible from anywhere within your shell.

Here are a few Windows native command-line tools I recommend.

There are a number of useful tools created by NirSoft.
I personally use nircmd for use with various command scripts as well as WirelessNetConsole, bluetoothcl, whosip and whoiscl.

Microsoft’s Systeminternals also has a few command-line utilities.
AccessChk, Coreinfo, Handle, ProcDump and PsTools Suite.

For handling compressed files.
7-Zip Command Line Version.
UnRAR for Windows

ImageMagick can be used to convert and create image files.

Git for source-code management.

MySQL Utilities.
SQLite portable database tools.

Some other potential sources for standalone utilities include.

Joeware Utilities.
NoNags.

Linux/Unix Terminal Command-line Tools for Windows

As a frequent Linux Bash shell user there are a number of tools I miss on Windows. Fortunately as much of Linux is open sourced many of those tools have been ported over. Unfortunately some of these ports are horribly out of date so they should be best avoided.

There are also a couple of popular Windows open source C compilers such as Cygwin and MinGW that contain ports of Linux terminal tools. But in my opinion their use are overkill and many of the ported tools are rather old.

My favourite collection of ported Linux tools for Windows is the still current GOW (GNU On Windows) by Brent Matzelle.
It includes a large collection of GNU terminal tools and programs including cURL, gawk, grep, ls, nano, tar, vim and many more.

Download the latest release of Gow and install it to a directory of your choosing. I prefer placing it into c:\terminal\gow\ instead of its default directory C:\Program Files (x86)\Gow.

Gow will automatically copy all its application and configure PATH settings so you can use it straight away after a ConsoleZ reload or a new tab.

Gow setup

Gow setup

Testing Gow

Testing Gow

One command Gow does not replicate is the Bash alias ll (double L). I have created a command script that replicates that functionality using Gow’s ls command. Save ll.cmd to your c:\terminal\cmd-scripts\ directory to use it.

ll In terminal

ll In terminal

Run PHP Scripts from the Windows Command-line

Download a copy of PHP For Windows from http://windows.php.net/download/.
Unzip the PHP package to a directory of your choosing. I will use c:\terminal\php\.
Without any arguments PHP does not do much in the terminal. So I created a command script to display the language version as the default behaviour. Download and place the php.cmd script into your c:\terminal\cmd-script\ directory and make any changes if needed.

To test your PHP configuration and the script run the command php.

php.cmd

php.cmd

Download the hi-php.php command line script and run it. It will print Hello, World! to the screen and quit.

php hi-php.php

php php-hi.php

php php-hi.php

Now to associate script files with the .php extension to run under our php.cmd script you do the following commands in a Run as Administrator session of ConsoleZ.

assoc .php=PHP.File
ftype PHP.File=c:terminal\cmd-scripts\php.cmd "%1" %*

assoc .php

assoc .php

Test your configuration and run php-hi.php.

You can refine this one step further so you don’t even have to type the .php file extension. Open the Rapid Editor and under System variables add the value .PHP to the PATHEXT variable. In a new ConsoleZ tab you should be able to run hi-php without the file extension.

php-hi without extension

php-hi without extension

Run Perl Scripts from the Windows Command-line

For Perl I use Strawberry Perl for Windows portable edition which can be downloaded from its website.
Unzip it to a directory of your choosing. I place it in c:\terminal\perl\.
Download and place the perl.cmd script into your c:\terminal\cmd-script\ directory and make any changes if needed.

To associate script files with the .pl extension to run under our perl.cmd script you do the following commands in a Run as Administrator session of ConsoleZ.

assoc .pl=Perl.File
ftype Perl.File=c:\terminal\cmd-scripts\perl.cmd "%1" %*

Download the hi-perl.pl command line script and run it. It will print Hello, World! to the screen and quit.

You can refine this one step further so you don’t even have to type the .pl file extension. Open the Rapid Environment Editor and under System variables add the value .PL to the PATHEXT variable. In a new ConsoleZ tab you should be able to run hi-perl without including the file extension.

hi-perl.pl

hi-perl.pl

Run Python Scripts and Programs from the Windows Command-line

The default Windows download of Python 3.4+ automatically configures itself to enable you to run Python scripts and programs from the Windows command-line.

Download and install a Python edition of your choice from https://www.python.org/downloads/.

To test that Python has installed itself correctly you can download and run hi-python.py.

hi-python.py

hi-python.py

Run Ruby Scripts and Programs from the Windows Command-line

At the time of writing Ruby is a little bit behind on the Windows platform. The current build of Ruby is at edition 2.1.x, while the recommended edition to use on Windows is 1.9.x. So with this in mind I recommend using the RubyInstaller for Windows which is a self-contained Ruby install.

Run the setup program and when prompted make sure you select both checkboxes for the options below.

Add Ruby executables to your PATH
Associate .rb and .rbw files with this Ruby installation

Install and setup Ruby

Install and setup Ruby

To test your Ruby installation download and run hi-ruby.rb.

hi-ruby.rb

hi-ruby.rb

Run NodeJS and Javascript from the Windows Command-line

Download and install the latest NodeJS release from http://nodejs.org/.
Node will create the required PATH variables for you. Unfortunately most Windows editions associate files with the NodeJS and JavaScript .js file extension to the legacy Windows Script Host application. It is probably best that you do not overwrite the Windows Script Host configurations and instead run all Node and JavaScript scripts using the node command.

node somescriptfile.js

To test your Node installation download and run hi-node.js.

node hi-node.js

node hi-node.js

node hi-node.js

Creating Interactive Scripting Shells

Another great feature of ConsoleZ is because it offers separate, tabbed environments you can use it as an interactive programming tool.

In ConsoleZ do the following.

Menu > Edit > Settings > Tabs > Add

In the Main tab change the Title: value to Python 3

Point the Shell: to the language interpreter. For a default Python 3.4 installation I have it pointed to c:\python3\python.exe.

Under Main you can set the Icon: value which usually should be the same as the Shell: value.

When done, press OK and you now have a new Interactive Python shell tab accessible from ConsoleZ.

Create a Python shell

Create a Python shell

For an interactive Ruby shell you need to set the Shell: value to c:\ruby193\bin\irb.bat (or wherever your Ruby installation is located). And you probably want to set the Icon to c:\ruby193\bin\ruby.exe.

Create a Ruby shell

Create a Ruby shell

PowerShell in ColesoleZ

While this guide mostly uses the standard CMD.EXE shell. Microsoft provides an alternative, feature rich and overall better shell known as PowerShell. PowerShell is an optional download for most consumer versions of Windows and offers a more powerful and complicated command-line. Fortunately PowerShell can also easily be incorporated into ConsoleZ.

PowerShell

PowerShell

Within ConsoleZ do the following.

Menu > Edit > Settings > Tabs > Add

Add a new tab and set the Title: to PowerShell.
Set the Shell: value to %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe
Set the Icon: to use that same file path.

Create a PowerShell profile

Create a PowerShell profile

If you want you can also create an administration version of this tab. Select your newly created PowerShell tab and click the Clone button. Change the Title: to Administrator PowerShell and the Icon: to use the PowerShell icon with Administrator shield. Finally make sure the Run as current user option also has the Administrator check-box checked.

Create a PowerShell administrator profile

Create a PowerShell administrator profile

End.

Congratulations, you’re done. The GitHub repository of the code in this article https://github.com/bengarrett/devtidbits/tree/master/post_1226.

Complete.

Complete.

KeePass 2, Password Management


With the recent admission from Adobe that it lost over 150 million user accounts from its online database in October. I thought it would be wise to go over my tool of choice for password management, KeePass.

In the case of the Adobe breach hackers managed to get the email addresses, passwords, credit/debit card details including expiry dates and more for every Adobe user account created on the system. One can not understate gravity of this breach. Losing banking details is bad enough but Adobe also used reversible encryption to store the user passwords. This means all 150 million passwords will eventually be unencrypted and passed around the Internet to multiple dubious organisations and persons.

That will be an immense problem as most people use the same login and password combination for multiple online accounts. That could potentially grant hackers, spammers and identity thieves access to hundreds of millions of online accounts with some simple trial and error.

As of writing (14th November 2013) a list of the top 100 passwords used within the system has been reversed by security researchers who obtained a copy of the leaked database. Nearly 2 million accounts used the password ‘123456’. Another 446,000 used ‘123456789’ and 346,000 used ‘password’.

Now we all know why people use simple passwords as they are easy to remember and simple to type. This has become more pressing as people use touch devices to reach online accounts that make the typing of complicated passwords difficult.

A major problem with this Adobe breach is that not only do the victims have to change the password of their Adobe account. They need to change all the passwords to other accounts that use the same login and password combination. Probably the first targets for this stolen login data will be the usual suspects, Banks, Ebay, Paypal, Facebook, Amazon, Google, Apple and Twitter.

This is why you should use a password management application such as KeePass. A password manager allows you to not only use strong, complicated passwords that you don’t need to manually type in at each request. It makes the use of unique passwords for online accounts simple. So if a security breach were to occur with another online service, a stolen unique password would be only useful for that compromised account.

KeePass – The Good, The Bad, The Ugly

In technology there is a constant conflicting battle of convenience verses security. The use of KeePass follows the path of security at the expensive of convenience and that can make its use complicated and confusing for new users.

The trick to using KeePass is to ignore the many choices it throws at you and focus on what you intend to use. I hope that this article will negate the potential steep learning curve that can come with early usage of KeePass.

One of many options that new users shouldn't bother with.

One of many options that new users shouldn’t bother with.

KeePass – Install

KeePass can be downloaded from http://keepass.info/download.html

Unfortunately the download page is as complicated as the program itself. As a Windows user you want the Professional Edition. Don’t worry, despite the name it is free and open-sourced. I would recommend the Portable KeePass (ZIP Package) download over KeePass (Installer EXE for Windows). As the portable edition makes it easier to make copies and backups your KeePass configurations.

When downloaded, right-click the KeePass-2.*.zip file and Extract All. Follow the wizard to unpack the file to where you want KeyPass stored. I usually unpack it and keep it in C:\Portable\KeePass\.

In the extracted directory you should see KeePass.exe which will launch the KeePass application. If you want to create a shortcut to your desktop, right-click the program and select Send to > Desktop (create shortcut). Otherwise launch the application. I’d recommend for Windows 7+ users to right-click the program in the Taskbar and select Pin this program to the taskbar.

Create a password database

When you run KeePass for the first time the application without a database is loaded. Create a new database by selecting the New button (or press Ctrl-N on your keyboard).

First time run of KeePass

First time run of KeePass

Press the new button

Press the new button

In the Create New Password Database dialogue navigate to where you want to store your KeePass database. You can store it in a directory that automatically synchronises online with a cloud service such as DropBox, Microsoft SkyDrive or Google’s Drive. This will backup your database which is very important and allow you to share it with KeePass compatible mobile applications.

I will create a database called DevTidbits.kdbx and store it in D:\Dropbox\KeyPassData\.

Create a new KeePass database

Create a new KeePass database

You will now be prompted for a Master password at the Create Composite Master Key dialogue. KeePass will prompt for this database access password at startup. So make sure it is not too difficult to type and something you will remember.

Create a secure key file for the database

For added security I recommend selecting the Key file / provider check-box and press the Create … button. This will further encrypt the database whereby it will need both the Master password and the key file before being accessible. If the key file is deleted or lost the database will not be usable.

Create Composite Master Key

Create Composite Master Key

For simplicity and the purposes of this tutorial I will name the key file DevTidbits.key and store it in my Downloads directory at C:\Users\Ben\Downloads\misc\. For added security you should give it a more obscure name and store it elsewhere on your hard drive or even on a USB stick. But make sure you keep multiple copies for backup!

Create a new key file

Create a new key file

Create Composite Master Key - Key file / provider

Create Composite Master Key – Key file / provider

The idea of a separate key file is that it be kept separate from your encrypted database and if you’re paranoid off the Internet altogether. That way if the database is ever lost or copied by people who shouldn’t have access. You can create and apply a new key to your KeePass database. Then permanently delete the old key file used by the comprised database to make it inaccessible.

At the Entropy Collection dialogue you are asked for some Random mouse input and Random keyboard input. Sporadically move your mouse cursor over the black and white texture in the mouse input until the Generated bits bar is green and reaches 256 Bits. Then in the text area of the Random keyboard input type in a large number of random characters. The purpose of this randomised data is to generate a unique key tied to your KeePass database that will never be replicated again.

Entropy Collection

Entropy Collection

Finally at the Create New Password Database – Step 2 dialogue in the General tab. Give your database a name at the Database name: input and a description at the Database description: text area.

Create New Password Database - Step 2

Create New Password Database – Step 2

Further information: KeePass – Composite Master Key documentation.

Configure columns to display passwords

Once done KeePass should load your new database and propagated with some sample groups and entries. As I have customised KeyPass earlier your results may look a little different to mine. For example my passwords are showing while by default KeyPass hides these. To customise your own display select View from the top menu and Configure Columns… To show passwords select Password and deselect Hide data using asterisks. This is not recommended if you are intending to use KeePass in a public environment such as an open plan office or cafe.

View and Configure columns

View and Configure columns

Configure Columns

Configure Columns

KeePass Basic Usage

Back at the KeyPass main screen I have highlighted the two main panes. The left pane which I marked with a 1 has the KeyPass groups. This pane operates like a computer directory system where you can store multiple password entries. The entry pane marked as 2 displays a list of the entries in the active group.

Sample Entries

Sample Entries

When you select a password entry you can right-click it to bring up a menu. The two most useful options in this menu are the Copy User Name (Ctrl-C) and Copy User Password (Ctrl-B). Take note of those two keyboard combinations as they will be in use often. Both options will copy the relevant data from the selected entry to your clipboard stored in the computer’s memory. In a web page login, password input or a text editor like Notepad you can Paste text or use the Ctrl-V keyboard combination to insert the clipboard data.

Menu to copy the entry's user name and password

Menu to copy the entry’s user name and password

Paste Clipboard text

Paste Clipboard text

For security the data will be automatically deleted from the computer’s memory after 2 minutes. You can change this countdown at any time by using the Tools > Options… top menu and adjusting the Clipboard auto-clear time (seconds) value.

Password in clipboard self-destruct countdown.

Password in clipboard self-destruct countdown.

Menu for Tools and Options...

Menu for Tools and Options…

Options, Clipboard auto-clear time in seconds

Options, Clipboard auto-clear time in seconds

Further information: KeePass – Using Stored Passwords documentation.

Add a new entry

In the groups pane (the one on the left) select a group such as eMail.

Select the eMail group

Select the eMail group

Now either use Ctrl-I (i for insert) or press the Add Entry button to bring up the Add Entry dialogue.

Add Entry button

Add Entry button

From here all fields are optional but I generally give each entry a Title, User name, Password and a URL for the site.

Add Entry dialogue

Add Entry dialogue

In the Password field you can either copy or type in your existing password or generate a new one using the Generate a password button.

Generate a password button

Generate a password button

The button will bring up a menu that will list some predetermined randomisers to create alpha-numeric passwords at a predetermined length. This is to help alleviate the issue where some websites have length restrictions on user account passwords.

The 40-Bit Hex Key will generate a short 10 character alpha-numeric password.

The 128-Bit Hex Key generates a standard 32 character alpha-numeric password.

The 256-Bit Hex Key generates a stronger 64 character alpha-numeric password though some websites may reject this length.

Select a hex key.

Select a hex key.

Remember if you decide to generate a new password for an existing web account. You will also need to login to the site and change its existing login password.

When you enter a password the Quality bar indicator lets you know the strength of the password based on the variety of characters and length.

Password quality bar

Password quality bar

Further information: KeePass – Password Generator documentation.

Complex password complications

A word of warning, you probably do not what to use a long complicated password for a service like Apple ID or Google Accounts as unfortunately they’re impractical. Both companies use unified passwords for all their products and services. Which means you could find yourself needing to enter these passwords with smart devices such as phones, tablets and digital media players that do not have keyboards or access to your KeePass database.

Add a group

To create a new group first select in the group pane where the group should go, either the database name or within an existing group.

Right-click to bring up the group menu and select Add group.

Add Group menu

Add Group menu

Give the group a distinct name, choose an icon if you wish and press OK. You now have an additional group to add or move entries into.

Add Group

Add Group

To move an entry into a group just select it then drag and drop. You can move groups around the same way.

Remember you can use KeePass for other data that needs security other than just website and network passwords. Some suggestions could be software registration keys and serial numbers. Wallet content such credit/debit card, social security, frequent flyer and other membership details. Private phone numbers. Private or public keys used for email, file and system encryption.

Save changes

By default KeePass does not automatically save any changes to your database.

You can tell the database has changes that need saving by the asterisk * next to the database file name. As shown in the title bar and highlighted in my screenshot by the orange arrow.

Unsaved database

Unsaved database

To save your changes press the Save button or use the Ctrl-S keyboard combination.

If you quit KeePass before saving the changes you will be prompted with Save database changes before exiting KeePass dialogue. There is a check-box there to Automatically save when closing/locking the database for future exits.

Save database changes before exiting KeePass?

Save database changes before exiting KeePass?

Use search to find entries

Once you find that the collection in the database is quite large you can use Quick Find to hunt for each entry. For this reason I always give the Titles to my entries descriptive names using both the company title and the product brand.

Quick Find search

Quick Find search

Otherwise it maybe difficult to quickly filter some searches. If I use example@hotmail.com as my default login for most websites and then search for ‘hotmail‘ for my Microsoft log-in data. All the entries with the example@hotmail.com User name will be in by the results. So I’d instead give my Hotmail entry the title of Microsoft Hotmail and use the search term ‘microsoft‘.

KeePass also offers a more fine-tuned Find dialogue that pops up with the Find button or the Ctrl-F keyboard combination.

Find

Find

You’re done

Congratulations that covers the basics of KeePass on Windows. There are many software ports for other platforms including mobile devices and JavaScript, some of which I hope to cover at a later date.

Run DOS on the Raspberry Pi – Use rpix86 to turn your Pi into a 1980s super-computer.


Please feel free to jump straight to the ‘DOS on the Pi‘ if you have no interest in a computer history lesson.

What is DOS?

DOS was a semi-open standard that included a common operating system for desktop computers in the 1980s and early 1990s. The hardware used Intel x86-compatible CPUs while the operating system for this architecture was DOS the Disk Operating System. Note that the DOS acronym was also used for a number of incompatible and unrelated computer systems such the Apple DOS and Atari DOS.

As DOS was a primitive, text only operating system that was easy to reverse-engineer. A number of companies made their own editions of DOS and for the most part they were all compatible with each other. The most common x86-edition of DOS was Microsoft‘s MS-DOS but IBM also had their own PC-DOS as did Digital Research with DR-DOS.

MS-DOS was Microsoft’s precursor to its Windows operating system. In the 1990s many editions of Windows were simply a layer of software designed to load up after MS-DOS or later contained MS-DOS built-in.

Off the shelf x86-compatible hardware combined with the DOS operating system became the prominent computer gaming system in North America from the late 1980s until Windows 95. Other regions such as Europe preferred cheaper, propriety systems such as the 8-bit Commodore 64, Amstrad CPC, Sinclair Spectrum. Or the equally as expensive but more user-friendly 16-bit Atari ST and Commodore Amiga lines. In these parts of the world x86-compatible computers only became popular after the early 1990s.

How did DOS come to be?

DOS started out as a clone of the early microcomputer operating system CP/M by Digital Research. CP/M was created in 1973 to run on the Intel Intellec-8 development system which happen to be the first microcomputer sold in the USA.

By the end of the 1970’s CP/M was the most popular microcomputer operating system for professionals and business. Once IBM decided to enter this burgeoning market they wanted to use CP/M on their future products but they couldn’t come to an amicable agreement with Digital Research.

So IBM asked Microsoft to develop a clone of CP/M for their first microcomputer that they dubbed the Personal Computer or the PC. But at this early stage of its life Microsoft had never developed an operating system but bluffed IBM to believe otherwise.

Microsoft discovered a small software company known as Seattle Computer Products who had a x86-compatible CP/M clone called 86-DOS. Microsoft signed a non-exclusive contract to market and distribute this operating system and on-sold it to IBM. Just weeks before IBM’s major PC public announcement Microsoft obtained near-exclusive rights for Seattle’s 86-DOS.

In August of 1981 IBM launched the IBM PC. A machine that bought mainstream attention to microcomputers and made them respectable in the eyes of the public. Every IBM PC sold was operating off a royalty-free copy of Seattle Computer Products 86-DOS operating system re-branded as PC-DOS.

IBM had always been a hardware company and believed there was little money in software. Apple to this day follows that same ethos. What IBM failed to realise was they had designed their machine with off the shelf parts that anyone could buy. The only thing other manufactures lacked to create their own PCs was the operating system. But due to IBM’s non-exclusive operating system agreement. Microsoft was free to re-brand their own version of IBM’s PC-DOS and to sell it to potential companies who wanted to make copycat IBM PCs. That small mistake by IBM launched Microsoft, who became one of the world’s largest companies.

Why DOS?

Many popular franchises, genres and software companies were first launched or popularised on DOS. Warcraft, Grand Theft Auto, Sid Meier’s Civilization, Monkey Island, X-Com, Need for Speed. The first person shooter, the graphic adventure, simulators, real-time strategy are some examples. So DOS would be well worth investigating for anyone who has an interest in retro-gaming from this era.

Why not DOS?

It must be said that the DOS ecosystem has to be the most complicated and arcane mainstream gaming system of all time. The same openness that made it popular was also what made it so horrible to use.

Backwards compatibility was a major benefit and selling point of the DOS ecosystem. But at the same time it held it back and made it much more complicated than necessary. By the late 1980s, its text based 1970s lineage was horribly primitive and outdated.

Hardware support was complex. Often users had to manually change and configure their operating system for each item of software they ran. What unfortunately started off as a simple and clean system for IBM in 1981 evolved into an ad hoc open platform that it was never designed for.

Worse was hardware driver support. For the most part new hardware had to be directly supported by software. This meant that a user’s existing game collection would often not support new hardware simply because it didn’t exist when the software was written.

Finally many game developers targeted the lowest common denominator hardware. This meant you could have had the latest and most expensive hardware yet it could have been ignored by new release software that was designed for a machine that was 5+ years old.

Fortunately much of this hardware complexity is now avoided when using software emulation such as rpix86 on the Raspberry Pi.

Snooper Troops supporting IBM’s Color Graphics Adapter  in 1982.

1984’s Troll’s Tale on IBM’s PCJr.

IBM’s Enhanced Graphics Adapter in use in 1986’s King’s Quest III.

IBM’s Multicolor Graphics Array in 1987 allowed photo like graphics in Moebius: The Orb of Celestial Harmony.

King’s Quest V from 1990 was one of the first titles to fully embrace IBM’s Video Graphics Array with upto 256 colours on-screen.

By 1992 some games such as Links 386 Pro had embraced hi-resolution Super VGA graphics.

DOS today?

DOS is still used on embedded systems that need a simple and small, text only operating system. Today most of the DOS community centres around the open source FreeDOS that offers modern tools as well as legacy compatibility.

DOS on the Pi.

At the time of writing in April 2013 the only acceptable DOS emulator for the Raspberry Pi is rpix86 by Patrick Aalto found at http://rpix86.patrickaalto.com.

What PC hardware does rpix86 emulate?

CPU: 80486 processor, including the protected mode features (for running DOS4GW games) but without virtual memory support. The emulation runs at a speed of around 20MHz 80486 (which equals a 40MHz 80386) machine.
Memory: 640KB of low memory, 4MB of EMS memory and 16MB of XMS memory.
Super-VGA graphics, with a maximum resolution of 640×480 with 256 colors.
SoundBlaster 2.0 (including AdLib-compatible FM sounds) sound card.

This combined hardware could have belonged to a very expensive high-end PC in the very late 1980s. Or a more affordable and common mid-range PC in the early 1990s. In that later era the audio would have been the machine’s weakest point. The graphics and memory would have been very good while the CPU would have been average.

My article DOSBox Sound Emulation talks about the huge variety of audio options on the x86-compatible platform while DOSBox Graphic and Machine Emulation covers the variety of graphic modes.

What operating systems does rpix86 support?

Unfortunately only the obscure 4DOS is supported by rpix86. 4DOS is a rather full featured and complex edition of DOS created by a small company out of Germany. While it was great for daily users of DOS based PCs back in the 1990s. It is an overkill for our purposes of PC gaming emulation on the Pi.

4DOS downloads http://www.4dos.info/v4dos.htm

Raspberry Pi set up.

This guide assumes that a clean install of Raspbian is in use on the Raspberry Pi.

Raspbian can be obtained from http://www.raspberrypi.org/downloads
A guide for installing Raspbian to an SD card for use with the Raspberry Pi is at http://elinux.org/RPi_Easy_SD_Card_Setup

raspi-config

raspi-config.

Start your Raspberry Pi with a new install of Raspbian and at the Raspbian Raspi-config configuration tool do the following.

First select update to make sure you’re using the latest version of the configuration tool.
Select expand_rootfs so the free space on the SD card is fully utilised.
Select change_timeszone to set your Pi’s time zone.
Select boot_behaviour and set Should we boot straight to desktop? as No.
Select change_pass to change the default pi user password.
Select <Finish> to exit.

Now in terminal if you ever need to return to Raspi-config.

sudo raspi-config

Now lets update Raspbian and any out-of-date software.

sudo apt-get upgrade -y

DOS set up.

We are going to use a directory called /dos located in our pi home directory /home/pi/ to store our rpix86 related files. For the commands shown below ~/ in Linux is treated a short-cut for the pi user home directory.

mkdir ~/dos
cd ~/dos

mkdir ~/dos

mkdir ~/dos

This /dos directory will also double up for a mock hard drive that rpix86 will use for DOS.

Download the latest version of rpix86 the DOS emulator.

wget http://rpix86.patrickaalto.com/rpix86.tar.gz

wget rpix86.tar.gz

wget rpix86.tar.gz

Now uncompress and run rpix86. Please note that rpix86 will not work over a remote terminal connection such as SSH or Telnet.

tar -xf rpix86.tar.gz
./rpix86

rpix86

rpix86 running under a remote terminal connection.

When prompted press Y to download the required 4DOS.COM command interpreter.

rpix86 unable to find 4dos.com

rpix86 unable to find 4dos.com.

Once complete your Raspberry Pi should be running a DOS prompt.

rpix86 startup

rpix86 startup.

Basics of DOS shell.

At its core a DOS shell is essentially a self-contained program that allows commands to be run from a text prompt. It lets you interact with your computer’s file system and some hardware. By interacting with the file system you can launch programs; move, copy or delete files and modify the directories.

The rest of the DOS operating system comes in the form of extra programs, applications and hardware drivers that usually reside in a subdirectory. These additions can vary between different releases and iterations of DOS.

DOS command prompt explained.

C:\>

C: Is the active drive. In DOS each floppy, CD, DVD or hard drive is assigned its own drive letter. The C: drive is usually reserved for the first hard drive while A: and B: are reserved for floppy drives.
\ Is the active directory. When a backlash is displayed the active directory is at the root of the drive.
> Is the beginning of the text prompt.
_ Is a blinking prompt that shows where the user’s typed input will display.

Commands.

Please run the following commands in the DOS prompt. Note that unlike Linux, DOS  is not case sensitive.

List the current directory.

DIR

Display some basic 4DOS information.

4DOS

Clear the screen.

CLS

Make a new directory named TEST.

MKDIR TEST

Copy the file 4DOS.COM to the TEST directory.

COPY 4DOS.COM TEST

List the content of the TEST directory.

DIR TEST

Change the active directory to TEST.

CD TEST

Delete the file 4DOS.COM

DEL 4DOS.COM

Change the active directory to root.

CD\

Remove the directory TEST.

RMDIR TEST

You can use the [Tab] key to auto-complete the names of files and directories that exist in the active directory.

4d [Tab]

You can cycle through previous commands typed at prompt by using the up and down arrow keys.

Quit rpix86.

EXIT

DOS commands.

DOS commands.

Dumb DOS names.

DOS uses the FAT file system. The legacy FAT16 file system was first introduced into MS-DOS 4.0 and used through the late 1980s and early 1990s. It is horribly dated but can still be read by modern operating systems.

Because of its age FAT and DOS has a number of particular but important to know traits.

  • In DOS file and directory names can be at most 8 characters long. An additional identifier extension can be given but that can only be up to 4 characters and the first character must always be a period.
  • File usage is identified by extensions, many of which are still in use today. FILE.TXT is a plain text document. FILE.MP3 is an MP3 audio track. FILE.AVI is a video file. FILE.HTM is a HTML document. DOS script files use the .BAT (batch processing) file extension. Program files use .EXE (execute) or the .COM (command) file extensions.
  • Files and directories are case insensitive.
  • The backlash \ character is used to separate directory and file paths.

Fetch and use DOS software.

For the purposes of this article I will only use games sourced from the excellent DOS Game Archive. It only offers DOS software that is legal to distribute and use without payment.

Under the Copyright laws of many countries, copyright on software remains active until 50 or 70 years after an author’s death. This is designed to protect any potential long-term income and the resale value of a product.

The majority of dated, commercial computer software has no financial value and is usually long out of print. This is more apparent with software created for obsolete systems such as DOS. Copyright of this type is rarely defended by its owners and is unofficially known as abandonware.

While the act the downloading and use of abandonware is classed as piracy and is officially illegal. I personally do not see any serious legal or moral issues with downloading and using out of print, obsolete commercial software. But you should use your own judgement.

Where to get games?

You can find legal to download DOS shareware, freeware and public domain games at the DOS Game Archive http://www.dosgamesarchive.com.

To find abandoned but technically illegal to download DOS software, type DOS abandonware into Google and go from there.

DOS Demo 1 – LucasArts, Day of the Tentacle Demo

The surreal and funny Day of the Tentacle was a huge success for LucasArts in 1993. This non-interactive demonstration is easy to run and shows the old DOS low resolution VGA graphics and SoundBlaster audio hardware at its best.

At the Raspbian Linux terminal prompt create the directory DOTTDEMO.

mkdir ~/dos/DOTTDEMO

Change the active directory to DOTTDEMO.

cd ~/dos/DOTTDEMO

Use wget to download the Day of the Tentacle demo from DOS Game Archive.

wget ftp://download.dosgamesarchive.com/tentacle.zip

Uncompress the demo file using unzip.

unzip tentacle.zip

Return back to the /dos directory and run rpix86.

cd ..
./rpix86

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the DOTTDEMO directory exists.

DIR

Change the active directory to DOTTDEMO.

CD DOTTDEMO

List the content of DOTTDEMO but only display items that use the extension .EXE which are execute programs.

DIR *.EXE

Run the program DOTTDEMO.EXE.

DOTTDEMO.EXE

Congratulations, hopefully you have a working Day of the Tentacle demo.

To exit the demo press [Alt] X and follow the prompt.

wget DOTT

wget tentacle.zip

DOS for DOTT

DOS for DOTT

DOTT DEMO

DOTTDEMO.EXE

DOS Demo 2 – Sierra Online, Space Quest 3.

From early 1980s to the 1990s Sierra Online were all about pushing the envelopes of audio and visuals on the home computer. Space Quest 3 in early 1989 was certainly doing that on the PC. Unfortunately but common for the era, this demo requires a rather complex set-up procedure.

At the Raspbian Linux terminal prompt create the directory SQ3DEMO.

mkdir ~/dos/SQ3DEMO

Change the active directory to SQ3DEMO.

cd ~/dos/SQ3DEMO

Use wget to download the Space Quest 3 demo from DOS Game Archive.

wget ftp://download.dosgamesarchive.com/spaceq3.zip

Uncompress the demo file using unzip.

unzip spaceq3.zip

Return back to the /dos directory and run rpix86.

cd ..
./rpix86

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the SQ3DEMO directory exists.

DIR

Change the active directory to SQ3DEMO.

CD SQ3DEMO

List the content of SQ3DEMO but only display items that use the extension .EXE which are execute programs.

DIR *.EXE

Run the program INSTALL.EXE to tell the demo of Space Quest 3 what hardware our DOS machine supports.

INSTALL.EXE

In the Sierra Game Setup/Installation Program.

Press [Enter] at the Sierra’s newest products .. notification.

Select EGA/VGA with RGB monitor.

Select AdLib Music Synthesizer Card.

Select IBM or IBM-compatible keyboard.

Press [Enter] at the NUMLOCK is OFF confirmation.

Select NO for the MicroSoft compatible mouse.

Press [Esc] to skip the install the game on your hard disk.

Press [Enter] at the Installation is complete prompt.

Back at the DOS prompt list the content of SQ3DEMO but only display items that use the extension .BAT which are script files. Many older Sierra games used batch script files to launch their programs.

DIR *.BAT

Run the batch script SQ3DEMO.BAT to launch the Space Quest 3 demo.

SQ3DEMO.BAT

Welcome to Scenes From Space Quest III.

To exit press [Esc] to bring up the in-game menu and then select File > Quit.

SQ3DEMO.BAT

SQ3DEMO.BAT

Demo 3 – Windmill Software Digger

Next up is the rather simple Digger created by Windmill Software in 1983. The basic graphics and primitive audio was amazing compared to many early games created for the original IBM PC machines of this era. This game looks to be inspired by the 1982 Universal arcade game Mr. Do! Currently there is some audio emulation issues with this game when used on rpix86.

At the Raspbian Linux terminal prompt create the directory DIGGER.

mkdir ~/dos/DIGGER

Change the active directory to DIGGER.

cd ~/dos/DIGGER

Use wget to download Digger from DOS Game Archive.

wget ftp://download.dosgamesarchive.com/digger.zip

Uncompress the game file using unzip.

unzip digger.zip

Return back to the /dos directory and run rpix86.

cd ..
./rpix86

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the DIGGER directory exists.

DIR

Change the active directory to DIGGER.

CD DIGGER

List the content of the directory DIGGER.

DIR

Run the program DIGGER.EXE.

DIGGER.EXE

Welcome to computer video gaming 1983 style. Use the arrow keys to control your character or to exit Digger press [F10] a couple of times.

DIGGER.COM

DIGGER.COM

Demo 4 – Non-working Games

At the moment rpix86 is still an immature product and your mileage for running various software will vary. Some DOS software can be temperamental such as Epic MegaGame’s Zone 66 from 1993. It demands that we do not load any DOS XMS memory drivers but currently we can not do this in rpix86.

At the Raspbian Linux terminal prompt create the directory ZONE66.

mkdir ~/dos/ZONE66

Change the active directory to ZONE66.

cd ~/dos/ZONE66

Use wget to download Zone66 from DOS Game Archive.

wget ftp://download.dosgamesarchive.com/zone66.zip

Uncompress the game file using unzip.

unzip zone66.zip

Return back to the /dos directory and run rpix86.

cd ..
./rpix86

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the ZONE66 directory exists.

CD ZONE66

List the content of ZONE66 but only display items that use the extension .EXE which are execute programs.

DIR *.EXE

Run the program ZONE66.EXE.

ZONE66.EXE

ZONE66.EXE

ZONE66.EXE

Disappointingly we receive an abort error message This program can not run with an XMS driver!!! because our DOS operating system is not configured correctly. As you can see gaming on PCs in this era often required more technical knowledge than today.

Fi

Well that is it for this tutorial. I will continue to update this entry as new revisions of rpix86 are released for the Raspberry Pi. Feel free to ask any questions in the comments below.

Use USB hard disk & flash drives with your Raspberry Pi


This was meant to be a brief article on how to mount an external USB drive, but it quickly spiralled out when I starting writing about all the nuances and potential issues one might run into. So I have a created a quick summary of the commands on how to mount a drive below. But I highly recommend reading the rest of the article as there are a number of potential pitfalls with the Pi and external USB drives that are addressed.

In-brief

To mount a USB drive:

sudo mkdir /mnt/usbdrive
sudo mount /dev/sda1 /mnt/usbdrive
ls /mnt/usbdrive

To list your file systems:

sudo fdisk -l
sudo mount -l
df -h

Before disconnecting a USB drive:

sudo umount /dev/sda1

Format a drive to EXT4

sudo mkfs.ext4 /dev/sda1 -L untitled

Add Apple OS X HFS+ read/write support

sudo apt-get install hfsutils hfsprogs hfsutils

Format a drive to HFS+

sudo mkfs.hfsplus /dev/sda1 -v untitled

Add Windows NTFS read/write support

sudo apt-get install ntfs-3g

Format a drive to NTFS

sudo mkfs.ntfs /dev/sda1 -f -v -I -L untitled

Add Windows/DOS FAT32 read/write support

sudo apt-get install dosfstools

Format a drive to FAT32

sudo mkfs.vfat /dev/sda1 -n untitled

In-depth

The Raspberry Pi is a great and flexible little device. But one of its main limitations is the storage options if you wish to use the device as a file or multimedia server. SD memory cards are cheap and common with low-end specifications but reach an affordability and storage cap when their sizes increase. A cheap USB powered external drive with many times more space can be had for a similar price to a top capacity SD memory card.

USB Power Problems

A major limitation for running a USB drive on a Raspberry Pi are the power requirements. The Universal Serial Bus specification states that to adhere to the standard, up to 0.5A (amps) can be drawn from a single port.

My old Samsung G2 Portable 640 hard drive requires 0.85A to work which is 0.35A above the USB2 specification. This is not an isolated example, many modern desktop PCs and laptops supply a greater amperage than the standard 0.5A to their USB ports to support devices such as portable USB hard drives. Unfortunately the Pi cannot power many external USD devices such as hard drives as its USB ports are restricted to the standard amperage.

To get around this problem you need a powered USB hub. You attach the hub device to the Pi’s USB port, plug the USB hard drive into one of the hub’s USB ports and insert the hub’s power supply into a walled power socket. My tiny Logitech ‘Premium 4-port’ USB hub can share 2.5A between 4 devices which is more than enough to power my Samsung drive.

I would also recommend against powering the Raspberry Pi off the same USB Hub as a USB drive. For me this caused interference where the drive would momentary lose power.

Mounting A Drive

When a drive is mounted, it connects to your Pi and Linux recognises it. The drive is given a directory where you are able to access and modify its content. These directories are known as mount points and can be given any name that works for you but they should be placed in /mnt.

I will call my mount point ‘usbdrive‘. First we need to create a mount point.

sudo mkdir /mnt/usbdrive

Linux has the /dev directory that is in use to store special files that allow access to the computer’s hardware. The /dev/sd* collection of files represent drives. Each drive connected to your Raspberry Pi is given a letter.

/dev/sda Would be your first connected drive.
/dev/sdb Would be your second drive.

To mount the drive to your mount point ‘usbdrive‘.

sudo mount /dev/sda1 /mnt/usbdrive

The numeric 1 at the end of /dev/sda is a requirement and tells Linux to mount the first partition.

Partitions are beyond the scope of this article, but you can learn more about that at the Ubuntu Community Docs.

mkdir mnt usbdrive

Disconnect / Unmount A Drive

It is always advisable that you unmount a USB drive before unplugging it from its power source. This forces all queued data to be written to the drive before it loses power.

sudo umount /dev/sda1

You may need to use the -f force option if the drive will not dismount.

sudo umount -f /dev/sda1

If you use the shutdown -P -h 0 command to power down your Pi you do not need to use unmount.

Disk File Systems

A disk file system is the method an operating system stores and reads data on a drive. There is an endless list of disk file systems out there as each operating system seems to have their own native but incompatible system.

Linux as a number of native file formats but generally today the most common is the EXT (Extended File System) series which include ext2, ext3 and ext4.

Apple OS X uses HFS+ (Hierarchical File System Plus) otherwise known as Mac OS Extended.

Modern Microsoft Windows systems mostly use NTFS (New Technology File System).

Legacy Microsoft Windows systems and ancient Microsoft DOS systems generally use a variation of the FAT (File Allocation Table) which includes FAT, VFAT, FAT32 and exFAT.

High CPU usage with the ntfs-3 driver

Excessive CPU usage with the ntfs-3 driver that slows Samba (smdb) transfers

xxx

With EXT4 the file transfer using Samba has an additional 250%+ CPU resource available for use

Disk File Systems Compatibility

EXT has native support in Linux and the Raspberry Pi. It has no official support in Windows. There are free third party drivers available for Windows offering limited read/write EXT support such as the open source EXT2FSD or EXT2Read. Apple OS X users need to use the commercial Paragon ExtFS to enable full EXT support.

To enable Linux EXT4 support:
It is turned on by default on the Raspberry Pi.

HFS+ has restricted support in Linux. It can read HFS+ formatted drives but can only write to them if journaling is disabled. Windows has no native HFS+ support but there are paid solutions such as Paragon HFS+ for Windows.

To enable Linux HFS+ support:
sudo apt-get install hfsutils hfsprogs hfsutils

FAT is probably the most supported file system but it is also the most limited. Linux, Windows and Apple OS X all support FAT, VFAT and FAT32. ExFAT otherwise known as FAT64 is native to modern Windows and Apple OS X but has no support in Linux due to patient incompatibilities.

To enable Linux FAT32 support:
sudo apt-get install dosfstools

NTFS has read only support in Linux and Apple OS X. Third party drivers are available to add write support including the commercial Paragon NTFS and the open source NTFS-3g.

To enable Linux NTFS support:
sudo apt-get install ntfs-3g

Performance Issues & Which Disk File System To Use?

As a non-scientific test I took a 4GB video file and copied it to various file systems using my Raspberry Pi and the USB hard drive.

The worst performer by far was the NTFS-3g driver for Linux NTFS read and write support.

The transfer that 4GB file from my Windows 7 PC to the NTFS formatted USB hard drive took around a minute or two. The same file from the Raspberry Pi’s SD memory card to the NTFS formatted USB drive took 30 minutes to write and 23 minutes to read!

Performance for EXT3, EXT4 and FAT32 were about the same at 12-14 minutes to both read and write. This suggests that there is a bottleneck with either the SD memory card or USB drivers and not the file system.

If your drives are mostly used by the Pi my recommendation would be to use EXT4 on your USB drives. EXT4 is mostly the same as EXT3 with some extra minor features but it is widely supported in the Linux world, plus it is backwards compatible with EXT3 and EXT2.

FAT32 is the most compatible file system but has a restrictive 4GB file size limit.

EXT2, HFS+ on Linux and FAT32 lack journaling support that makes them prone to errors when used on portable drives. As these file systems can’t elegantly recover if they unexpectedly lose power.

FAT32, NTFS can not store Linux file or user permissions.

Format A Drive

To change the file system of a drive you need to format it. Linux allows you to format any supported disk format using the mkfs tool.

In the examples below you will notice an option followed by ‘untitled‘. These are optional volume labels to name your drive.

First you must unmount the drive you wish to format.

sudo umount /dev/sda1

To format a drive to EXT3 (Linux):
sudo mkfs.ext3 /dev/sda1 -L untitled

To format a drive to EXT4 (Linux):
sudo mkfs.ext4 /dev/sda1 -L untitled

To format a drive to HFS+ (Mac OS X):
sudo mkfs.hfsplus /dev/sda1 -v untitled

To format a drive to FAT32 (DOS and legacy Windows):
sudo mkfs.vfat /dev/sda1 -n untitled

To format a drive to NTFS (Windows):
sudo mkfs.ntfs /dev/sda1 -f -v -I -L untitled

I have applied a few options here that I will explain.
-f Fast Format. Due to the poor performance of 3g.ntfs on the Pi I highly recommend using the less CPU intensive fast format mode.
-v Verbose. By default the NTFS status output is limited so this lets you know what is happening.
-I Disable Windows Indexing. This improves the write performance of the drive but it will mean Windows Search queries used on this drive will take longer.

Format NTFS

Format NTFS

Format HFS+

Format HFS+

Format FAT32 (vfat)

Format FAT32 (vfat)

Format EXT4

Format EXT4

Automatically Mount A Drive

To simplify the process of mounting a drive you can add the drive’s information to the fstab settings file located in /etc/. I would recommend taking a look at the Ubuntu FSTAB community page for a deeper understanding of this file.

First run nano to edit fstab. The -Bw options tell nano to backup the file and not to use any line-wrap.

sudo nano -Bw /etc/fstab

You should already see some existing entries. Do NOT change these as the two /mnt/mmcblk0p entries are there to mount the SD card.

Add the following to the bottom of the file.

/dev/sda1 /mnt/usbdisk auto defaults,user 0 1

These are explained:

/dev/sda1 Is the location of the drive to mount.
/mnt/usbdisk Is the mount point, which is the folder to access the content of the drive.
auto Is the file system type, here you can set ‘auto‘ or force a file system type such as ext2, ext3, ext4, hfsplus, ntfs, vfat.
defaults,user Are mount options. You normally need to only supply ‘defaults‘. Though there are some others that maybe useful such as ‘ro‘ for read-only or ‘user‘ to enable write permission for all users. Use a non-spaced comma to separate multiple options.
0 A binary value used for debugging. It is best to keep this set at zero.
1 Pass number for a file system check at boot. ‘0‘ (zero) to disable or ‘2‘ to enable.

Save the changes to fstab.

[Ctrl] x
Y at the Save modified buffer prompt.
[Enter] for the File name to Write: /etc/fstab prompt.

nano etc fstab

The drive will mount at boot as long as it is attached to the Pi. If you want to mount the drive after you have plugged it in use mount with the automatic option.

sudo mount -a

Using a Point-to-Point Tunnelling Protocol, Virtual Private Network (PPTP VPN) client on a Raspberry Pi


Why use a VPN?

VPN or a Virtual Private Network is a common way of securing an Internet connection using encryption.

Basically a VPN sets a trusted, designated server on the Internet to act as a man in the middle. As a VPN client, all your web traffic gets encrypted and directed to this designated machine. It decrypts and then reroutes the traffic to its intended destination. This gives you a both benefits and disadvantages.

Having an encrypted connection between yourself and a trusted VPN server means your Internet traffic is mostly safe from snoopers. That could be anyone on the same Wi-Fi network as you, your ISP or even big-brother governments.

If you are behind a government or corporate firewall, the VPN could be used to reach blacklisted websites. As the firewall will only see that your traffic is directed to the designated VPN server.

A VPN also allows you to hide the point of origin of your Internet request from your destination. As the designated server intercepts and decrypts your Internet traffic before it resents it. This fools the destination to believe the Internet request came from the designated server and not from you.

If a website or service has region restrictions placed on it to allow only users from particular countries access. A VPN server hosted in a permitted country allows you to bypass this form of restriction.

The most obvious disadvantage in using a VPN server is that adding an extra layer of Internet bureaucracy slows down your overall Internet experience. The encryption requires more processing time and it consumes more bandwidth. Plus adding a man the middle means your Internet traffic isn’t going to be traversing across the Internet via the quickest possible route.

About this project

For this project I am going to carry out a VPN client connection on a Raspberry Pi without using any GUI tools. This could be used with a headless or server Raspbian Pi set-up. It should also work fine on a Ubuntu based Linux system.

Also this tutorial will use the most common VPN protocol PPTP, known as the Point-to-Point Tunnelling Protocol. Its encryption is not as secure or as safe as some other VPN options but it is the most supported of the protocols out there.

Software installs and updates

First in a Pi shell we need to update our repositories, our Pi operating system and install PPTP for Linux.

sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install -y pptp-linux

sudo apt-get install pptp-linux

Setup and configuration

The pptp-linux installation includes the very useful Perl script pptpsetup that is found at /usr/sbin/pptpsetup.

List the help options for pptpsetup.

pptpsetup --help

pptpsetup has 6 options to create a PPTP connection configuration.

pptpsetup
--create [tunnel name]
--server [vpn server]
--username [vpn server username]
--password [vpn server password]
--encrypt (optional)
--start (optional)

pptpsetup --help

Example usage of pptpsetp.

sudo pptpsetup --create examplela --server pptp-la.example.com --username exampleuser --password examplepassword --start

The --create option is the ‘tunnel’ name you wish to give your connection. It can not have spaces or weird characters and should be something that is descriptive of the VPN server it will connect to such as a geographical location.

--server Is the address of the VPN server to connect to.

--username Is your assigned user name used to authorise access to the VPN server.

--password Is the password required by the user name.

--encrypt Toggles encryption for all communication between your Pi and the VPN server. While this setting is optional, most VPN servers will not accept a connection unless encryption is in use.

--start Will connect to the VPN after creating your new connection configuration.

For the rest of this tutorial I will use my real world example of pptpsetup. Replace any references to my proxpnuk configuration with your own VPN server configuration.

sudo pptpsetup --create proxpnuk --server pptp-uk1.proxpn.com --username xxxx --password xxxx --encrypt --start

pptpsetup realworld

If you have a Connection termination but you are sure your server address, user name and password are correct, try using the --encrypt option.

pptpsetup creates a configuration file that gets placed in /etc/ppp/peers/ which is a superuser protected directory so you will need to run sudo to view its content.

sudo ls -l /etc/ppp/peers/
sudo cat /etc/ppp/peers/proxpnuk

cat proxpnuk

You can see your point-to-point protocol connection using ifconfig -s. It should be listed under ppp0.

ifconfig -s

pptpsetup can also be used to delete an existing configuration file.

sudo pptpsetup --delete [tunnel name]

pptpsetup delete

Usage

PPTP for Linux also installed two Bash scripts that are used to connect and disconnect to your PPTP VPN server. You can use the -h option to see the scripts help text.

/usr/bin/pon and /usr/bin/poff are scripts that call the /usr/sbin/pppd application. pppd known as the Point-to-Point Protocol Daemon is a complicated program but if needed its manual with an endless list of options can be found at http://ppp.samba.org/pppd.html

pon -h
poff -h

pon and poff -h

If connected let us disconnect our current ppp connection using the poff script.

sudo poff -a
ifconfig -s

Ifconfig should only list eth0 and lo under the Iface column.

ifconfig -s

Let us reconnect again.

sudo pon proxpnuk updetach
ifconfig -s

sudo pon proxpnuk updetach

To see your connection settings for troubleshooting.

sudo pon proxpnuk debug dump

To save the debug output to a file and view it.

sudo pon proxpnuk debug dump > ~/proxpnuk-debug.txt
cat ~/proxpnuk-debug.txt

pon proxpnuk dump proxpnuk-debug.txt

PPTP for Linux has a log file that can also be useful for troubleshooting. It is found at /var/log/messages. The tail command normally displays the last 10 lines of text file but when it’s combined with the -f follow option it displays the most recent text appended to the file.

tail -f /var/log/messages

Press [Ctrl] C to exit tail.

tail -f var log messages

Test for a VPN connection

I use the traceroute command to test for an active VPN connection. First make sure there is no ppp0 connection.

sudo poff -a
ifconfig -s

Then run traceroute to a well-known website such as google.com.

traceroute google.com

traceroute google.com no vpn

Make a note of the number of hops. In my case it took 8 hops for a request from my Raspberry Pi to reach Google’s servers.

Now if we enable our ppp0 connection and try again, you will probably discover that connecting to Google takes exactly the same number of hops.

sudo pon proxpnuk updetach
traceroute google.com

traceroute google.com no route

This probably means the internet traffic sent and received from the Pi is not using our VPN connection.

We need to create a route to tunnel our Internet traffic through our VPN. The route command at /bin/route is perfect to carry out this task.

sudo route add -net "0.0.0.0/0" dev "ppp0"

The -net 0.0.0.0/0 sets the target network of the route to be the default route. dev toggles a device to reroute our traffic to. While PPP connection number one ppp0 is that device.

To check if our route was successful use netstat with the -a all sockets option and feed the output into grep to filter and display only the lines that contain the string "/var/run/pptp/". These lines signify our PPTP routes.

netstat -a | grep "/var/run/pptp/"

route add

Test your connection again and you should have a greater number of hops than last time.

traceroute google.com

traceroute google.com with route

That meant the connection to Google.com from your Pi had rerouted through your VPN server. Congratulations.

VPN at boot

To automatically connect your Pi to a VPN server at start-up I’d recommend placing the following script into /etc/rc.local. This file runs all commands within as a superuser, at the end of a Pi boot but before the user login prompt. I use the -B option with the nano text editor to backup the rc.local file before making any changes to it.

sudo nano -B /etc/rc.local

Make sure you insert any script before the exit 0 line and after the hashed # comments.

vpn="on"
if [ $vpn = on ]; then
printf "\nVPN connection to ProXPN UK\n"
pon proxpnuk updetach
printf "Add Internet traffic route through ppp0\n"
sudo route add -net "0.0.0.0/0" dev "ppp0"
printf "Netstat output of all PPTP sockets\n"
netstat -a | grep "/var/run/pptp/"
fi

nano etc rc.local

The script is pretty self-explanatory and I have contained it within a conditional statement to disable if needed. To do this simply change the vpn="on" variable to something else such as vpn="off".

In nano press the [Ctrl] X keys to prompt for a Save modified buffer? query. Answer Y at the prompt to save and press enter to keep the current File Name to Write.

You don’t need to reboot to test your changes to rc.local.

sudo /etc/rc.local

Executes the script same way as it would with a reboot.

 

sudo etc rc.local

fi

I am happy to reply to any questions about this post but I am no expert on the subject of VPN and can not help in setting up a VPN server or using other client protocols such as OpenVPN. Extracting this information on PPTP for Linux was quite time-consuming.

Ubuntu Linux Shell Tips Part I


I love the Linux shell. While cryptic and horribly daunting for a first time user the text only command line offers an amazing amount of flexibility and power without the need to actually develop or code. Yet unlike a graphical user interface of which I don’t tend to like in Linux. The text shell is fast and allows you to remotely connect to a headless device with the bare minimum of configuration.

Take the Raspberry Pi, it only needs a plugged-in Ethernet cable and a micro-USB power cable. And this credit-card size computer can be controlled with Linux shell from your favourite Windows PC, Mac, tablet or mobile phone.

This series of entries is going to be a run-down of most of the useful Linux shell commands and configuration options that I use daily on my Linux boxes and images.

Understanding the Prompt

shell prompt

ben@developers-tidbit:/$ _

ben Is the logged in user.
@ At symbol.
developers-tidbit Is the server name. On some systems the server’s IP address maybe listed.
: Divider.
/ Is the current path. ~ Means the current path is the home directory.
$ Dollar states that the user is a standard user. It is replaced by a hash # if the user is a root administrator.
_ Is the cursor to show the point of keyboard input, text placement.

Linux supports auto-completion at prompt, so cl [Tab] [Enter] will finish the rest of the command clear and launch it for you. Auto-complete also works with directories, files and apt-get repositories. Everything in Linux is case-sensitive so this means Clear, clear and CLEAR are all treated as different commands.

The prompt also keeps a history of the previously used commands. The up and down arrow keys cycle through these.

Directories

cd Changes the active directory.
cd / Goes to the root of the file system. Like the C:\ on a Windows system.
cd ~ Goes to your personal home directory.
cd /var Goes into the directory var.

cd and ls

ls Lists information about the files and is probably one of the more useful tools in the Linux toolbox.

ls Lists the active directory.
ls / List the root of the file system.
ls /var List the content of the directory /var.
ls -l / List the root of the file system using a long detail format.

ll Ubuntu has an alias (a short-cut) for the ls -l command.

ll var

ls -lh / Lists the root of the file system using a long detail format that humanises the file size.
Alternatively ll -h / will display the same results.

ll var humanised

To only list the directories of the active directory a hack will be required. It lists all the items returned by the ls command that end with a forward slash / directory tag.

ls -d */ List the directories of root.
ls -d /var/*/ Lists the directories of /var.

list only directories

Chaining Commands; Running Multiple Commands in Sequence

The semicolon character ; allows the running of multiple commands in a sequence. The following changes the active directory to home, lists the content using a detail list, then changes the active directory to root.

cd ~ ; ls -lh ; cd /

chained commands

Viewing Text Files

Linux is highly dependant on plain text files for its configuration and settings so it would be natural that it includes many tools to handle these. The few common tools to display the content of a text file are cat, more, less.

cat Is short for concatenate and prints the file on-screen.
more Operates the same as cat except that it pauses the printing of the file at the end of each page.
less Operates the same as cat except you can use the up / down arrow keys or the Pg Up / Pg Down keys to scroll through the document.
To quit from both less and more press the Q key.

cat /etc/mime.type
more /etc/mime.type
less /etc/mime.type

more mime-types

To clean the screen use the clear command.

To display multiple files in a sequence more than one file can be supplied to cat.

cat file1 file2 file3

Another useful choice is the -n option that appends line numbers to the file in view.

cat -n /etc/mime.type

cat line numbers

To remove the display of double-line spacing use the -s option.

The head tool allows the partial display of a file.

head -n [number of lines to display]

head -n 29 /etc/mime.type

29 lines of mime-types

The use of a negative number will display the complete file except the last X lines.

The tail tool is the reverse of head, it displays the footer of a file.

tail -n 3 /etc/mime.type

Tail has as a really useful option that allows you to follow the most recent changes that have been append to a text file. This is an excellent tool for reading only the most recent entries in a large log file. You may need to press Ctrl C to exit tail.

cat /var/log/syslog
clear
tail -f /var/log/syslog

tail f syslog

Some configuration files such as the Linux user and group settings consist of comma-delimiter data. To extract data from these files you can use the cut tool. By default cut uses the tab character as a delimiter but this maybe changed with the -d "[delimiter]" option. The -f option with a list or a range of numbers provides the columns for display.

cut -d ":" -f "[columns]" [file]

Lists the first 3 columns in the group settings file.

cut -d ":" -f "1,2,3" /etc/group
cut -d ":" -f "1-3" /etc/group
cut -d ":" -f "1,2-3" /etc/group

cut etc group

To search within a file the grep tool is most useful.

grep "[search term]" [file]

A few grep options.
-i Will ignore letter cases in the [search term], so "Abc", "abc", "ABC" will be considered matches for grep "abc" [file]
-c Returns a count of the number of search term matches rather than a summary of the finds.
-n Includes line numbers in the summary of the finds.
-m [limit value] Only returns the first X number of results.

grep -i "system v" /var/log/boot.log
grep -in "system v" /var/log/boot.log
grep -inm1 "system v" /var/log/boot.log
grep -ic "system v" /var/log/boot.log

grep example

In-built Help

Linux shell has a number of help tools while you are in terminal. The quickest way to discover what a command’s purpose is the use of the whatis tool.

whatis more

Some terminal commands include a basic usage help which appears when you run the command with no options.

more

whatis

Many commands have an extended help which is trigged with the --help option.

whatis --help

help long

Finally most commands have usage manuals that are accessible at any time with the tool man [command]. Reading manuals through the terminal is pretty tedious so fortunately you can find these same texts online at http://manpages.ubuntu.com/manpages/. Use the website’s search box to find the tool’s manual.

man more

man more

Or read it at http://manpages.ubuntu.com/manpages/precise/en/man1/more.1.html

Piping and Input/Output redirection

This may sound a little abstract but it is easier to grasp when explained. Linux shell gives you the very powerful ability to take the output of one command and feed it into another.

ls --help

With the ls --help there is more text then what the terminal can display. A pipe using the | character will feed the output text into a secondary tool such as less or more. To exit either tool press the Q key.

ls --help | more
ls --help | less

ls help with more

It’s not just help text that you can pipe. Any text output could be fed into another tool if it is supported.

ls -lh / | more

ls with more

Output text can also be saved to a file using the greater-than > character. In the example below the output of ls is saved to result.txt that will be created in the Linux temporary directory.

ls -lh / > /tmp/result.txt
ls -l /tmp/result.txt

ls saved to result.txt

less /tmp/result.txt

The greater-than > character will always overwrite an existing file with the new text content. Double greater-than >> will append the text to the end of the file.

echo "Hello world." > /tmp/hi.txt
cat /tmp/hi.txt
echo "Test 1, 2, 3." >> /tmp/hi.txt
cat /tmp/hi.txt

appending text example

The less-than < character will redirect the file content into a command.

less < /tmp/result.txt

Easy Text editing

To edit text files from Linux shell I believe the easiest to use tool is nano.

nano /tmp/hi.txt

nano editing hi.txt

Like in less you can use the arrow keys or Pg Up / Pg Down to scroll through the text. The left and right arrow keys can reposition the cursor. Any text typed will be inserted at active position of the cursor.

To cut a line of text press F9.
To cut a paragraph of text use F9 multiple times.
To paste text press F10.
To go to row position press Ctrl _.
For a help that unfortunately is slightly cryptic press Ctrl G. In the help text the ^ symbol means the Ctrl key while the M- symbol means Alt key.

To exit nano press Ctrl X.

When you exit nano you will be asked to Save modified butter (ANSWERING “No” WILL DESTROY CHANGES) ? To save the changes to file type otherwise type and the file will remain unedited and any changes lost.

Piping output text into nano doesn’t work as expected.

ls -lh / | nano

ls to nano error

Nano expects to read text sourced from a file. To force nano to read text from standard input (aka stdin which is the Linux term for piped or streamed text) a dash - is used for the input file name.

ls -ln / | nano -

ls output in nano

To get some basic statistics of a text file the wc word count tool is useful. wc without any options displays the number of lines, words and the byte count for the file. The -w option counts the words, the -m counts the number of characters while the -l counts the number of lines.

wc /proc/cpuinfo
wc -w /proc/cpuinfo
wc -m /proc/cpuinfo
wc -l /proc/cpuinfo

wc cpuinfo

That is the end of Part I of my Linux shell tips. The 2nd part that will be online at a later date will cover administration, file, network system and user tools.