Helpers and common usage for tar

tar is a feature rich but often confusing archiving tool most commonly used on Linux and BSD systems. One of the great benefits of tar over other more modern tools such as 7z, RAR or ZIP is that it is open source, free and platform agnostic. Even more importantly for Linux users tar archives preserve user and file permissions allowing for easy directory backups and restoration.

Rsync – Helpers and common usage

rsync is a widely used tool for synchronisation so to keep copies of a file on multiple computers the same. Because of its flexibility it has become the defacto standard on Linux and other similar systems. While newer protocols and tools such as Dropbox and BitTorrent Sync overlap with and improve on some of rsync’s capability. Rsync is still relevant today both for syncing and for other function such as local directory duplication and large file copying.

Backup your MySQL/MariaDB database easily using Percona XtraBackup

Over the years I have looked around for easy to use MySQL database backup tools that I can quickly and reliably automate. I’ve tried many unsatisfactory solutions but finally I have found something I am happy with, Perscona’s XtraBackup.

Unfortunately for Windows users this is a Linux only solution.

Percona is an enterprise focused database company with the aim of delivering their clients faster and more reliable MySQL solutions. Fortunately for the rest of us they also believe in open source and offer a number of free tools suitable for all MySQL and MariaDB users in addition to their own MySQL server platform.

Below is a Markdown formatted cheat sheet that I wrote which enables Linux users to easily back-up their MySQL compatible databases. With simple instructions on how to create, restore and automate backups. While the instructions were written with Ubuntu/Debian in mind they should be useful for all other distributions with the usual adjustments.

Percona XtraDB Software page.
Percona XtraDB downloads (not needed for Ubuntu/Debian users).

Gist bengarrett /

Make your Windows Command Prompt, Linux-like

Updated on 2015, March 25. Added information on rsync, OpenSSH clients and updated some links plus io.js.

In this entry I will explain my method of improving the Windows Command Prompt and its command-line. To turn it from a crude shell prompt into an ANSI coloured interface with a useful set of shell commands that can commonly be found on Linux and Mac terminals. We will also integrate a number of optional popular programming languages and features. This will require some existing knowledge of Windows and the command-line. But before we get into that let’s go on a trip down memory lane for some context.

Historical Context

(you can skip this if you wish)

In the ancient days of personal computing during the mid to late 1970s and the early days of Microsoft. There was a popular text only operating system called CP/M. By today’s standards it was rather crude and rudimental but at the time it was popular in business due to its relative ease of use.

Years later when Microsoft delivered to IBM its first PC operating system. It became apparent that the Microsoft operating system MS-DOS was a derivative of this earlier CP/M system.

I bring this up because to this day the default and most common Windows command-line the Command Prompt cmd.exe. Can trace its lineage back to this 1970s operating system from the defunct Digital Research.

After Windows became popular Microsoft merged both its MS-DOS shell and Windows overlay together into a operating system known as Windows 95. So for the longest time until the release of Windows 2000 and XP. Most consumers used a version of Windows that had a MS-DOS Command Prompt as a critical component of the operating system.

With Windows 2000, XP and later operating systems the MS-DOS functionality was pushed to the wayside and has been neglected ever since. Since the release of PowerShell in 2006 this neglect deepened. In fact the current iteration of the Command Prompt in Windows 8.1 is not much different to that which was found in Windows 95 nearly twenty years ago.

Windows 95 MS-DOS prompt
Windows 95 MS-DOS prompt
Windows 8.1 Command Prompt
Windows 8.1 Command Prompt

Windows Command Prompt

So let us start with the default Command Prompt.

In Windows use the Run feature (usually this can be done with the Windows key + R) and type in CMD. This should launch the default Command Prompt application which should be similar to the prompt above.

The Command Prompt CMD.EXE is the application that we use to input and receive feedback from the command-line shell. The shell is a collection of programs that allows you to interact with the Windows filesystem and operating system.

This is an important distinction because we are going to replace the Command Prompt but not the underlying command-line shell.

Replace the Command Prompt CMD.EXE with ConsoleZ

First create a directory on the root of your hard drive to contain all your command-line tools. I will use c:\terminal but some other directory name suggestions could be c:\prompt or c:\cp or c:\shell. I do recommend that you keep the directory name short and use a complete word without any spaces.

Created c:\terminal directory
Created c:\terminal directory

I am going to use an open sourced Command Prompt replacement application called ConsoleZ by Christophe Bucher.  Which itself is a fork of a more popular replacement application known as Console2 by Marko Bozikovic.

Go to the ConsoleZ download page and select the x86 for the 32-bit or amd64 for the 64-bit edition. Save the download to your c:\terminal directory and unzip it.

Download ConsoleZ
Download ConsoleZ

For all the rest of the article I am going to use and reference the 64-bit edition.

ConsoleZ is a portable application which means it does not need to be installed to work. This means you can easily archive and backup the whole c:\terminal directory into a zip file or copy it to a USB stick or a secondary computer without losing any of your customisations or settings.


Customize ConsoleZ

With ConsoleZ you have an alternative Command Prompt. I do the following customisations to give it a less cluttered look but you can pick and choose which ever you want.

Menu > View > Toolbar off
Menu > View > Status Bar off

ConsoleZ View adjustments
ConsoleZ View adjustments

Menu > Edit > Settings > Console

Change the Windows size Rows value if you want a longer command-line interface by default.

Change the Buffer size Row value if you want to keep a larger backlog of command-line output. By default 500 lines of text are stored to memory but increasing this value is useful if you want to display and scroll through large logs or text files. I usually increase it to its maximum value of 32766.

Windows and Buffer sizes
Windows and Buffer sizes

Menu > Edit > Settings > Appearance > Styles
Window Transparency > Alpha > Active Window

I set the application transparency as a low value of 20 but Windows 8+ users may not wish to use this aesthetic effect.

Customize the ConsoleZ font

Next up is I adjust the font selection which can be an important personal choice depending on your requirements and screen size. The default font is Courier New at a size of 10.

Menu > Edit > Settings > Appearance > Font

I change Name: to use Lucida Console in regular with a Size: of 10. But play around until you find a font and size combination that you like.

Font Selection
Font Selection

Command and Batch Scripts

In c:\terminal create a new subdirectory named cmd-scripts ie c:\terminal\cmd-scripts. This will contain my custom text based scripts that use the file extension .bat or .cmd. Modern Windows scripts use the .cmd (Command) extension while the .bat (Batch) is a MS-DOS legacy convention that functions exactly the same.

Colourise Your Command-line with ANSI Colour

In the 1980s and 1990s Windows and MS-DOS supported a widely used feature called ANSI escape sequences that allowed additional functionality such as colour text. But this support was dropped in more recent Windows editions. So I will show you how to re-implement ANSI escape sequence support using Jason Hood’s excellent ANSICon shell overlay.

Visit the ANSICon website at Download the file and save it to c:\terminal.

Now if you run the command type "ANSI Prompt Colours.txt" you should see a whole lot of garbled text dumped to the ConsoleZ window. These are ANSI escape sequences combined with plain text that Windows does not know how to interpret.

ANSI Esc garbled
ANSI Esc garbled

Download then save ANSICon to your c:\terminal directory and unzip it into its own ansicon directory. I keep it in c:\terminal\ansicon.

Use notepad or notepad++ to create a new file called shell-colour.cmd. Copy and save the following code hosted on my Github account to the shell-colour.cmd file.

If you are using 32-bit Windows or 32-bit ConsoleZ you should replace the following code c:\terminal\ansiconx64\ansicon.exe with c:\terminal\ansiconx86\ansicon.exe.

Back in ConsoleZ we will set the shell-colour.cmd as our default shell. This is a hack that loads ANSICon in addition to the default cmd.exe shell to give us ANSI escape sequence support without replacing the underlying shell.

Menu > Edit > Settings > Console

Under Shell: add c:\terminal\cmd-scripts\shell-colour.cmd

Console Settings shell-colour
Console Settings shell-colour

Now load a new tab.

Menu > File > New Tab > Console 2

Or relaunch ConsoleZ. In c:\terminal display the ANSI Prompt Colours.txt using the following commands.

cd terminal
type "ANSI Prompt Colours.txt"

If you see coloured text congratulations you now have ANSI escape sequence support.

ANSI Esc working
ANSI Esc working

Customise and Colourise the Prompt

The default Windows text input prompt only lists the active drive and current directory. As a frequent Linux Bash shell user I like a bit more flare and information to my prompt.

Use notepad or notepad++ to open your existing shell-colour.cmd file in c:\terminal\cmd-scripts. Copy and save the following code hosted on Github to the shell-colour.cmd file.

Notepad++ editing shell-colour
Notepad++ editing shell-colour

This code adds a couple of new commands. The echo command displays text to your command-line while the prompt command customises the input text prompt. The strings that are wrap within percentage symbols % are environment variables that are accessible from the shell. They allow you to display tidbits of information that are stored by Windows to the computer memory. A complete list of environment variables can be found at

The prompt command has some rather cryptic ANSI escape sequences that introduce colours. The $E string is a prompt argument to display escape characters which conveniently is needed by the ANSI escape sequences as a trigger. The [number;number;40m is a code sequence used to trigger an effect. A list of ANSI escape effects and colours is listed on Pueblo.

Reload ConsoleZ or open up a new tab to apply the changes. You should see a more information pack and colourful input text prompt.

The first part of the prompt in green displays the USER @ DOMAIN while the second part in blue displays the active drive and path.

Colour prompt
Colour prompt

Set a Default Directory at Launch

By default ConsoleZ sets the active directory to the location of its application which in my case is c:\terminal\ConsoleZ.x64. But you can easy change this by doing the following.

Menu > Edit > Settings > Console

Set Startup dir: to the path of your choosing such as c: or you can use a Windows environmental variable such as %userprofile%.

Startup directory setting
Startup directory setting

Create a Run as Administrator tab

A neat feature of ConsoleZ is the ability to create shell tabs with different user account permissions including those of an Administrator. As there are times when less restricted access is required to interact with parts of the Windows and its settings.

In ConsoleZ do.

Menu > Edit > Settings > Tabs > Add

In the Main tab change the Title: value to something more meaningful such as Administrator Console.

Point the Icon: to C:\Windows\System32\imageres.dll and select the yellow and blue shield icon which represents Run as administrator.

Under Shell make sure the Run as current user option is selected and the Administrator checkbox is checked.

Press OK when done. You now have a new Run as administrator tab accessible from ConsoleZ.

Admin tab
Create an Administrator Console tab

Configuring Paths

To run most tools and software from the command-line the active directory has to be the same as the location of the tool.

For example if I am in C:and try to run our script shell-colour.cmd that is located in c:\terminal\cmd-scripts. The shell returns the error “’shell-colour.cmd’ is not recognized as an internal or external command, operable program or batch file”. Which basically means the shell could not find the command I was trying to run.

Shell-colour not found
Shell-colour.cmd not found

But by using the Windows environment variable called %PATH% we can add additional directories that the shell will scan in when it searches for the existence of programs.

Configuring the %PATH% variable in Windows is a bit messy using the default Windows Control Panel option. So I prefer to use a 3rd party portable tool called Rapid Environment Editor by Oleg Danilov.

Download either the 64-bit or 32-bit editions, unzip and run the editor.

Rapid Environment Editor
Rapid Environment Editor

The pane on the left contains a list of editable Windows system variables. While the pane to the right has variables that are restricted to your current Windows user account. You may notice there is a PATH= variable for both the system and the user panes.

I generally use the system PATH for my c:\terminal path links. But to modify this you need to run Rapid Editor in administrator mode.

Restart as administrator
Restart as administrator

By default Windows stores multiple values of the PATH in a single string separated by semicolons ;. Fortunately Rapid Editor allows you to list and edit each individual PATH entry.

PATH expanded
PATH expanded

To add new directories to the PATH, right-click the PATH= string in Rapid Editor and select Add value or use the Alt+Ins keyboard combination.

Add value
Add value

Press F7 or select Insert directory path… Then point it to your c:\terminal\cmd-script directory. You should now have a new entry in your PATH list. Save the changes by pressing the Save icon or using Ctrl+S.

Insert directory path
Insert directory path

Now for changes to have an effect you have to either restart ConsoleZ or open up a new tab. To test that the PATH modifications are active, issue a cd command to return to the directory root and try running the shell-colour command again. If it works then congratulations you can now run any custom scripts placed into c:\terminal\cmd-script from anywhere in your command-line.

PATH modification works
PATH modification works (yellow highlight was added by myself)

Launch Windows Notepad++ or Notepad from the Command-line

While I love using command-line shells I generally prefer a GUI when it comes to programming or editing text files. I use the following script to to edit a file from the command-line. It launches Notepad++ and opens the file supplied but you can use any text editor including Windows Notepad located in C:\Windows\System32\notepad.exe.

Download and save edit.cmd to your c:\terminal\cmd-scripts directory.

In a new session of ConsoleZ test the edit script with the following commands.


edit "ANSI Prompt Colours"

Edit command
Edit command
EDIT command launches notepad++
EDIT command launches notepad++

To regain your prompt either close Notepad++ or press Ctrl+c in ConsoleZ.

Congratulations the core of this guide is complete. The remainder paragraphs are optional so you can pick and mix which topics you wish to implement. These include adding extra Windows and Linux tools; installing and running Node.js, Perl, PHP, Python or Ruby scripts from the command-line.

Useful Windows Command-line Tools

Besides the complete collection of commands and command-line programs built into Windows that you can discover at and There are quite a number of useful third-party command-line programs that are available for the Windows platform. I like to keep these contained in a single directory at c:\terminal\bin. Bin is an abbreviation for binary an alternative term for a program file.

Create the directory c:\terminal\bin and then using the processes covered in ‘Configuring Paths’ add c:\terminal\bin to your PATH variable. Restart ConsoleZ and so that any programs placed into c:\terminal\bin should be accessible from anywhere within your shell.

Here are a few Windows native command-line tools I recommend.

There are a number of useful tools created by NirSoft.
I personally use nircmd for use with various command scripts as well as WirelessNetConsole, bluetoothcl, whosip and whoiscl.

Microsoft’s Systeminternals also has a few command-line utilities.
AccessChk, Coreinfo, Handle, ProcDump and PsTools Suite.

For handling compressed files.
7-Zip Command Line Version.
UnRAR for Windows

ImageMagick can be used to convert and create image files.

Git for source-code management.

MySQL Utilities.
SQLite portable database tools.

Some other potential sources for standalone utilities include.

Joeware Utilities.

Linux/Unix Terminal Command-line Tools for Windows

As a frequent Linux Bash shell user there are a number of tools I miss on Windows. Fortunately as much of Linux is open sourced many of those tools have been ported over. Unfortunately some of these ports are horribly out of date so they should be best avoided.

There are also a couple of popular Windows open source C compilers such as Cygwin and MinGW that contain ports of Linux terminal tools. But in my opinion their use are overkill and many of the ported tools are rather old.

My favourite collection of ported Linux tools for Windows is the still current GOW (GNU On Windows) by Brent Matzelle.
It includes a large collection of GNU terminal tools and programs including cURL, gawk, grep, ls, nano, tar, vim and many more.

Download the latest release of Gow and install it to a directory of your choosing. I prefer placing it into c:\terminal\gow instead of its default directory C:\Program Files (x86)\Gow.

Gow will automatically copy all its application and configure PATH settings so you can use it straight away after a ConsoleZ reload or a new tab.

Gow setup
Gow setup
Testing Gow
Testing Gow

One command Gow does not replicate is the Bash alias ll (double L). I have created a command script that replicates that functionality using Gow’s ls command. Save ll.cmd to your c:\terminal\cmd-scripts directory to use it.

ll In terminal
ll In terminal


Finally any useful terminal needs a copy of OpenSSH for remote logins. While there are many out of date variants of OpenSSH on Windows. I personally use OpenSSH for Windows ported by which is the most up to date port I have found. Simply download the current ‘New’ version from their website, the download link is named setupssh-6.[version].exe and run the setup installer. The installer should automatically configure the %PATH% for you.

To test OpenSSH just output its version. If it doesn’t match the version number listed on site (6.7 p1 as of writing) then you probably have one or more copies of OpenSSH on your system that are taking president in your %PATH%. Tools such as Cygwin or Git can have their own out of date ports of OpenSSH.

ssh -V

ssh version
ssh -V


The popular Linux synchronisation tool rsync also has a Windows port.

ITeF!x offers a commercial package of the daemon/server with a GUI client and a free edition of the command prompt client.

The free edition comes without an installer. So I just unzip it to C:\terminal\cwrsync and use Rapid Environment Editor to add a path to point to it.

To test that rsync is working.

rsync --version

rsync --version
rsync –version

Run PHP Scripts from the Windows Command-line

Download a copy of PHP For Windows from
Unzip the PHP package to a directory of your choosing. I will use c:\terminal\php.
Without any arguments PHP does not do much in the terminal. So I created a command script to display the language version as the default behaviour. Download and place the php.cmd script into your c:\terminal\cmd-script directory and make any changes if needed.

To test your PHP configuration and the script run the command php.


Download the hi-php.php command line script and run it. It will print Hello, World! to the screen and quit.

php hi-php.php

php php-hi.php
php php-hi.php

Now to associate script files with the .php extension to run under our php.cmd script you do the following commands in a Run as Administrator session of ConsoleZ.

assoc .php=PHP.File
ftype PHP.File=c:\terminal\cmd-scripts\php.cmd "%1" %*

assoc .php
assoc .php

Test your configuration and run php-hi.php.

You can refine this one step further so you don’t even have to type the .php file extension. Open the Rapid Editor and under System variables add the value .PHP to the PATHEXT variable. In a new ConsoleZ tab you should be able to run hi-php without the file extension.

php-hi without extension
php-hi without extension

Run Perl Scripts from the Windows Command-line

For Perl I use Strawberry Perl for Windows portable edition which can be downloaded from its website.
Unzip it to a directory of your choosing. I place it in c:\terminal\perl.
Download and place the perl.cmd script into your c:\terminal\cmd-script directory and make any changes if needed.

To associate script files with the .pl extension to run under our perl.cmd script you do the following commands in a Run as Administrator session of ConsoleZ.

assoc .pl=Perl.File
ftype Perl.File=c:\terminal\cmd-scripts\perl.cmd "%1" %*

Download the command line script and run it. It will print Hello, World! to the screen and quit.

You can refine this one step further so you don’t even have to type the .pl file extension. Open the Rapid Environment Editor and under System variables add the value .PL to the PATHEXT variable. In a new ConsoleZ tab you should be able to run hi-perl without including the file extension.

Run Python Scripts and Programs from the Windows Command-line

The default Windows download of Python 3.4+ automatically configures itself to enable you to run Python scripts and programs from the Windows command-line.

Download and install a Python edition of your choice from

To test that Python has installed itself correctly you can download and run

Run Ruby Scripts and Programs from the Windows Command-line

At the time of writing Ruby is a little bit behind on the Windows platform. The current build of Ruby is at edition 2.1.x, while the recommended edition to use on Windows is 1.9.x. So with this in mind I recommend using the RubyInstaller for Windows which is a self-contained Ruby install.

Run the setup program and when prompted make sure you select both checkboxes for the options below.

Add Ruby executables to your PATH
Associate .rb and .rbw files with this Ruby installation

Install and setup Ruby
Install and setup Ruby

To test your Ruby installation download and run hi-ruby.rb.


Run io.js or Node.js and Javascript from the Windows Command-line

Download and install the latest io.js release from or its predecessor Node.js release from Both io.js and Node will create the required PATH variables for you. Unfortunately most Windows editions associate files with the io.js, Node and JavaScript .js file extension to the legacy Windows Script Host application. It is probably best that you do not overwrite the Windows Script Host configurations and instead run all Node and JavaScript scripts using the node or iojs command.

iojs somescriptfile.js
node somescriptfile.js

To test your Node installation download and run hi-node.js.

node hi-node.js

node hi-node.js
node hi-node.js

Creating Interactive Scripting Shells

Another great feature of ConsoleZ is because it offers separate, tabbed environments you can use it as an interactive programming tool.

In ConsoleZ do the following.

Menu > Edit > Settings > Tabs > Add

In the Main tab change the Title: value to Python 3

Point the Shell: to the language interpreter. For a default Python 3.4 installation I have it pointed to c:\python3\python.exe.

Under Main you can set the Icon: value which usually should be the same as the Shell: value.

When done, press OK and you now have a new Interactive Python shell tab accessible from ConsoleZ.

Create a Python shell
Create a Python shell

For an interactive Ruby shell you need to set the Shell: value to c:\ruby193\bin\irb.bat (or wherever your Ruby installation is located). And you probably want to set the Icon to c:\ruby193\bin\ruby.exe.

Create a Ruby shell
Create a Ruby shell

PowerShell in ConsoleZ

While this guide mostly uses the standard CMD.EXE shell. Microsoft provides an alternative, feature rich and overall better shell known as PowerShell. PowerShell is an optional download for most consumer versions of Windows and offers a more powerful and complicated command-line. Fortunately PowerShell can also easily be incorporated into ConsoleZ.


Within ConsoleZ do the following.

Menu > Edit > Settings > Tabs > Add

Add a new tab and set the Title: to PowerShell.
Set the Shell: value to %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe
Set the Icon: to use that same file path.

Create a PowerShell profile
Create a PowerShell profile

If you want you can also create an administration version of this tab. Select your newly created PowerShell tab and click the Clone button. Change the Title: to Administrator PowerShell and the Icon: to use the PowerShell icon with Administrator shield. Finally make sure the Run as current user option also has the Administrator check-box checked.

Create a PowerShell administrator profile
Create a PowerShell administrator profile


Congratulations, you’re done. The GitHub repository of the code in this article


KeePass 2, Password Management

With the recent admission from Adobe that it lost over 150 million user accounts from its online database in October. I thought it would be wise to go over my tool of choice for password management, KeePass.

In the case of the Adobe breach hackers managed to get the email addresses, passwords, credit/debit card details including expiry dates and more for every Adobe user account created on the system. One can not understate gravity of this breach. Losing banking details is bad enough but Adobe also used reversible encryption to store the user passwords. This means all 150 million passwords will eventually be unencrypted and passed around the Internet to multiple dubious organisations and persons.

That will be an immense problem as most people use the same login and password combination for multiple online accounts. That could potentially grant hackers, spammers and identity thieves access to hundreds of millions of online accounts with some simple trial and error.

As of writing (14th November 2013) a list of the top 100 passwords used within the system has been reversed by security researchers who obtained a copy of the leaked database. Nearly 2 million accounts used the password ‘123456’. Another 446,000 used ‘123456789’ and 346,000 used ‘password’.

Now we all know why people use simple passwords as they are easy to remember and simple to type. This has become more pressing as people use touch devices to reach online accounts that make the typing of complicated passwords difficult.

A major problem with this Adobe breach is that not only do the victims have to change the password of their Adobe account. They need to change all the passwords to other accounts that use the same login and password combination. Probably the first targets for this stolen login data will be the usual suspects, Banks, Ebay, Paypal, Facebook, Amazon, Google, Apple and Twitter.

This is why you should use a password management application such as KeePass. A password manager allows you to not only use strong, complicated passwords that you don’t need to manually type in at each request. It makes the use of unique passwords for online accounts simple. So if a security breach were to occur with another online service, a stolen unique password would be only useful for that compromised account.

KeePass – The Good, The Bad, The Ugly

In technology there is a constant conflicting battle of convenience verses security. The use of KeePass follows the path of security at the expensive of convenience and that can make its use complicated and confusing for new users.

The trick to using KeePass is to ignore the many choices it throws at you and focus on what you intend to use. I hope that this article will negate the potential steep learning curve that can come with early usage of KeePass.

One of many options that new users shouldn't bother with.
One of many options that new users shouldn’t bother with.

KeePass – Install

KeePass can be downloaded from

Unfortunately the download page is as complicated as the program itself. As a Windows user you want the Professional Edition. Don’t worry, despite the name it is free and open-sourced. I would recommend the Portable KeePass (ZIP Package) download over KeePass (Installer EXE for Windows). As the portable edition makes it easier to make copies and backups your KeePass configurations.

When downloaded, right-click the KeePass-2.*.zip file and Extract All. Follow the wizard to unpack the file to where you want KeyPass stored. I usually unpack it and keep it in C:\Portable\KeePass\.

In the extracted directory you should see KeePass.exe which will launch the KeePass application. If you want to create a shortcut to your desktop, right-click the program and select Send to > Desktop (create shortcut). Otherwise launch the application. I’d recommend for Windows 7+ users to right-click the program in the Taskbar and select Pin this program to the taskbar.

Create a password database

When you run KeePass for the first time the application without a database is loaded. Create a new database by selecting the New button (or press Ctrl-N on your keyboard).

First time run of KeePass
First time run of KeePass
Press the new button
Press the new button

In the Create New Password Database dialogue navigate to where you want to store your KeePass database. You can store it in a directory that automatically synchronises online with a cloud service such as DropBox, Microsoft SkyDrive or Google’s Drive. This will backup your database which is very important and allow you to share it with KeePass compatible mobile applications.

I will create a database called DevTidbits.kdbx and store it in D:\Dropbox\KeyPassData\.

Create a new KeePass database
Create a new KeePass database

You will now be prompted for a Master password at the Create Composite Master Key dialogue. KeePass will prompt for this database access password at startup. So make sure it is not too difficult to type and something you will remember.

Create a secure key file for the database

For added security I recommend selecting the Key file / provider check-box and press the Create … button. This will further encrypt the database whereby it will need both the Master password and the key file before being accessible. If the key file is deleted or lost the database will not be usable.

Create Composite Master Key
Create Composite Master Key

For simplicity and the purposes of this tutorial I will name the key file DevTidbits.key and store it in my Downloads directory at C:\Users\Ben\Downloads\misc\. For added security you should give it a more obscure name and store it elsewhere on your hard drive or even on a USB stick. But make sure you keep multiple copies for backup!

Create a new key file
Create a new key file
Create Composite Master Key - Key file / provider
Create Composite Master Key – Key file / provider

The idea of a separate key file is that it be kept separate from your encrypted database and if you’re paranoid off the Internet altogether. That way if the database is ever lost or copied by people who shouldn’t have access. You can create and apply a new key to your KeePass database. Then permanently delete the old key file used by the comprised database to make it inaccessible.

At the Entropy Collection dialogue you are asked for some Random mouse input and Random keyboard input. Sporadically move your mouse cursor over the black and white texture in the mouse input until the Generated bits bar is green and reaches 256 Bits. Then in the text area of the Random keyboard input type in a large number of random characters. The purpose of this randomised data is to generate a unique key tied to your KeePass database that will never be replicated again.

Entropy Collection
Entropy Collection

Finally at the Create New Password Database – Step 2 dialogue in the General tab. Give your database a name at the Database name: input and a description at the Database description: text area.

Create New Password Database - Step 2
Create New Password Database – Step 2

Further information: KeePass – Composite Master Key documentation.

Configure columns to display passwords

Once done KeePass should load your new database and propagated with some sample groups and entries. As I have customised KeyPass earlier your results may look a little different to mine. For example my passwords are showing while by default KeyPass hides these. To customise your own display select View from the top menu and Configure Columns… To show passwords select Password and deselect Hide data using asterisks. This is not recommended if you are intending to use KeePass in a public environment such as an open plan office or cafe.

View and Configure columns
View and Configure columns
Configure Columns
Configure Columns

KeePass Basic Usage

Back at the KeyPass main screen I have highlighted the two main panes. The left pane which I marked with a 1 has the KeyPass groups. This pane operates like a computer directory system where you can store multiple password entries. The entry pane marked as 2 displays a list of the entries in the active group.

Sample Entries
Sample Entries

When you select a password entry you can right-click it to bring up a menu. The two most useful options in this menu are the Copy User Name (Ctrl-C) and Copy User Password (Ctrl-B). Take note of those two keyboard combinations as they will be in use often. Both options will copy the relevant data from the selected entry to your clipboard stored in the computer’s memory. In a web page login, password input or a text editor like Notepad you can Paste text or use the Ctrl-V keyboard combination to insert the clipboard data.

Menu to copy the entry's user name and password
Menu to copy the entry’s user name and password
Paste Clipboard text
Paste Clipboard text

For security the data will be automatically deleted from the computer’s memory after 2 minutes. You can change this countdown at any time by using the Tools > Options… top menu and adjusting the Clipboard auto-clear time (seconds) value.

Password in clipboard self-destruct countdown.
Password in clipboard self-destruct countdown.
Menu for Tools and Options...
Menu for Tools and Options…
Options, Clipboard auto-clear time in seconds
Options, Clipboard auto-clear time in seconds

Further information: KeePass – Using Stored Passwords documentation.

Add a new entry

In the groups pane (the one on the left) select a group such as eMail.

Select the eMail group
Select the eMail group

Now either use Ctrl-I (i for insert) or press the Add Entry button to bring up the Add Entry dialogue.

Add Entry button
Add Entry button

From here all fields are optional but I generally give each entry a Title, User name, Password and a URL for the site.

Add Entry dialogue
Add Entry dialogue

In the Password field you can either copy or type in your existing password or generate a new one using the Generate a password button.

Generate a password button
Generate a password button

The button will bring up a menu that will list some predetermined randomisers to create alpha-numeric passwords at a predetermined length. This is to help alleviate the issue where some websites have length restrictions on user account passwords.

The 40-Bit Hex Key will generate a short 10 character alpha-numeric password.

The 128-Bit Hex Key generates a standard 32 character alpha-numeric password.

The 256-Bit Hex Key generates a stronger 64 character alpha-numeric password though some websites may reject this length.

Select a hex key.
Select a hex key.

Remember if you decide to generate a new password for an existing web account. You will also need to login to the site and change its existing login password.

When you enter a password the Quality bar indicator lets you know the strength of the password based on the variety of characters and length.

Password quality bar
Password quality bar

Further information: KeePass – Password Generator documentation.

Complex password complications

A word of warning, you probably do not what to use a long complicated password for a service like Apple ID or Google Accounts as unfortunately they’re impractical. Both companies use unified passwords for all their products and services. Which means you could find yourself needing to enter these passwords with smart devices such as phones, tablets and digital media players that do not have keyboards or access to your KeePass database.

Add a group

To create a new group first select in the group pane where the group should go, either the database name or within an existing group.

Right-click to bring up the group menu and select Add group.

Add Group menu
Add Group menu

Give the group a distinct name, choose an icon if you wish and press OK. You now have an additional group to add or move entries into.

Add Group
Add Group

To move an entry into a group just select it then drag and drop. You can move groups around the same way.

Remember you can use KeePass for other data that needs security other than just website and network passwords. Some suggestions could be software registration keys and serial numbers. Wallet content such credit/debit card, social security, frequent flyer and other membership details. Private phone numbers. Private or public keys used for email, file and system encryption.

Save changes

By default KeePass does not automatically save any changes to your database.

You can tell the database has changes that need saving by the asterisk * next to the database file name. As shown in the title bar and highlighted in my screenshot by the orange arrow.

Unsaved database
Unsaved database

To save your changes press the Save button or use the Ctrl-S keyboard combination.

If you quit KeePass before saving the changes you will be prompted with Save database changes before exiting KeePass dialogue. There is a check-box there to Automatically save when closing/locking the database for future exits.

Save database changes before exiting KeePass?
Save database changes before exiting KeePass?

Use search to find entries

Once you find that the collection in the database is quite large you can use Quick Find to hunt for each entry. For this reason I always give the Titles to my entries descriptive names using both the company title and the product brand.

Quick Find search
Quick Find search

Otherwise it maybe difficult to quickly filter some searches. If I use as my default login for most websites and then search for ‘hotmail‘ for my Microsoft log-in data. All the entries with the User name will be in by the results. So I’d instead give my Hotmail entry the title of Microsoft Hotmail and use the search term ‘microsoft‘.

KeePass also offers a more fine-tuned Find dialogue that pops up with the Find button or the Ctrl-F keyboard combination.


You’re done

Congratulations that covers the basics of KeePass on Windows. There are many software ports for other platforms including mobile devices and JavaScript, some of which I hope to cover at a later date.

Run DOS on the Raspberry Pi – Use rpix86 to turn your Pi into a 1980s super-computer.

Please feel free to jump straight to the ‘DOS on the Pi‘ if you have no interest in a computer history lesson.

What is DOS?

DOS was a semi-open standard that included a common operating system for desktop computers in the 1980s and early 1990s. The hardware used Intel x86-compatible CPUs while the operating system for this architecture was DOS the Disk Operating System. Note that the DOS acronym was also used for a number of incompatible and unrelated computer systems such the Apple DOS and Atari DOS.

As DOS was a primitive, text only operating system that was easy to reverse-engineer. A number of companies made their own editions of DOS and for the most part they were all compatible with each other. The most common x86-edition of DOS was Microsoft‘s MS-DOS but IBM also had their own PC-DOS as did Digital Research with DR-DOS.

MS-DOS was Microsoft’s precursor to its Windows operating system. In the 1990s many editions of Windows were simply a layer of software designed to load up after MS-DOS or later contained MS-DOS built-in.

Off the shelf x86-compatible hardware combined with the DOS operating system became the prominent computer gaming system in North America from the late 1980s until Windows 95. Other regions such as Europe preferred cheaper, propriety systems such as the 8-bit Commodore 64, Amstrad CPC, Sinclair Spectrum. Or the equally as expensive but more user-friendly 16-bit Atari ST and Commodore Amiga lines. In these parts of the world x86-compatible computers only became popular after the early 1990s.

How did DOS come to be?

DOS started out as a clone of the early microcomputer operating system CP/M by Digital Research. CP/M was created in 1973 to run on the Intel Intellec-8 development system which happen to be the first microcomputer sold in the USA.

By the end of the 1970’s CP/M was the most popular microcomputer operating system for professionals and business. Once IBM decided to enter this burgeoning market they wanted to use CP/M on their future products but they couldn’t come to an amicable agreement with Digital Research.

So IBM asked Microsoft to develop a clone of CP/M for their first microcomputer that they dubbed the Personal Computer or the PC. But at this early stage of its life Microsoft had never developed an operating system but bluffed IBM to believe otherwise.

Microsoft discovered a small software company known as Seattle Computer Products who had a x86-compatible CP/M clone called 86-DOS. Microsoft signed a non-exclusive contract to market and distribute this operating system and on-sold it to IBM. Just weeks before IBM’s major PC public announcement Microsoft obtained near-exclusive rights for Seattle’s 86-DOS.

In August of 1981 IBM launched the IBM PC. A machine that bought mainstream attention to microcomputers and made them respectable in the eyes of the public. Every IBM PC sold was operating off a royalty-free copy of Seattle Computer Products 86-DOS operating system re-branded as PC-DOS.

IBM had always been a hardware company and believed there was little money in software. Apple to this day follows that same ethos. What IBM failed to realise was they had designed their machine with off the shelf parts that anyone could buy. The only thing other manufactures lacked to create their own PCs was the operating system. But due to IBM’s non-exclusive operating system agreement. Microsoft was free to re-brand their own version of IBM’s PC-DOS and to sell it to potential companies who wanted to make copycat IBM PCs. That small mistake by IBM launched Microsoft, who became one of the world’s largest companies.

Why DOS?

Many popular franchises, genres and software companies were first launched or popularised on DOS. Warcraft, Grand Theft Auto, Sid Meier’s Civilization, Monkey Island, X-Com, Need for Speed. The first person shooter, the graphic adventure, simulators, real-time strategy are some examples. So DOS would be well worth investigating for anyone who has an interest in retro-gaming from this era.

Why not DOS?

It must be said that the DOS ecosystem has to be the most complicated and arcane mainstream gaming system of all time. The same openness that made it popular was also what made it so horrible to use.

Backwards compatibility was a major benefit and selling point of the DOS ecosystem. But at the same time it held it back and made it much more complicated than necessary. By the late 1980s, its text based 1970s lineage was horribly primitive and outdated.

Hardware support was complex. Often users had to manually change and configure their operating system for each item of software they ran. What unfortunately started off as a simple and clean system for IBM in 1981 evolved into an ad hoc open platform that it was never designed for.

Worse was hardware driver support. For the most part new hardware had to be directly supported by software. This meant that a user’s existing game collection would often not support new hardware simply because it didn’t exist when the software was written.

Finally many game developers targeted the lowest common denominator hardware. This meant you could have had the latest and most expensive hardware yet it could have been ignored by new release software that was designed for a machine that was 5+ years old.

Fortunately much of this hardware complexity is now avoided when using software emulation such as rpix86 on the Raspberry Pi.

Snooper Troops supporting IBM’s Color Graphics Adapter  in 1982.
1984’s Troll’s Tale on IBM’s PCJr.
IBM’s Enhanced Graphics Adapter in use in 1986’s King’s Quest III.
IBM’s Multicolor Graphics Array in 1987 allowed photo like graphics in Moebius: The Orb of Celestial Harmony.
King’s Quest V from 1990 was one of the first titles to fully embrace IBM’s Video Graphics Array with upto 256 colours on-screen.
By 1992 some games such as Links 386 Pro had embraced hi-resolution Super VGA graphics.

DOS today?

DOS is still used on embedded systems that need a simple and small, text only operating system. Today most of the DOS community centres around the open source FreeDOS that offers modern tools as well as legacy compatibility.

DOS on the Pi.

At the time of writing in April 2013 the only acceptable DOS emulator for the Raspberry Pi is rpix86 by Patrick Aalto found at

What PC hardware does rpix86 emulate?

CPU: 80486 processor, including the protected mode features (for running DOS4GW games) but without virtual memory support. The emulation runs at a speed of around 20MHz 80486 (which equals a 40MHz 80386) machine.
Memory: 640KB of low memory, 4MB of EMS memory and 16MB of XMS memory.
Super-VGA graphics, with a maximum resolution of 640×480 with 256 colors.
SoundBlaster 2.0 (including AdLib-compatible FM sounds) sound card.

This combined hardware could have belonged to a very expensive high-end PC in the very late 1980s. Or a more affordable and common mid-range PC in the early 1990s. In that later era the audio would have been the machine’s weakest point. The graphics and memory would have been very good while the CPU would have been average.

My article DOSBox Sound Emulation talks about the huge variety of audio options on the x86-compatible platform while DOSBox Graphic and Machine Emulation covers the variety of graphic modes.

What operating systems does rpix86 support?

Unfortunately only the obscure 4DOS is supported by rpix86. 4DOS is a rather full featured and complex edition of DOS created by a small company out of Germany. While it was great for daily users of DOS based PCs back in the 1990s. It is an overkill for our purposes of PC gaming emulation on the Pi.

4DOS downloads

Raspberry Pi set up.

This guide assumes that a clean install of Raspbian is in use on the Raspberry Pi.

Raspbian can be obtained from
A guide for installing Raspbian to an SD card for use with the Raspberry Pi is at


Start your Raspberry Pi with a new install of Raspbian and at the Raspbian Raspi-config configuration tool do the following.

First select update to make sure you’re using the latest version of the configuration tool.
Select expand_rootfs so the free space on the SD card is fully utilised.
Select change_timeszone to set your Pi’s time zone.
Select boot_behaviour and set Should we boot straight to desktop? as No.
Select change_pass to change the default pi user password.
Select <Finish> to exit.

Now in terminal if you ever need to return to Raspi-config.

sudo raspi-config

Now lets update Raspbian and any out-of-date software.

sudo apt-get upgrade -y

DOS set up.

We are going to use a directory called /dos located in our pi home directory /home/pi/ to store our rpix86 related files. For the commands shown below ~/ in Linux is treated a short-cut for the pi user home directory.

mkdir ~/dos
cd ~/dos

mkdir ~/dos
mkdir ~/dos

This /dos directory will also double up for a mock hard drive that rpix86 will use for DOS.

Download the latest version of rpix86 the DOS emulator.


wget rpix86.tar.gz
wget rpix86.tar.gz

Now uncompress and run rpix86. Please note that rpix86 will not work over a remote terminal connection such as SSH or Telnet.

tar -xf rpix86.tar.gz

rpix86 running under a remote terminal connection.

When prompted press Y to download the required 4DOS.COM command interpreter.

rpix86 unable to find
rpix86 unable to find

Once complete your Raspberry Pi should be running a DOS prompt.

rpix86 startup
rpix86 startup.

Basics of DOS shell.

At its core a DOS shell is essentially a self-contained program that allows commands to be run from a text prompt. It lets you interact with your computer’s file system and some hardware. By interacting with the file system you can launch programs; move, copy or delete files and modify the directories.

The rest of the DOS operating system comes in the form of extra programs, applications and hardware drivers that usually reside in a subdirectory. These additions can vary between different releases and iterations of DOS.

DOS command prompt explained.


C: Is the active drive. In DOS each floppy, CD, DVD or hard drive is assigned its own drive letter. The C: drive is usually reserved for the first hard drive while A: and B: are reserved for floppy drives.
\ Is the active directory. When a backlash is displayed the active directory is at the root of the drive.
> Is the beginning of the text prompt.
_ Is a blinking prompt that shows where the user’s typed input will display.


Please run the following commands in the DOS prompt. Note that unlike Linux, DOS  is not case sensitive.

List the current directory.


Display some basic 4DOS information.


Clear the screen.


Make a new directory named TEST.


Copy the file 4DOS.COM to the TEST directory.


List the content of the TEST directory.


Change the active directory to TEST.


Delete the file 4DOS.COM


Change the active directory to root.


Remove the directory TEST.


You can use the [Tab] key to auto-complete the names of files and directories that exist in the active directory.

4d [Tab]

You can cycle through previous commands typed at prompt by using the up and down arrow keys.

Quit rpix86.


DOS commands.
DOS commands.

Dumb DOS names.

DOS uses the FAT file system. The legacy FAT16 file system was first introduced into MS-DOS 4.0 and used through the late 1980s and early 1990s. It is horribly dated but can still be read by modern operating systems.

Because of its age FAT and DOS has a number of particular but important to know traits.

  • In DOS file and directory names can be at most 8 characters long. An additional identifier extension can be given but that can only be up to 4 characters and the first character must always be a period.
  • File usage is identified by extensions, many of which are still in use today. FILE.TXT is a plain text document. FILE.MP3 is an MP3 audio track. FILE.AVI is a video file. FILE.HTM is a HTML document. DOS script files use the .BAT (batch processing) file extension. Program files use .EXE (execute) or the .COM (command) file extensions.
  • Files and directories are case insensitive.
  • The backlash \ character is used to separate directory and file paths.

Fetch and use DOS software.

For the purposes of this article I will only use games sourced from the excellent DOS Game Archive. It only offers DOS software that is legal to distribute and use without payment.

Under the Copyright laws of many countries, copyright on software remains active until 50 or 70 years after an author’s death. This is designed to protect any potential long-term income and the resale value of a product.

The majority of dated, commercial computer software has no financial value and is usually long out of print. This is more apparent with software created for obsolete systems such as DOS. Copyright of this type is rarely defended by its owners and is unofficially known as abandonware.

While the act the downloading and use of abandonware is classed as piracy and is officially illegal. I personally do not see any serious legal or moral issues with downloading and using out of print, obsolete commercial software. But you should use your own judgement.

Where to get games?

You can find legal to download DOS shareware, freeware and public domain games at the DOS Game Archive

To find abandoned but technically illegal to download DOS software, type DOS abandonware into Google and go from there.

DOS Demo 1 – LucasArts, Day of the Tentacle Demo

The surreal and funny Day of the Tentacle was a huge success for LucasArts in 1993. This non-interactive demonstration is easy to run and shows the old DOS low resolution VGA graphics and SoundBlaster audio hardware at its best.

At the Raspbian Linux terminal prompt create the directory DOTTDEMO.

mkdir ~/dos/DOTTDEMO

Change the active directory to DOTTDEMO.

cd ~/dos/DOTTDEMO

Use wget to download the Day of the Tentacle demo from DOS Game Archive.


Uncompress the demo file using unzip.


Return back to the /dos directory and run rpix86.

cd ..

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the DOTTDEMO directory exists.


Change the active directory to DOTTDEMO.


List the content of DOTTDEMO but only display items that use the extension .EXE which are execute programs.


Run the program DOTTDEMO.EXE.


Congratulations, hopefully you have a working Day of the Tentacle demo.

To exit the demo press [Alt] X and follow the prompt.

wget DOTT

DOS Demo 2 – Sierra Online, Space Quest 3.

From early 1980s to the 1990s Sierra Online were all about pushing the envelopes of audio and visuals on the home computer. Space Quest 3 in early 1989 was certainly doing that on the PC. Unfortunately but common for the era, this demo requires a rather complex set-up procedure.

At the Raspbian Linux terminal prompt create the directory SQ3DEMO.

mkdir ~/dos/SQ3DEMO

Change the active directory to SQ3DEMO.

cd ~/dos/SQ3DEMO

Use wget to download the Space Quest 3 demo from DOS Game Archive.


Uncompress the demo file using unzip.


Return back to the /dos directory and run rpix86.

cd ..

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the SQ3DEMO directory exists.


Change the active directory to SQ3DEMO.


List the content of SQ3DEMO but only display items that use the extension .EXE which are execute programs.


Run the program INSTALL.EXE to tell the demo of Space Quest 3 what hardware our DOS machine supports.


In the Sierra Game Setup/Installation Program.

Press [Enter] at the Sierra’s newest products .. notification.

Select EGA/VGA with RGB monitor.

Select AdLib Music Synthesizer Card.

Select IBM or IBM-compatible keyboard.

Press [Enter] at the NUMLOCK is OFF confirmation.

Select NO for the MicroSoft compatible mouse.

Press [Esc] to skip the install the game on your hard disk.

Press [Enter] at the Installation is complete prompt.

Back at the DOS prompt list the content of SQ3DEMO but only display items that use the extension .BAT which are script files. Many older Sierra games used batch script files to launch their programs.


Run the batch script SQ3DEMO.BAT to launch the Space Quest 3 demo.


Welcome to Scenes From Space Quest III.

To exit press [Esc] to bring up the in-game menu and then select File > Quit.


Demo 3 – Windmill Software Digger

Next up is the rather simple Digger created by Windmill Software in 1983. The basic graphics and primitive audio was amazing compared to many early games created for the original IBM PC machines of this era. This game looks to be inspired by the 1982 Universal arcade game Mr. Do! Currently there is some audio emulation issues with this game when used on rpix86.

At the Raspbian Linux terminal prompt create the directory DIGGER.

mkdir ~/dos/DIGGER

Change the active directory to DIGGER.

cd ~/dos/DIGGER

Use wget to download Digger from DOS Game Archive.


Uncompress the game file using unzip.


Return back to the /dos directory and run rpix86.

cd ..

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the DIGGER directory exists.


Change the active directory to DIGGER.


List the content of the directory DIGGER.


Run the program DIGGER.EXE.


Welcome to computer video gaming 1983 style. Use the arrow keys to control your character or to exit Digger press [F10] a couple of times.


Demo 4 – Non-working Games

At the moment rpix86 is still an immature product and your mileage for running various software will vary. Some DOS software can be temperamental such as Epic MegaGame’s Zone 66 from 1993. It demands that we do not load any DOS XMS memory drivers but currently we can not do this in rpix86.

At the Raspbian Linux terminal prompt create the directory ZONE66.

mkdir ~/dos/ZONE66

Change the active directory to ZONE66.

cd ~/dos/ZONE66

Use wget to download Zone66 from DOS Game Archive.


Uncompress the game file using unzip.


Return back to the /dos directory and run rpix86.

cd ..

Now in the rpix86 DOS prompt list the content of the DOS root to make sure the ZONE66 directory exists.


List the content of ZONE66 but only display items that use the extension .EXE which are execute programs.


Run the program ZONE66.EXE.



Disappointingly we receive an abort error message This program can not run with an XMS driver!!! because our DOS operating system is not configured correctly. As you can see gaming on PCs in this era often required more technical knowledge than today.


Well that is it for this tutorial. I will continue to update this entry as new revisions of rpix86 are released for the Raspberry Pi. Feel free to ask any questions in the comments below.

Use USB hard disk & flash drives with your Raspberry Pi

This was meant to be a brief article on how to mount an external USB drive, but it quickly spiralled out when I starting writing about all the nuances and potential issues one might run into. So I have a created a quick summary of the commands on how to mount a drive below. But I highly recommend reading the rest of the article as there are a number of potential pitfalls with the Pi and external USB drives that are addressed.


To mount a USB drive:

sudo mkdir /mnt/usbdrive
sudo mount /dev/sda1 /mnt/usbdrive
ls /mnt/usbdrive

To list your file systems:

sudo fdisk -l
sudo mount -l
df -h

Before disconnecting a USB drive:

sudo umount /dev/sda1

Format a drive to EXT4

sudo mkfs.ext4 /dev/sda1 -L untitled

Add Apple OS X HFS+ read/write support

sudo apt-get install hfsutils hfsprogs hfsutils

Format a drive to HFS+

sudo mkfs.hfsplus /dev/sda1 -v untitled

Add Windows NTFS read/write support

sudo apt-get install ntfs-3g

Format a drive to NTFS

sudo mkfs.ntfs /dev/sda1 -f -v -I -L untitled

Add Windows/DOS FAT32 read/write support

sudo apt-get install dosfstools

Format a drive to FAT32

sudo mkfs.vfat /dev/sda1 -n untitled


The Raspberry Pi is a great and flexible little device. But one of its main limitations is the storage options if you wish to use the device as a file or multimedia server. SD memory cards are cheap and common with low-end specifications but reach an affordability and storage cap when their sizes increase. A cheap USB powered external drive with many times more space can be had for a similar price to a top capacity SD memory card.

USB Power Problems

A major limitation for running a USB drive on a Raspberry Pi are the power requirements. The Universal Serial Bus specification states that to adhere to the standard, up to 0.5A (amps) can be drawn from a single port.

My old Samsung G2 Portable 640 hard drive requires 0.85A to work which is 0.35A above the USB2 specification. This is not an isolated example, many modern desktop PCs and laptops supply a greater amperage than the standard 0.5A to their USB ports to support devices such as portable USB hard drives. Unfortunately the Pi cannot power many external USD devices such as hard drives as its USB ports are restricted to the standard amperage.

To get around this problem you need a powered USB hub. You attach the hub device to the Pi’s USB port, plug the USB hard drive into one of the hub’s USB ports and insert the hub’s power supply into a walled power socket. My tiny Logitech ‘Premium 4-port’ USB hub can share 2.5A between 4 devices which is more than enough to power my Samsung drive.

I would also recommend against powering the Raspberry Pi off the same USB Hub as a USB drive. For me this caused interference where the drive would momentary lose power.

Mounting A Drive

When a drive is mounted, it connects to your Pi and Linux recognises it. The drive is given a directory where you are able to access and modify its content. These directories are known as mount points and can be given any name that works for you but they should be placed in /mnt.

I will call my mount point ‘usbdrive‘. First we need to create a mount point.

sudo mkdir /mnt/usbdrive

Linux has the /dev directory that is in use to store special files that allow access to the computer’s hardware. The /dev/sd* collection of files represent drives. Each drive connected to your Raspberry Pi is given a letter.

/dev/sda Would be your first connected drive.
/dev/sdb Would be your second drive.

To mount the drive to your mount point ‘usbdrive‘.

sudo mount /dev/sda1 /mnt/usbdrive

The numeric 1 at the end of /dev/sda is a requirement and tells Linux to mount the first partition.

Partitions are beyond the scope of this article, but you can learn more about that at the Ubuntu Community Docs.

mkdir mnt usbdrive

Disconnect / Unmount A Drive

It is always advisable that you unmount a USB drive before unplugging it from its power source. This forces all queued data to be written to the drive before it loses power.

sudo umount /dev/sda1

You may need to use the -f force option if the drive will not dismount.

sudo umount -f /dev/sda1

If you use the shutdown -P -h 0 command to power down your Pi you do not need to use unmount.

Disk File Systems

A disk file system is the method an operating system stores and reads data on a drive. There is an endless list of disk file systems out there as each operating system seems to have their own native but incompatible system.

Linux as a number of native file formats but generally today the most common is the EXT (Extended File System) series which include ext2, ext3 and ext4.

Apple OS X uses HFS+ (Hierarchical File System Plus) otherwise known as Mac OS Extended.

Modern Microsoft Windows systems mostly use NTFS (New Technology File System).

Legacy Microsoft Windows systems and ancient Microsoft DOS systems generally use a variation of the FAT (File Allocation Table) which includes FAT, VFAT, FAT32 and exFAT.

High CPU usage with the ntfs-3 driver
Excessive CPU usage with the ntfs-3 driver that slows Samba (smdb) transfers
With EXT4 the file transfer using Samba has an additional 250%+ CPU resource available for use

Disk File Systems Compatibility

EXT has native support in Linux and the Raspberry Pi. It has no official support in Windows. There are free third party drivers available for Windows offering limited read/write EXT support such as the open source EXT2FSD or EXT2Read. Apple OS X users need to use the commercial Paragon ExtFS to enable full EXT support.

To enable Linux EXT4 support:
It is turned on by default on the Raspberry Pi.

HFS+ has restricted support in Linux. It can read HFS+ formatted drives but can only write to them if journaling is disabled. Windows has no native HFS+ support but there are paid solutions such as Paragon HFS+ for Windows.

To enable Linux HFS+ support:
sudo apt-get install hfsutils hfsprogs hfsutils

FAT is probably the most supported file system but it is also the most limited. Linux, Windows and Apple OS X all support FAT, VFAT and FAT32. ExFAT otherwise known as FAT64 is native to modern Windows and Apple OS X but has no support in Linux due to patient incompatibilities.

To enable Linux FAT32 support:
sudo apt-get install dosfstools

NTFS has read only support in Linux and Apple OS X. Third party drivers are available to add write support including the commercial Paragon NTFS and the open source NTFS-3g.

To enable Linux NTFS support:
sudo apt-get install ntfs-3g

Performance Issues & Which Disk File System To Use?

As a non-scientific test I took a 4GB video file and copied it to various file systems using my Raspberry Pi and the USB hard drive.

The worst performer by far was the NTFS-3g driver for Linux NTFS read and write support.

The transfer that 4GB file from my Windows 7 PC to the NTFS formatted USB hard drive took around a minute or two. The same file from the Raspberry Pi’s SD memory card to the NTFS formatted USB drive took 30 minutes to write and 23 minutes to read!

Performance for EXT3, EXT4 and FAT32 were about the same at 12-14 minutes to both read and write. This suggests that there is a bottleneck with either the SD memory card or USB drivers and not the file system.

If your drives are mostly used by the Pi my recommendation would be to use EXT4 on your USB drives. EXT4 is mostly the same as EXT3 with some extra minor features but it is widely supported in the Linux world, plus it is backwards compatible with EXT3 and EXT2.

FAT32 is the most compatible file system but has a restrictive 4GB file size limit.

EXT2, HFS+ on Linux and FAT32 lack journaling support that makes them prone to errors when used on portable drives. As these file systems can’t elegantly recover if they unexpectedly lose power.

FAT32, NTFS can not store Linux file or user permissions.

Format A Drive

To change the file system of a drive you need to format it. Linux allows you to format any supported disk format using the mkfs tool.

In the examples below you will notice an option followed by ‘untitled‘. These are optional volume labels to name your drive.

First you must unmount the drive you wish to format.

sudo umount /dev/sda1

To format a drive to EXT3 (Linux):
sudo mkfs.ext3 /dev/sda1 -L untitled

To format a drive to EXT4 (Linux):
sudo mkfs.ext4 /dev/sda1 -L untitled

To format a drive to HFS+ (Mac OS X):
sudo mkfs.hfsplus /dev/sda1 -v untitled

To format a drive to FAT32 (DOS and legacy Windows):
sudo mkfs.vfat /dev/sda1 -n untitled

To format a drive to NTFS (Windows):
sudo mkfs.ntfs /dev/sda1 -f -v -I -L untitled

I have applied a few options here that I will explain.
-f Fast Format. Due to the poor performance of 3g.ntfs on the Pi I highly recommend using the less CPU intensive fast format mode.
-v Verbose. By default the NTFS status output is limited so this lets you know what is happening.
-I Disable Windows Indexing. This improves the write performance of the drive but it will mean Windows Search queries used on this drive will take longer.

Format NTFS
Format NTFS
Format HFS+
Format HFS+
Format FAT32 (vfat)
Format FAT32 (vfat)
Format EXT4
Format EXT4

Automatically Mount A Drive

To simplify the process of mounting a drive you can add the drive’s information to the fstab settings file located in /etc/. I would recommend taking a look at the Ubuntu FSTAB community page for a deeper understanding of this file.

First run nano to edit fstab. The -Bw options tell nano to backup the file and not to use any line-wrap.

sudo nano -Bw /etc/fstab

You should already see some existing entries. Do NOT change these as the two /mnt/mmcblk0p entries are there to mount the SD card.

Add the following to the bottom of the file.

/dev/sda1 /mnt/usbdisk auto defaults,user 0 1

These are explained:

/dev/sda1 Is the location of the drive to mount.
/mnt/usbdisk Is the mount point, which is the folder to access the content of the drive.
auto Is the file system type, here you can set ‘auto‘ or force a file system type such as ext2, ext3, ext4, hfsplus, ntfs, vfat.
defaults,user Are mount options. You normally need to only supply ‘defaults‘. Though there are some others that maybe useful such as ‘ro‘ for read-only or ‘user‘ to enable write permission for all users. Use a non-spaced comma to separate multiple options.
0 A binary value used for debugging. It is best to keep this set at zero.
1 Pass number for a file system check at boot. ‘0‘ (zero) to disable or ‘2‘ to enable.

Save the changes to fstab.

[Ctrl] x
Y at the Save modified buffer prompt.
[Enter] for the File name to Write: /etc/fstab prompt.

nano etc fstab

The drive will mount at boot as long as it is attached to the Pi. If you want to mount the drive after you have plugged it in use mount with the automatic option.

sudo mount -a