Category Archives: Linux

Check File Copies with Linux Scripts

OpenSuSE logoFor several years, I’ve had an original Buffalo LinkStation NAS as my main fileserver. Being on 24×7, it’s gone through several fans and at least one hard disk, but it’s now time to retire it in favour of a LinkStation Duo which will give me more space, RAID capabilities and faster transfer speeds.

Naturally, as my main fileserver, it’s backed up. However when I copy the files to the new Duo, how do I know that they’ve all copied correctly and none have been missed? There are hundreds of thousands of files and checking each one by hand would be pointless.

Linux has lots of tools that tell me how much disk space is used, such as du, df and filelight, but they don’t always report back consistently between filesystems. Mostly for reasons of speed, they report the total size of the file blocks used to store a file and as block sizes can vary between filesystems, the total number of blocks used for a set of files will be different. For example, I have two folder sets that I know are identical and du -s on one reports 210245632 and on the other 209778684.

Fortunately, there’s an extra command line flag that will change the behaviour to take longer and sum the actual bytes. In this case, du -sb will return 214813009920 bytes on both filesystems. On the whole, I can be reasonably confident that if the total number of bytes used is the same between two filesystems, then all the files have copied correctly.

But what if the total number of bytes don’t match? How can I find the missing or truncated file? After thinking and tinkering, what I want is to get a list of files with each filesize from the old and new filesystems and then compare the two. And here’s how you do it (each section here goes on one line).

find /home/old_folder -type f -printf '%s %p\n' |
sed 's/\/home\/old_folder\///' | sort > old.txt
find /home/new_folder -type f -printf '%s %p\n' |
sed 's/\/home\/new_folder\///' | sort > new.txt
diff -wy --suppress-common-lines old.txt new.txt

If you aren’t used to Linux, this can look a bit scary, but it’s not really. The first two lines create the text files with all the files and the third line compares the two. The the first two lines are much the same in that they do the same commands but on different filesystems. There are three sections, find to list all the files, sed to chop the directory path off, and sort to get all the files in some sort of order. Here are some explanations.

find – finds files
/some/folder – where to start finding files
-type f – only interested in files (not directories or links)
-printf ‘%s %p\n’ – only print the filesize (%s) and the full pathname (%p) on each line (\n)

sed – processes text
‘s/x/y/’ – means replace x with y. In our instance, it’s replacing the leading folder path with nothing. It looks worse than it is because the slashes in the path need to be escaped by a preceeding backslash, so you get these \/s everywhere.

sort – sorts text
> file.txt – copy everything into a text file.

One of the clever things about Unix-like operating systems is that you can pass information from one command to another using a pipe. That’s represented by the | symbol, so the find command gets the information on files and files sizes, passes it to sed to tidy up which then in turn passes it on to sort before sending it to a file.

After running this set of commands on the old and new filesystems, all that needs done is to compare the files. Let’s look at the third and final command.

diff – compares files line-by-line
-w – ignore whitespace (spaces and tabs)
-y – compare files side-by-side
–suppress-common-lines – ignore lines that are the same
old.txt new.txt – the two files to compare

So what might the output of the diff be? If all the files copied correctly, you’d get absolutely nothing. Other possibilities are that the file partially copied or didn’t copy at all. Here’s what the output might be like.

598 i386/compdata/epson3.txt | 500 i386/compdata/epson3.txt
598 i386/compdata/onstream.txt <

The numbers at the beginning of the entry are the number of bytes, so the first line shows that the epson3.txt is only 500 bytes long in the new file but 598 in the old. The second line shows that onstream.txt is present in the old file but not in the new, as the arrow points to the old file.

To close the story, did I find that I had lost any files? Yes, I did. I discovered a couple of small files that hadn’t copied at all because of non-standard characters in their filenames. The filenames were acceptable to Windows but not Linux and I’d used my Linux PC to do the copying. Fortunately, the files were saved and all the scripting was worth it.

Read more on Linux at Geek News Central.

Network Switches and Data Transfer Speeds

I recently upgraded my home network from 100 Mb/s to 1 Gb/s by replacing the switches. The main house switch is an unmanaged 1U rack-mounted switch, with a second desktop switch. Out of pure interest, I took the opportunity to do a little bit of speed testing to see how much of a difference upgrading the switches made in terms of actual data transfer speeds.

A few basics to avoid confusion  – b/s is bits per second and B/s is bytes per second. All of the reported figures will be in MB, so converting b/s to B/s:
Fast Ethernet = 100 Mb/s = 12.5 MB/s
Gigabit Ethernet = 1 Gb/s = 125 MB/s

100 Mb/s and 1 Gb/s refer to the speed of the underlying technology but data transfers at these rates are never achieved because of protocol overheads and such. As a baseline, if I write a large file (8 GB) to my PC’s local disk, I get a data transfer of between 50-55 MB/s.

On my network, I have two Buffalo Linkstation NAS devices, one with a Fast Ethernet interface and one with a Gigabit Ethernet interface. 2 GB’s worth of data would be written to each of these devices with different Ethernet switches in place to see what actual data transfer speeds would be achieved. The following Linux command was used five times in each situation and the result averaged.

time dd if=/dev/zero of=testfile bs=16k count=16384
Switch Model Data Rate to Fast NAS Data Rate to Gigabit NAS
1U Rack
Dynamode SW240010-R(Fast) 6.2 MB/s 8.6 MB/s
TP-Link TL-SG1016 (Gigabit) 6.4 Mb/s 21.4 MB/s
D-Link DES-1008D (Fast) 6.2 MB/s 8.6 MB/s
Netgear GS605 (Gigabit) 6.5 MB/s 21.1 MB/s

I also carried out two further tests:

  1. With Gigabit Ethernet only, I wrote to both NAS devices at the same time. The data transfer speeds were unaffected.
  2. I connected the two Gigabit Ethernet switches in series and wrote to the NAS. Transfer speeds were reduced by 1 MB/s on the Gigabit NAS to 20 MB/s. The change on the Fast Ethernet NAS was minimal.

There are several things that can be deduced from the information shown in the table above and the other tests.

  1. Actual data transfer rates are considerably less than the theoretical maximums.
  2. There’s no performance difference between rack-mounted and desktop switches.
  3. The write speed of the NAS can be a limiting factor.
  4. Gigabit Ethernet switches give large improvements with Gigabit Ethernet devices.
  5. Gigabit Ethernet switches give small improvements even with Fast Ethernet devices.
  6. Keep the number of switches in the network path to a minimum.


The Best Ever SuSE Linux – v11.4

OpenSuSE 11.4 was released back on 11 March so this weekend I took the plunge and upgraded my main PC from 11.3 to 11.4. And less than two hours later, I had the best ever SuSE running on my PC.  Here’s how I got on…

SuSE offer two methods of upgrading, the first being an on-line update and the second being the more traditional iso image download, burn and boot. I chose the latter as the guidance on SuSE’s website suggested that this would be more reliable. It also means that if the upgrade does fail and I needed to carry out a complete install from scratch, I already had the media to hand. Before booting from the DVD to upgrade, I backed up all the user files from the home partition and made copies of the important files – fstab, hosts, passwd, groups, auto.nas and so on.

Booting from the DVD, the installer goes through the usual licensing screen before analysing the existing system. As I had 11.3 previously installed, the installer gave me the option to upgrade, which I choose. After more analysis, it gives a summary of the changes required before asking permission to proceed – which I gave.

About 35 minutes and 250-odd packages later, the PC rebooted, loaded Linux and displayed the login screen. I entered my username and password, and the screen faded to the X desktop, with all my icons and widgets still there. Sweet!

Even more surprisingly, all the 3D window effects worked out of the box. That’s never happened before – normally you have to download drivers from nVidia or ATi before all the graphic goodness works smoothly. To be fair 11.3 was a “nearly” release. While the applications and tools worked, the 3D effects were a bit hit or miss. Sometimes they worked, sometimes they didn’t. But 11.4 hits it on the head.

The 3D eye candy is very slick. I run a 3 x 2 virtual desktop and the scrolling between the desktops is super-smooth, making it feel like one giant desktop. Windows glide in and out as they open and close. But by far my favourite effect is when you have overlapping windows and you want to bring one to the foreground. The upper window slides down the screen and then slips behind the lower window, bringing it to the front. Think of taking off the top sheet from a pile of paper and putting it to the back. So cool.

I’ve taken a couple of screenshots but (a) it’s really hard to catch the window closing when pressing the PrtScn button and (b) there’s no sense of the animation.

To finish off the installation, I added the ubiquitous Packman repository to load up all the unofficial multimedia goodies, such as DVD playing and video encoders.

Although it’s only been a few days, I’ve not encountered any problems at all with 11.4 and I’ve discovered that several of 11.3’s bugs have been fixed. Most of the major packages have been updated and has been replaced by LibreOffice (which is a whole sorry story in itself). Everything seems to be working fine.

If you want to try SuSE without messing with your current setup, there are live DVDs available for download. I run the KDE desktop rather than Gnome.

While it may be a little premature, I think this is the best SuSE ever.

OpenSuSE Linux 11.4 Released

The latest version of OpenSuSE Linux, 11.4, has just been released and it’s chock full of new features. The replacement for, LibreOffice, gets its first major outing, KDE gets bumped to 4.6 and Gnome comes in at 2.32.

There’s a also a pile of updates to applications, including Empathy, RhythmBox, Amarok, Totem, Evince and Shotwell. For developers, GTK 3 is included so Gnome applications can be upgraded to the new framework.

I’m running 11.3 so I’ll be downloading from the mirrors tonight and upgrading over the weekend. I’m looking forward to the new eye candy provided by the KDE Plasma Desktop Workspaces. Ok, so I’m shallow.

If you want to try OpenSuSE, there’s a live version as well, in both KDE and Gnome flavours. Give it a whirl.

Goodbye Ubuntu, Welcome Back SuSE

Some of you may recall that early last summer, I left my long-term Linux partner OpenSuSE for Canonical’s Ubuntu – the post is here. I thought it was going to be forever but I’m afraid it hasn’t worked out and SuSE has taken me back.

The original problem was that I couldn’t get SuSE 11.2 to install on new hardware and while Ubuntu 10.04 happily installed onto the new motherboard and harddrives, it’s been the legacy hardware that has been the root of the problem. Specifically, applications that wanted to access my SCSI scanner had to run as root, I completely failed to pull DV video from a video camera over Firewire and I couldn’t configure, never mind watch, my TV card. Scouring the newsgroups, I wasn’t alone. Perhaps naively, I thought that these problems would be fixed with Ubuntu 10.10 but alas, they persisted.

During the Christmas holidays I’d finally had enough – I can’t remember what finally caused me to snap but I downloaded OpenSuSE 11.3, burnt the DVD and rebooted. This time I didn’t encounter any of the previous problems from 11.2 and the installation went smoothly. It was like coming home – everything worked. Scanner – check, DV – check, TV – check.  And although returning to KDE desktop from Gnome, I have decided to keep some of the Gnome-based apps in preference to the KDE equivalents. gPodder is now my default podcatcher and Amarok has been replaced by RhythmBox.

It’s interesting times for OpenSuSE. In November, parent company Novell was sold to Attachmate but allegedly it’s business as usual. Version 11.4 will be out in a few weeks too.

There’s no doubt that some parts of Ubuntu were very seductive, such as package management, but I’m sorry Ubuntu…you’ve been dumped.

Amarok & KDE Crash Reporting – FAIL

Sigh. I love Linux but there are times when you realise it’s never going to take over the world…

I was working with Amarok, but the program crashed completely every time it hit a certain file in my audio library. This wasn’t a big deal but being a helpful soul, when I was presented with the option to send crash information back to the coders, I clicked on “Ok.”  And this is where it all went wrong.

First of all, after showing the stack trace (whatever that is), the crash handling dialog tells me that it’s not much use without the debug symbols, but the package to do that isn’t installed.  Did I want to install the necessary package?  So I said, “Yes,” still being a helpful soul.

Next, an error pops up saying that it can’t find the package and could I add a repository via the package manager?  Of course, the error message doesn’t tell me either the package that’s needed or the name of the repository needed. Being an ever-helpful soul, I figured out by myself that I need to enable the debug repository in the package manager, after which the crash handler was able to load the package and add the debug symbols. Hurrah!

So I hit  “Next” and I get presented with a username and password dialog for the KDE bug database. Apparently I can only log crashes if I’ve registered with the bug database. At this point I gave up being a helpful soul and closed the dialog.

So, for Amarok and KDE developers, here’s a clue. If you want feedback from your users on what’s going wrong with your applications, don’t make it so hard to give the information. Having agreed to give the feedback, that should be it, job done. I should not have to install a package, configure a repository and get a username for some website I’m never going to visit.

Even Dr Watson wasn’t this stupid.

WobZIP, An Online Unzipper

Have you ever downloaded some data off the ‘net only to find it’s in a compressed or archive file format that your PC doesn’t have a helper app for? Or you’re fixing up a friend’s PC, you download some drivers and ditto, you can’t get them unpacked?

If so, you’ll be interested in WobZIP. It’s a web site where you can upload an archive file and it will uncompress it for you.  Once uncompressed, you can either download the files one by one, or else the site will bundle the files back up into a zip archive for you to download.

The site is still in beta but claims to support the following archive formats – 7z, zip, gzip, bzip2, tar, rar, cab, iso, arj, lzh, chm, z, cpio, rpm, deb and nsis.  Obviously quite a few of those formats are Unix and Linux, but there’s a fair collection of DOS / Windows ones too.  As it’s a website, it doesn’t care what OS you’re running either. From the FAQ, WobZIP uses the open source 7-zip program as the decompression engine.

Cleverly, there’s also a feature to unpack or uncompress from a URL so you don’t always have to download to your PC and then upload back to WobZIP – you can just enter the URL and it will go and get the file for you.  Also, it will scan the unpacked files for viruses.

Put this site in your bookmarks.  You may not need it right now, but you will one day.

Ubuntu 10.10 Released 10/10/10

The latest version of the Ubuntu Linux distribution, 10.10 will be released tomorrow (if all goes to plan). Otherwise known as the Maverick Meerkat, this release focuses on improving the desktop experience and stability rather than radically updating it.

As usual, the kernel has been updated along with the Gnome desktop and there has been one change to the default apps (Shotwell for F-Spot in photo management) but apart from that, it’s pretty much upgrades and improvements.  Allegedly boot times have been improved as well, but 10.04 already booted pretty quickly.

If you haven’t figured out from the post title, Ubuntu releases aren’t numbered by simply incrementing versions.  The numbers are the year and month that software was released in thus October 2010 is 10.10.  The last release, Lucid Lynx, came out in April of this year so is 10.04.

As a further joke, this release is coming out on 10/10/10 which in binary is 42, homage to Douglas Adams’ answer to the meaning of life, the universe and everything.

I’ll report back on how my upgrade goes.

Shuttle Adds Blu-ray To XS35 Series

If you haven’t seen how small Shuttle’s XS35 series of ultra small PCs are, then you need to look more closely at the picture on the left.  The PC is the unit stuck to the back of the monitor.

The XS3510M is now available with a Blu-ray player making this a full hi-def media device.  Powered by Intel’s dual-core Atom D510 coupled with Nvidia’s Ion graphics, it’s capable of 1080p playback via HDMI.

At only 38mm thick, it’s just a little bit bigger than an external USB 3.5″ HDD unit and yet there’s both the Blu-ray / DVD writer combo drive and a 500GB disk drive (I assume that it’s of the 2.5″ variety) crammed in there, along with a 4-in-1 card reader on the front.  Round the back, there are four USB ports for your mice and keyboards.  11n Wi-fi is built in too.

Shuttle also touts the low power credentials, claiming a maximum of 29W, which would hardly get my standard desktop PC started.  Power is supplied via an external power brick, much like a laptop.

There are no fans in the XS35 series and all cooling comes from air circulating through the many holes in the case.  Consequently, the PC has to be stood upright, not on its side, but it makes the system totally quiet apart from the gentle whirr of the Blu-ray drive.

Windows 7 comes pre-installed with the XS3510M but the range has also been tested with Novell’s OpenSuSE Linux.

Prices start at £171 for barebones systems and around £650 for the unit featured above (depending on options, exchange rate, etc.)  Brochure (.pdf) available here.

All pictures courtesy of Shuttle.

Goodbye SuSE, Hello Ubuntu

Some of you will know that I’m a Linux user from my previous posts.  I’m not a distro-tart: I tend to stick with a particular Linux distribution rather than swap between the different flavours.  I started out over 15 years ago with an early Slackware release before moving to Red Hat around version 5 (1998?).  I then hopped over to SuSE for 7.3 (2001?) and have been with SuSE ever since.  After I moved to SuSE Linux, I ditched Ms Windows and never looked back.

A few weeks ago, I decided to upgrade my PC for a bit more oomph.  It was basically a complete rebuild as my motherboard was still AGP graphics, so everything’s new – new ASUS mobo, AMD AM3 processor, DDR3 memory,  all SATA drives, ATI (PCI-e) graphics, the works.  None of this stuff was cutting-edge but none of it was complete rubbish either.  All the gear had arrived in the post and Sunday afternoon was free – I was ready to go.

New drives meant new install of SuSE 11.2….or not.  First of all, I couldn’t get SuSE to see the SATA drives until I changed some BIOS settings.  Then the bootloader wouldn’t install unless I only had one HDD and disconnected the two mirrored drives.  And even then, when I rebooted after the initial install, I got buckets of SATA data errors and the OS failed to boot.

After tinkering with BIOS settings and kernel modules for a couple of hours I gave up.  Life’s too short.  I bunged in a Linux Format magazine coverdisk that had Ubuntu 10.04 on it.  Twenty minutes later, I had a working system, with all my drives, including the RAID mirror and it had also installed the proprietary drivers for the graphics card.  No fuss, no muss.  Boy, was I relieved!

I sometimes feel that it’s a bit churlish to complain about an OS which is essentially free, i.e. no cost.  In my defence, I have previously supported SuSE and Red Hat by buying the box sets, but in this day and age, if Linux wants to have any chance to succeed against the Redmond machine, it has to just work.  I’d consider myself somewhere between a power user and an expert and if I can’t get it to work on a fairly ordinary system, there’s no hope.

I’ll probably replace Ubuntu (which uses the Gnome desktop) with Kubuntu (which uses KDE) tonight to keeps things a bit more familiar but I’m afraid it’s goodbye SuSE, hello Ubuntu.