Encrypted SMB for Linux and Windows 10

Filed under: TechNotes, Ajax — lars @ 09:57:00 am

I recently tried to secure (ie transport encryption) access to my local Samba server running on a Debian box running Samba 4.2.  By default all Samba traffic outside password exchanges is unencrypted, meaning file contents and directory listings can be 'sniffed' over the network.

Getting this working end-to-end involved a little more trial-and-error than I would have hoped.

Enabling encryption on the Samba server seems easy enough by editing /etc/samba/smb.conf and adding:

server max protocol = SMB3 #Want to ensure Win8.1+ compatible encryption is enabled (now default) smb encrypt = required #Force encryption on
lanman auth = no #Disable old insecure authentication (now default)

However I immediately started getting cryptic errors from my Linux-based clients like Kodi and Nautilus File Manager, along the lines of "File exists".  Very helpful.

It turns out clients like nautilus typically use libsmbclient to connect to SMB shares.  With some trial and error testing, I was able to successfully connect to my share using smbclient with *either* the '-e' or '-m SMB3' parameters:

smbclient -e -m SMB3 //<hostname_or_ip>/<share_name> -U <username>

The '-e' tells the smbclient to attempt an encrypted connection, however Nautilus would still fail.  It turns out I need to edit the smb.conf on the Linux client machines in order to make these settings the default:

smb encrypt = required client min protocol = SMB2 #Tried SMB3 here, but caused errors connecting client max protocol = SMB3

This change to the default smbclient behaviour appeared to take effect as soon as smb.conf was saved.  (It's worth noting that some linux machines have an smb.conf file under the <user_home>/.smb/ directory, and this may need to be edited too.)

I tested one of my Windows 10 devices and it could connect without issue.  (Why Windows 10 can connect without issue when encryption is on, but linux/samba clients need trial-and-error configuration changes is beyond me.)

The final step was to fire up wireshark and confirm that I was indeed now getting encrypted SMB.  I could see that prior to these changes, plaintext details of files were visible in the packet stream and after the changes these details were encrypted:

wireshark screenshot

I noted my username was still transferred in plaintext.  Figuring out whether I can prevent that is a job for another day.

Smaller compressed tar archives

Filed under: Ajax — lars @ 06:15:00 am

I've seen some benchmarks recently showing just how much smaller 7z compression can be than traditional zip/gzip/bzip.  This motivated me to update some regular archiving scripts to switch from gzip compression to 7z.

Normally to create a compressed tar archive under Linux, you do something like this:

tar cvzf <Destination archive>.tar.gz <source folder>

To use 7z instead, you first need to make sure it's installed - which on Ubuntu/Debian based systems you can do as follows:

sudo apt-get install p7zip p7zip-full

Then you can pipe your tar output through 7z:

tar cf - <source folder> | 7z a -mx=9 -si <Destination archive>.tar.7z

It really works!  Some of my text-heavy directories were 40% smaller using 7z "Ultra" compression.

The above command uses the following parameters.

cf -> create archive
- -> output to stdout instead of a file

a -> Add to archive
-mx=9 -> Use "ultra" compression (slower but smaller)
-si -> Read from stdin instead of from a file/folder.

On unix filesystems, it's recommended to use tar and 7z in combination in this way because 7z by itself will not preserve file permissions and ownership.

Using Amazon Glacier and handling 408 RequestTimeoutException errors

Filed under: TechNotes, AWS — lars @ 08:30:00 pm

Lots of people have problems with their Amazon Glacier uploads timing out.  For some, switching to a different AWS region may help.  Otherwise, the key to dealing with timeouts is to use multipart uploads.

I use the glacier-cmd command-line tool.  It seems to be popular, well maintained, and have a slightly more coherent set of commands for managing your uploads than the standard AWS client.  The multi-part approach I used to work around timeouts is explained below.

Set up

First you will need to install and configure glacier-cmd as per the instructions on GitHub.

Create a file vault if you don't already have one.  This is a container in which all your uploads will reside:

glacier-cmd mkvault [VaultName]

Uploads and resuming

Then begin an upload, specifying a small upload part-size to preserve as much of your upload as possible if there is a failure:

glacier-cmd upload [VaultName] ./filename --description "file description" --partsize 1

You may get the dreaded Timeout exception:

boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)

You should be able to see the failed upload if you run the listmultiparts command:

glacier-cmd listmultiparts [VaultName]
|        MultipartUploadId         | ArchiveDescription |       CreationDate       | PartSizeInBytes |                          VaultARN                          |
| AbCdEfghIhijklm_...zxZxZXzxZ00XZxx | file_description   | 2016-11-05T10:26:00.000Z |     1048576     | arn:aws:glacier:us-east-1:012617398452:vaults/[VaultName] |

You can then resume the upload by referring to the MultipartUploadId from the output above:

glacier-cmd upload --resume --uploadid "AbCdEfghIhijklm_...zxZxZXzxZ00XZxx" [VaultName] ./filename --description "file description" --partsize 1

If the upload fails again, you can keep resuming and eventually it should complete.  For me, this happened frequently enough to warrant running the command in a simple bash-script loop:

for ((n=0;n<10;n++)); do    /usr/local/bin/glacier-cmd upload --resume --uploadid "AbCdEfghIhijklm_...zxZxZXzxZ00XZxx" [VaultName] ./filename --description "file descripion" --partsize 1 ;   done

Clean up

If you have made multiple attempts to upload the same file, you may find there are 'orphaned' file fragments that will show up in the listmultiparts command:

glacier-cmd listmultiparts [VaultName]

You can run abortmultipart to delete them:

glacier-cmd abortmultipart [VaultName] AbCdEfghIhijklm_...zxZxZXzxZ00XZxx

Listing your successfully uploaded files

Unfortunately, viewing an up-to-date list of your uploaded files isn't as simple as a 'dir' or 'ls' command.  Retrieving the vault inventory is a job that AWS run in the background and notify you on completion.  You run it like this: 

glacier-cmd inventory [VaultName]
|           Header          |            Value                 |
|           Status          | Inventory retrieval in progress. |
|           Job ID          | AbCdEfghIhijklm_...zxZxZXzxZ00XZxx |
| Job started (time in UTC) |     2016-11-18T23:23:00.000Z      |

If you have set up notifications for Glacier via the AWS console, you may get an email/sms or other notification on completion.  You can also check on the status of this running job using the listjobs command:

glacier-cmd listjobs [VaultName]
|                          VaultARN                         |        Job ID               | Archive ID |       Action       |        Initiated         |   Status   |
| arn:aws:glacier:us-east-1:012617398452:vaults/[VaultName] | AbCdEfghIhijklm_...zxZxZXzxZ00XZxx |    None    | InventoryRetrieval | 2016-11-18T23:23:00.000Z | InProgress |
 Once the job completes, which may take up to 24 hours, you can then use the inventory command again to view the up-to-date file listing:
glacier-cmd inventory [VaultName]
Inventory of vault: arn:aws:glacier:us-east-1:012617398452:vaults/[VaultName]
Inventory Date: 2016-12-22T12:00:00Z

|              Archive ID            | Archive Description |       Uploaded       |           SHA256 tree hash         |    Size    |
| AbCdEfghIhijklm_...zxZxZXzxZ00XZxx |    file one desc    | 2016-11-05T21:00:00Z | AbCdEfghIhijklm_...zxZxZXzxZ00XZxx | 725500000  |
| AbCdEfghIhijklm_...zxZxZXzxZ00XZxx | file two desc | 2016-11-16T07:00:00Z | AbCdEfghIhijklm_...zxZxZXzxZ00XZxx | 7413500000 |
+------------------------------------+---------------------+----------------------+------------------------------------+------------+ This vault contains 2 items, total size 10.2 GB.


That's it.  Good luck using AWS Glacier for your own archive storage!

Ripping scratched audio CDs in Linux

Filed under: Ajax — lars @ 04:36:00 pm

I recently tried to rip some old audio CDs that had quite a few scratches. Lately I've been doing my CD-ripping using Asunder on Ubuntu. It works well enough, but I found for the scratched CDs the progress is horribly slow, to the point where it seems like it would take days to rip a CD.

I tried a couple of recommended options to find a fast way just to 'skip' the unreadable blocks, like ddrescue and safecopy, however these either errored or were just as slow. But I had some success with a command-line tool called cdda2wav (www.cdda2wav.de).

Here's my usage:

sudo cdda2wav -vall -D /dev/sr0 speed=4 cddb=0 -B

Full-Duplex audio recording on Linux, for Skype and other VoiP

Filed under: TechNotes, Linux — lars @ 12:31:00 pm

I've recently had the need to record a phone-call made via Skype.  This need can come up for lots of reasons, for example to keep a record of information you provide over the phone when taking out an insurance policy.  There are a number of means to achieve this, but I've used a generic method of recording both PC and microphone audio using PulseAudio - which is applicable to any application or VoiP service.  There are other guides on how to do this, but I didn't find any of them as easy to follow or complete as I'd like, so I've written my own.

Remember - there may be legal restrictions on call recording in your location.  You may need to inform the other party if they are being recorded.  So use this guide at your own risk - I am no expert on matters of law.

Firstly, you need to install PulseAudio Volume Control.  This is a nice utility that allows mapping between audio-devices, in order to stream your PC's sound to your recording application.  On Ubuntu, you can install it like so:

sudo apt-get install pavucontrol

Now, run PulseAudio Volume Control - it should now be in your Applications menu somewhere.  You should see some out-of-the-box devices like 'Built-in Analogue Audio Stereo' which represents your PC sound hardware.  We first want to make sure that none of your key devices are 'muted', which would definitely inhibit recording!  Go to the Input Devices tab - Show "Monitors" (from the drop-down at the bottom) and make sure 'Monitor of Built-in Audio Analogue Stereo' is not muted, as this is a virtual device that allows recording of the sound output of your PC.  Also make sure your microphone input under 'Built-in Analogue Audio Stereo' is not muted, as you will need this to record your side of the conversation.

Now we need to drop to the command-line.  These commands below will create a 'sink' and two 'loopbacks':

pactl load-module module-null-sink sink_name=myaudiosink
pactl load-module module-loopback; pactl load-module module-loopback

The 'sink' is a device that allows combining of multiple audio streams, and you need this to record both the PC audio and microphone input at the same time (ie, both sides of your Skype conversation).  The 'loopback' is a way of looping audio from one of the 'real' hardware devices into your sink.

Now, return to PulseAudio Volume Control.  Under the 'Playback' tab, set the two new Loopbacks to 'Null Output', as this is your sink.  Then, under the 'Recording' tab, set the two new Loopbacks to 'Built-in Audio Analogue Stereo' and 'Monitor of Built-in Audio Analogue Stereo' respectively, (it doesn't matter which is which), as this will route your PC and microphone audio through the loopbacks.

Now, start Gnome Sound Recorder or the audio recording app of your choice, and press record.  The recording app will appear in the 'Record' tab of PulseAudio Volume Control.  It should be recording from the 'Monitor of Null Output' to capture all audio going in to your sink, so set it to do this if it is not.

Then, run Skype (or whatever app you want to record audio from).  Once you start a call, Skype should appear under both the Recording tab and Playback tab in PulseAudio Volume Control.  It should be recording and playing back to Built-in Audio Analogue Stereo, which would be as per normal.

Through the magic of PulseAudio - Gnome Sound Recorder should be recording all of the microphone input plus any audio that's coming from the other end of your Skype conversation!  Below are screenshots of PulseAudio Volume Control, in the state that it should look while successfully recording your Skype call:

PluseAudio Volume Control screenshot

Note that the above settings will not be maintained through a reboot.  You'll need to add them to your PulseAudio config to do this - and you'll need to search elsewhere for how this is done.


Whilst experimenting with this set up, I found I messed up my PulseAudio configuration at one point, and PulseAudio started crashing.  If you find you have any problems with your PulseAudio config, it's possible to delete the pulse-audio config local to your user account and then re-create it using the following two commands:

rm -rf ~/.pulse
pulseaudio -k

I hope you find this guide useful.  And remember to check the legality of what you're going before you try this out on a real call!

powered by  b2evolution