I have a lot of data and I would like to keep this data safe. For example, I have way too many pictures going back way too long, and I would like to keep these forever!

That means I need a good data backup strategy, and for this I have mostly settled on a combination of tools and scripts. Perhaps I'll write about my backup strategy in detail in the future. For this post it will suffice to say that the tool which handles the backup is duplicity (bonus, encrypted!) and one of the remote locations it backs up to is rsync.net, which I highly recommend.

So far, rsync.net can accept my data at the fastest rate I can push it. However this typically means that the rest of my Internet experience suffers when backups are running as my bandwidth is being saturated.
I had a little look around and came across a nice little tool called tricke which can limit the download/upload bandwidth of a process, all in userland!

It seems to be quite configurable but all I had to do was prepend 'trickle -u 800' to my backup scripts to limit duplicity to using at most 800Kb/s of my upload bandwidth. I quickly verified this with nethogs. Perfect!
This was posted on Fri 08 Jan 2016 (3 years, 5 months ago) by Ryan McConville
So QUB have begun rolling out a new WiFi solution across the campus, deprecating qub_sec.
The settings for _QUB_WIFI have changed, so for convenience sake I'll list them here.

EAP Method (Authentication): PEAP
Phase 2 Authentication: MSCHAPv2
Identity: student number
Anonymous Identity: leave blank
Wireless Password: your password

Unlike before, after the first time you connect to _QUB_WiFi you need to sign in via some sort of web based portal. Going straight to Google or Facebook won't work due to HTTPS, so try example.com in order to be redirected to the _QUB_WiFi sign in page.

I've tested this on my Linux computer and Android phone.
This was posted on Thu 07 Jan 2016 (3 years, 5 months ago) by Ryan McConville
While I do have off-site backups for linkiepie.com, why not take the safety of your data into your own hands?

Here I will quickly document a simple method of backing up all your link data stored in LinkiePie using the API. As mentioned in a previous post you can access your API key in the settings page (https://linkiepie.com/settings/).

Using this API key you can issue a request to access a JSON blob of all your data. Let's use curl in a simple shell script to download it and save it to your home directory.
#!/usr/bin/env sh

curl -s "https://linkiepie.com/api/v1/links/?username=YOUR_USERNAME&api_key=YOUR_API_KEY&format=json&limit=0&offset=0" -o "$HOME/linkiepie_backup.json"
This will save a JSON file containing all of the links (and associated data) you have added to LinkiePie. This includes data such as the URL and extracted summary text of each link.

Depending on how much you use LinkiePie this may be quite a large file. I like to use an awesome tool called jq for exploring and extracting useful information from JSON files. Here's an example of using jq for extracting just the URLs from this file.

cat linkiepie_backup.json | jq '.objects[].url'

You can add the above sh script to cron for automatic periodic backups.

Easy :)
This was posted on Wed 06 Jan 2016 (3 years, 5 months ago) by Ryan McConville
Using dbus and ConsoleKit it's easy to power off your computer from the terminal. This is useful when you're not using a desktop like GNOME or Unity and you want to shutdown without superuser privileges.

dbus-send --system --print-reply --dest="org.freedesktop.ConsoleKit" /org/freedesktop/ConsoleKit/Manager org.freedesktop.ConsoleKit.Manager.Stop

Of course it's probably a good idea to create an alias for this rather long command.

alias turnoff='dbus-send --system --print-reply --dest="org.freedesktop.ConsoleKit" /org/freedesktop/ConsoleKit/Manager org.freedesktop.ConsoleKit.Manager.Stop'

For the sake of completeness you can also restart and suspend your computer using dbus and ConsoleKit.

To restart use:
dbus-send --system --print-reply --dest="org.freedesktop.ConsoleKit" /org/freedesktop/ConsoleKit/Manager org.freedesktop.ConsoleKit.Manager.Restart

To suspend use:
dbus-send --system --print-reply --dest="org.freedesktop.UPower" /org/freedesktop/UPower org.freedesktop.UPower.Suspend


This was posted on Sun 30 Nov 2014 (4 years, 6 months ago) by Ryan McConville
I wanted to use the key combination 'Fn+F2' to lock my computer with xscreensaver, but wasn't sure how to represent this key combination in the awesome wm rc.lua configuration file. Unfortunately you can't just insert combinations using 'Fn' like you would 'Ctrl', instead you must use the relevant 'Fn' combination keycode.

It turns out that you can use a tool called xev to print out events such as keypresses that occur in a X11 window. Running xev and pressing the desired key combination will output to a terminal information for this event, including the keycode. Doing this I discovered that 'Fn+F2' translated into keycode 160.

Adding the following to the key binding section of my rc.lua worked great.
awful.key({}, "#160", function () awful.util.spawn("xscreensaver-command -lock") end),
This was posted on Fri 31 Oct 2014 (4 years, 7 months ago) by Ryan McConville
A quick and easy way get a readable version of a webpage directly from your terminal is to use the dump argument of the lynx text based web browser.

This is great for times when you are working from your terminal and want to view a website, for example to view a link received via IM, but would prefer not to leave your terminal to do so.

While tools like curl and wget will return the HTML file, including the HTML tags, lynx will parse it and return a much more readable version.

The following will output a readable version of this webpage directly to your terminal via stdout.
lynx -dump https://ryanmcconville.com/blog/

lynx

Sometimes you may want to save a readable copy of a webpage to a text file to read later. This can be done by redirecting stdout to a file.
lynx -dump https://ryanmcconville.com/blog/ > blog.txt

Lynx can also be used interactively for browsing the web by dropping the dump argument. For a program that predates the world wide web, it works suprisingly well on many modern websites.
This was posted on Thu 30 Oct 2014 (4 years, 7 months ago) by Ryan McConville
EDIT: qub_sec has been deprecated and replaced by _QUB_WiFi. You can find the settings for _QUB_WiFi here.

The settings for the Queen's University Belfast wireless network can be hard to find (for me at least) and when actually found the settings are buried in some PDF file, so I'm posting them here for convenience.

EAP Method: TTLS
Phase 2 Authentication: PAP
Identity: student number
Anonymous Identity: leave blank
Wireless Password: your password

You can optionally choose to use the CA Certificate to verify the authenticity of the qub_sec connection. I'm hosting a copy here for my own conveience .

The names of the fields above are from my Android phone, and they are similar using NetworkManager on Linux. I haven't tried on iOS but it's probably similar too.
This was posted on Sun 24 Aug 2014 (4 years, 9 months ago) by Ryan McConville
Tags: Tech Notes
Sometimes I am running experiments on a remote computer but doing the analysis of the experiments on
a local computer. Rather than manually copying or periodically syncing the results data for analysis, it's easier to mount the remote directory locally so that the remote files can be treated as if they are on the local filesystem. SSHFS (SSH Filesystem)
is one such tool that allows you to do this. All you need is SSH access to the remote server and SSHFS installed on the client.

The following instructions were tested on Ubuntu 12.04.
apt-get install sshfs
Now that sshfs is installed we can use it to mount the remote filesystem locally. First we will make a directory where we would like the remote filesystem mounted.
mkdir ~/rfs
Now we use sshfs to mount the remove filesystem...
sshfs user@dest:/dir/anotherdir ~/rfs
..and lets check the contents of the remote directory we just mounted.
ls ~/rfs
If you want to unmount the remote filesystem you can do so with the following...
fusermount -u ~/rfs
This was posted on Wed 03 Sep 2014 (4 years, 9 months ago) by Ryan McConville
If you prefer Debian over Ubuntu (and Ubuntu derivatives) but like the appearance of Xubuntu, then here’s some steps for making Xfce on Debian Wheezy look a lot more like Xubuntu. These steps are far from complete, but they make it look close enough to the default appearance of Xubuntu for me.

I’m going to assume you already have Xfce installed. I typically do this during the Debian install procedure but if you didn't do that you can still replace the default GNOME desktop environment with Xfce. Here’s a nice overview of how to do both. The key to the appearance of Xubuntu is the Greybird Xfce theme that it uses by default. I prefer to get the latest version directly from the Github repository of The Shimmer Project rather than relying on potentially older packaged versions.

First let's get the theme itself.
wget https://github.com/shimmerproject/Greybird/archive/master.zip
unzip master.zip
su -c 'mv Greybird-master /usr/share/themes/'
rm master.zip


Now, let's get the Greybird icons.
wget https://github.com/shimmerproject/elementary-xfce/archive/master.zip
unzip master.zip
su -c 'mv elementary-xfce-master/* /usr/share/icons'
rm master.zip


The next step is to enable the theme. First go to Settings->Appearance and select 'Greybird-master' under the 'style' tab. Then under the ‘Icons’ tab select 'elementary Xfce dark'. Under the 'fonts' tab I typically enable anti-aliasing, set hinting to slight and set the sub-pixel order to RGB.

After doing this, you should be seeing a theme that looks similar to Xubuntu, but not quite. The next step is to go to Settings->Window Manager and under 'style' select 'Greybird-master'.

After doing this I find that my Debian installation looks close enough to Xubuntu for me, and certainly much better than the rather dated default Xfce theme.

screenshot
This was posted on Sat 19 Jul 2014 (4 years, 11 months ago) by Ryan McConville
The default setting on many SSH servers is to disconnect clients who have been inactive for a set period of time. If you ever left an SSH client running without interacting with it for a while, only to find a message such as Connection reset by peer, then this is likely the cause. I find the easiest way to prevent this is to configure your SSH client to send a null packet to the SSH server, almost like a heartbeat. Edit your ssh_config file setting ServerAliveInterval to:
ServerAliveInterval 60
and ServerAliveCountMax to
ServerAliveCountMax 3
ServerAliveInterval tells your SSH client to send a message every 60 seconds to the SSH server thereby preventing the server from disconnecting you for being idle. I chose the value of 60 but you can increase or decrease this as required. ServerAliveCountMax is the number of times to send the null packet and receive no response, before giving up.
This was posted on Wed 25 Jun 2014 (4 years, 11 months ago) by Ryan McConville
Tags: Tech Notes
When working with an SQLite3 database I occasionally forget a column name that I need to know. This tends to be followed by being unable to remember how to find out column names (and other information) for a table. Future me, here's how to do it.

If you know the table name, great. If not, you can ask SQLite3 to list all tables with the following command:
.tables

Once you know the table name, you can find the table information with the following:
PRAGMA table_info(table-name);

Each column in the table is printed out on a new row as follows:
cid|name|type|notnull|dflt_value|pk

A real world example:
3|date_retrieved|TIMESTAP|0||0

More information on this can be found here: http://www.sqlite.org/pragma.html#pragma_table_info
This was posted on Tue 24 Jun 2014 (4 years, 11 months ago) by Ryan McConville
Tags: Tech Notes
As part of my research I tend to be working with relatively large files. One 150GB file I work with particularly often involves moving an updated version of it from one directory to another. On my Ubuntu desktop this causes everything to become unresponsive for the duration of the transfer.

One solution to this is ionice which lets you set the IO priority of processes. It's kind of like regular nice but for IO specifically. I found that the follow command, which sets the scheduling class to be 'Idle', let my desktop remain completely responsive while the transfer was taking place.

ionice -c 3 cp $src_file $dst_directory
This was posted on Tue 24 Jun 2014 (4 years, 11 months ago) by Ryan McConville
Continuing in my effort to create blog posts of old bits of code I can find lying around on my computer and internal git server, I decided to post this small C++ project. It logs the percentage of battery left at specific intervals into a SQLite3 database. Again, this is one of my first C++ projects so I'm not exactly confident it's done to a high standard. :)

Nonetheless, the code is included below and a the git repository on bitbucket can be found here: https://bitbucket.org/ryanmcconville/windows-battery-stats/

#include "stdafx.h"
#include "iostream"
#include "sstream"
#include "time.h"
#include "vector"
#include "iostream"
#include "Windows.h"
#include "fstream"
#include "Shlobj.h"
#include "stdio.h"
#include "WbemCli.h"
#include "sqlite3.h"

using namespace std;

//get the users home directory
wstring GetHomeDirectory(){
PWSTR path;
SHGetKnownFolderPath(FOLDERID_Profile, 0, NULL, &path);
return path;
}

void WriteDefaultConfiguration(char *defaulttime){
ofstream config_file;
config_file.open("config.ini", ios::out);
config_file.write(defaulttime, strlen(defaulttime));
config_file.close();
}

int ConvertToMilliseconds(int interval){
return ((interval * 60) * 1000);
}

int ReadConfigurationFile(){
char defaulttime[7] = "10";
char line[7];
ifstream config_file;
config_file.open("config.ini",ios::in);
if(!config_file.is_open()){
WriteDefaultConfiguration(defaulttime);
config_file.open("config.ini",ios::in);
if(!config_file.is_open()){
return ConvertToMilliseconds(atoi(defaulttime));
}
}
config_file.get(line, 10);
int interval;
interval = atoi(line);
int interval_milli = ConvertToMilliseconds(interval);
return interval_milli;
}

sqlite3* CreateDatabase(){
sqlite3 *db;
char *err = 0;

int rc = sqlite3_open("battery_stats.sqlite", &db);

if(rc){
fprintf(stderr, "cant open database %s\n", sqlite3_errmsg(db));
}

char *sql = "CREATE TABLE IF NOT EXISTS stat(" \
"date DATETIME NOT NULL," \
"battery_percentage INT NOT NULL," \
"ac_connected INT NOT NULL);";

sqlite3_stmt *statement;
if(sqlite3_prepare_v2(db, sql, -1, &statement, 0) == SQLITE_OK){
sqlite3_step(statement);
}
sqlite3_close(db);

return db;
}

int _tmain(int argc, _TCHAR* argv[]){

//hide console window
HWND hWnd = GetConsoleWindow();
ShowWindow(hWnd, SW_HIDE);
int interval_time = ReadConfigurationFile();
sqlite3 *db = CreateDatabase();

while(true){
SYSTEM_POWER_STATUS status;
::GetSystemPowerStatus(&status);//get battery life
int per = status.BatteryLifePercent;
//get time
time_t rawtime;
struct tm* printtime;
time(&rawtime);
printtime = localtime(&rawtime);
string timenow = asctime(printtime);
//create structure to hold percentage as a string
char strPer[4];
//copy int percentage to string
itoa(per, strPer, 10);
//open textfile to write to
ofstream batteryStats;
batteryStats.open(GetHomeDirectory() +L"/battery_stats.txt", ios::app);
int AC = (int) status.ACLineStatus;
string acconnected;
if(AC){
acconnected = "AC connected";
}else{
acconnected = "AC disconnected";
}

if(!batteryStats.is_open()){
cout<< "Error";
}else{
batteryStats << "\n" << timenow < }

sqlite3_stmt *statement;
char *sql = "INSERT INTO stat (date, battery_percentage, ac_connected) VALUES(?, ?, ?);";
if(sqlite3_prepare_v2(db, sql, strlen(sql), &statement, 0) == SQLITE_OK){
sqlite3_bind_text(statement, 1, asctime(printtime),strlen(asctime(printtime)), 0);
sqlite3_bind_int(statement, 2, per);
sqlite3_bind_int(statement, 3, AC);

sqlite3_step(statement);
sqlite3_finalize(statement);
} else {
fprintf(stderr, "cant update database %s\n", sqlite3_errmsg(db));
}

batteryStats.close();
//sleep for specified time
Sleep(interval_time);
}
}

This was posted on Tue 24 Jun 2014 (4 years, 11 months ago) by Ryan McConville
A few years ago, when learning Python, I wrote a little script that will retrieve and save to disk the current bing.com background image. The directory where the image is saved is the first argument. You can run it via cron if you use a Linux based OS or via Windows Task Scheduler if you use, well, Windows.

I did notice that if you access bing.com from a USA IP address then you have the ability to retrieve a higher resolution 1920x1200 image, whereas from the UK (and likely elsewhere) you can only access the 1366x768 image.

import re
import urllib2
import os
import shutil
import sys

bing_url = 'http://www.bing.com'
save_location = sys.argv[1]

p = re.compile('/az/.+?(jpg)')
resp = urllib2.urlopen(bing_url).read()

image_url = p.search(resp).group()
if '_EN-US' in image_url:
res = image_url.split('_')
extension = image_url.split('.')[-1]
image_url = '_'.join(res[:-1]) + '_1920x1200.' + extension

try:
img = urllib2.urlopen(bing_url + image_url)
except urllib2.HTTPError as e:
print e
sys.exit(-1)

image_name = image_url.split('/')[-1]

if not os.path.exists(save_location):
os.makedirs(save_location)
with open(save_location + image_name, 'wb') as f:
shutil.copyfileobj(img, f)


Edit: I have since updated this script with some nice new changes. You can find that post here: https://ryanmcconville.com/blog/post/bing-image-retrieval-v2/
This was posted on Mon 23 Jun 2014 (4 years, 11 months ago) by Ryan McConville
Last week my Queen's University Belfast (QUB) IMAP settings stopped working. After tracking down that this was the reason I wasn't receiving new emails, I discovered the following IMAP and SMTP settings which now work. I have tested these settings with Outlook, Thunderbird and my Android phone. I assume they will also work with iOS and OSX clients too.

Note that there seems to be two different types of email accounts currently in use within QUB, but the following should work for 'Office 365' accounts, i.e. the new ones.

IMAP
Server name: outlook.office365.com
Port: 993
Encryption method: SSL

SMTP
Server name: smtp.office365.com
Port: 587
Encryption method: STARTTLS

User name: studentnumber@ads.qub.ac.uk.


Staff Accounts
Here are the settings that work for me to get staff email to work over IMAP/SMTP.
IMAP
Server name: owa.qub.ac.uk
Port: 993
Encryption method: SSL/TLS

SMTP
Server name: smtp.qub.ac.uk
Port: 587
Encryption method: STARTTLS

User name: staff number
This was posted on Mon 23 Jun 2014 (4 years, 11 months ago) by Ryan McConville
Tags: Tech Notes