Strange Adventures In…

…Ansible: Creating Logical Volumes With Ansible

2021-06-17T22:07:13+00:00

Ansible is often overlooked for managing the more physical aspects of the devices it is used to configure, but it has some powerful tools for handling lower level duties, such as disks. Recently I was working on some old-fashioned VMWare virtual machines and discovered that a lot of the guidance for managing logical volumes in Ansible is dated so here are my modernized notes on taking a disk from device to volume. This is NOT a primer on LVM itself, an understanding of LVM is recommended before trying to implement these steps.

Create a new logical volume group (docs)

- name: Create a volume group on top of /dev/sdb with physical extent size = 32MB
  community.general.lvg:
    vg: vg.storage
    pvs: /dev/sdb1
    pesize: 32

Create a logical volume in your logical volume group (docs)

- name: create logical volume
  community.general.lvol:
    vg: vg.storage
    lv: data
    size: 10g

Create a filesystem on your new volume (docs)

- name: Create a ext4 filesystem on the data volume
  community.general.filesystem:
    fstype: ext4
    dev: /dev/vg.storage/data

Create a folder to mount our new volume to (docs)

- name: Create directory /data if does not exist
  ansible.builtin.file:
    path: /data
    state: directory
    mode: '0755'

Mount our new volume to /data (docs)

- name: mount the logical volume to /data
  ansible.posix.mount:
    path: /data
    src: /dev/vg.storage/data
    fstype: ext4
    state: mounted

These instructions were tested on Ubuntu 20.04 servers with Ansible 2.11 but should work on any Linux flavor that supports LVM.

…CLI: Configure Weechat for Libera.Chat with CertFP

2021-05-19T22:07:13+00:00

Freenode is dead, it’s weird to say it, but it is. I am not going to go into detail around the circumstances but the folk behind Freenode have moved on to a new IRC network, Libera.Chat. For those of you looking to hook into this new community with a TUI I highly recommend WeeChat for your CLI IRC needs and here is a quick how to on how to hook it up with CertFP for easy authentication.

Install WeeChat, on MacOS this can be done with homebrew, for other operating systems follow the guides on WeeChat’s site

brew install weechat
Launch WeeChat ( weechat ) and add the Libera.Chat server to your config

/server add liberachat irc.libera.chat/6697 -ssl
We can now connect to Libera.Chat for the first time

/connect liberachat
Once connected it is time to claim our nickname (you will have connected with a default nickname that is likely not available). To do so we first need to switch to the nickname to see if it is free

/nick <YOUR_NICKNAME>
If it is free you will see your nickname change to the one you selected, if not try another. Once you have selected your nickname successfully we need to register it

/msg NickServ REGISTER <YOUR_PASSWORD> <YOUR_EMAIL@EXAMPLE.COM>
Check your email and copy and paste in the command from that email to complete the registration of your nickname. At this point you have registered your nickname and can log into it with the password specified in step 5. For ease and security though we are going to switch to CertFP so we can use a cert for auth so for now disconnect from the server with /disconnect
In a new terminal window create a new certificate with openssl

openssl req -x509 -new -newkey rsa:4096 -sha256 -days 1096 -nodes -out libera.pem -keyout libera.pem
Create the certificate directory for WeeChat

mkdir ~/.weechat/certs
Copy your new certificate into our new directory

mv libera.pem ~/.weechat/certs

Switching back to our WeeChat window we now need to configure WeeChat to use our certificate

/set irc.server.libera.addresses irc.libera.chat/6697
/set irc.server.liberachat.ssl on
/set irc.server.liberachat.ssl_verify on
/set irc.server.liberachat.ssl_cert %h/certs/libera.pem
/set irc.server.liberachat.sasl_mechanism external
/set irc.server.liberachat.nicks <MY_NICKNAME>
/save

Now we log into Libera.Chat using our password for (hopefully) the last time and register our certificate fingerprint
```
/connect
/msg NickServ IDENTIFY <YOUR_NICKNAME> <YOUR_PASSWORD>
/msg NickServ CERT ADD
```
Finally disconnect and reconnect to test your new, password free, login
```
/disconnect
/connect
```

…iOS: Loading Discord Web on iOS Safari

2021-04-13T22:07:13+00:00

Discord have stopped iOS users from accessing NSFW servers using their app. They blame Apple, I call bull, but rather than play the blame game here is a quick and simple circumvention technique that you may find useful to bypass this unnecessary censorship.

Open up Safari on your iOS device
Go to the Discord homepage
By default this page will only allow you to download the app. Open up the page options and press “Request Desktop Site”

A login button will now be displayed, enjoy Discord uncensored.

…CLI: Protonmail -> offlineimap -> notmuch -> Mutt

2021-03-15T19:54:13+00:00

Protonmail is a great service, after using it for over a year I cannot see myself returning to GMail or moving to another email service in the near future. Out of the box however it misses a few key features:

Offline backups
Desktop client compatibility
The ability to search the message body (by design I know, but still frustrating)

With a little tinkering these are all solvable problems, and using CLI tools (mostly).

Protonmail Bridge

Out of the gate we need to be able to connect to Protonmail using standard protocols, like IMAP, which are not available by default. To achieve this we need a paid Protonmail account so we have access to Protonmail bridge. You shouldn’t need a whole lot of guidance to get this up and running, but the official documentation can be found here.

Once the bridge is up and running you will have IMAP and SMTP services running locally with the bridge translating those two protocols to Protonmail’s API.

Why not hydroxide

Hydroxide is a third party alternative to the official bridge. I tried it out but it has issues, most notably the inability to move emails between folders, so I stuck to Bridge.

Offlineimap

Offlineimap is going to be synchronizing our mailbox to a local directory, fulfilling our requirement for an offline backup as well as making our mail available to our local client. Install it using your package manager (brew for me) brew install offlineimap and create a configuration file in your home directory called .offlineimaprc, using this template:

# ~/.offlineimaprc
[general]
accounts = main
pythonfile = ~/.offlineimap.py

[Account main]
localrepository = main-local
remoterepository = main-remote

# full refresh, in min
autorefresh = 0.2

# quick refreshs between each full refresh
quick = 10

# update notmuch index after sync
postsynchook = notmuch new

[Repository main-local]
type = Maildir
localfolders = ~/.mail

# delete remote mails that were deleted locally
sync_deletes = yes

[Repository main-remote]
type = IMAP
remoteport = 1143
remotehost = 127.0.0.1
remoteuser = $YOUR_EMAIL@protonmail.com
# remotepass = $SOME_PASS
remotepasseval = get_keychain_pass(account="$YOUR_EMAIL@protonmail.com", server="imaps://localhost")
keepalive = 60
holdconnectionopen = yes

# delete local mails that were deleted on the remote server
expunge = yes

# sync only these folders
folderfilter = lambda foldername: foldername in ['INBOX', 'Archive', 'Sent', 'Trash']

# SSL
ssl = no
starttls = yes
sslcacertfile = ~/.protonbridge.pem

Most of this file is pretty boring but there are some pieces to be aware of:

Credential Management

Skip this by uncommenting remotepass and entering the bridge password, but know that means you are storing a credential in plain text and that comes with all the risks you think it does and more. Don’t forget to comment out pythonfile and remotepasseval if you do this

My solution to this is MacOS focused, if you need a solution for another OS I would check out gpg. We do not want to store credentials in plain text, lets use the Keychain to store the password instead. Open up Keychain Access, select the “login” chain and select File -> New Password Item:

Keychain Item Name = imaps://localhost
Account Name = $YOUR_EMAIL@protonmail.com
Password = $BRIDGE_PASSWORD

We will use this entry in various ways in our configuration, but for offlineimap we need to craft a little python to get the password. Create .offlineimap.py in your home directory and copy in the following code

# ~/.offlineimap.py

import re, commands
def get_keychain_pass(account=None, server=None):
    params = {
        'security': '/usr/bin/security',
        'command':  'find-generic-password',
        'account':  account,
        'server':   server
    }

    command = "%(security)s %(command)s -a %(account)s -s %(server)s -w" % params
    outtext = commands.getoutput(command)
    return outtext

This is a modified version of a python script found here

This python script is imported at pythonfile and the function is used by remotepasseval to get our password from the Keychain when it’s needed.

Certificates and TLS

We need to provide a certificate file for this to work correctly due to Proton Bridge using a self signed cert. Once the bridge is running we can get the certificate by running openssl s_client -starttls imap -connect 127.0.0.1:1143 -showcerts. Copy -----BEGIN CERTIFICATE----- through to ----END CERTIFICATE----- and paste it into a new file in your home directory called .protonbridge.pem.

Notmuch

Notmuch is new to my setup so I do not have much to say about it at this time (ha). I will write more about it later but for now we will use a simple setup. Install with your package manager of choice, for example brew install notmuch and run notmuch setup to get a basic config going.

Mutt (or Neomutt)

I use Neomutt instead of Mutt so all of the configurations assume Neomutt’s directory structure, they should be pretty similar though. Install Neomutt using your trusty package manager, brew install neomutt perhaps. Next we create a configuration file for Neomutt at ~/.config/neomutt/neomuttrc and copy in the following:

## Accounts
# Variables
set my_user = "$YOUR_EMAIL@protonmail.com"
set my_pass = `security find-generic-password -a YOUR_EMAIL@protonmail.com -s imaps://localhost -w`

# Protonmail

set realname = "Firstname Lastname"
set smtp_url = smtp://$my_user:$my_pass@127.0.0.1:1025
set ssl_force_tls
set ssl_starttls
set from = "$YOUR_EMAIL@protonmail.com"
set envelope_from
set use_from = "yes"

set mbox_type=Maildir
set folder=~/.mail/
set record=+Sent
set postponed=+Drafts
set trash=+Trash
set mail_check=2 # seconds

set virtual_spoolfile # use first defined virtual-mailbox as spool
virtual-mailboxes "INBOX" "notmuch://?query=folder:INBOX"
virtual-mailboxes "Archive" "notmuch://?query=folder:Archive"
virtual-mailboxes "Sent" "notmuch://?query=folder:Sent"
virtual-mailboxes "Trash" "notmuch://?query=folder:Trash"

This is a snippet from my neomuttrc that handles mail access and SMTP, I will publish my full configuration at some point, but this will get you started

Things to note here are set my_pass which is once again grabbing our password from the Keychain and the use of virtual-mailboxes to leverage notmuch, even if we are not doing much with it in this simple configuration.

Startup

Now we have everything in place we can get it running. Make sure Proton Bridge is running then execute offlineimap in a terminal to start the synchronization process. Once it is done offlineimap will call notmuch to index the emails thanks to postsynchook = notmuch new in its configuration file. After the initial sync is done (which may take a while) you can launch neomutt and start working on your email!

Assuming everything works you probably want Proton Bridge and offlineimap to run on startup, which I will leave up to you and your OS.

…Blogging: Going Static

2019-01-16T18:54:13+00:00

For the past three years I have run this blog off of the Ghost CMS platform, and it has been good to me. I backed Ghost on Kickstarter way back in its infancy, I stuck with it through the good times and the bad, sadly however my time with the platform has come to an end. But why?

Ease

No matter what CMS you pick they all have one thing in common, they need care and feeding, as do their components. When you commit to a CMS you take on a bunch of responsibilities, the CMS itself, the underpinning technologies, the database, all of these things need attention to keep them running smoothly.

I also used a custom theme, which meant that every release I was having to tweak my code to keep it compatible with Ghost. Did I have to use a custom theme? Of course not, but I wanted to and that meant a lot of time hacking in fixes to account for new features I would never use anyway.

By switching to a static site I was greatly reducing the number of dependencies I had to manage as well as reducing the amount of tinkering required to keep my site current.

Security

The vast majority of modern web site security issues can be boiled down to two main culprits. The first is that people dont keep things up to date, and in the world of the CMS we have a lot of moving parts that need to be patched, not least of which is the CMS itself. With my new static site all I have to do is keep the core OS up to date along with a single extra package, the web server.

The second culprit is the trend towards interactivity. Back in The Good Old Days (tm) web sites were simple delivery mechanisms much like this one. Now it is percieved that we must accept user input in some way, be that a login, comments, or more complex web application functionality. Wherever we accept input however that input can be exploited by malicious actors for nefarious means. By returning to a static platform I have negated a whole slew of potential security issues in a single move.

Environmetal concerns

For Ghost I operated a fairly minimal custom theme, this was designed to help the environment a little with each page load by forgoing the fancy bells and whistles of many sites and the CPU cycles needed to render them. Whilst this project will likely die with this move to static content you can read more about it here.

The CMS and associated infrastructure was still absorbing more natural resources than strictly necessary however, and by waving it goodbye I can save even more CPU cycles than ever before.

Hello Jekyll

So this is goodbye Ghost, but hello Jekyll. I will likely do a later post on why I selected Jekyll over the large number of other trendy site generators out there, but for now welcome to our brave new world!

…40k: Picking up the hobby again

2018-07-14T03:37:54+00:00

The last time I played Warhammer 40000 I was 15 years old give or take. We played about once a month on a pool table in my friend’s basement with half painted armies, makeshift terrain and more house rules than i care to remember. Fast forward a few years and I have a friend who dearly wants to pick up the game again and thats enough of an excuse for me. The problem, i need an army ready in a months time.

Army Selection

A hurried rule book purchase and speed read caught me up on the lore (which has seen some pleasant progression since I last played and holy crap Eye of Terror everywhere!) My first task was picking an army to build and paint in one month. Here are the guidelines i set for myself:

This army is not forever, do not get too hung up on it
I need to acquire ~1000 points of it in one month
The army needs to be easy to paint in the alotted time
Xenos or heretic only, everyone plays Imperium, try something different

The Candidates

Necrons

My gut reaction was to go Necron. They were practically non existent when I last played and it was fun seeing how they had grown into the lore, more importantly they checked all of my selection criteria. I did not love the idea though, not sure why. They stayed on the shortlist but I kept looking.

Dark Angels Fallen

I have always wanted to play an army of Fallen, I love the lore behind them and Dark Angels models are great. The catch, there was no way I was painting an army that ornate in time with my rusty modelling skills. This idea is on the back burner, but does not fit the criteria for now.

Craftworld Aeldari (Eldar)

Their name may have changed, and they may be one craftworld short compared to when I last played, but do they fit the criteria? At first I figured not, but then I remembered the fluff for Iyanden. Wraith models seemed to be easy to paint in bulk, and I had always wanted to give the Eldar a shot.

Next Steps

After settling on an Iyanden war host I ordered the Codex and the Start Collecting box (which handily is loaded with Wraith stuff). Next stop, testing paint schemes.

…Azure: Replacing SSL Certificates on Application Gateways

2017-03-19T00:52:30+00:00

There is a strange gap in the documentation for Azure Application Gateways around that famous Microsoft afterthought, SSL offload. Specifically, I am talking about what to do when an SSL certificate expires which if you are following best practices will be every year or two.

The general advice is to perform a rip and replace of the listener, which is a downtime inducing event on not really practical in a busy production environment. Luckily we can remedy the issue with no outage with a few lines of PowerShell, but it would be nice if Microsoft would put this in the portal for all to use freely!

The Script

The Powershell to achieve the replacement is simple and looks like this:

First, we grab our Application Gateway and pop it into a variable. Next, we grab our certificate and do the same. One important note here is the Certificate Name MUST match the name of the certificate attached to your Application Gateway (you can find the name in the portal and in the resource explorer). Finally, we commit the change to our Application Gateway, a process that will take around 15 minutes so do not panic!

Disclaimer

Use this code at your own risk. It is provided without warranty, guarantee and if used incorrectly could hose all your backups and this would not be my fault!

…Immigration: Adjustment of Status from L1-B (I-130 & I-485)

2017-01-12T03:42:37+00:00

This post is a brief PSA. For anyone adjusting from an L1-B visa to Green Card via meeting a wonderful American guy/girl (i.e. marriage) your package to USCIS should look a little like this:

Packet Cover

Cover letter for the entire packet (example)
Completed G-1145 (e-notification registration)

First Folder (I-130)

Cover letter for I-130 (example)
Payment as per the list of fees found here
Completed form I-130 with attached evidence:
- Copy of Petitioner’s U.S. birth certificate
- Copy of Petitioner’s U.S. passport biographic page
- Copy of the marriage certificate
- Copy of any divorce certificates (if applicable)
- One passport sized photo of the Petitioner (placed in a zip lock bag and attached to a blank sheet of paper)
- One passport-sized photo of the Applicant (placed in a zip lock bag and attached to a blank sheet of paper)
One completed G-325A filled out by the Petitioner
One completed G-325A filled out by the Applicant with attached evidence:
- Copy of the Applicant’s foreign birth certificate
- Copy of the Applicant’s visa page and associated passport stamps
Evidence of a bonafide marriage which can include any of the items listed here

Second Folder (I-485)

Cover letter for I-485 (example)
Payment as per the list of fees found here (include payment for both I-485 and the biometric appointment in one check)
Form I-485, Application to Adjust Status with the following attached:
- Copy of Applicant’s foreign birth certificate
- Copy of Applicant’s foreign passport including L-1B visa and stamps (all pages)
- Copy of Applicant’s latest I-94
- Two passport-sized photos of the Applicant (placed in a zip lock bag and attached to a blank sheet of paper)
I-693, Medical Examination of the Beneficiary completed by a civil surgeon, in a sealed (unopened) envelope
G-325A, Biographic Information filed by the Applicant
I-864, Affidavit of Support filed by Petitioner and the following supporting documents:
- Copy of Petitioner’s US Passport as proof of citizenship
- Copy of Petitioner’s last 3 years of tax returns
I-765, Application for Employment Authorization with the following attached:
- Copy of Applicant’s Passport
- Copy of Applicant’s existing Visa
- Copy of Applicant’s foreign birth certificate
- Copy of I-797C from previous work authorization
- Two passport-sized photos of the applicant (placed in a zip lock bag and attached to a blank sheet of paper)
I-131, Application for Travel Document (Advance Parole) with following attached:
- Explanation for Form I-131 filing (example)
- Two passport-sized photos of the applicant (placed in a zip lock bag and attached to a blank sheet of paper)

Take the two folders, put them in one envelope and ship them off to the Chicago lockbox. All this information is provided with zero warranty, YMMV!

…Azure: Redirecting HTTP to HTTPS with Azure Application Gateways and SSL Offload

2016-08-23T18:41:18+00:00

OK, someone at Microsoft really dropped the ball with this one. HTTP to HTTPS redirection is one of those things that everyone needs to do, and in fact everyone should do (seriously, there is no excuse for unencrypted HTTP now, stop it).

This is even stranger when you consider the fact that all the Application Gateway actually consists of is a specially configured IIS VM set. And to do this natively in IIS is really quite easy. Until Microsoft fix this little faux pas however there is a simple workaround using a small Ubuntu VM and a little reconfiguration of our Application Gateway.

Concept

What we are trying to achieve is summarized in this diagram.

We will be setting up two ports on our application gateway, one for plain traffic (port 80) and one for HTTPS traffic (port 443). When the gateway receives HTTP traffic it will forward it to an Ubuntu server running nginx where the request will be redirected to HTTPS. HTTPS traffic will be decrypted at the gateway and passed to our IIS box as normal.

How To

For the purposes of this guide I will assume you have already configured an Application Gateway with SSL offload and the backend IIS (or other web server) to match. I will focus on building the Ubuntu 16.04 VM and configuring nginx, as well as modifying the Application Gateway configuration.

Building the redirect server

Provision a new Ubuntu VM from the Azure Portal. It does not have to be particularly large, remember it will exist solely to redirect traffic.
Once the machine is provisioned and you have logged in lets get everything up to date.

sudo apt-get update && sudo apt-get upgrade
Next we install nginx.

sudo apt-get install nginx
With nginx installed we can now start getting it configured. As we will be using this box for redirects only we can simply modify the default configuration.

sudo nano /etc/nginx/sties-available/defuault
Delete the contents of this file and replace them with the following.

server { listen 80 default_server; listen [::]:80 default_server; server_name _; return 301 https://$host$request_uri; }
Restart nginx to apply the new configuration.

sudo service restart nginx

Configuring the Application Gateway

Log into the Azure Portal and find the Application Gateway that you wish to modify.
Under Settings select Backend Pools and add a new pool with your redirect server configured as the target IP.
Select listeners and create a new listener on port 80 for HTTP if you do not have one already.
Select Rules and create a new basic rule which takes your newly created listener and forwards it to your new backend pool.

And thats it. I agree, this is a “hack” at best and Microsoft need to add a simple “Redirect HTTP to HTTPS” button somewhere in there. But if you need this capability now, this works!

…Powershell: Deleting old files from Azure Blob Storage

2016-08-17T18:54:13+00:00

Backing up to Azure Blob Storage is great, especially if like me you use Ola Hallengren’s amazing (and free) jobs to backup MS SQL Server (and if you don’t, I recommend you do, find out more here). What is not quite so awesome is cleaning up old files. Blobs do not behave like files, and certainly if you use Ola’s jobs setting the cleanup time when using a URL for the backup destination will cause the backup to fail. There is however hope, because we can do our cleanup using PowerShell!

The Code

The Gist below contains the code at time of writing. For the latest and greatest version keep an eye on the Github repo.

Save this with a .ps1 extension and schedule it as you see fit.

The Detail

We capture 7 arguments to make this script tick:

storageAccountName - The name (NOT the URL) of our Azure Storage Account
storageAccountKey - The key for the Storage Account
containerName - The name of the container we want to clean up
age - The maximum age of the files we want to keep, anything older will be removed
errorSMTPServer - The SMTP server to use to send an email if something goes wrong
errorSourceAddress - The from email of any error messages
errorDestAddress - The to email of any error messages

When working with Blob Storage we have to first build the storage context ($context in the above script) We do this by running New-AzureStorageContext and passing in the storage account name and storage account key we capture as parameters. Interestingly, this cmdlet does not verify this inforamtion, it just build a context object which we can then pass to other Azure storage cmdlets.

The try/catch block then executes the Get-AzureBlobStorage cmdlet using our storage context from above and also adding the container to search. The results are filtered for anything older than the specified age and finally the filtered results passed to Remove-AzureStorageBlob for immediate destruction.

Should anything go wrong the catch block grabs the error and sends it via email using the details provided.

Disclaimer

Use this code at your own risk. It is provided without warranty, guarantee and if used incorrectly could hose all your backups and this would not be my fault!