Persistent ssh-agent on Bash on Ubuntu on Windows

Posted by Dave Eddy on 18 Oct 2017 - tags: tech

After installing Bash on Ubuntu on Windows I realized some interesting side effects related to how processes and daemons in the Unix environment are handled. Running a process in the background, or daemonizing a process, will work so long as there is a Bash session open on Windows. Once the last window is closed, all of the processes are cleaned up and killed.

I use ssh keys for authentication when connecting to remote servers which requires the use of ssh-agent. I can run this program manually and it will work so long as there is at least one bash session running on my computer, but once I close the last window the ssh-agent is killed and my keys are unloaded. I've found a couple guides online regarding ssh-agent and WSL specifically, but most of them assume the keys you are using are not password protected.

To remedy this situation, I managed to find a way to create a hidden terminal session that runs ssh-agent in foreground mode when I login to my computer which persists through sleeps and hibernations. This way, ssh-agent will run and stay running from the moment I login until the moment I logout (which is almost never, unless I reboot).

Read More...


Install Bash on Ubuntu on Windows

Posted by Dave Eddy on 17 Oct 2017 - tags: tech

Last week I made a big change in my life. In 2008, I switched off of using Windows (and even Linux) as my main Operating System for my laptop and haven't looked back... until this year. Last week I finally made the jump and bought a Microsoft Surface Pro.

surface

I wanted something like an iPad, but that didn't feel like a crippled version of a computer. Instead, I wanted something that was a powerful computer that could double as a tablet (touchscreen and removable keyboard mostly).

Note: Some, if not all, of this guide may be deprecated with the release of the Windows 10 Fall Creators Update in 2017.

Read More...


Raspberry PI WiFi Hang Fix

Posted by Dave Eddy on 17 May 2017 - tags: tech

If you have a Raspberry Pi on WiFi then you are probably aware of the struggles with it randomly disconnecting. If anything goes wrong with the WiFi (access point restarts, IP address changes, you look at the pi the wrong way, etc.) the WiFi will drop and never fix itself. I've written a script called net-dev-kick to kick the network device if a ping check fails.

https://github.com/bahamas10/net-dev-kick

There are a lot of small scripts floating around forums that do what net-dev-kick does (albeit, poorly and with a lot of assumptions) so I decided to clean it all up and make a simple yet robust program to deal with this situation.

This script attempts to ping a server that should always be up (use your router or something similar) and if it fails, it restarts the interface given as $1.

net-dev-kick is intended to be used on the raspberry pi to fix wlan0, but can theoretically be used on any machine to fix any interface. It's best used as a cronjob that runs every 5 or so minutes.

Read More...


Now Open for iOS

Posted by Dave Eddy on 08 Sep 2016 - tags: tech

Use Now Open to find places around you that are currently open

I created What's Open 3 years ago, but got a cease and desist for the name and pulled it from the market. Now, I have an LLC in place and a Trademark filed for Now Open, so it's back up under a new name!

Open this app to see a map that automatically zooms to your current location and shows every place around you that is open - Complete with "food" and "pizza" buttons!

  • Find all places around you that are currently open
  • See how long you have until a business closes (ie. 30 minutes left!)
  • Search for keywords like "pasta", "Chinese", "bars", etc.
  • Easily open your favorite maps app for navigation
  • Quickly call any place you find
  • Share places found on Facebook and Twitter
  • Eat at new restaurants, visit new locations!
  • Great for finding late night spots


Automatic ZFS Snapshots and Backups

Posted by Dave Eddy on 05 Dec 2015 - tags: tech

I recently made a new storage server to replace my old one to keep up with my growing space requirements (I think 40T should hold me over for a while!). I store all of my movies, music, tv shows, etc. on it, as well as all of my backups. All of my laptops and desktop computers also backup to this server using rsync.

While it's all stored on SmartOS using the ZFS filesystem in a raid setup that can handle 2 or more drive failures without data loss, it still worries me because it is all stored in one physical location: my closet. If there is a fire or some other disaster like that, all of my data could potential be lost.

To remedy this, I've repurposed the server I replaced (my old storage server) to be an off-site backup server that is used solely for ZFS receive. This server now runs FreeBSD, which you can read about in my blog post here

Automatic Snapshots

Before diving into my off-site backup solution, the first thing to talk about is how I handle automatic ZFS snapshots, and also removing snapshots as they get too old.

zfs-snapshot-all

https://github.com/bahamas10/zfs-snapshot-all

Recursively snapshot all zpools

I use this program to snapshot all zpools on my new storage server automatically in cron. My crontab looks something like this:

Read More...


ZFS Zpool Encryption with Geli on FreeBSD

Posted by Dave Eddy on 04 Dec 2015 - tags: tech

I recently set up a server at my buddies house for remote, off-site, backups using ZFS send/recv. Since SmartOS is lacking any sort of encryption for ZFS I instead used FreeBSD with geli. This way, I get the benefits of ZFS incremental send and receive for doing backups, with the security of knowing my data is encrypted on disk.

To ensure maximum security, I made sure the passphrase for unlocking the drives was not stored anywhere on the server itself, and instead would require manual intervention on my part to decrypt the drives anytime the server is rebooted. To do this, I created 2 pools:

  • zroot - the main pool, mounted at /, unencrypted
  • paper - the pool used for backups, mounted at /paper, encrypted

The server will boot into the operating system automatically, and from there I must run a script to unlock the drives and import the paper pool.

PS: The zpool is called paper because the server is called paperdyne. That name was cleverly derived from dataDyne (my storage server) and @papertigerss (my buddy whose house this server now lives).

Setup

Install FreeBSD

The first step, of course, is to install FreeBSD. Go to the website, make a bootable USB (or DVD, or CD, or whatever) and install it.

Read More...


dataDyne2 - 40T Hypervisor

Posted by Dave Eddy on 22 Nov 2015 - tags: tech

It's been 5 years since I created my first storage server and was now approaching full capacity - I was at 90% used out of 9.5TB available. I built a new server with over 40TB usable to replace this server, and will be using the old server for offsite backups eventually.

The new server will run SmartOS like the server it is replacing, and will run almost all of the same zones as the original

Parts

Server parts

Rack and case parts

Specs

The new build has 20x 4TB SAS drives, 32GB of ECC ram, and an Intel Xeon Quad Core 3.3GHz processor.

Memory

# prtconf | grep Memory
Memory size: 32740 Megabytes

Processor

# psrinfo -vp
The physical processor has 4 cores and 8 virtual processors (0-7)
  The core has 2 virtual processors (0 4)
  The core has 2 virtual processors (1 5)
  The core has 2 virtual processors (2 6)
  The core has 2 virtual processors (3 7)
    x86 (GenuineIntel 306A9 family 6 model 58 step 9 clock 3300 MHz)
      Intel(r) Xeon(r) CPU E3-1230 V2 @ 3.30GHz

Read More...


Graphite Quick-Start on SmartOS

Posted by Dave Eddy on 14 Oct 2015 - tags: tech

Graphite is an open source system for graphing data. It manages storing data (in a database format called "whisper"), as well as graphing data (with a project called "graphite-web").

These instructions are fairly platform-independent, and should work on any Unix operating system - the only SmartOS specific bits will be at the end to create SMF services that ensure the services will start at boot.

Graphite Example

The above graph is the temperature of my garage, as monitored by a Raspberry Pi, over a 1 month period.

Overview

There are 3 projects that will be used to setup the graphite server:

  1. carbon - stats receiving daemon (called carbon-cache) that manages storage
  2. whisper - database file format used by carbon to store data
  3. graphite-web - web interface to compose graphite graphs

Prerequisites and Dependencies

To get started, create a user for the graphite services to run as:

groupadd graphite
useradd graphite

Read More...


Sending Email on the Command Line

Posted by Dave Eddy on 24 Aug 2015 - tags: tech

Sending email on the command line is nothing new; In fact, it is not even all that exciting. When I got Nagios up and running at home, I needed an easy way to have it email me whenever there was an alert. When I worked at Voxer as an Operations Engineer I created a program to allow Nagios to generate HTML emails - now all I needed at home was to allow outbound email.

This, however, turned out to be more difficult than I thought, mostly because I was (and largely, still am) ignorant to the intricacies of the various email protocols in use today on the Internet. Joyent released a blog post when I was setting this all up that went over the various options for sending email from a SmartOS server. One of the options they mentioned but didn't choose to implement really caught my eye: Google's free SMTP server. Since my main email address is a gmail address (well, Google Apps), this route seemed perfect.

gmailx

gmailx was born! From the documentation:

Send email easily on the command line without running a server

This program is basically mailx but hardwired for Gmail (hence the name). Usage is simple:

Read More...


SmartOS Pkgsrc Caching Proxy

Posted by Dave Eddy on 19 Jul 2015 - tags: tech

I have 2 SmartOS servers at home with a total of 17 zones running persistently - a majority of these provisioned with the latest LTS support release of pkgsrc: 2014Q4. All zones are setup to download a similar set of "bootstrap" packages to make them ready to be used. Packages like gcc, git, etc. I pull to all of my zones upon creation.

The problem, however, is my home internet is not the fastest - it usually can pull packages at around 100KB/s. git and gcc alone are responsible for over 100MB of compiled binary data, meaning these initial package downloads can take over 15 minutes per new zone.

A possible solution to speed up initial package downloads I investigated was to rsync the entire pkgsrc tree to a local server, and host it over an internal HTTP server. This, however, required a massive amount of storage dedicated to packages I am almost guaranteed to never use. I could have just pulled 2014Q4, but then I would have been required to add new releases whenever I wanted to upgrade, and keep around the old packages until all my old zones have been upgraded.

Instead, I created a Node.JS HTTP caching server to proxy GET and HEAD requests to Joyent's pkgsrc server, and cache the data on the local filesystem. The requests will be proxied if the file does not exist locally, and will be dual-written to the requesting client, and the local filesystem. Subsequent requests for the same resource will be streamed from the local filesystem without every making an outbound request.

fs-caching-server

https://github.com/bahamas10/node-fs-caching-server

The program is written to be generic - it allows it to act as a caching proxy to any website, and also allows the user to specify a regex to test a URL when deciding if the request should be cached, or just proxied directly with no cache.

Read More...


Newer Posts 1 of 4 Older Posts »