Welcome to Pete Brown's 10rem.net

First time here? If you are a developer or are interested in Microsoft tools and technology, please consider subscribing to the latest posts.

You may also be interested in my blog archives, the articles section, or some of my lab projects such as the C64 emulator written in Silverlight.

(hide this)

The Great Migration (or why I’m not interested in playing home IT guy anymore)

Pete Brown - 10 July 2012

This is the story of how I moved my infrastructure out of my house and into the hands of people who do this for a living.

In addition to six laptops (two for me, one for my wife, one for each kid, one for my CNC machine) several PCs, and countless other connected screens (tablets, phone, game consoles, smart tv, STB, blu ray player, and more), I actively run three rack servers. One is the database server, one is the web server, and the third is both email and domain server (yeah, I know).

  • I run my own domain server
  • I run my own mail server
  • I run my own web server and its associated database server

This is all connected to a FIOS for business account, with multiple static IPs, and pretty decent upstream and downstream bandwidth (50MBps down, 20+ up - at this moment, speakeasy reports I'm getting 43 down and 32 up, and that's with people hitting my site etc.). I pay through the nose for that service, and managing it is a pain because it is separated from all my other Verizon accounts - without any online access for billing. Verizon systems aren't set up to have FIOS business accounts without an associated business phone number. Like business is the only entity that could use a static IP.

Here's an older post talking about my infrastructure, complete with a few photos.

image image

Cooling for this rack (and it does need cooling) is performed by one nasty window shaker air conditioner in my basement window, plus one fan to help direct the air into the front of the servers. (Actually, several fans, as I burn them out once a year or so). The AC unit in that window is ugly and noisy.

image

I've never figured out what the AC running all that time costs me, but I'm sure it's not cheap. Not to mention that those IBM servers themselves, complete with dual power supplies blowing a serious amount of heat out of the back, probably use a ton of electricity as well.

So there's a cost issue. There's also the small matter of a giant server rack taking up a ton of space in my shop room - the room I want to convert into my new home office. You may think of a server rack as mostly a vertical floor-to-ceiling affair, but they're around 4' deep and 2' wide as well. They blow hot air out the back, so need clearance behind, and have a swinging door in the front, so clearance is required there as well.

Oh, and the noise! On a quiet night, you can hear those x345 fans coming through the central air vents in every room in the house. Each of my dual-Xeon xSeries rack severs has eight 2u high speed fans blowing the hot air out the back. The noise around the house is low-level, but it's there, kind of a jet engine noise in the background. Sure, I could insulate the bare AC ductwork in that room, but that's just extra work I don't need. In any case, I couldn't possibly put enough insulation in that room to cut down on the fan noise to make it usable as a home office.

image

But holy crap is it major geek cred to have a server rack in your basement! Now I'll just need to replace it with something more geeky, but requiring less maintenance. Suggestions welcome :)

The Storm

Every summer and almost every winter, we end up with an outage situation. Usually the power is out for a day or so, but this time, it was much longer - six days without power, seven without internet (plus a blip a few days later that caused internet access to drop out again for the better part of a day). To be clear, I'm not out in the boonies - I'm in a suburb located between Washington, DC, Annapolis, MD and Baltimore, MD. This was a serious storm that caused widespread damage. It even has its own Wikipedia entry now. BGE posted a bunch of photos of the damage and their repair efforts as well (look at the stuff starting June 29, 2012).

image

That photo is not from my street, but I saw lots of streets similar to this. I was driving home from MADExpo (with my 6yo son in the back) when I hit the storm just south of Bowie, MD. I've never driven through anything like that in my life. Scary. Branches and limbs flying around the road. Torrential rain. 70-80mph winds. Scary. I had to drive over a tree top and into a ditch to get to my house since the other ways in were blocked with bigger trees. At one point, lightning struck so close to the car that everything turned white, the lights lit up and the DVD player / sound system let out a huge BZZZZZZTTT.

Other than some smaller dents on the top of my car from the drive up, we had a few trees down on our property, but had no other property damage. Electronics didn't do quite so well, though.

Storm Damage

When you run your infrastructure from your house, and the power goes out, you start bouncing emails, and no one can get to your web site. Importantly, search engine crawlers see your site as down as well. I'm sure that 404ing all my URLs for a week hurt rankings, and more.

In addition to blowing the FIOS power supply and battery backup (it did this all around the area - the Verizon tech knew what the problem was with only a glance) The storm also caused one of my drives in my database server to fail. The drive was one of four in a RAID 1E array which contained family photos and videos plus the database that ran my site. Unfortunately, I hadn't realized that a second drive had been automatically marked as defunct quite some time earlier. When I looked at the log, it was marked as part of a "predictive failure analysis". That meant that the ServeRAID 5i controller thought the drive was likely to fail, so it took it offline.

Losing two drives in a 4 drive array meant the logical drive was down. Ugh.

image

I had regular backups of the site and database. However, the database backed up to the same RAID array it was on. I would simply copy the .bak file to my local machine on a regular basis. Unfortunately, with MADExpo, TechEd, and other travel, that local copy was almost a month old. My local backup of photos was older by a few months, so I lost some of those as well.

(BTW, that ServeRAID management app: The 90s called and they want their ugly 4 bit color, chamfered-corner, Java-style, UI back.)

My workstation backup solution is more robust. I have dedicated WD backup drives handling it, doing nightly backups. Server-based solutions were (when I investigated) significantly more expensive because they assumed only rich corporations ran servers, so I just handled the off-machine backup manually.

Email

For the better part of a decade, I've run my own mail server in-house using the Lumisoft MailServer source code. This is a free and open source .NET mail server. It's not complicated, and works really well. One downside is I never got web access working, and I had pop3 secured so it could only be accessed on my network. That meant that I could never access my home email while traveling. A minor inconvenience, but annoying.

More importantly, while I was away at MADExpo, someone in Russia started using my server to relay spam. I was pretty sure I had checked to make sure it wasn't an open relay. I even ran some online tools which verified that I'm not running an open relay.

So how were the spammers getting around this? I'm not sure if it was a bug in the mail server, or simply some IT thing I don't quite understand, but they were able to send the spam as long as it had my email address in the "from" field. (Note that I changed my domain name in the pasted log)

# Fields: SessionID SessionStartTime RemoteEndPoint AuthenticatedUser LogType LogText
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "220 ESMTP Windows 3.11 Mail Server"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "EHLO mail.irr1tat3pvowel.com"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-spaces.ru"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-PIPELINING"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-SIZE 32000000"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-VRFY"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-ETRN"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-ENHANCEDSTATUSCODES"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 8BITMIME"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "MAIL FROM:<pmbrown@irr1tat3pvowel.com> SIZE=45521"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 2.1.0 Ok"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "RCPT TO:<Grodno92@spaces.ru>"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 2.1.5 Ok"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "DATA"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "354 End data with <CR><LF>.<CR><LF>"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "Binary data, sent 45524 bytes."
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 2.0.0 Ok: queued as 9CF0EC66F7EE"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "QUIT"

(I changed the spelling of my email address and domain name above to help prevent crawlers from posting my correct address)

Umm, yeah, that's reporting itself as Windows 3.11. I have no idea if it really is or not (bonus retro points for the spammer) or simply that their spam software reports itself as WFWG 3.11 for grins.

But note that it's sending it from my email address. If I turn on SMTP authentication on the server, no one can send me email (strange, I know, but I tried it). If I leave it off, anyone can send email purporting to be from anyone at the domain. There was no middle ground with that server; I'm surprised it took this long for the spammers to exploit that machine. I tried to fix that for several days, but had zero luck.

The pattern was common: I'd figure out a way to shut it down (or in the case of the storm, that happened as result of power loss). Shortly after it came back up, I'd get a couple relayed messages sent to an @spaces.ru account, and then the floodgates would open and I'd start getting relayed spam from many different IP addresses - all with me in the "from" field. All the bounced email would go to my inbox.

Solution

I considered a number of different hosts. My criteria was:

  • Retain existing email addresses
  • Support several addresses plus aliases
  • Works with Outlook
  • Mobile and/or web access
  • Free or very very cheap

So today I moved all my email accounts over to GMail. You can have up to 10 email addresses, with your custom domain, for free. That's enough for the family, and with the aliases, enough for the service-specific accounts I've created for things like paypal, ebay, etc. (helps cut down on phishing).  Once you go over 10 users you start to pay a fee per user per month, and it makes more sense to investigate other services like Office 365.

As I wrote the original draft of this post (waiting for my MX records to update), I was still getting bounced spam messages in my inbox. Who knows how many blacklists I'm on now. Grrr. My home mail server is now turned off and everything goes through GMail.

We'll see how well GMail works for me. Both my wife and I use Outlook for managing email, and GMail concepts like tags and filters don't really match well with folders and other Outlook concepts. Because I kept the same email addresses, I'll be free to change to yet another provider in the future should I need to.

UPDATE: 7/10. I'm taking a closer look at Office 365 right now as it works more like I do. In particular, I (and my wife) both like to work from Outlook, and GMail and Outlook don't quite mesh. You can force some things, but Filters/Tags, calendaring and more just aren't doing it for me. I've been an Outlook user since Day 1.

Web Site

I'm in the process of migrating my site off-premises as well. More on that in another post. Right now, the lead contender is Azure Web Sites (not the free one, as I get enough traffic to hit that limit quite quickly).

End Result

The end result of all this will be to get out of the home IT business. No more server rack, no more dedicated AC unit, no more on-premises email, web, or other servers. Probably no home domain server either. IT: It was fun while it lasted, but I think it's time we decided to just be friends.

       
posted by Pete Brown on Tuesday, July 10, 2012
filed under:        

7 comments for “The Great Migration (or why I’m not interested in playing home IT guy anymore)”

  1. Jeff Putzsays:
    I hosted sites out of my house for a couple of years (2001 to 2003) with a T-1, when it was the "cheapest" way I could host. It seemed pretty cool at the time, but even two servers and a router impacted my electric bill, to say nothing of the fan noise. I was glad to get the stuff out of my house.

    From that point on, I rented a box, with costs starting at $400/month and coming down ever since. Eventually I ditched mail serving and went to Google Apps (aside from a couple of brief outages, never a complaint). Azure pricing keeps coming down, and honestly, I'd like to get my stuff there entirely, eventually. I hate feeding a server. I look forward to just deploying to something and never again thinking about patches and firewalls and what not.
  2. Damien Guardsays:
    When you move your website consider moving the static binaries etc. to a cheap CDN like MaxCDN ($50 a year for 1TB) and then you won't be paying $$$ for data transfers on your dynamic ..NET/Umbraco host.

    [)amien
  3. Marc Drossaerssays:
    Hi, Making the same move right now. Email went to live.com (manage domains at domains.live.com). This works fine with Outlook. Resources for the websites are being transferred to SkyDrive. The only thing I hadn't figured out yet was a website to serve Silverlight applications from. Azure Web Sites seems something to explore further. Thanks for the tip! Ciao.
  4. Tonysays:
    Live is great. I recommend it to all my clients and use it for my business account. Unfortunately, I still use hosted email for my family account because I too use aliases with multiple recipients and you can't do that with hotmail. Aside from that, it's great. EAS, hotmail interface, calendars contacts, etc.
  5. G. Andrew Duthiesays:
    Sorry to hear about all the pain from the storm, but I think you'll be much happier taking the IT admin hat off, if only partially.

    When I was running my own company, I ran the website for the company on a webserver in my basement on a fractional T1, because I wanted the comfort of knowing the hardware was right next to me, where I had full control. It was hugely expensive, and as you've discovered, there are people whose job it is to make this reliable and secure, and we should generally let them do that. It's cheaper and better that way. :-)
  6. Chucksays:
    One thing to consider is backup. You could use a hosted solution like Carbonite or you can run a Windows Home server. I have a Dell 2950 in the basement (nice and quiet unless it gets hot) backing up all my computers daily. The hardest thing is doing an off-site backup but I pay enough monthly fees already. I was at the MADExpo and had a hard time getting back to Colorado due to flight cancellations.
  7. JEmlaysays:
    Interesting read. Thanks for sharing!

    However, for the life of me, I can't see why you're running 3 physical machines. I know my eye sight is going bad but I swear I see 4 machines in that pic. Anywho, VM those bad boys! HyperV is free. Trade all 4 of those into one beefy HP machine on eBay. I had to watch for a while however I scored myself a G6 DL370 for $1200. With that came more then enough memory, drive and CPU power to run 6 fully loaded server VMs. I run just about everything from that and it require no more cooling then any other PC in my house so you can ditch all that cooling nonsense as well. In fact my server sits in the garage where I ported the rear to the outside. just gotta keep up on those front filters!

    I use it to get VMs ready for new installations for other companies. So any point in time I could have AD, Exchange, Web and various other app servers running along side my own servers.

Comment on this Post

Remember me