This is the story of how I moved my infrastructure out of my
house and into the hands of people who do this for a living.
In addition to six laptops (two for me, one for my wife, one for
each kid, one for my CNC machine) several PCs, and countless other
connected screens (tablets, phone, game consoles, smart tv, STB,
blu ray player, and more), I actively run three rack servers. One
is the database server, one is the web server, and the third is
both email and domain server (yeah, I know).
- I run my own domain server
- I run my own mail server
- I run my own web server and its associated database server
This is all connected to a FIOS for business account, with
multiple static IPs, and pretty decent upstream and downstream
bandwidth (50MBps down, 20+ up - at this moment, speakeasy reports
I'm getting 43 down and 32 up, and that's with people hitting my
site etc.). I pay through the nose for that service, and managing
it is a pain because it is separated from all my other Verizon
accounts - without any online access for billing. Verizon systems
aren't set up to have FIOS business accounts without an associated
business phone number. Like business is the only entity that could
use a static IP.
Here's an
older post talking about my infrastructure, complete with a few
photos.
Cooling for this rack (and it does need cooling) is performed by
one nasty window shaker air conditioner in my basement window, plus
one fan to help direct the air into the front of the servers.
(Actually, several fans, as I burn them out once a year or so). The
AC unit in that window is ugly and noisy.
I've never figured out what the AC running all that time costs
me, but I'm sure it's not cheap. Not to mention that those IBM
servers themselves, complete with dual power supplies blowing a
serious amount of heat out of the back, probably use a ton of
electricity as well.
So there's a cost issue. There's also the small matter of a
giant server rack taking up a ton of space in my shop room - the
room I want to convert into my new home office. You may
think of a server rack as mostly a vertical floor-to-ceiling
affair, but they're around 4' deep and 2' wide as well.
They blow hot air out the back, so need clearance behind, and have
a swinging door in the front, so clearance is required there as
well.
Oh, and the noise! On a quiet night, you can hear those
x345 fans coming through the central air vents in every room in the
house. Each of my dual-Xeon xSeries rack severs has eight
2u high speed fans blowing the hot air out the back. The noise
around the house is low-level, but it's there, kind of a jet engine
noise in the background. Sure, I could insulate the bare AC
ductwork in that room, but that's just extra work I don't need. In
any case, I couldn't possibly put enough insulation in that room to
cut down on the fan noise to make it usable as a home office.
But holy crap is it major geek cred to have a server
rack in your basement! Now I'll just need to replace it with
something more geeky, but requiring less maintenance. Suggestions
welcome :)
The Storm
Every summer and almost every winter, we end up with an outage
situation. Usually the power is out for a day or so, but this time,
it was much longer - six days without power, seven without internet
(plus a blip a few days later that caused internet access to drop
out again for the better part of a day). To be clear, I'm not out
in the boonies - I'm in a suburb located between Washington, DC,
Annapolis, MD and Baltimore, MD. This was a serious storm
that caused widespread damage. It even has its own Wikipedia entry now. BGE
posted a bunch of photos of the damage and their repair
efforts as well (look at the stuff starting June 29, 2012).
That photo is not from my street, but I saw lots of streets
similar to this. I was driving home from MADExpo (with my 6yo son
in the back) when I hit the storm just south of Bowie, MD. I've
never driven through anything like that in my life. Scary. Branches
and limbs flying around the road. Torrential rain. 70-80mph winds.
Scary. I had to drive over a tree top and into a ditch to get to my
house since the other ways in were blocked with bigger trees. At
one point, lightning struck so close to the car that everything
turned white, the lights lit up and the DVD player / sound system
let out a huge BZZZZZZTTT.
Other than some smaller dents on the top of my car from the
drive up, we had a few trees down on our property, but had no other
property damage. Electronics didn't do quite so well, though.
Storm Damage
When you run your infrastructure from your house, and the power
goes out, you start bouncing emails, and no one can get to your web
site. Importantly, search engine crawlers see your site as down as
well. I'm sure that 404ing all my URLs for a week hurt rankings,
and more.
In addition to blowing the FIOS power supply and battery backup
(it did this all around the area - the Verizon tech knew what the
problem was with only a glance) The storm also caused one of my
drives in my database server to fail. The drive was one of four in
a RAID 1E array which contained family photos and videos plus the
database that ran my site. Unfortunately, I hadn't realized that a
second drive had been automatically marked as defunct quite some
time earlier. When I looked at the log, it was marked as part of a
"predictive failure analysis". That meant that the ServeRAID 5i
controller thought the drive was likely to fail, so it took it
offline.
Losing two drives in a 4 drive array meant the logical drive was
down. Ugh.
I had regular backups of the site and database. However, the
database backed up to the same RAID array it was on. I would simply
copy the .bak file to my local machine on a regular basis.
Unfortunately, with MADExpo, TechEd, and other travel, that local
copy was almost a month old. My local backup of photos was older by
a few months, so I lost some of those as well.
(BTW, that ServeRAID management app: The 90s called and
they want their ugly 4 bit color, chamfered-corner, Java-style, UI
back.)
My workstation backup solution is more robust. I have dedicated
WD backup drives handling it, doing nightly backups. Server-based
solutions were (when I investigated) significantly more expensive
because they assumed only rich corporations ran servers, so I just
handled the off-machine backup manually.
Email
For the better part of a decade, I've run my own mail server
in-house using the
Lumisoft MailServer source code. This is a free and open source
.NET mail server. It's not complicated, and works really well. One
downside is I never got web access working, and I had pop3 secured
so it could only be accessed on my network. That meant that I could
never access my home email while traveling. A minor inconvenience,
but annoying.
More importantly, while I was away at MADExpo, someone
in Russia started using my server to relay spam. I was
pretty sure I had checked to make sure it wasn't an open relay. I
even ran some online tools which verified that I'm not running an
open relay.
So how were the spammers getting around this? I'm not sure if it
was a bug in the mail server, or simply some IT thing I don't quite
understand, but they were able to send the spam as long as it had
my email address in the "from" field. (Note that I changed my
domain name in the pasted log)
# Fields: SessionID SessionStartTime RemoteEndPoint AuthenticatedUser LogType LogText
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "220 ESMTP Windows 3.11 Mail Server"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "EHLO mail.irr1tat3pvowel.com"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-spaces.ru"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-PIPELINING"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-SIZE 32000000"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-VRFY"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-ETRN"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250-ENHANCEDSTATUSCODES"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 8BITMIME"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "MAIL FROM:<pmbrown@irr1tat3pvowel.com> SIZE=45521"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 2.1.0 Ok"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "RCPT TO:<Grodno92@spaces.ru>"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 2.1.5 Ok"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "DATA"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "354 End data with <CR><LF>.<CR><LF>"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "Binary data, sent 45524 bytes."
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" "<<<" "250 2.0.0 Ok: queued as 9CF0EC66F7EE"
"c2f1f7c4-ea4f-455e-8328-a634e3a6d24e" "7/8/2012 6:13:36 AM" "77.109.85.101:25" "" ">>>" "QUIT"
(I changed the spelling of my email address and domain
name above to help prevent crawlers from posting my correct
address)
Umm, yeah, that's reporting itself as Windows 3.11. I have no
idea if it really is or not (bonus retro points for the spammer) or
simply that their spam software reports itself as WFWG 3.11 for
grins.
But note that it's sending it from my email address. If I turn
on SMTP authentication on the server, no one can send me email
(strange, I know, but I tried it). If I leave it off, anyone can
send email purporting to be from anyone at the domain. There was no
middle ground with that server; I'm surprised it took this long for
the spammers to exploit that machine. I tried to fix that for
several days, but had zero luck.
The pattern was common: I'd figure out a way to shut it down (or
in the case of the storm, that happened as result of power loss).
Shortly after it came back up, I'd get a couple relayed messages
sent to an @spaces.ru account, and then the floodgates would open
and I'd start getting relayed spam from many different IP addresses
- all with me in the "from" field. All the bounced email would go
to my inbox.
Solution
I considered a number of different hosts. My criteria was:
- Retain existing email addresses
- Support several addresses plus aliases
- Works with Outlook
- Mobile and/or web access
- Free or very very cheap
So today I moved all my email accounts over to GMail. You can
have up to 10 email addresses, with your custom domain, for free.
That's enough for the family, and with the aliases, enough for the
service-specific accounts I've created for things like paypal,
ebay, etc. (helps cut down on phishing). Once you go over 10
users you start to pay a fee per user per month, and it makes more
sense to investigate other services like Office 365.
As I wrote the original draft of this post (waiting for my MX
records to update), I was still getting bounced spam messages in my
inbox. Who knows how many blacklists I'm on now. Grrr. My home mail
server is now turned off and everything goes through GMail.
We'll see how well GMail works for me. Both my wife and I use
Outlook for managing email, and GMail concepts like tags and
filters don't really match well with folders and other Outlook
concepts. Because I kept the same email addresses, I'll be free to
change to yet another provider in the future should I need to.
UPDATE: 7/10. I'm taking a closer look
at Office 365 right now as it works more like I do. In particular,
I (and my wife) both like to work from Outlook, and GMail and
Outlook don't quite mesh. You can force some things, but
Filters/Tags, calendaring and more just aren't doing it for me.
I've been an Outlook user since Day 1.
Web Site
I'm in the process of migrating my site off-premises as well.
More on that in another post. Right now, the lead contender is
Azure Web Sites (not the free one, as I get enough traffic to hit
that limit quite quickly).
End Result
The end result of all this will be to get out of the home IT
business. No more server rack, no more dedicated AC unit, no more
on-premises email, web, or other servers. Probably no home domain
server either. IT: It was fun while it lasted, but I
think it's time we decided to just be friends.
