Glo Networks Technical Blog (Glo Blog)

As promised in “Our New Private Cloud Platform“, I’m about to divulge all our secrets. Or at least some of them. In vague deal.

I should warn you that this is a blog post aimed at technical people, who have some knowledge of Hyper-V clusters already, so if you’re looking at this from a “users” point of view you may get very lost, very fast. I’m not going to explain every little detail, because quite frankly we’d be here all day.

I feel I should start off by defining our “Private Cloud”. Cloud is term that thrown about a lot recently by marketing staff, and for that reason technical staff need to use it in front of boards and in front of the decision makers. To us techies it may be frustrating, however it’s the world we live in. If you’re uninitiated it’s a very broad term that covers:

• Infrastructure as a Service (IaaS) – Server and networking hardware, possibly server OS, such as Amazon’s AWS,
• Software as a Service (SaaS) – Software provided by a remote system, such as Google Apps (Mail, Docs, etc.), or Dropbox,
• Platform as a Service (PaaS) – Normally software infrastructure for developers to rapidly build software, such as force.com, Google’s AppEngine, or parts of Azure.

Whilst we also provide SaaS products, in this instance our small “Private Cloud” we see as an IaaS offering. It’s a small, 2 node, Hyper-V cluster that we use to run our own and customer’s systems.

As a rough outline our cluster consists of:

• 1x HP Procurve 2810-24G as our main switch,
• 1x Juniper SRX210 acts as our firewall and gateway device for some portions of our network,
• 1x IBM x3250 M3 acts as our physical Active Directory Domain Controller, and also hosts our Data Protection Manager (DPM) 2010 and System Center Virtual Machine Manager (SCVMM) virtual machine under Hyper-V,
• 2x IBM x3550 M3 act as our Hyper-V nodes,
• 1x IBM DS3512 acts as our shared storage,
• 1x QNap TS-459U+ acts as our short term backup storage,
• Several USB hard drives for off-site backup that are routinely swapped.

We’re aware that there are some issues with this design; single switch, single firewall and only 2 Hyper-V nodes. However the importance here is why we chose some of these things and why we don’t care as much right now (this was a significant investment for our small company);

1. Granted all hardware does die. In the event that a switch does we can get one on-site reasonably quickly if we needed to, however we’re yet to have a HP Procurve die on us since we’ve started business,
2. Single firewall is something that we worry about, but we’ve chosen Juniper as they are easily clustered,
3. Provided that you don’t over subscrbe 2 Hyper-V Nodes should be sufficient, however additional nodes can be introduced to the cluster easily in the future.

So whilst we are aware of the problems, I believe that we’ve engineered the system in such a manner that we’re able to introduce new hardware easily, upgrade the existing hardware, and provide some additional redundancy, including multiple switches with multi-chassis LACP links.

We’ve not built this system to compete with Amazon’s amazing AWS, however we have built it with 3 goals in mind:

1. Extensibility,
2. To use as a small reference design,
3. To virtualise our own systems more redundantly. The fact we’re able to host customer’s systems as well is a nice perk.

I won’t take you through the process of setting up your Hyper-V cluster, but I will cover a few bits and bobs that we feel a techy should be aware of before walking into a project like this, but might forget when looking at the big picture.

Clustered Shared Volume, or CSV, is the magic that makes the shared storage work. It’s a clever file system that allows multiple nodes to share the same storage. We’re yet to deploy a CSV using FC so we’re unsure if this is true for FC as well, however in the instance of both DAS and iSCSI what happens is the following;

1. The master node takes control of the storage,
2. All other nodes are notified of this, and effectively redirect all storage requests for the shared storage to the master node, over the network.

It should be clear from this that your choice of network card and switch are very important.

CSVs are not supported by Microsoft for any other use other than Hyper-V clusters. So don’t go getting any ideas.

Jumbo frames on your networking gear is a must. Generally speaking a Jumbo Frame is any ethernet frame that exceeds 1500 bytes, however they’re commonly also used as a naming convention for frames of 9000-9600 bytes (+/- 14 bytes for the header, depending on your switch(es)/NIC configuration language). If you don’t remember how IP and ethernet interact I suggest you go and refresh your memory very quickly. You should recognise the importance of having Jumbo Frames enabled very quickly; it should provide higher performance in situations where large payloads are being transmitted frequently.

At present we’re using Microsoft’s DPM 2010 to backup. The major gotcha that we didn’t see was that DPM 2010 on a Domain Controller is basically a no-no:

For a DPM server that is installed on a domain controller, only protection of data sources local to the DPM server is supported. You cannot install agents on other computers to configure protection.

SCVMM (System Center Virtual Machine Manager) 2008 R2 needs some polish. We’ve had to dive into the database once already. Don’t be afraid of it.

Other than that the project went exceedlingly smoothly. There are a few features that I wish Hyper-V had, in comparison to VMWare and Xen. And I really do wish that there were more, cheaper, graphics cards out there for RemoteFX.

However, that’s just something to plan for as a future project.

Posted by Karl in Applications, General, Hardware, Operating Systems | 2 Comments »

There’s a Microsoft TechNet article called ‘Ten Immutable Laws Of Security’. It discusses ten security problems that the Microsoft Security Center feel cannot be ‘fixed’ by software. These are persistent problems that won’t be solved by an update from the software manufacturer but by following best practice and using good judgement. The ten laws are as follows:

Law #1: If a bad guy can persuade you to run his program on your computer, it’s not solely your computer anymore.
Law #2: If a bad guy can alter the operating system on your computer, it’s not your computer anymore.
Law #3: If a bad guy has unrestricted physical access to your computer, it’s not your computer anymore.
Law #4: If you allow a bad guy to run active content in your website, it’s not your website any more.
Law #5: Weak passwords trump strong security.
Law #6: A computer is only as secure as the administrator is trustworthy.
Law #7: Encrypted data is only as secure as its decryption key.
Law #8: An out-of-date antimalware scanner is only marginally better than no scanner at all.
Law #9: Absolute anonymity isn’t practically achievable, online or offline.
Law #10: Technology is not a panacea.

While the entire article is interesting and raises several very good points today we’d like to focus on Law #5 – Weak passwords trump strong security.

Passwords can be and frequently are a pain in the neck but they are a necessary evil. Currently they are simply the best way to verify the person accessing the computer/program is who they say they are.

That isn’t to suggest that they are the only way. Several other methods have been tried; facial recognition, finger print scanners and RSA Keys are just a few such methods. They all have their strengths and weaknesses but for the most part where they fail when compared to passwords is ease and convenience of use.

The problem with this is ease and convenience are the downfall of secure passwords. A short, simple and easy to remember password won’t stand up against an attempt to crack it for long. For a password to be secure it needs to complex and not easy to guess, and as such the securest passwords are generally randomly generated strings of letters, numbers and symbols, the longer the string the better.

But the average person will struggle to recall a 64 character string of characters that means nothing to them, so instead they trade off better security in favour of convenience. A quick read through lists of the most popular passwords reveals that many people will chose overly simple or easy to guess passwords such as ‘123456’ or simply ‘password’.

Somehow a compromise must be made.  Passwords should be as long and as complicated as possible, while still being memorable. One suggestion we at Glo Networks can offer is, rather than just a word, why not use a phrase for your password? Switch some letters for numbers/symbols and make sure it’s a phrase that will stick with you whilst not being obvious.

Here’s a quick example of how you could change a phrase to a decent passphrase:

‘spend a penny’ becomes ‘Spend4penEE’

Using this method it becomes much easier to create a longer password that you will quickly remember every time!

Posted by James in Desktops and Laptops, General, Networking, Servers | No Comments »

Sometimes SCVMM 2008 [R2] has a problem, and it becomes out of sync with the underlying Hyper-V cluster. Usually a quick refresh of the cluster and you’re stored.

Sadly this is not always the case. In the instance where you see duplicate virtual machines you have a bit of an issue on your hands as you can’t migrate that virtual machine to the Hyper-V cluster node that also has the “missing” version of itself.

If you delve deeper, and check the Failover Cluster Manager, you should see that there is only one copy. With that verified the answer is to break out the SQL to fix SCVMM.

1. Ideally you’ll want to stop the SCVMM service (VMMService)
2. Download and install SQL Studio Manager, or if you’re really hardcore use osql. If you’ve used the default database then you’ll want to connect to COMPUTERNAME\MICROSOFT$VMM$. Otherwise it’ll be where ever it was installed.
3. Now there are “2″ solutions we’ve found for this problem. As always backup your database first.
   • We found by digging into the database that the virtual machines instances are stored in tbl_WLC_VObject and have an ObjectState equal to 220. Deleting these very quickly removes the VM. However it does potentially leave other references behind.
   • The technet way, which we’d recommend, (see http://technet.microsoft.com/en-us/library/ff641854.aspx), which looks at the same table, however also searches all other tables for references.
4. Restart the SCVMM service and you should be laughing all the way to the bank

Just incase Technet ever decides to go away, we’ve put the script from the page referenced above, below.

BEGIN TRANSACTION T1

DECLARE custom_cursor CURSOR FOR
SELECT ObjectId from
dbo.tbl_WLC_VObject WHERE [ObjectState] = 220
DECLARE @ObjectId uniqueidentifier
OPEN custom_cursor
FETCH NEXT FROM custom_cursor INTO @ObjectId
WHILE(@@fetch_status = 0)
BEGIN
DECLARE vdrive_cursor CURSOR FOR
SELECT VDriveId, VHDId, ISOId from
dbo.tbl_WLC_VDrive WHERE ParentId = @ObjectId
DECLARE @VDriveId uniqueidentifier
DECLARE @VHDId uniqueidentifier
DECLARE @ISOId uniqueidentifier

OPEN vdrive_cursor
FETCH NEXT FROM vdrive_cursor INTO @VDriveId, @VHDId, @ISOId
WHILE(@@fetch_status = 0)
BEGIN
DELETE FROM dbo.tbl_WLC_VDrive
WHERE VDriveId = @VDriveId
if(@VHDId is NOT NULL)
BEGIN

DELETE FROM dbo.tbl_WLC_VHD
WHERE VHDId = @VHDId
DELETE FROM dbo.tbl_WLC_PhysicalObject
WHERE PhysicalObjectId = @VHDId
END
if(@ISOId is NOT NULL)
BEGIN

DELETE FROM dbo.tbl_WLC_VCOMPort
WHERE HWProfileId = @ObjectId
DELETE FROM dbo.tbl_WLC_HWProfile
WHERE HWProfileId = @ObjectId
DELETE FROM dbo.tbl_WLC_VMInstance
WHERE VMInstanceId = @ObjectId
DELETE FROM dbo.tbl_WLC_VObject
WHERE ObjectId = @ObjectId
FETCH NEXT FROM custom_cursor INTO @ObjectId
END
CLOSE custom_cursor
DEALLOCATE custom_cursor
COMMIT TRANSACTION T1

Posted by Karl in Applications, Servers | No Comments »

Since the birth of Glo Networks we’ve been virtualising servers and desktops. Those of us who worked together before Glo Networks was started have been virtualising since early 2003. It’s fair to say that we love virtualising stuff. Several of us virtualise our own home systems.

It saves you money on power, hardware and in some cases because of how Microsoft’s licensing works, we’ve actually reduced the number of licenses that some of our customers had to buy. In early 2007, we started hosting systems and services online – both ours and one of our very first customer’s. It could be said that we were at the forefront of “cloud computing”, mere weeks before the term was coined.

Recently we came to terms with the fact that our platform needed a good redesign. It had grown with us and our customers, but it was coming a bit unwieldy to maintain. Last month we put in our new solution – a small Hyper-V cluster, backed up by Microsoft’s DPM 2010, powered by IBM X Series x3550 M3′s, an IBM x3250 M3, an IBM DS3512 disk system, a QNAP TS-459+ and HP Procurve 2800 series switch.

We’re still migrating our services and customers to it, but boy does it make a difference. With all our hardware consolidated and updated we’re seeing more responsive, more manageable and more highly available systems.

Would we do anything different? Probably, but that’s the way we are at Glo Networks; always striving for something better and trying to push the limit available to us at that time.

If you’re interested we’ll be following this up with a more technical post about how we setup our cluster.

Posted by Karl in Cool Stuff, General, Servers | 1 Comment »

Microsoft have announced they are updating pricing for Hosted BlackBerry Service for Exchange Online. Soon this service will be available free of charge to BPOS subscribers (new and existing).  The Hosted BlackBerry service offers increased functionality compared to the other, already free option: BlackBerry Internet Service.

Previously BPOS users who access their email via the free BlackBerry Internet Service were limited to wireless email access, pushed from the mobile operator. Other information, such as contacts and calendar data had to be updated through a wired connection.

Hosted Blackberry services allows wireless access to e-mail, calendar, tasks and contacts with global address list (GAL) integration, and device management such as device wipe and password reset.

By scrapping the current Hosted BlackBerry services pricing structure and moving towards a free service for BPOS subscribers, Microsoft will be hoping to see businesses that have come to like and possibly rely on their BlackBerry looking on the BPOS (and Office 365 in the future) offerings more favourably.

Posted by James in Applications, General, Small Business Software | 1 Comment »