20 Jan

How we get things done for the City of Chicago

The checks and balances of government offered NIE an opportunity to assist with the City of Chicago’s migration to Office 365. The City of Chicago – Committee on Finance runs systems that are separate from the main City of Chicago network. As their systems are separate from the City of Chicago, but play an integral role, they needed a qualified, accomplished, and reputable Microsoft partner to handle their migration to Office 365. It was business as usual for the Committee on Finance as NIE worked diligently and accurately to ensure their Active Directory environment populated Office 365 allowing them single point of maintenance going forward. Coupled with top notch migration tools to ensure all of the Committee on Finance’s data was migrated quickly and completely to Office 365, users were able to continue working during the process and after the migration without any issues. Office 365 allowed Committee on Finance to streamline their back end while leaving the end user experience for the most part unchanged with additional services available should they chose to use them. Even though we added additional functionality and all the services that Office 365 offers like spam filtering, malware protection, click-to-run Office suite with anywhere access to their documents; change management of the system became easier as our implementation centralized management in Active Directory. The end result is a streamline implementation with single point of maintenance, reliability of Exchange email, and the services they need to effectively manage their integral and important part in the running of our fine City of Chicago.

 

Share this
11 Jan

Do you procrastinate your network maintenance? Disaster and Recovery. Success Story – this time….

What happened and what led to the disaster?

2 Virtual Servers went offline unexpectedly. The physical host has not been maintained for 1.5 years. No firmware updates, no hardware health checks, etc. This happened to a break and fix customer (those customers call for support only when something does not function properly and do not engage into monthly maintenance contracts)

How did we recover and how much time it saved?

The customer made a very smart decision to purchase a Datto Disaster Recovery and Continuity unit from us a few months back. 20 minutes from contacting of our Support Hotline the network was up and running again. We were able to spin up the servers on the Datto unit while working on a permanent fix. The physical server was repaired and the Virtual Server rolled back from the Datto backup unit by Saturday. If it was not for the Datto Business Continuity feature the customer would had been down for 2 business days. I will let the business owners do the math…

Detailed Description:

  1. Sometime on Thursday the Dell host had an error, this caused servers to go offline.
  2. Initially we identified this as a hard drive issue, so we then decided to turn on backups on your Datto so you could work while we resolved the issue.
  3. After contacting Dell, engineers initially thought just one drive was bad (and had this dispatched to your office), but since the server continued to throw random errors, they suggested we update all firmware, and if possible VMware, to the latest versions.
  4. Updates were started at the end of the business day Thursday, and wrapped up Friday morning.
  5. After updating, the hard drives were stress tested, and we ran a drive consistency test.  Dell reviewed results and logs, which actually confirmed the problem was not drives or anything like that (in fact, now all the drives showed up healthy).  The root cause was determined to be server firmware and raid controller firmware that were out of date (these require manual updates), and Dell had released newer firmware.  As a result, the RAID controller identified what it thought was disk fault, and took multiple disks “offline”, which triggered the event on Thursday.  Updated firmware now showed information correctly.
    NOTE: Dell advised that going forward we update firmware at a minimum once a year, and if possible every 6 months.  This is up to you of course.  The process takes 1-2 hours (less time if done more frequently—in fact, we did do firmware updates on this server, but the last time was June 2014, so over a year and a half ago).
  6. Since Dell had already shipped a drive, then recommended we might as well replace it, so at least one new drive is in the system.  Old drive to be shipped back in pre-paid box.
  7. With Edward’s permission, we updated host ESXi from version 5.0 to the latest 5.5.0 Update 3b.
  8. At End of Business on Friday, we ran a final backup of DC1 and TS3 on the Datto, shut them down, and began the restore process back to the Dell host server.
  9. We checked on restore process periodically until it concluded late in the evening, we restored backup settings, and confirmed servers were operational.  This did require re-activation of Windows Server, Microsoft Office, etc. (these can be lost when the server detects what it thinks is “new hardware”).
Share this

© 2017 Network it Easy, Inc. All rights reserved.