Patching Requirements
Windows Server patches, hotfixes and service
pack is critical for compliance, service level agreement and security
purposes. Keeping an operating systems and application up to date is the
key to align your infrastructure with latest software. Patches and
hotfixes also enable you to prevent any security breaches and malware
infection.
Windows Patch Classification
The following are strongly recommended patches:
- Critical
- Security
- Definition Updates for malware
- Service packs
Windows Product Classification
It is highly recommended that you patch Windows
Servers, Windows Clients, Office, Applications (Silverlight, .Net
Framework, SQL, Exchange, SharePoint, FF TMG).
Patching Groups
Consultants should take time to test the
patches in a non-production environment prior to being deployed to
production. This will help to gauge the impact of such changes. Ideally
you will have the following patching groups:
1. UAT (UAT1, UAT2, etc)
2. Test Environment (Test1, Test2, etc)
3. Development Environment (Dev1, Dev2 etc)
4. Production (Prod1, Prod2, etc)
If you have clustered environment like SQL,
Exchange and SharePoint then create Prod1, prod2 group and place each
node on each group.
Change Management
System administrators should maintain a log,
written or electronic, of all changes to the operating environment, to
include hardware, system security software, operating system, and
applications. Prior to any changes being implemented on a system, the
system administrator should receive approval of stakeholders.
Backup
Why am I discussing backup with patching best
practice? In case of emergency you can rollback completely and restore a
server to its original state if necessary. It is very important that
servers be backed up on a regular basis. Depending on the use of the
server, it may be adequate to backup the server once per week. A backup
of a more critical environment may be needed daily, and possibly
continuously. The backup program provided with Windows is capable of
backing up to virtually any writable media, which can include network
drives provided by a server in another physical location. This program
is also capable of scheduling backups which can ensure backups occur on a
regular interval.
Microsoft strongly recommends that you create
the following backups before you install an update rollup, service pack
and patch on Exchange and SQL:
- A full backup of all databases on the server.
- A full backup of transaction log and log backup
- A system state backup of the server.
- A snapshot of virtualized exchange server. Delete snapshot after successful patching and updating.
Application Compatibility
Read release notes of each hotfixes you are
going to apply so that you are compliant with the application installed
on the server. Consult with application vendor before applying service
pack to any server if the server is hosting specific business
application. Consult with application engineer about the importance of
server patching. Inform and educate application engineer as much as
possible to avoid conflict of interest.
Documentation
Documentation released with the updates is usually in the form of web pages, attached Word documents and
README.TXT files. These should be printed off and attached to change control procedures as supporting documentation.
Back out Plan
A back-out plan will allow the system and
enterprise to return to their original state, prior to the failed
implementation. It is important that these procedures are clear, and
that contingency management has tested them, because in the worst case a
faulty implementation can make it necessary to activate contingency
options. Historically, service packs have allowed for uninstalling, so
verify there is enough free hard disk space to create the uninstall
folder. Create a back out plan electronically and attach with change
management software.
User Notifications
You need to notify helpdesk staff and support
agencies of the pending changes so they may be ready for arising issues
or outages.
Consistency across Servers
Always install the same service packs or hotfixes to each SQL server node, Exchange DAG member and Domain Controller.
Routine Maintenance Window
A scheduled maintenance window must be agreed
with business so that application outage and server reboot can maintain a
respectable Service Level Agreement (SLA). If you have a large
infrastructure with thousands of servers and many regions working round
the clock then you must consider application dependencies. A patching
schedule can be considered in between every Friday of every month at
6:00 P.M. Friday to 6:00 A.M Monday. Setup maintenance window in system
center or deadline for WSUS to make sure patches are applied when you
want instead of when patch is available. In this way you will have a
complete control over change windows approved by change advisory board
(CAB). Do not allow end users to update patches on their client machine
according to their wishes and happiness! then user will never install
any patch.
Patching Tools
I strongly recommend that you spend few $$$ to
buy Microsoft System Center 2012 to manage and deploy Windows patches,
service pack and hotfixes. However you can use Windows Server Update
Services (WSUS) as poor man’s patching solutions.
Patching DMZ server can be accomplished using WSUS offline patching solutions available for free to download from
http://download.wsusoffline.net/.
Automate, Automate and Automate!
Automated patch management using System Center
could enable a single IT administrator to access a pre-populated patch
policy. He then could execute the command and with the press of a single
button, download the patches from Microsoft’s website, install them on a
test machine and test for compatibility issues. Meanwhile, an automatic
inventory check could search for systems with the affected software,
wake them up, check their readiness and push the verified patches out to
waiting machines. The patches would then be automatically installed on
each system, and they’d reboot as necessary. The final step is an
automated report on the status of the remediated devices.
Standardize Patch Management Processes
Standardized patch management processes could
allow for daily assessment and remediation of client devices and weekly
assessment and remediation for servers. Reports can then be generated to
validate system status on a weekly or bi-weekly schedule. A systems
monitoring task that used to take days now takes minutes, and patches
are deployed more completely and consistently across the entire IT
environment. A single IT administrator can proactively manage thousands
of systems tasks in the same amount of time it took an entire team to do
the tasks manually.
Reboot Windows Computer
Some application may require reboot of server
before patching such as RSA Secure Console. However most of the server
must be rebooted after patching. Do not suppress reboot after patching
in any circumstances or you will have a messy environment and broken
clusters.
X86 and X64 Windows Systems
The most prominent 32-bit application you’re
likely to see on a 64-bit Windows system is Office. In this sort of
situation System Center benefits most because you can adjust and make
decision based on architecture and compliance as well. You can approve
patches based on “Needed and Not Installed”. If a server or client need
update it will install if not then it will not installed. It’s safe to
do so.
Antivirus and Antispyware
Servers are vulnerable to many forms of attack.
Implementation and standardization of security methods should be
developed to allow early and rapid deployment on servers. It’s important
that a Windows server be equipped with a latest centrally managed
Antivirus program. Antivirus update must be scheduled with the same
maintenance window to update antivirus with latest definition.
Audit Practices
Servers have a powerful auditing feature built
in. Typically, server managers would want the auditing system to capture
logins, attempted logins, logouts, administrative activities, and
perhaps attempts to access or delete critical system files. Auditing
should be limited to gathering just the information that is needed, as
it does require CPU and disk time for auditing to gather information.
Log Management software should be used, if possible, for ease of
managing and analysing information. Report can be generated from Systems
Center and WSUS as proof of patching cycle.
Log Retention
Servers keep multiple logs and, by default, may
not be set to reuse log file entries. It is a good practice to expand
the size of the allowed log file and to set it to reuse space as needed.
This allows logging to continue uninterrupted. How far back your log
entries go will depend on the size of the log file and how quickly you
are accumulating log data. If your server environment is critical, you
may wish to ensure that the log file size is sufficient to store about
30 days of logging information, and then rotate log files once per month.
Installing Updates on a single Exchange Server
Download Exchange Update from Microsoft Download Center. Record Current Exchange Version information
Check for publisher’s certificate revocation
1. Start Internet Explorer.
2. On the
Tools menu, click
Internet Options.
3. Click the
Advanced tab, and then locate the
Security section.
4. Clear the
Check for publisher’s certificate revocation check box, and then click
OK.
5. After the update rollup installation is complete, select the
Check for publisher’s certificate revocation option.
Pre-check before installing
1. Determine which update rollup packages are installed on your Exchange server roles
2. Determine whether any interim updates are installed
3. Review interim updates
4. Obtain the latest update rollup package
5. Apply on a Test Exchange Server
Install Exchange Update
1. Ensure that you have downloaded the
appropriate rollup to a local drive on your Exchange servers, or on a
remote network share.
2. Run the Windows Installer *.msp Setup file that you downloaded in step 1.
Install Exchange Update on DAG Member
To update all DAG members, perform the
following procedures on each DAG member, one at a time. Set the member
server in maintenance mode using this PowerShell Command.
.
StartDagServerMaintenance.ps1
Install the update rollup
1. Close all Exchange management tools.
2. Right-click the Exchange update rollup file (.msp file) you downloaded, and then select
Apply.
3. On the
Welcome page, click
Next.
4. On the
License Terms page, review the license terms, select
I accept the License Terms, and then click
Next.
5. On the
Completion page, click
Finish.
Once installed exit from maintenance mode run the
StopDagServerMaintenance.ps1 script. Run the following command to re-balance the DAG, as needed
.
RedistributeActiveDatabases.ps1 -DagName
-BalanceDbsByActivationPreference -ShowFinalDatabaseDistribution
When the installation is finished, complete the following tasks:
- Start the Services MMC snap-in, and then verify that all the Exchange-related services are started successfully.
- Log on to Outlook Web App to verify that it’s running correctly.
- Restore Outlook Web App customizations, and then check Outlook Web App for correct functionality.
- After the update rollup installation is
complete, select the Check for publisher’s certificate revocation option
in Internet Explorer. See “Certificate Revocation List” earlier in this
topic.
- Check Exchange 2010 version information
- View Update rollup in Control Panel>Programs and Features
Patching Microsoft Failover Cluster
You can install Windows service packs on
Windows Server Failover Cluster nodes using the following procedure.
Administrative privilege is required to perform the following tasks.
Procedure to install Windows service pack or hotfixes in Windows Server 2003:
- Check the System event log for errors and ensure proper system operation.
- Make sure you have a current backup and
updated emergency repair disk for each system. In the event of corrupt
files, power outage, or incompatibility, it may be necessary to revert
back to the state of the system prior to attempting to install the
service pack/hotfixes.
- Expand Node A, and then click Active Groups. In the left pane, right-click the groups, and then click Move Group to move all groups to Node B.
- Open Cluster Administrator, right-click Node A, and then click Pause Node.
- Install the service pack on Node A, and then restart the computer.
- Check the System event log for errors. If you find any errors, troubleshoot them before continuing this process.
- In Cluster Administrator, right-click Node A, and then click Resume Node.
- Right-click Node B, and then click Move Group for all groups owned by Node B to move all groups to Node A.
- In Cluster Administrator, right-click Node B, and then click Pause Node.
- Install the service pack on Node B, and then restart the computer.
- Check the system event log for errors. If you find any errors, troubleshoot them before continuing this process.
- In Cluster Administrator, right-click Node B, and then click Resume Node.
- Right-click each group, click Move Group, and then move the groups back to their preferred owner.
Procedure to install Windows service pack or hotfixes in Windows Server 2008 and Windows Server 2012:
- Check the event log for errors and ensure proper system operation.
- Make sure you have a current backup and
updated emergency repair disk for each system. In the event of corrupt
files, power outage, or incompatibility, it may be necessary to revert
back to the state of the system prior to attempting to install the
service pack/hotfixes.
- On Node A, Expand Services and Applications, and then click the service or application
- Under Actions (on the right), click Move this service or application to another node, then choose the node or select Best possible.
- In the Failover Cluster Manager snap-in, right-click Node A, and then click Pause.
- Install the service pack/hotfixes on Node A, and then restart the computer.
- Check the event log for errors. If you find any errors, troubleshoot them before continuing this process.
- In Failover Cluster Manager snap-in, right-click Node A, and then click Resume.
- Under Actions (on the right), click Move this service or application to another node, then choose the node.
Note:
As the service or application moves, the status is displayed in the
results pane (in the center pane). Follow the Step 9 and 10 for each
service and application configured on the cluster.
- Install the service pack/hotfixes on Node B, and then restart the computer.
- Check the event log for errors. If you find any errors, troubleshoot them before continuing this process.
- From the Failover Cluster Manager snap-in, right-click Node B, and then click Pause.
- In Failover Cluster Manager, right-click Node B, and then click Resume.
- Right-click each group, click Move Group, and then move the groups back to their preferred owner.
You can use the following PowerShell Cmdlet to accomplish the same.
1. Load the module with the command:
Import-Module FailoverClusters
2
. Suspend (Pause) activity on a failover cluster nodeA:
Suspend-ClusterNode nodeA
3
. Move a clustered service or application (a resource group) from one node to another:
Get-ClusterNode NodeA | Get-ClusterGroup | Move-Cluster Group
4. Resume activity on nodeA that was suspended in step 5:
Resume-ClusterNode nodeA
5. Move a clustered service or application (a resource group) from one node to another:
Get-ClusterNode NodeB | Get-ClusterGroup | Move-Cluster Group
6
. Suspend (Pause) activity on other failover cluster node:
Suspend-ClusterNode nodeB
7. Resume activity on nodeB that was suspended in step 10 above:
Resume-ClusterNode nodeB
Conclusion
It is critical that when service packs,
hotfixes, and security patches are required to be installed, that these
best practices be followed.
Bottom line
1. Read all related documents.
2. Use a change control process.
3. Apply updates that are needed.
4. Test patches and hotfixes on test environment.
5. Don’t get more than 2 service packs behind.
6. Target non-critical servers first.
7. Service Pack (SP) level consistency.
8. Latest SP instead of multiple hotfixes.
9. Apply only on exact match.
10. Subscribe to Microsoft email notification.
11. Always have a back-out plan.
12. Have a working Backup and schedule production downtime.
13. Consistency across Domain Controllers and application servers.
Published By
S.G.Godwin Dinesh.MCA
Sr.System Administrator