We recently engaged in a conversation with our team of experts regarding their ongoing server hardening project. We inquired about the obstacles encountered during manual hardening procedures and asked if they’d be willing to explain the underlying reasons for issues that arise when automation is not employed. Their latest encounter with a client provided a valuable opportunity to further expound on strategies to mitigate these challenges.
Q: What challenges do customers face when they’re hardening their systems manually?
Team: We recently helped a customer with the deployment of a Windows 2012 failover cluster. We were able to configure a 3 node cluster in a matter of few hours but to my surprise, the final 'litmus test' of bringing the Network name up failed. Due to this, we could not bring the SQL Server Resource online. From both the cluster and the event log it was quite evident that we were hitting an Authentication issue:
Log Name: System
Source: Microsoft-Windows-FailoverClustering
Event ID: 1207
Task Category: Network Name Resource
Level: Error
Keywords:
User: SYSTEM
Computer: ********
Description:
Cluster network name resource ‘Cluster Name’ cannot be brought online. The computer object associated with the resource could not be updated in domain ********** for the following reason:
Unable to get Computer Object using GUID.
The text for the associated error code is: Logon failure: unknown user name or bad password
What was the Root Cause?
Team: Upon investigation, it turned out to be an issue with the security privilege 'Access this computer over the network'. By default on Windows Server 2012 the following groups have rights to access this computer from network GPO:
- Administrators
- Backup Operators
- Users
- Everyone
On the problem server(s), the Everyone group was missing; and adding it back resolved the issue. Why was this needed? The Cluster Name Object (CNO), service accounts of applications that use cluster and the computer object of each node would need to have permission to access the computer from the network.
Windows testing does not include test cases where the default setting has been changed. Therefore we cannot predict how cluster would behave after removing the default groups from the above GPO.
When did you have that 'aha!' moment?
Team: Over a cup of coffee, we asked the customer to share the reason behind this change. We were later informed of a "Hardening Document" implemented religiously before putting the servers into production. In our experience, such documents tend to be dangerous because they were written in the late 1990's and in many cases they have not been reviewed or updated for the modern OS versions.
What advice would you give to someone that takes on the labor-intensive job of manual hardening?
Team: Implementing manual hardening steps can introduce fragility into the system, making it vulnerable to failure from even minor environmental changes. It is difficult to predict when such failures might occur, so it is important to conduct thorough testing in an environment that closely resembles the production environment. This testing should consider the duration and coverage to ensure that all aspects of the system remain functional. It is important to note that passing testing for a certain number of days does not guarantee immunity from problems on subsequent days.
To summarize:
- Use of Security compliance manager or Security Configuration Wizard is the only supported way for Windows server hardening.
- Have a layered security approach that disallows direct access to the servers.
- Monitor your servers using a centralized auditing system.
- Manual Hardening is not at all recommended as detailed previously.
- Please ensure that Windows server 2003 hardening settings are not applied on newer operating system like Windows server 2008 R2, Windows server 2012
- Verify the source of the hardening document being followed in your organization
Common Mistakes:
| Setting | Description | More information |
| "Do not allow storage of credentials or .NET Passports for network authentication" | Task scheduler is widely in a lot of organizations and you should be aware that ENABLING the below GPO can cause scheduled tasks to fail in certain situations | The new version of Task Scheduler uses Windows Credential Manager to store the credentials of the account that is specified to perform a task. If the Network access: Do not allow storage of passwords and credentials for network authentication policy is enabled and applied, Since Credential Manager cannot store the credentials you will end with Task Scheduler related issues. |
| Windows Firewall Service being kept in the disabled state | Keeping the Firewall service as disabled causes the WMI calls to fail resulting in the failure many applications. Stopping the service associated with Windows Firewall with Advanced Security is not supported by Microsoft. | A natural question is:
What is the right way to disable Firewall? You can either disable it from the Windows Firewall Control Panel (firewall.cpl), OR Run the below commands and disable firewall for all profiles Netsh.exe AdvFirewall Set CurrentProfile State Off Netsh firewall set opmode mode = disable profile = ALL |
Address Space Layout Randomization
Previous versions of Windows used to load core processes into specific memory locations upon system startup, which made them vulnerable to certain exploits that targeted these known memory locations. However, with the introduction of ASLR in Windows 2008 and subsequent releases, memory locations used by system files and programs are now randomized, making it significantly more challenging for attackers to guess the location of a targeted process. By combining ASLR with DEP, a powerful security feature, attackers face a formidable obstacle when attempting to execute code reliably by exploiting vulnerabilities.
Are there specific reasons why hardening servers should be automated?
Team: Server hardening is a crucial security practice that involves implementing measures to protect servers from potential attacks and vulnerabilities. Therefore, hardening servers is necessary to reduce the risk of data breaches, cyber attacks, and other security incidents that can have devastating consequences for businesses.
Also when a client is applying the CIS Benchmarks, there are hundreds of pages of documentation and is incredibly time-consuming. By implementing automated server hardening measures, organizations can ensure that their servers are secure, reliable, and resilient to attacks continually without worrying about missed documented benchmarks. Automating the hardening process can ensure:
- Consistency: Automated server hardening ensures that the hardening process is consistent across all servers, reducing the risk of human error and ensuring that all servers have the same level of protection.
- Time-saving: Automating the hardening process saves time and effort compared to manual hardening, especially when dealing with large numbers of servers or systems.
- Scalability: Automated server hardening can be easily scaled to accommodate a growing number of servers, without requiring additional manpower or resources.
- Reliability: Automated server hardening provides a reliable and repeatable hardening process, eliminating the risk of errors or omissions that may occur with manual hardening.
- Compliance: Automated server hardening can help organizations meet compliance requirements by ensuring that all servers are hardened in a consistent and auditable manner.
- Rapid Response: Automated server hardening can be quickly implemented in response to new threats or vulnerabilities, ensuring that systems are protected in a timely manner.
