NIST server hardening: Guide for NIST 800-123

The National Institute of Standards and Technology (NIST) creates guidelines and best practices to meet the needs of industrial, public, and federal agencies in Cybersecurity. NIST SP 800-123 Guide to General Server Security contains recommendations on how to secure your servers. Deviating from established NIST server hardening standards exposes vulnerabilities that attackers are quick to exploit.

In this article, you will learn how to plan a new server deployment and securely configure the server’s OS based on these NIST SP 800-123 server hardening guidelines.

What You Will Learn

• What is NIST 800-123
• Why you must identify a new server’s role, the services it provides, and the data it stores.
• Guidelines for selecting which users can access the server, the privileges assigned to users, and how to authenticate them.
• What server applications to install and how to secure them
• Why you must remove unnecessary services, applications, and network protocols

What Is NIST 800-123?

NIST server hardening standards refer to the guidelines and best practices for specific configuration settings and controls to mitigate vulnerabilities. Achieving security and compliance requires implementing server hardening as an essential prerequisite.

NIST 800-123 describes the steps for securing and maintaining network server security. Designed to complement broader frameworks like NIST 800-53 and the NIST Cybersecurity Framework (CSF), it offers general advice and guidelines on how you should approach this mission. It aims to assist organizations in understanding the fundamental activities they need to undertake to secure their servers. Regulations such as HIPAA, HITRUST, CMMC, and many others rely on those recommendations, demanding organizations to enforce and comply with the guide.

Now, we will examine NIST Guidelines for:

• Initial Server Setup
• Securing Server Operating Systems

NIST Guidelines for Initial Server Setup

The initial step involves planning the installation and deployment of the operating system (OS) and other components, crucial for securing and maintaining the server’s integrity from the outset:

Define the Server’s Role and Services

• Determine what the server will do (e.g., host web apps, store sensitive data).
• Identify which network services it will provide (e.g., HTTP, FTP, SMTP, NFS).
• List the required server-side, client-side, and support software.

Understand the User Environment

• Define who will use the server and from which hosts or systems.
• Specify privilege levels for each group of users and support hosts.
• Decide whether the server will be managed locally, from internal networks, or from external networks.

Plan Authentication and Access Controls

• Choose how users will authenticate (e.g., passwords, smart cards).
• Ensure that authenticated data will be encrypted and protected in transit.

Select the Right Server Application

Choose applications that prioritize security, even if that reduces functionality.

Choose a Hardenable Operating System

Ensure your OS supports the following:

• Restrict administrative/root-level activities to authorized users.
• Enforce granular access controls over server data.
• Allow OS configuration management and service disabling.
• Log key server activity for intrusion detection.
• Use a host-based firewall to control traffic.
• Support strong authentication protocols and modern encryption algorithms.

NIST Guidelines for Securing the Server Operating System

Appropriately configuring the server’s underlying OS is a critical part of securing the system against numerous security issues. Each organization needs to configure its servers according to its security requirements. The techniques for securing different types of OSs can vary greatly. NIST published generic procedures relevant to most OS.

After planning and installing the OS, NIST offers three issues that need to be addressed when configuring the server OS:

Step 1: Minimize the Attack Surface by Removing Unnecessary Components

The ideal state will be to install the minimal OS configuration and then add, remove, or disable services, applications, and network protocols. Removing unnecessary components is better than just disabling them. Only disabling will allow an attacker with the proper access to change the settings and enable the object. Human errors might also lead to configuration drift and expose the organization to unnecessary vulnerabilities.

Remove Unnecessary Service to Strengthen Security

• Each service added to the host increases the risk of leveraging it to access and compromise the server. When it comes to functionality versus security, less is more.
• Removing services may even improve the server’s availability in cases of defective or incompatible services.
• Reducing services will decrease the number of logs and log entries. Therefore, detecting suspicious behavior becomes easier.

Services and Applications to Review for Deactivation

• File and printer sharing services such as NetBIOS file and printer sharing, NFS, FTP.
• Wireless networking services.
• Remote control and remote access programs, especially those without strong encryption in their communication, such as Telnet.
• Directory services such as LDAP and NIS.
• Web servers and services.
• Email services such as SMTP.
• Language compilers and libraries.
• System development tools.
• System and network management tools and utilities, such as SNMP.

Step 2: Configure OS and User Authentication

Users who can access the server may range from a few authorized employees to the entire Internet community. To control access to the server, the server administrator should configure the OS to authenticate users by requiring proof that they can perform the intended actions. Enforcing authentication methods involves configuring parts of the OS, firmware, and applications on the server.

To ensure the appropriate user authentication is in place, take the following steps:

1. Remove or Disable Unneeded Default Accounts

OS default configuration can include guest accounts, administrator accounts, or root-level accounts. For machines containing sensitive information, it is recommended to disable access to guest accounts.

In addition, allow access to accounts associated with local and network services that truly require it. Accounts that need to access the server should protect their access by changing the name (don’t leave the default ‘Administrator’ name) and applying the organizational password policy.

2. Disable Non-Interactive Accounts

Disable accounts (and the associated passwords) that need to exist but do not require an interactive login.

3. Create the User Groups

Assigning individual accounts their required rights is complex when the number of users becomes too large to manage. The solution to this challenge is to assign users to different groups and assign the required rights to the group.

4. Create the User Accounts

Create only necessary accounts and permit the use of shared accounts only when there is no better option. Server administrators should also have an ordinary user account if they are also one of the server’s users.

5. Configure Automated Time Synchronization

Unsynchronized time zones between the client host and the authenticating server can cause several authentication protocols (such as Kerberos) to stop functioning. To prevent this, you must configure the server to synchronize the system time with a reliable time server automatically. Typically, the time server is internal to the organization and uses the Network Time Protocol for synchronization.

6. Check the Organization’s Password Policy

The organization’s password policy should include references regarding the minimum password length, the requirement for a mix of characters (complexity), how often passwords need to be changed (aging), whether users can reuse a password, and who’s allowed to change or reset a password.

This should also include any proof before initiating a change, such as how passwords should be stored. Passwords shouldn’t be stored unencrypted on the server. In addition, administrators should have different passwords for their server administrator account and for their other administrator accounts.

7. Configure Computers to Prevent Password Guessing

Automated password-guessing tools (network sniffers) allow unauthorized users to gain access relatively easily. There are two options to cope with those tools. If you can’t use option one, use option 2.

1. Configure the OS to increase the period between login attempts each time there’s a login failure.
   
2. Deny login after a limited number of failed attempts. In case of multiple failures, the account will then be locked for a period of time or until a user with appropriate authority reactivates it.
   
   Note: Implementing this recommendation may prevent some attacks, but can also lead to a Denial of Service condition. An attacker can use failed login attempts to control user access. The risk of DoS utilizing this method is greater if the server is externally accessible, as the attacker might know or guess the account name.

All failed login attempts, whether via the network or console, should be logged. If the server doesn’t need to be administered remotely, disable the option to log in from the network for administrators or root-level accounts.

8. Install and Configure Other Security Mechanisms to Strengthen Authentication

Servers containing sensitive information should strengthen authentication methods using biometrics, smart cards, client/server certificates, or one-time password systems.

Organizations should implement the latest authentication and encryption technologies, such as SSL/TLS, SSH, or virtual private networks, while using IPsec or SSL/TLS to protect passwords when communicating on untrusted networks. Using those methods will reduce the likelihood of man-in-the-middle and spoofing attacks.

Step 3: Restrict Access Through Resource Permissions and Auditing

You can specify access privileges for files, directories, devices, and other computational resources. Here are some examples of how a server administrator can reduce security breaches:

• Denying read access to files and directories helps to protect the confidentiality of information.
• Denying write (modify) access can help protect the integrity of information.
• Limiting the execution of system-related tools to authorized system administrators can prevent configuration drift. It can also restrict the attacker’s ability to use those tools to attack the server or other hosts in the network.
• Audit in order to monitor attempts to access protected resources.

These are the most basic issues one should consider in order to protect a server. The practical part of each step includes hundreds of specific actions affecting each object in the server OS. Building the right policy and then enforcing it is a rather demanding and complex task. Special resources should be invested in it, including money, time, and human knowledge.

Key Takeaways

• Identifying the new server’s role, the services it provides, and the data it stores
• Deciding which users can access the server, the privileges assigned to users, and how to authenticate them
• Determining which server applications to install and how to secure them
• Removing unnecessary services, applications, and network protocols
• Committing to maintaining server security with An Ongoing Process

Automate NIST Server Hardening with CHS by CalCom

Server hardening is an endless process as the infrastructure and security recommendations constantly change. Implementing and automating robust security controls through server hardening is essential to establishing a secure baseline.

Calcom Hardening Suite (CHS) is the perfect solution for this painful issue. CHS will transform your hardening project to be effortless while ensuring that your servers are constantly hardened to adapt to the dynamic nature of the infrastructure.