First of all I am recommending that before start to apply following actions on the live environment try it on test environment.
1. Made full backup set from site server database and reporting services database.
2. If you have a chance to keep existing server, install new OS on another server and/or take image of exist OS in case if you need to rollback.
3. You could use Backup Recovery Solution in Configuration Manager - Scheduling the Backup Task
a. In the Configuration Manager console, click Administration.
b. In the Administration workspace, expand Site Configuration, and then click Sites.
c. Select the site in which you want to enable the site backup maintenance task.
d. On the Home tab, in the Settings group, click Site Maintenance Tasks.
e. Click Backup Site Server and then click Edit.
f. Select Enable this task, and then click Set network Paths to specify the backup destination.
g. Start SMS_SITE_BACKUP services.
4. The success of the backup task can be verified in the following ways:
a. Review the timestamp on the SMSBKUP.LOG file that the Backup Site Server maintenance task created in the backup destination folder. Verify that the timestamp has been updated with a time that coincides with the time when the Backup Site Server maintenance task was last scheduled to run. Review the log files for errors.
b. In the Component Status node in the Monitoring workspace, review the status messages for SMS_SITE_BACKUP. When site backup is completed successfully, you see message ID 5035, which indicates that the site backup was completed without any errors.
c. When the Backup Site Server maintenance task is configured to create an alert if backup fails, you can check the Alerts node in the Monitoring workspace for backup failures.
d. In <ConfigMgrInstallationFolder>\Logs, review Smsbkup.log for warnings and errors. When site backup is completed successfully, you see Backup completed with a timestamp and message ID STATMSG: ID=5035.
5. Back Up Custom Reporting Services Reports (if you have)
When you have modified predefined or created custom Reporting Services reports, creating a backup for the report server database files is an important part of your backup strategy. The report server backup must include a backup of the source files for reports and models, encryption keys, custom assemblies or extensions, configuration files, custom SQL Server views used in custom reports, custom stored procedures, and so on.
6. Backup Content Files
The content library in Configuration Manager is the location where all content files are stored for software updates, applications, operating system deployment, and so on. The content library is located on the site server and each distribution point. The Backup Site Server maintenance task does not include a backup for the content library or the package source files. When a site server fails, the information about the content library files is restored to the site database, but you must restore the content library and package source files on the site server.
• Content library: The content library must be restored before you can redistribute content to distribution points. When you start content redistribution, Configuration Manager copies the files from the content library on the site server to the distribution points. The content library for the site server is in the SCCMContentLib folder, which is typically located on the drive with the most free disk space at the time that the site installed. For more information about the content library, see Introduction to Content Management in Configuration Manager.
• Package source files: The package source files must be restored before you can update content on distribution points. When you start a content update, Configuration Manager copies new or modified files from the package source to the content library, which in turn copies the files to associated distribution points. You can run the following query in SQL Server to find the package source location for all packages and applications: SELECT * FROM v_Package. You can identify the package source site by looking at the first three characters of the package ID. For example, if the package ID is CEN00001, the site code for the source site is CEN. When you restore the package source files, they must be restored to the same location in which they were before the failure. For more information about updating content, see the Update Content on Distribution Points section in the Operations and Maintenance for Content Management in Configuration Manager topic.
Verify that you include both the content library and package source locations in your file system backup for the site server.
7. Built your new OS with same IP, Hostname make sure all windows updates installed and install SQL server.
8. Start SCCM setup and select Recover the site server using an existing backup .
9. Re-Enter User Account Passwords
After a site server recovery, passwords for the user accounts specified for the site must be re-entered because they are reset during the site recovery. The accounts are listed on the Finished page of the Setup Wizard after site recovery is completed and saved to C:\ConfigMgrPostRecoveryActions.html on the recovered site server.
10. Setup reporting services.