In case you have installed the DPM agent manually, on a workgroup computer and you are facing issues when trying to attach it to the DPM server:

At first, check the following steps:

- Does pinging the Hostname from both sides work?
- Can you try to use the FQDN instead of the hostname for the setdpmserver command ?

Then, backup your Hosts an LmHosts files (located in the %SystemRoot%\System32\Drivers\Etc folder on a Windows computer) from both the DPM and the protected server.

Then, try to add the NetBIOS name of the DPM server, to the Hosts file on the affected remote server.
The refresh the DPMRA service and check the issue again.

If that is not enough (after monitoring the issue for some time) you need to update the Hosts file, on both servers > refresh the DPMRA service on the Protected server >
check the agent status on the DPM console..

If that fails too, then you will need to use the LMhosts file of the protected server:
Rename the extension from .Sam to simply lmhosts (you need to back-it-up first) > then add the static entry for the DPM server > then start the network discovery services.
Check the issue again.

More information you may find here:
Installing Protection Agents on Computers in a Workgroup or Untrusted Domain - http://technet.microsoft.com/en-us/library/hh757995.aspx

After installing the agent, you need to run SetDpmServer and specify the local user credentials which would be used for authentication.
A local user account will be created and the DPM protection agent would be configured to use this account for authentication.

Syntax:
SetDpmServer.exe -dpmServerName <serverName> -isNonDomainServer -userName
<userName> [-productionServerDnsSuffix <DnsSuffix>]

 Parameter

Description

-IsNonDomainServer

Specifies that this server is in a
  workgroup or an untrusted domain.

-UserName

Creates an NT user account with the
  specified username for this server to communicate with DPM server. This
  option should be used along with -IsNonDomainServer.

-ProductionServerDnsSuffix

In case there are multiple DNS
  suffixes configured for this server, ProductionServerDnsSuffix
  represents the DNS suffix which DPM server will use to communicate with this
  server.

-DpmServerName

Name
  of the DPM server. FQDN if DPM server and protected computer are accessible
  to each other using FQDNs. NETBIOS if DPM server and protected computer are
  accessible to each other using NETBIOS names.

If it fails on these ways too, you will need to collect the DPM logs (msdpmcurr.errlog or, the log from the time you have the last repro) so we can investigate further.