Event ID 3 from Resourcemanager: No such user

Problem

A MIMWAL workflow fails, and the ForeFront Identity Manager event log records Event ID 3 from Microsoft.Resourcemanager with the message:

GetCurrentUserFromSecurityIdentifier: No such user [DOMAIN\USER], [SID]

image

The WAL event log was not particularly helpful in this case, it just threw out generic exceptions:

Event ID 40405: WAL (2.17.0927.0): 09/27/2018 14:20:47.3127: The type initializer for ‘Microsoft.ResourceManagement.WebServices.Client.ResourceManagementClient’ threw an exception.

Analysis

The Workflow in question tries to execute a PowerShell script, and we spent quite a lot of time troubleshooting the script to no avail. In retrospect, the problem is actually stated quite directly in the first event from the MIM/FIM log: “No such user”.

The user mentioned is the service account for the FIM Service, and both the username and SID is correct. Thus the error message did not really make any sense, I was not able to figure out why MIM could not find the user in AD when both the sAMAccountName and SID were correct. Wherein lies the problem. I was thinking backwards. The problem was not in AD, but in the FIMService database. For some reason the service account was not registered with a SID in the database. To troubleshoot you have to run some queries against the FIMService database.

Find the ObjectKey for the user

USE FIMService;
SELECT [ObjectKey],[DomainAndAccountName]
FROM [FIMService].[fim].[DomainAndAccountName]
WHERE DomainAndAccountName = 'DOMAIN\USER';
ObjectKey            DomainAndAccountName
-------------------- ----------------------------------------------------
12345                DOMAIN\USER

(1 row affected)

We find the object key “12345” and place it into the where-clause for the next query.

Find the SID

USE FIMService;
SELECT  [UserObjectKey] ,[SecurityIdentifier]
  FROM [FIMService].[fim].[UserSecurityIdentifiers]
  WHERE UserObjectKey = '12345';
UserObjectKey        SecurityIdentifier
-------------------- ----------------------

(0 rows affected)

This query returns no result. It should return the SID for the user in hexadecimal format.

Solution

Add the SID <-> ObjectKey mapping. As usual, make sure that you understand these steps before you execute them.

  • Get the hexadecimal SID. You can get it from AD Users and Computers in advanced mode.
  • Open your user, look at the attribute list and find the ObjectSID attribute.
  • View the attribute in hexadecimal form. It should look something like 01 05 00 00 00 and so on.
  • Remove the spaces using your favorite text editor, and add a 0x prefix to indicate that this is a hex value.
  • Your result should look like this: 0x010500000000000512345234504560734063457AFCDEBB69EE0000
  • Run the following query, inserting the username and hexadecimal SID
USE FIMService;
INSERT INTO  [FIMService].[fim].UserSecurityIdentifiers VALUES ('12345', 0x010500000000000512345234504560734063457AFCDEBB69EE0000);

To verify, run the query against UserSecurityIdentifiers again (the one that returned 0 rows) and verify that you now get a response mapping your ObjectKey to your SID. If you are lucky, your workflow is now working as expected. If you are not so lucky, you should at least get a different error message…

Redirecting the root site to the MIM Portal

Problem

When you install the MIM Portal, the root site of your portal will display a rather glum page devoid of anything immediately useful. It looks like this:

image

This is not the portal you are looking for… The one you want is located at /IdentityManagement. Now, there are many ways to work around this. You may publish through some kind of load balancer and add the redirect there, you can install IIS URL Rewrite and fiddle around with the settings for a while, but the most elegant solution I have come across so far is to change the WelcomePage in SharePoint.

Solution

Source: http://konab.com/redirect-identitymanagement-site-spf-2013/.

 

Notes

  • I use an http link in the example as this is from a lab setup, but you should of course use SSL in production. HTTP to HTTPS redirect is another issue (for another post).
  • I have not tested this on SharePoint 2016, but it I see no reason why it shouldn’t work.

 

Action plan

  • Start by opening the SharePoint 2013 Management Shell.
  • Enter the following commands, replacing the web application name with the URL for your MIM Portal.
$webapp = Get-SPWeb http://portal.mim.local
$root = $webapp.RootFolder
$root.WelcomePage = "IdentityManagement/default.aspx"
$root.Update()
  • The change should take effect immediately.

MIM: Sharepoint central administration returns 404

Problem

After an unscheduled reboot of some of my production MIM 2016, SharePoint central administration just returned a 404 error. As the reboot was caused by a massive network outage, pinpointing this specific error took some time.

 

Analysis

Rebooting the servers brought the MIM portal up again (after some time), but Sharepoint Central administration was still down. Rummaging around in the ULS log I came across this message, the first error message logged after trying to load the management site:

Failed to get document content data. System.TypeInitializationException: The type initializer for ‘Cobalt.MetricsBase`1’ threw an exception. —> System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

at System.Linq.Expressions.Expression.Parameter(Type type, String name)

at Cobalt.MetricsBase`1..cctor() –

— End of inner exception stack trace —

at Cobalt.MetricsBase`1..ctor()

at Microsoft.SharePoint.SPFileStreamHostBlobStore..ctor(SPFileStreamStore spFileStreamStore, Config config)

at Microsoft.SharePoint.SPFileStreamManager.CreateCobaltStreamContainer(SPFileStreamStore spfs, ILockBytes ilb, Boolean copyOnFirstWrite, Boolean disposeIlb)

at Microsoft.SharePoint.SPFileStreamManager.SetInputLockBytes(SPFileInfo& fileInfo, SqlSession session, PrefetchResult prefetchResult)

at Microsoft.SharePoint.CoordinatedStreamBuffer.SPCoordinatedStreamBufferFactory.CreateFromDocumentRowset(Guid databaseId, SqlSession session, SPFileStreamManager spfstm, Object[] metadataRow, SPRowset contentRowset, SPDocumentBindRequest& dbreq, SPDocumentBindResults& dbres)

at Microsoft.SharePoint.SPSqlClient.GetDocumentContentRow(Int32 rowOrd, Object ospFileStmMgr, SPDocumentBindRequest& dbreq, SPDocumentBindResults& dbres)

And this one followed:

Could not get DocumentContent row: 0x80131534.

Some searching brought up this discussion on Technet that contained the solution. It worked like a charm. As it is currently past 3am I have not made any attempt to discover why this happended, and why this solution works.

Update 2018-08-18: J. Qvarnström on the MIM FB Group suggested that the problem could be caused by .Net framework patches from June 2018. I checked, and there was indeed some .Net patches installed recently. In this particular case KB4099639 and KB4099635.

Update 2018-08-23: As suggested by M Kaufman on the MIM FB Group I tried to remove SCOM APM. That allowed me to remove the LoaderOptimization registry setting. SCOM APM has been installed since the server was created years ago according to our SCOM team, so something else must have triggered the problem. The .Net framework updates mentioned above are my primary suspects. That being said, SCOM APM is not supported on Sharepoint servers so it should be removed in any case.

Solution

As usual, do not perform these steps in production if you do not understand them. These steps should be performed on the MIM Portal servers.

Alternative 1: Disable SCOM APM

  • Aquire a copy of your current SCOM msi, usually called MOMAgent.msi and place it on the server.
  • Run the following command from an administrative command prompt:
msiexec.exe /fvomus "MOMagent.msi" NOAPM=1
  • Restart the server

Alternative 2: Change the .Net framework LoaderOptimization

If alternative 1 did not help, change the LoaderOptimization to 1. Be aware that this is a sledgehammer approach, but it is highly likely to get your system back up and running. Further investigation into the root cause is recommended.

  • Locate HKLM\SOFTWARE\Microsoft\.NETFramework
  • Add a new DWORD value called LoaderOptimization.
  • Set the value to 1. See MSDN for documentation.
  • Perform an iisreset.

image

About the UserAccountControl attribute

Intro

When working with MIM you will sooner or later have to deal directly with the UserAccountControl Active Directory attribute. This attribute defines account options, and we use it most prevalently to enable and disable users, but there are a lot of other options as well. These options are stored in a binary value as bit flags, where each bit defines a specific function.

Bit number 1 (or 2 if you are not used to zero-based numbering) defines whether or not an account is enabled. Bit number 9 defines an account as a normal account. Thus, a normal disabled account will have bits 1 and 9 set to one. As long as no other bits are set, the decimal value is 2^9 + 2^1 = 514 or (0010 0000 0010). If we enable the account, the value is 2^9 = 512 (0010 0000 0000).

In MIM we are usually presented with decimal values. These are easier to read, but not necessarily easier to understand.

Continue reading “About the UserAccountControl attribute”

MIM: The Portal cannot connect to the middle tier using the web service interface

Problem

After installing the MIM Service and Portal successfully, you are greeted by a portal that never loads and eventually displays a generic 503-error or a “Service not available notice”.

image

Analysis

This is a list of things I checked while trying to smoke out the badger causing this issue:

  • IIS bindings, even though I tested this prior to running the installer
  • The enormous setup log (verbose logging).
  • IISRESET.
  • Sharepoint alternate access mappings, also checked and found to be working prior to the installation.
  • Service status, both the FIM service and the Sharepoint services were running.
  • Restarted the server (have you tried turning it off and on again?).
  • FIM Event log, empty

And then I finally had the bright idea to check the application event log. It looked like the remnants of a great battle, only red and yellow messages in sight:

image

I dug in and found this one particularly interesting, Event 10 from Microsoft.ResourceManagement.PortalHealthSource:

The Portal cannot connect to the middle tier using the web service interface.  This failure prevents all portal scenarios from functioning correctly.


The cause may be due to a missing or invalid server url, a downed server, or an invalid server firewall configuration.


Ensure the portal configuration is present and points to the resource management service.

SNAGHTML2be3c711

I suddenly remembered that the load balancer was not yet configured and went to check the DNS records for the MIM urls. As I suspected, they were pointing to the load balancer, but the load balancer did not know where to redirect the traffic and thus did nothing.

Solution

For once, a simple solution without much of a risk factor:

  • Change the DNS record for the load balanced addresses, in this case the MIM Service server address to point directly to one of the portal servers.
  • Perform an IISRESET on the portal servers

I could of course fix the load balancer as well, but that requires a minion with access, and as the local time is 00:18 on a Saturday I will just add it to the list of things to fix later.

Administrator locked out of the MIM Portal after initial MA sync

Problem

After the first MIM Portal / Service management agent sync run the initial portal administrator account (the one used during portal installation) is locked out of the portal. The error message “Unable to process your request” and “The requestor of this operation is invalid” is displayed when you try to log in:

image

 

Analysis

For some reason, the User mapping is removed from the FIMService database. The query SELECT * FROM [FIMService].[fim].[UserSecurityIdentifiers] returns 0 rows. at this point, one row should be returned, lining the default admin UserObjectKey (2340 at time of writing) with the SID for the account used to install the MIM Service.

I found the solution with help from this post: http://dloder.blogspot.no/2011/12/administrator-locked-out-of-fim-portal.html.

In short, use Extended Events or SQL Profiler to find the ObjectID and corresponding ObjectKey.

SELECT *  FROM [FIMService].[fim].[Objects]
  WHERE ObjectKey = '2340'

ObjectKey	ObjectTypeKey	ObjectID
2340	24	7FB2B853-24F0-4498-9534-4E10589723C4

The ObjectKey and ObjectID for the first administrator account seems to be hard-coded into the FIMService database. This conclusion is based on the fact that I got the same values as those from a fresh MIM 2016 install as those listed in a post from 2011.

What remains is to re-establish the link between the FIMService Object and the AD SID (user).

Update 2017.09.20: Further analysis strongly indicates that the root cause of the problem is lack of a filter in the MIM/FIM Service MA during the initial sync run. There should be a filter in the MA preventing synchronization of the primary administrator account (the account used during installation) and the Built-in Synchronization account.

Solution

The usual warning: This solution details commands that should be understood before they are executed in a production environment. If the solution looks like gibberish, seek help before you continue. You may need a DBA to interpret the commands. And remember backups.

Get you SID in hexadecimal form. You can get it from AD Users and Computers in advanced mode. Open your user, look at the attribute list and find the ObjectSID attribute. View it in hexadecimal form. It should look something like 01 05 00 00 00 and so on. Remove the spaces using your favorite text editor, and add a 0x prefix to indicate that this is a hex value. Your result should look like this:

0x010500000000000512345234504560734063457AFCDEBB69EE0000

Execute the following SQL command against your FIMService database:

 
insert into [FIMService].[fim].UserSecurityIdentifiers 
values (2340, 0x010500000000000512345234504560734063457AFCDEBB69EE0000)

Then, perform an IISRESET. You should now be able to log in to the portal again.

MIM LAB7: Testing Run profiles and populating data

This post is part of a series. The chapter index is located here.

In this post we will create run profiles and initialize the MAs.

Continue reading “MIM LAB7: Testing Run profiles and populating data”

MIM LAB6: The AD MA and Run profiles

This post is part of a series. The chapter index is located here.

In this lab we will configure the firs AD management agent and set up run profiles.

Continue reading “MIM LAB6: The AD MA and Run profiles”

MIM LAB5: The MIM Service / Portal Management Agent

This post is part of a series. The chapter index is located here.

In this post we will install and configure the MIM Portal / Service management agent.

Continue reading “MIM LAB5: The MIM Service / Portal Management Agent”

MIM LAB 4: Installing the MIM Portal / MIM Service

This post is part of a series. The chapter index is located here.

In this post we install and configure the MIM Portal / Service.

Be aware that I had to make some changes to things I did in previous labs to make this work. I hope I have included all the details, but I have yet to re-run a complete install to test it. Continue reading “MIM LAB 4: Installing the MIM Portal / MIM Service”