Problem
After an unscheduled reboot of some of my production MIM 2016, SharePoint central administration just returned a 404 error. As the reboot was caused by a massive network outage, pinpointing this specific error took some time.
Analysis
Rebooting the servers brought the MIM portal up again (after some time), but Sharepoint Central administration was still down. Rummaging around in the ULS log I came across this message, the first error message logged after trying to load the management site:
Failed to get document content data. System.TypeInitializationException: The type initializer for ‘Cobalt.MetricsBase`1’ threw an exception. —> System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)
at System.Linq.Expressions.Expression.Parameter(Type type, String name)
at Cobalt.MetricsBase`1..cctor() –
— End of inner exception stack trace —
at Cobalt.MetricsBase`1..ctor()
at Microsoft.SharePoint.SPFileStreamHostBlobStore..ctor(SPFileStreamStore spFileStreamStore, Config config)
at Microsoft.SharePoint.SPFileStreamManager.CreateCobaltStreamContainer(SPFileStreamStore spfs, ILockBytes ilb, Boolean copyOnFirstWrite, Boolean disposeIlb)
at Microsoft.SharePoint.SPFileStreamManager.SetInputLockBytes(SPFileInfo& fileInfo, SqlSession session, PrefetchResult prefetchResult)
at Microsoft.SharePoint.CoordinatedStreamBuffer.SPCoordinatedStreamBufferFactory.CreateFromDocumentRowset(Guid databaseId, SqlSession session, SPFileStreamManager spfstm, Object[] metadataRow, SPRowset contentRowset, SPDocumentBindRequest& dbreq, SPDocumentBindResults& dbres)
at Microsoft.SharePoint.SPSqlClient.GetDocumentContentRow(Int32 rowOrd, Object ospFileStmMgr, SPDocumentBindRequest& dbreq, SPDocumentBindResults& dbres)
And this one followed:
Could not get DocumentContent row: 0x80131534.
Some searching brought up this discussion on Technet that contained the solution. It worked like a charm. As it is currently past 3am I have not made any attempt to discover why this happended, and why this solution works.
Update 2018-08-18: J. Qvarnström on the MIM FB Group suggested that the problem could be caused by .Net framework patches from June 2018. I checked, and there was indeed some .Net patches installed recently. In this particular case KB4099639 and KB4099635.
Update 2018-08-23: As suggested by M Kaufman on the MIM FB Group I tried to remove SCOM APM. That allowed me to remove the LoaderOptimization registry setting. SCOM APM has been installed since the server was created years ago according to our SCOM team, so something else must have triggered the problem. The .Net framework updates mentioned above are my primary suspects. That being said, SCOM APM is not supported on Sharepoint servers so it should be removed in any case.
Solution
As usual, do not perform these steps in production if you do not understand them. These steps should be performed on the MIM Portal servers.
Alternative 1: Disable SCOM APM
- Aquire a copy of your current SCOM msi, usually called MOMAgent.msi and place it on the server.
- Run the following command from an administrative command prompt:
msiexec.exe /fvomus "MOMagent.msi" NOAPM=1
- Restart the server
Alternative 2: Change the .Net framework LoaderOptimization
If alternative 1 did not help, change the LoaderOptimization to 1. Be aware that this is a sledgehammer approach, but it is highly likely to get your system back up and running. Further investigation into the root cause is recommended.
- Locate HKLM\SOFTWARE\Microsoft\.NETFramework
- Add a new DWORD value called LoaderOptimization.
- Set the value to 1. See MSDN for documentation.
- Perform an iisreset.
