Problem
When trying to update the NVM/Firmware on an intel X710 SFP+ network interface card running on Windows the process is interrupted with the error message below:
OS layer initialization failed
This card was made by Intel and mounted in a HP ProLiant DL380 gen9. It was chosen over a genuine HP-approved card due to supply chain issues. I was trying to install version 8.5, the latest available from Intel at the time of writing.
Analysis
There is some component that is not available. I suspect a hardening issue, as someone installed an ginormous amount of hardening GPOs some time ago.
A process monitor trace shows that the tool loads pnputil and tries to install some drivers with varying degrees of success. Specifically it appears to be looking for iqvsw64e in miscellaneous temporary folders.
It has been a while since last time I read a Procmon output, but as far as I can tell the process is not successful. The files are included with the package and self-identifies as an “Intel Network Adapter Diagnostic Driver”.
Hypothesis: the nvm updater needs the diagnostic drivers to communicate with the adapter, but something blocks the installation.
I do not have access to test this, but I am pretty sure that there is a GPO blocking the installation. I tested the previous version from 2018 that had been installed successfully on the server, and it now fails with the exact same error. The next step would be to start disabling hardening GPOs, but as I do not have the access to do that directly on this server, I gave up and started looking for a workaround. Some hours later I found one.
Workaround
As per usual, if you do not fully understand the consequences of the steps in the action plan below, seek help. This could brick your server, which is a nuisance when you have 20, but a catastrophe if you have only one and the support agreement has expired.
Prerequisites
- HP ProLiant DL380 Gen9 (should work on all currently supported HPE ProLiant DL series servers).
- Windows Server 2016. Probably compatible with 2012-2022, but I have yet to test that.
- Other HP-approved Intel SFP+ network adapters mounted in the same server, in my case cards equipped with Intel 560-series chips. Could work with other Intel adapters as well.
- A copy of a not too old Service Pack for ProLiant iso for your server or similar. The SPP for DL380 gen10 has been tested, and I can confirm that it works for this purpose even though it will refuse to install any SPP updates.
- A valid HPE support contract is recommended
- A towel is recommended.
- A logged-in ILO remote console or local console access.
- The local admin password for the server.
Action plan
NB: If you are not planning an SPP update as part of this process, or if you are unable to obtain one, see the update below for an alternative approach. You need an active support contract to download SPP packages, but individual cp-s are available.
- Install the intel drivers that correspond with your firmware version. Preferably both should be the latest available.
- Be aware that this will disconnect your server from the network if all your adapters are intel adapters. Usually only temporarily, but if it does not come back online, you may have to reconfigure your network adapters using ILO or a local console.
- Reboot the server.
- Extract the NVM update
- In an administrative command shell, navigate to the Winx64 folder.
- Try running nvmupdatew64e.exe and verify the error message.
- Mount the SPP iso.
- Run launch_sum.bat from the iso as admin.
- In the web browser that appears, accept the cert error and start a localhost guided update:
- While the inventory is running, switch back to the command shell and keep trying to start the nvm update.
- This will take som time, so do not give up and remember your towel.
- Suddenly, this will happen:
- Update all cards that have an update available.
- Reboot the server. You may complete the SPP update before you reboot.
Update: an alternative method
As the HP SPP is a fairly large download to haul around, I kept looking for a mor lightweight workaround. If you are going to install an SPP anyway, using it makes sense, but if you are using other methods for patching your servers it is a bit overkill to use at 10GB .iso to install a 70kb temporary driver. Thus a new plan was hatched.
- Instead of the SPP iso, get a hold of the cpnnnn update package for you HP approved Intel-based network card. For my x560 card, the current cp is cp047026.
- Extract the files to a new folder. I have not tested whether or not it is possible to extract the package without the card being installed, but it appears to be a branded winzip self-extracter or similar so I expect it to work.
- Inside your new folder you will find a file called ofvpdfixW64e.exe. Run it from an administrative command shell.
- Wait for it to finish scanning for adapters.
- You should now be able to start nvmupdatew64e.exe and upgrade your X710.
As we can see, the tool detects both the HP approved and Intel original adapters. The tool is designed for a different purpose, but that is not important. All we need is a tool that will load the diagnostic drivers and thus enable our Intel updater to function. The package also contains rfendfixW64e.exe, another fixup tool that will load the driver. The HP branded firmware update tool (HPNVMUpdate.exe) may also load the driver in some scenarios. I guess what I am saying is try all of them if one is not working. And make sure to wait for the scan to complete before you try starting nvmupdatew64e.exe.
Also, make sure to install the PROSet drivers. I have had trouble getting this to work without them.
Why this works
The HP SPP is using a branded version of the Intel NVM updater. This updater is using the same driver mentioned above, at least for the 560-series of chips. It is running in a different host process and is thus able to circumvent the hardening that blocks the installation of the driver from the intel tool directly. When the SPP inventory process is querying Intel network adapters, the driver is loaded and keeps running until you reboot the server. You may be able to get this working even without any other Intel adapters, but I have not tested this scenario. It all depends on how the SPP inventory process runs.
Verify the result
You can verify the result using the Intel-supplied powershell commandlets. They are installed together with the PROSet driver package. You activate them by running this command:
Import-Module -Name "C:\Program Files\Intel\Wired Networking\IntelNetCmdlets\IntelNetCmdlets"
And you list the VNM versions running the next command. Be aware that HP branded adapters may not respond to this command and will be listed as not supported. These commands may be relatively slow to respond, this is normal.
Get-IntelNetAdapter | ft -Property Name, DriverVersion, ETrackID, NVMVersion
