File server memory leak?

Problem

On two of our fileservers (Windows 2008R2) we noticed an increase in memory usage over time. It would start out at say 1.5GiB after a boot, and then slowly work it’s way up to 6GiB, that was the server’s allocated amount of memory (vmware). This being a busy file server due to hosting our user profiles for citrix, we tried increasing the memory allocation to 8GiB. Sadly, this only had the effect that reaching 99% memory usage took longer time after a reboot. After a day or two it would be back up. Further investigation revealed that it also affected performance. Backup took 18 hours for 800GiB, and once in a while it would just give up. Testing also revealed that profile access was sometimes slow.

Continue reading “File server memory leak?”

Folder copy with logging in Powershell, and a bit about scripts in general

Problem

I have been trying for some time to find an easy method for keeping my scripts up to date on my servers. I could of course use robocopy or something like that, but I wanted something written in PS. I figured I would learn something along the way, and I also had hopes that this would be fairly easy to accomplish. It would seem I was wrong on the being easy part, or perhaps I have over-engineered it slightly Smilefjes som blunker.

Not wanting to re-invent the wheel, I summoned the powers of the closest search engine to get up with some samples I could build on. I am a bit prejudiced towards scripts from the web in general, as I usually find most scripts longer than a few lines have some logical bugs in them. Scripts are, in general, easy to get started with, but it is very difficult to produce robust scripts. I have debugged countless VB and Powershell scripts (both my own and those of others) who were working fine in the lab, and perhaps also in production, but suddenly they cease to function as expected. Usually this is caused by some simple logical error appearing due to changes in the environment the script is running in, but from time to time you come across some obscure scripting engine bug you have to program around. And of course you have the pure idiotic fails, such as creating vb-scripts requiring “On error resume next” at the top of the script. Those are usually doomed by design and can take days to debug. Since I am fairly proficient in C# I usually just write a small utility .exe instead, thus circumventing many of the problems altogether. Once you have spent 4 hours debugging an error caused by a misspelled variable name in the middle of a 200 line script, you start dreaming about the wonders of explicit variable declaration and intellisense. Anyways, I think that is enough ranting for one post.Smilefjes

Continue reading “Folder copy with logging in Powershell, and a bit about scripts in general”

How to list all PSTN calls in LYNC/ OCS 2007R2

UPDATE 2011-10-20

Revised the queries and confirmed that it also works for Lync.

Problem

I needed a list of all calls leaving and entering the organization via the Mediation servers IP trunk for invoice control. We suddenly detected a large increase in the volume of external calls through our OCS, and wanted to make sure the calls were actual calls and not misconfigured call redirection or something else. We had already deployed the monitoring role, but it was not logging any calls. This was due to an MSMQ error, which meant that the monitoring server could not receive any log messages. I fixed that and waited a couple of days for the data to accumulate. Then I had to find a report to give me the data I wanted. The default OCS reporting services reports allows me to list all the calls for a specific user, but I wanted to list all PSTN calls for all users.

Solution

First I tried to modify the included reports, but they were closely linked to several stored procedures, who again called other stored procedures and gave me far to much data anyhow. So I gave up that approach and started looking at the tables in the database. At http://technet.microsoft.com/en-us/library/dd819961(office.13).aspx I found a nice reference of all the QOE database tables. This enabled me to construct the following query:

USE [QoEMetrics]
GO

select us.URI as CallerURI, us2.URI as CalleeURI, StartTime, EndTime, 
datediff(Second, StartTime, EndTime) as DurSeconds ,
CONVERT(varchar(6), datediff(Second, StartTime, EndTime)/3600)
+ ':' + RIGHT('0' + CONVERT(varchar(2), (datediff(Second, StartTime, EndTime) % 3600) / 60), 2)
+ ':' + RIGHT('0' + CONVERT(varchar(2), datediff(Second, StartTime, EndTime) % 60), 2) as DurHMS, CallerEndpoint, CalleeEndpoint

from Session se
join [User] us on se.CallerURI = us.UserKey
join [User] us2 on se.CalleeURI = us2.UserKey
where (CalleeEndpoint = 1 or CalleeEndpoint = Null or CallerEndpoint = 1 or CallerEndpoint = Null) AND DialogCategory = 0 AND StartTime Between '2011-03-08' and '2011-03-09'

The where-statement is where the magic happens. You have to tailor this to your needs.

First it checks the endpoints, which identifies the caller and callee type. The endpoint ID’s are found in the Endpoint table. In my case, ID 1 translates to the mediation server. If you have more than one mediation server, you will have to add entries for all of them. I have also found the mediation server to have more than one ID, probably caused by changes made to the topology. A PSTN call will have the mediation server as an endpoint if the number is registered in Active Directory. Otherwise, the endpoint will be null. If the endpoint has a value other than null or the mediation server’s id, it refers to another computer running the OCS client. The endpoint table lists the computer names.

The second part of the where statement defines the timeframe from 00:00 on the first date to 00:00 on the second date. StartTime Between ‘2011-03-08’ and ‘2011-03-09’ will return all calls made between 2011.03.08 00:00:00 and 2011.03.09 00:00:00.

Running this query will list the calls made in the specified timeframe where one of the parties is on the PSTN. If you want to list outbound calls only, use this version:

USE [QoEMetrics]
GO

select us.URI as CallerURI, us2.URI as CalleeURI, StartTime, EndTime, 
datediff(Second, StartTime, EndTime) as DurSeconds ,
CONVERT(varchar(6), datediff(Second, StartTime, EndTime)/3600)
+ ':' + RIGHT('0' + CONVERT(varchar(2), (datediff(Second, StartTime, EndTime) % 3600) / 60), 2)
+ ':' + RIGHT('0' + CONVERT(varchar(2), datediff(Second, StartTime, EndTime) % 60), 2) as DurHMS, CallerEndpoint, CalleeEndpoint

from Session se
join [User] us on se.CallerURI = us.UserKey
join [User] us2 on se.CalleeURI = us2.UserKey
where (CalleeEndpoint = 1 or CalleeEndpoint = Null) AND DialogCategory = 0  and StartTime Between '2011-03-08' and '2011-03-11'

Enable logging for Windows Firewall (2008R2)

When troubleshooting problems with the internal Windows firewall it might be beneficial to know exactly what traffic is being blocked. One can of course just turn the firewall off to test if things start working, and then search for documentation for the failing application or service. Sadly, such an approach causes security issues during testing, and documentation is often not complete as to which ports an application actually depends on. The firewall log makes it somewhat easier to troubleshoot without having to disable the firewall completely.

Configuration

Start with bringing up the firewall properties from the Windows Firewall from the Advanced Security mmc snap-in:

image

You can configure logging for each of the profiles (domain, public and private). By default they all log to the same file, %windir%\SYSTEM32\Logfiles\firewall\pfirewall.log. It might be smart to use different log files if you have connections on more than one profile, e.g. if you have one lan and one wan adapter. Logging dropped packages only is recommended, as logging successful connections will fill up the log quickly on a busy server.

image

I would recommend turning logging of when troubleshooting is finished and leaving the log size limit at 4 096 KiB. If you specify a different folder than the default folder you must make sure that the firewall service have the necessary file system permissions. Unlike the w3svc log the firewall log is limited to two files, the main .log and a .old file. This ensures that the disk is not filled with firewall log files, and translates to a maximum disk space allocation of 2 times the size limit.

Analysis

The log files are space delimited, and can be imported into a spreadsheet for analysis, but it is easier to use a specialized log analyzer such as Sawmill (large professional tool) or ZedLan Firewall Log Analyser (freeware).

Exchange 2010 Dag won’t go online

Problem

After installation of Exchange 2010 SP1 Rollup5 I discovered that one of my networks was listed as partitioned in EMC. I ran several test without being able to find a cause for this. I am able to ping the addresses both ways, and there are no routes from network 1 to network 2. I even ran “Validate this cluster” (network only) successfully. Further investigation established that the cluster core resources were offline as well:

SNAGHTML5c562012

After a lot of fruitless searching I came across this Technet blog: http://blogs.technet.com/b/timmcmic/archive/2010/05/12/cluster-core-resources-fail-to-come-online-on-some-exchange-2010-database-availability-group-dag-nodes.aspx , who describes this situation and how to resolve it. But it also claims that the error is fixed in SP1. Since I’ve been at SP1 since installation I was skeptical, but I tried it anyway. As I suspected it didn’t help, but a comment from MattP_75 put me on the right track to a solution.

The problem is related to one or both of the networks not allowing client connections. I even found it in the eventlog, event 1223 from FailoverClustering.

SNAGHTML5c5c3660image

When I tried to change it in Failover Cluster management as the event suggested, it just bounced back to not allowing client access. To get it to stick, I had to change it in the registry.

I have no idea why this happens, but several of the comments on the article mentioned above talk about backup agents, mostly backup exec which I don’t have on my servers.

Solution

This is what I did to resolve the issue:

  • Shut down one of the nodes to ensure quorum
  • Change the role value to 3 on all cluster networks
    image
  • Get the core resources online
  • Restart the other node
  • Check the registry on both nodes to verify

I have had this happen again when I restart the cluster node that is hosting the Public Folders database, but it doesn’t happen every time.

Update 2011.11.23:

Microsoft recently released a hotfix which might be related to this error, kb2550886. According to the Exchange Team blog this is highly recommended for Exchange DAG’s running on Windows 2008R2, and they describe a scenario resembling the problem mentioned above. I have not verified that this update solves the problem permanently, but it is most certainly worth installing at your earliest convenience if you haven’t done so already.

Unable to access local drive(s)

Problem

On a Windows 2008 or 2008 R2 server administrators are unable to browse the contents of local drives while logged on to the server either directly at the console or via remote desktop. Access to the same drive using a network share works fine. UAC is turned on, and the local administrators group have full control access to the drive(s) in question. You get an “Access denied” error in Windows Explorer even when running in an elevated process (administrator mode).

The problem also affects Windows Vista and 7.

Analysis

If you try to access the drive using a program other than Windows Explorer, you can access the drive as long as the program is running in an elevated session. The problem seems to affect Windows Explorer alone, but I am not sure about that. What I have been able to establish though, is that it only affects users who are members of the local “Administrators” group. If a user has explicit access or access through another group, everything works as expected.

I detected the problem while migrating files and permissions from an old 2003 server to a new one running 2008 R2, and I think it is related to the local “Users” group not being granted access to the drive. Not denied, just removed from the root acl on the drive.

Solutions

  • Add explicit access to the drive for the administrative users that need access
  • Turn off UAC (not recommended)
  • Create a new group called Local_Admin_Access or something like that, add the local administrators group as a member, and give the new group full control of the drive.
  • Give the local group “Interactive” full control of the drive. This grants access to any user who have local logon permissions and are currently logged on to the server.

Presentasjon på ekstern skjerm

Hvordan man får powerpoint til å vise presentasjonen på en bestemt skjerm, ikke bare på primærskjerm når man  har satt opp mer enn en skjerm på maskinen i utvidet modus.

Slå på utvidet modus dersom det samme bildet vises på prosjektoren og på hovedskjermen:

Trykk [Win] + P , velg utvide

image

I powerpoint, trykk lysbildefremvisning, deretter Vis på: og velg den skjerm du vil ha presentasjonen på.

image

List users in OU not enabled for Lync

get-csaduser -filter {Enabled -ne $True} -OU "ou=orgunit,dc=domain,dc=topleveldomain"| sort SamAccountName |ft SamAccountName, UserPrincipalName

Lists users contained in the specified organizational unit that are not enabled for Lync in a table ordered by username (ascending).

Useful for finding users not listed in Lync Server Control Panel. The reason for this is most likely that the user has an AD primary email address (UserPrincipalName) in a domain that is not Sip enabled.

SNAGHTML295c7788

This can happen if your AD domain is not the same as your sip/e-mail domain. To correct this, you could either change the primary email address in AD(best option) and try again when AD is synced, or you could manually specify a sip address in the correct domain (Enable-CSUser –SipAddress “sip:x@y.tld” …)

Some clients unable to log on externally

Problem

Some users complain about not being able to log in to Lync externally. The domain logon is looping, even though they have entered the password correctly and a check confirms that their account is not locked out or disabled. Additionally, Event ID 14614 from LS Protocol Stack is logged on the FrontEnd server when the users try to log on:

SNAGHTML198fc660

Solution

The solution is mentioned further down in the event log message: “Ensure that the “Network security: Minimum session security for NTLM SSP based (including secure RPC) clients” policy settings on the computers from which users log on are the same as “Network security: Minimum session security for NTLM SSP based (including secure RPC) servers” policy settings on this server.” This translates to a configuration error in the policy mentioned. On Server 2008 R2 and Win7 the default setting is require 128-bit encryption, on earlier versions of windows both 128 bit encryption and NTLMv2 is off. To solve the problem, either turn it off on the server side (not recommended) or turn it on at the client side. This setting can be pushed as a computer GPO.

In my case the user experiencing the problem was actually running Win7, but said computer had joined a Windows 2003 server domain which had probably pushed the wrong setting. Changing the setting was all it took to fix it, no restart of either Lync nor Windows was required.

image

Avinstallere CCM klienten

Dersom du har fått installert System Center Configuration Manager klienten på en maskin som ikke skal/bør ha slik klient, feks. på et databasecluster, kan det være greit å kunne fjerne den igjen. Først kan det være en ide å sjekke om den faktisk er installert. Den ligger nemlig ikke i legg til/fjern programmer, men man kan lett identifisere den ved å se etter tjenesten SMS Agent Host:

image

For å fjerne den igjen kjøres følgende kommando i et elevated command prompt:

ccmsetup.exe /uninstall

Hvor denne filen ligger er dog en annen sak. Det avhenger av hvordan CCM er installert, og hva slags OS man har. På 32bit er det vanligvis en mappe kalt %SYSTEMROOT%\System32\CCMSetup, på 64 bit os kan det variere litt, men vanligvis en av følgende:

  • %SYSTEMROOT%\System32\CCMSetup
  • %SYSTEMROOT%\CCMSetup
  • %SYSTEMROOT%\SysWOW64\CCMSetup

Om man ikke finner den kan man referere til et nettverksshare på CCM serveren. Hva det heter må få fra de ansvarlige for løsningen (dersom det ikke er en selv), eller man kan være utspekulert søke opp filene selv Winking smile