Short Tom Blog

On a day to day basis, I'm troubleshooting SharePoint, SQL and Windows Server issues.  Amongst some of my Trinity Colleagues, I have become well known for my 1st port of call fix for any issue, the TSUF.

Let me explain, TSUF stands for Tom Short Universal Fix.  This actually means a server reboot, a power cycle, the 1 finger power button salute, “turning it off and on again”.

Having come from a support background, starting off on 1st line, I know the power of a reboot.  Many a in the past time I have uttered the priceless words “Have you tried turning off and back on again?” and many times it has worked.

So now I am trying to spread the gospel of the TSUF outside my close circle of colleagues.

The TSUF Fixes Everything*

I would like to let people know that I’m a serious IT Consultant\Engineer and always carefully evaluate and investigate any issues before acting the appropriate manner.  It just seems that many times the TSUF is applied and resolves the issue. 

I have done some research and I can confirm that my studies have shown that when the TSUF is applied, 75% of the time it works every time*.

*not actually correct. 

On my last SharePoint deployment, I came across a very peculiar issue.

When I opened the SharePoint Central Admin (SCA) site, selected the Operations tab and the Services on Server link was missing. when I typed in the URL directly (http://SCA/_admin/Server.aspx) I was prompted with an access denied page. Strange as I had farm administrator permissions??? I also noticed other links were missing, like Outgoing e-mail settings.

Bit of background, the SharePoint environment as all built on Windows Server 2008 R2 with UAC enabled to the highest level, requiring username and password to run all UAC tasks.

So I worked out, I had just opened Internet Explorer and clicked a favourite to SCA rather than via the official Central Admin link (START > Programs > Microsoft Office Server > SharePoint 3.0 Central Admin).

This link runs PSCONFIGUI.exe to query the SharePoint Config database to see where the SCA is running and which is a UAC task. opening it via the official Central Admin link gave me all the SCA options again.

For my Latest SharePoint Extranet deployment, I had a lot of issues with basic SharePoint functionality which I have never had issues with before.

• I could only upload a file up to 28MB in size, despite having the file upload limit set to 100MB (following these instructions from Microsoft - KB925083 & KB94481).
• Office Integration didn't work (checkout didn’t work and “Edit in Office” opened the document in Read-Only mode, which is pretty useless for a for a collaboration tool).

My SharePoint site was publishing via ISA Server 2006, so made use of persistent cookies to enable the Office integration to work.  I had tested this in my test environment and both the file upload limit and office integration worked as expected.

Whilst troubleshooting the issue in the live environment, one of my Colleagues realised a difference in my Test and Live environments.  The Live environment web servers had URLSCAN installed.  Another one of my Colleagues, tasked with performing server hardening of the live environment, had previously installed URLSCAN on my SharePoint web servers to secure IIS.  When I was asked about URLSCAN before it was installed, having not heard of it, I did a quick bit of research which led me to the conclusion it shouldn’t cause any issues. How badly I was wrong :(

To resolve the issue, we removed URLSCAN from the web servers and accepted the risk. The SharePoint file upload and office integration worked as expected.

Lessons Learnt

• SharePoint and URLSCAN just don’t get along, without the correct configuration and thought.
• Make sure your Test environment exactly matches the configuration of your Live environment.
• Take more time to research things you don’t know.

Issue

Following a fairly innocent reboot of all my SharePoint farm servers, all SharePoint site content was not displayed and I go the following error message. 

Interestingly though, all the content under /_layouts/ was displayed with no issues.  This made me think there must have been an issue with the content databases rather than the SharePoint servers themselves.

 Reason

After much investigation (crawling through logs and internet searches), I determined (with the help of Matt Groves) that the SharePoint aware anti-virus (Symantec 5.1 for SharePoint) had stopped and left the VirusStatus code for the master page and several page layouts in anti-virus limbo. 

When a virus is detected or when the service that connects SharePoint to the antivirus scan engine becomes unavailable, the antivirus scanner sets the VirusStatus code to a non-zero and non-null value.  This makes SharePoint believe the files have a virus and won’t display it (pretty disastrous when its the master page!). 

The scan engine became unavailable because the temporary licence for the Symantec 5.1 for SharePoint had expired. I believe the reboot stopped the scan engines from running, the license may have expired days or weeks ago.

You may also have an issue if you remove your SharePoint aware anti-virus.

Solution

1. Run the following SQL Query against your content database(s) to determine if they have have files stuck in AV limbo

SELECT * FROM Docs

WHERE (VirusStatus > 0) AND (VirusStatus IS NOT NULL)

(see http://support.microsoft.com/kb/928169 for more detail)

This will return all files which SharePoint believes are viruses.

• Enable your SharePoint aware anti-virus (I did this by applying a new license code to all the SharePoint servers).  If you’ve uninstalled it, reinstall your AV, fix the content and follow products recommended removal process.
• Run a manual AV scan on SharePoint using your SharePoint aware AV.
• Run the SQL Query again to make sure no virus items are returned.  if they are, run your manual scan again and try and determine if you have a  virus.