Who’s afraid of the BIG BIG SPList?

During a Project intervention, I came accross something that just might sound very, very, VERY scary: big sharepoint lists.

There is a mystical number of 2000 ListItems, but this number is not correct. This number is a reference for ListItems per View and not per list. Well, in my scenario, my list was 65.000 ListItems and the client expects to have 150.000+ ListItems. Well, by first thought was “where the hell is the fire escape, let me out, let me out…”. Just kidding . CAML queries to this list were instant thus, no problems there. The problem was around a routine that had to process each and every one of the items, breaking security inheritance and setting a set of a new permission groups.

The routine was in place, but took about 2 seconds to process each ListItem.

Yep, that’s about 36 hours of processing time…!!

Some code refactoring, cleaning, optimization, a lot of adrenalin J and SP1 (don’t ask…), we got the process down to 16 records per second. That’s about 1 hour and 8 minutes.

Considering this process runs in a job and during the night, it allows us to estimate a secure processing time of less that 8 hours for the 150k lists.

So, the main idea is:

• Install SP1, rollups and the latest fixes (beware that some of them may add some glitches)
• Keep you code neat, simple and as atomic as possible
• Profile: profile your code, get execution metrics to you code
• Focus on the operations that are executed the most. Those must be PERFECT!
• String concatenation using “+” is a BIG NO NO. Use string.Format or StringBuilder instead

Besides that, treat big lists as exciting opportunities to refresh your knowledge on writing good quality code J.

bK00L & Have Fun (I know I DO! (sometimes…))

Search customization: the Path property

Search is a powerfull component of the sharepoint architecture. During the customization process, it is frequent to use manage properties to fine tune your results, in my scenario, even to the list level.
Managed properties are like dynamic column names that you can map to internal columns (collection of). After configured, you can use a managed property to filter content by adding it to the where clauses of your SQL statement. I’ll try to post a detailled information on how to create and use managed properties.

For the scope (sorry, couldn’t resist…) of this post, I would like to share something that kept me going for a couple of hours:

• there is a core managed property called “Path”
• this managed property normally contains the URL to the item (page, listitem, document depending on the type of the element)

The tricky part was to find out that if for some reason the item (in this case a document of an unkown extension to the crawller) is not indexed, the Path property contains a link not to the item/element itself, but to the DispForm.aspx that contains the metadata of the document.

Hope this saves you the time I lost…

Cheers.

“Code blocks are not allowed in this file”

If you come accross this error, here is the information you need to solve it:

Cause: you are running code snippets inside a file that is not allowed to do so by default.

Afected files: i.e. Master pages and Page layouts

Solution: you have to add an explicit authorization for the file(s) where you want to do so. The question is: do you really want to do it? You should take the best approach which is to add a code behind file (you can find how to do it here: Adding Code-Behind Files to Master Pages and Page Layouts in SharePoint Server 2007). If you still want to do it, just add a tag like this to your <PageParserPaths>:

<PageParserPath VirtualPath=”/_catalogs/masterpage/MyMaster.Master” CompilationMode=”Always” AllowServerSideScript=”true” />

Keep having fun

Sharepoint config store

While the web.config may be a nice place to store your configuration information, you need to edit it manualy if you need to change a configuration key. This operation must be repeated as many times as the number of machines you have on your farm. Thus it is not practical and it does not make sense. Other approaches include SQL table, sharepoint lists and so on.

Chris O’Bryan came up witn an implementation of a config store, based on a SPList and with the basic methods to access them.

The project is available from codeplex and you can find the approach described here: A better Config Store for SharePoint sites .

Great work Chris!

It is out: SPDisposeCheck

It is finally out: . You can get it from here: http://code.msdn.microsoft.com/SPDisposeCheck.

For those of you that haven’t heard about it, this is a tool that integrates with Visual Studio, and checks your assemblies for Sharepoint Disposable objects not being disposed. This is a must. Use it frequently during the development and NEVER, and I do mean NEVER, put your code into productyion before evaluating it with this tool.

The problematic around disposing objects can be found in articles like Best Practices: Using Disposable Windows SharePoint Services Objects.

Get it RIGHT NOW!

Debuging WSS 3/MOSS 2007

In all these years on developping, I have found that one of the main productivity factors is the ability to find the cause of the errors. Once you know where they are happening, solving is usually faster. This knowledge is an ongoing process that you build upon, and the longer you work on a certain tecnology/platform/programming language the faster you will be able to work your way arount it. In order to help you to achieve this state, I have put together some topics that are really just common knowledge, but that would have saved me some hard time when I have started to develop on Sharepoint. The debugging tips that I normally use are:

1. Turn off Custom Errors: This tag basically controls what error pages your web site will show when an error occurs.  In most cases you would not want to show a full stack trace to the world when an error occurs so I would opt for setting this to On for most of the time and RemoteOnly when you want to debug something.  If it is a dev only system though knock yourself out and set it to Off.  You can set this tag in the <system.web> tag though it should already exist for you to modify.  More info on the tag here.
   • On: This will always show a simple error has occured page and will not show details.
   • RemoteOnly: This will show a simple error has occured page to remote clients and a full stack trace to a browser from the local server.
   • Off: This shows a full stack trace to everyone when an error occurs.
   • Ex: <customErrors mode=”RemoteOnly” />   

2. Enable stack trace: in your web.config file enable the CallStack and AllowPageLevelTrace parameters
       <SafeMode MaxControls=”200″ CallStack=”true” DirectFileDependencies=”10″ TotalFileDependencies=”50″ AllowPageLevelTrace=”true”>   

3. Add an HttpModule to handle exceptions of type “An unknow error has occured.”: Here is a post whith that approach.  
4. Attach the debugger to the process W3WP (be sure to attach to the right one as there may be more than one. When in doubt, attach to all of them. Don’t forget to copy the *.pdb to the location of your dll (GAC or Bin). Some tools (i.e. WSPBuilder) have shortcuts to help you in this process.  Use the command iisapp.vbs to check which application pool is running he site you want to debug.
5. Attach the debugger to the process owstimer: if you need to debug workflows, for example, you will need to attach the debugger to this process.  
6. Use windbg: when the going gets tough, go right to the bone. Some links on windbg:

• Windows Debuggers: Part 1: A WinDbg Tutorial
• Debugging Tools for Windows – Overview

How about you? Any other hints? Looking forward to hear from you!

Have fun!

Profiles and Forms Based Authentication: Connecting the hard way

Recently I had to store some user information that made sense to be present in the use profile. The plot got thicker as I added the FBA (Forms Based Authentication) variable to this formula. Why? Because out of the box the only way to create the profiles is to import them from the active directory. Although you can find connector for LDAP, and BDC they only work in append mode to add extended information to the already created profiles.

Don’t panic! there’s a solution for this. The trick is to create the profiles through object model. There is a project in codeplex called “MOSS 2007 Profile Import Tool“, that provides you with the source for a connector. The implementation is based on a command line tool that can be called on demand or included in some scheduler mechanism. It is up to you if you want this flavour or for example, want to have some fun with sharepoint timer jobs .

No big news here, but can keep you from throwing some time out the window, trying to find out why the FBA profiles aren’t available.

Sharepoint development: how I’m doing it

 As a consultant and with a lot of friends in partner companies, I constantly come across people willing to start developing on top of Sharepoint (MOSS/WSS) platform. I agree it is not an easy task. The object model is huge, you have several ways of achieving an objective, but usually there is one that works better than the others. The absence of a full tool set, originated a big (AND GREAT!!!) community workforce that developed a set of tools to achieve those tasks that could only be done through object model. On top of this, there are serious challenges in debugging, accessibility, but mainly in understanding how all the pieces fit together.

This week, a friend of mine told me he was beginning to develop in Sharepoint and if I knew Sharepoint Designer. At that point I thought to myself “Ok, there is a message that is not reaching everybody. How to start developing.”.

Before some of you start throwing rocks because of the above reference to the designer, don’t get me wrong. I use it too! I use it for creating my master pages and layouts, that I then export and pack inside the WSP solution in visual studio. Don’t stress (yet ). I’ll go through the process ahead…

Disclaimer: This is by no means a suggestion that sharepoint development must be done this way or that this is the best way of doing it. I’m simply showing how I do my developments. This method has proven ME to be a good approach and even so, every new project I change a couple of things to evolve and ajust to each project reality.

Therefore, there is no guarentee that this approach will work for you. Use it at your own risk. I will not be responsible for any unforseen actions/reactions.

In short: This works for me, feel free to use it if you like it, but don’t come after me if it doesn’t work for you  

</Disclaimer>

I will do this in a serie of posts where I will try to show you:

1. My environment: What I use
2. How I structure my work
3. How I install my work
4. Which are my prime references

 

I hope this helps! Coments, questions, suggestions are all welcome. 

Office 2007 SP2 is comming

From the Office Sustained Engineering blog, there is the announcement of the SP2 combined (client and server) release for late 1st quarter of 2009. The post (Announcing Service Pack 2 (SP2) for the 2007 Microsoft Office System ), reveals some details of the package namelly the server components:

•  Performance and manageability improvements to variations in Enterprise Content Management (ECM) including STSADM commands for repairing links between source and target pages

·         Improvements around processing status approvals from Office Project Web Access into Office Project Professional 2007

·         Improvements to read-only content databases and index rebuild timer jobs in Windows SharePoint Services 3.0

Baddly behaved Publishing controls? Hmm…

This is the second time this kind of errors happened in projects where I am. You add a couple of scripts (js) and suddenly your publishing controls start breaking, not rendering, etc. Well, start with the basics: think of accessibility and if you do, you can’t use scripts. Anyway, if this kind of behaviour happens to you, start by commenting not standard js and css files. If by any remote chance you used mootools… that’s probably the one !