Who’s afraid of the BIG BIG SPList?

3 04 2009

During a Project intervention, I came accross something that just might sound very, very, VERY scary: big sharepoint lists.

There is a mystical number of 2000 ListItems, but this number is not correct. This number is a reference for ListItems per View and not per list. Well, in my scenario, my list was 65.000 ListItems and the client expects to have 150.000+ ListItems. Well, by first thought was “where the hell is the fire escape, let me out, let me out…”. Just kidding :-D. CAML queries to this list were instant thus, no problems there. The problem was around a routine that had to process each and every one of the items, breaking security inheritance and setting a set of a new permission groups.

The routine was in place, but took about 2 seconds to process each ListItem.

Yep, that’s about 36 hours of processing time…!!

Some code refactoring, cleaning, optimization, a lot of adrenalin J and SP1 (don’t ask…), we got the process down to 16 records per second. That’s about 1 hour and 8 minutes.

Considering this process runs in a job and during the night, it allows us to estimate a secure processing time of less that 8 hours for the 150k lists.

So, the main idea is:

  • Install SP1, rollups and the latest fixes (beware that some of them may add some glitches)
  • Keep you code neat, simple and as atomic as possible
  • Profile: profile your code, get execution metrics to you code
  • Focus on the operations that are executed the most. Those must be PERFECT!
  • String concatenation using “+” is a BIG NO NO. Use string.Format or StringBuilder instead

Besides that, treat big lists as exciting opportunities to refresh your knowledge on writing good quality code J.

bK00L & Have Fun (I know I DO! (sometimes…))



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: