During a Project intervention, I came accross something that just might sound very, very, VERY scary: big sharepoint lists.
There is a mystical number of 2000 ListItems, but this number is not correct. This number is a reference for ListItems per View and not per list. Well, in my scenario, my list was 65.000 ListItems and the client expects to have 150.000+ ListItems. Well, by first thought was “where the hell is the fire escape, let me out, let me out…”. Just kidding :-D. CAML queries to this list were instant thus, no problems there. The problem was around a routine that had to process each and every one of the items, breaking security inheritance and setting a set of a new permission groups.
The routine was in place, but took about 2 seconds to process each ListItem.
Yep, that’s about 36 hours of processing time…!!
Some code refactoring, cleaning, optimization, a lot of adrenalin J and SP1 (don’t ask…), we got the process down to 16 records per second. That’s about 1 hour and 8 minutes.
Considering this process runs in a job and during the night, it allows us to estimate a secure processing time of less that 8 hours for the 150k lists.
So, the main idea is:
- Install SP1, rollups and the latest fixes (beware that some of them may add some glitches)
- Keep you code neat, simple and as atomic as possible
- Profile: profile your code, get execution metrics to you code
- Focus on the operations that are executed the most. Those must be PERFECT!
- String concatenation using “+” is a BIG NO NO. Use string.Format or StringBuilder instead
Besides that, treat big lists as exciting opportunities to refresh your knowledge on writing good quality code J.