Monday, August 27, 2012

Best Practice Alternative to overcome List view lookup threshold limitations

Recently, once again i have come across the List view lookup threshold limitation while designing a custom webpart. I can not use more than 8 lookup fields(it includes user fields too) in a single list view or while fetching the list data using SPQuery.

There is configuration in Central adminstration however as per Best practice it is not at all advisable. It hamper the Server cpu very badly. Please finds links below to know more about it.
http://sympmarc.com/2012/07/23/sharepoints-list-view-lookup-threshold-and-why-we-dont-change-it/
http://technet.microsoft.com/en-us/library/cc262813.aspx

So there are few options available to overcome this.

Option 1 - Reduce number of lookup fields in the view. (This is not possible all the time as per requirement)
Option 2 - Increase List View Look up Threshold for that webapplication in central admin. (This is not advisable as it hampers Server CPU badly.)

So i have implemented following option. Option 3-

- While query list data split them in to two or more query groups such that you dont query more than 8 lookup fields in a single query.
- Get different data sets based on different SPQuery. Just make sure you have ID column in all the query groups.
- Merge them using LINQ

Please find the implementations as below -
- While query list data split them in to two or more query groups such that you dont query more than 8 lookup fields in a single query.
 protected void Page_Load(object sender, EventArgs e)
        {

               SPList splistObject = objSPWeb.Lists["ListName"];
                    SPQuery spqPart1 = new SPQuery();
                    spqPart1.ViewFields = @"<FieldRef Name='Title' />
   <FieldRef Name='LookupFieldName1' />
   <FieldRef Name='LookupFieldName2' />
   <FieldRef Name='LookupFieldName3' />
   <FieldRef Name='LookupFieldName4' />
   <FieldRef Name='LookupFieldName5' />
   <FieldRef Name='LookupFieldName6' />
   <FieldRef Name='LookupFieldName7' />
    <FieldRef Name='ID' />";
                    spqPart1.ViewFieldsOnly = true;

                    SPQuery spqPart2 = new SPQuery();
                    spqPart2.ViewFields = @"<FieldRef Name='LookupFieldName8' />
   <FieldRef Name='LookupFieldName9' />
   <FieldRef Name='LookupFieldName10' />
   <FieldRef Name='LookupFieldName11' />
   <FieldRef Name='LookupFieldName12' />
   <FieldRef Name='LookupFieldName13' />
   <FieldRef Name='ID' />";
                    spqPart2.ViewFieldsOnly = true;


                      SPListItemCollection splicPart1 = splistObject.GetItems(spqPart1);
                        DataTable dtListPart1 = splicPart1.GetDataTable();

                        SPListItemCollection splicPart2 = splistObject.GetItems(spqPart2);
                        DataTable dtListPart2 = splicPart2.GetDataTable();

                        dtOppInfo = MergeDataTablesVertically(dtListPart1, dtListPart2);

}

- Get different data sets based on different SPQuery. Just make sure you have ID column in all the query groups.

- Merge them using LINQ
public DataTable MergeDataTablesVertically(DataTable table1, DataTable table2)
        {
            DataTable retTable = new DataTable();
            var dtMergeVertically =
    from part1 in table1.AsEnumerable()
    join part2 in table2.AsEnumerable() on part1.Field<Int32>("ID") equals part2.Field<Int32>("ID")
    select new
   {
       Title = part1.Field<String>("Title"),
       LookupFieldName1 = part1.Field<String>("LookupFieldName1"),
       LookupFieldName2 = part1.Field<String>("LookupFieldName2"),
       LookupFieldName3 = part1.Field<String>("LookupFieldName3"),
       LookupFieldName4 = part1.Field<String>("LookupFieldName4"),
       LookupFieldName5 = part1.Field<String>("LookupFieldName5"),
       LookupFieldName6 = part1.Field<String>("LookupFieldName6"),
       LookupFieldName7 = part1.Field<String>("LookupFieldName7"),
       ID = part1.Field<Int32>("ID"),


       LookupFieldName8 = part2.Field<String>("LookupFieldName8"),
       LookupFieldName9= part2.Field<String>("LookupFieldName9"),
       LookupFieldName10= part2.Field<String>("LookupFieldName10"),
       LookupFieldName11= part2.Field<String>("LookupFieldName11"),
       LookupFieldName12= part2.Field<String>("LookupFieldName12"),
       LookupFieldName13= part2.Field<String>("LookupFieldName13"),
   };
            retTable = ConvertToDataTable(dtMergeVertically);
            return retTable;

        }


        public DataTable ConvertToDataTable<T>(IEnumerable<T> varlist)
        {
            DataTable dtReturn = new DataTable();

            // column names
            PropertyInfo[] oProps = null;

            if (varlist == null) return dtReturn;

            foreach (T rec in varlist)
            {
                // Use reflection to get property names, to create table, Only first time, others will follow
                if (oProps == null)
                {
                    oProps = ((Type)rec.GetType()).GetProperties();
                    foreach (PropertyInfo pi in oProps)
                    {
                        Type colType = pi.PropertyType;

                        if ((colType.IsGenericType) && (colType.GetGenericTypeDefinition() == typeof(Nullable<>)))
                        {
                            colType = colType.GetGenericArguments()[0];
                        }

                        dtReturn.Columns.Add(new DataColumn(pi.Name, colType));
                    }
                }

                DataRow dr = dtReturn.NewRow();

                foreach (PropertyInfo pi in oProps)
                {
                    dr[pi.Name] = pi.GetValue(rec, null) == null ? DBNull.Value : pi.GetValue
                    (rec, null);
                }

                dtReturn.Rows.Add(dr);
            }
            return dtReturn;
        }

No comments: