Welcome to MSDN Blogs Sign in | Join | Help
Tip 8 - How to write 'WHERE IN' style queries using LINQ to Entities

Imagine if you have a table of People and you want to retrieve only those whose the Firstname is in a list of interesting firstnames. This is trivial in SQL, you write something like this:

SELECT * FROM People
WHERE Firstname IN ('Alex', 'Colin', 'Danny', 'Diego')

A SQL IN is equivalent to a LINQ Contains

In the LINQ (to objects) world there is no 'IN' so you need to reverse the order like and use the Contains method:

var names = new string[] { "Alex", "Colin", "Danny", "Diego" };

var matches = from person in people
        where names.Contains(person.Firstname)
        select person;

Notice that semantically we've gone from:

value.IN(set)

in SQL to

set.Contains(value)

In LINQ. The result however is the same.

Support for Contains in .NET 3.5 SP1 vs .NET 4.0

IEnumerable<T>.Contains(T t)will be supported in EF in .NET 4.0, so you can write queries like the LINQ query above in the next version of EF.

Unfortunately though it doesn't work in .NET 3.5 SP1: LINQ to Entities complains when it encounters a LINQ expression like this because it doesn't know how to translate calls to Contains to SQL.

But what about all our .NET 3.5 SP1 users out there? What can they do in the meantime?

Workaround for .NET 3.5 SP1

Well there is a workaround courtesy of Colin one of the big brains on the EF team.

The essence of the workaround is to recognize that you can rewrite the above query like this:

var matches = from person in people
        where person.Firstname == "Alex" ||
              person.Firstname == "Colin" || 
              person.Firstname == "Danny" ||
              person.Firstname == "Diego"

        select person;

Certainly this is more 'wordy' and a pain to write but it works all the same.

So if we had some utility method that made it easy to create these kind of LINQ expressions we'd be in business.

Well that is exactly what Colin did in our forums a while back, with his utility method in place you can write something like this:

var matches = ctx.People.Where(
        BuildOrExpression<People, string>(
           p => p.Firstname, names
        )
);

This builds an expression that has the same effect as:

var matches = from p in ctx.People
        where names.Contains(p.Firstname)
        select p;

But which more importantly actually works against .NET 3.5 SP1.

Extra Credit...

If you've read this far, well done!

Here is the plumbing function that makes this possible:

public static Expression<Func<TElement, bool>> BuildOrExpression<TElement, TValue>(
        Expression<Func<TElement, TValue>> valueSelector, 
        IEnumerable<TValue> values
    )
{     
    if (null == valueSelector) 
        throw new ArgumentNullException("valueSelector");

    if (null == values)
        throw new ArgumentNullException("values"); 
 

    ParameterExpression p = valueSelector.Parameters.Single();

    if (!values.Any())   
        return e => false;

    var equals = values.Select(value =>
        (Expression)Expression.Equal(
             valueSelector.Body,
             Expression.Constant(
                 value,
                 typeof(TValue)
             )
        )
    );

   var body = equals.Aggregate<Expression>(
            (accumulate, equal) => Expression.Or(accumulate, equal)
    ); 

   return Expression.Lambda<Func<TElement, bool>>(body, p);
}

I'm not going to try to explain this method, other than to say it essentially builds a predicate expression for all the values using the valueSelector (i.e. p => p.Firstname) and ORs those predicates together to create an expression for the complete predicate.

For more information on the technique Colin uses here mosey on over to his blog and read this.

Posted: Thursday, March 26, 2009 12:47 AM by AlexJ
Filed under: , ,

Comments

Meta-Me said:

Hopefully if you're reading this you've noticed that I've started a series of Tips recently. The Tips

# March 25, 2009 8:04 PM

Martin Laufer said:

Hi Alex,

unfortunately in T-SQL there is no easy way to express a recordset of constants like the array initializer in C#. The analogous way would be insertrs into a temporary table.

With that in mind one can rewrite the query with the IN operator using an equi-join. As far as I remember, LINQ-2-SQL was able to transform a query like

var matches = from person in people

             join name in names on person.Firstname equals name

             select person;

into the query with the IN operator. I wonder if EF isn't capable of that. Do I miss something?

Best regards

Martin

# March 27, 2009 2:23 PM

AlexJ said:

This query isn't supported in EF in .NET 3.5 SP1

But it will be supported (along with Contains()) in .NET 4.0 in the same way L2S supports it today, namely by translation to IN (...)

Trivia:

Actually the way this is actually supported is that the EF produces a list of OR predicates in our Canonical Query Trees (provider agnostic trees), and then the SqlClient provider, recognizes that all those predicates share the same parameter on the same side, which means it can translate it into a SQL IN (...)  

Alex

# March 27, 2009 7:48 PM

smnbss said:

# March 27, 2009 9:05 PM

dahlbyk said:

I think you want Expression.OrElse (logical OR) rather than Expression.Or (bitwise OR).

Also, rather than considering Any() a special case, I would use DefaultIfEmpty():

<pre>var equals = values.Select(value =>

       (Expression)Expression.Equal(

            valueSelector.Body,

            Expression.Constant(

                value,

                typeof(TValue)

            )

       )

   ).DefaultIfEmpty(

       Expression.Constant(false)

   );</pre>

Cheers ~

Keith

# August 26, 2009 6:14 PM

AlexJ said:

Keith,

The reason I use Expression.Or(..) is that no matter what we do we can't convince the database to do a short-circuiting evaluations, so from the EF's perspective Or and OrElse are identical. So by using Or we 'produce' CLR expressions who's intent can actually be satisfied by the database!

Nice spot though!

Alex

# August 26, 2009 8:06 PM

Jeff said:

I made a suggestion of how the 2100 parameter limit could be addressed using xml parameters on the Microsoft .Connect site but it looks like they are not interested in this solution.

https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=475984&wa=wsignin1.0

I really think we need a decent way of handling large mufti valued parameters which doesn't require hacking the expression tree or changing our queries to call some other workaround method.

# September 16, 2009 12:20 PM

Ngoc said:

Hi Alex,

How to write a LINQ query like this SQL statement in Entity Framework (with the above workaround)?

SELECT * FROM People

WHERE Id = 123 OR Firstname IN ('Alex', 'Colin', 'Danny', 'Diego')

# September 25, 2009 12:59 PM

Abrar said:

Hi Alex,

I am trying to fit the below Buildexpression function in my join query. My query is

var q = from l in blendedItems

join v in blendedItemsView

on l.LineItemID equals v.LineItemID

where v.Status == _readyStatus

&& strArray.Contains(v.BillingGroupID)

select l;

I am able to do this in a saperate statement like this:

var query = dao.LineItemInterface.Where(BuildContainsExpression<LineItemInterface, string>(l => l.BillingGroupID, strArray));

Thanks in advance.

Abrar

# November 10, 2009 12:30 PM

AlexJ said:

@Abrar

I don't think I understand what you are asking for. Can you be a bit more explicit for me? I.e. can you write the code you WANT to be able to write?

Cheers

Alex

# November 10, 2009 1:10 PM

Abrar said:

Thanks for your prompt reply.

I have a join query which has a contains function with comma saperated list as an array.

As .Net Framework 3.5 sp1 doesn't support Contains function, I am using BuildContainsExpression function by Colin.

Using this i am able to retrieve values in a simple way but i am not able to fit this function in my Join query.

Given below is my Join query and the simple BuildContainsExpression function

My Join Query

------------------

var q = from l in blendedItems

join v in blendedItemsView

on l.LineItemID equals v.LineItemID

where v.Status == "Blended"

&& strArray.Contains(v.BillingGroupID)//should go here

select l;

My Simple IN query

------------------

var query = dao.LineItemInterface.Where(BuildContainsExpression<LineItemInterface, string>(l => l.BillingGroupID, strArray));

I want to fit the Simple query into my Join query.

Should fit in the commented line ("should go here").

Thanks in advance.

Abrar

# November 10, 2009 1:24 PM

Maria Dixit said:

Hi Alex,

I have a "WHERE IN" query which works fine with one condition, now i want to add one more condition in my existing query.

Can you please help me out in writing the second condition.

my second condition is "WHERE cust.Region == _strRegionName"

below is my query with first condition with "WHERE IN or BuildContainsExpression" which is working fine, i want to add second condition in this query:

var query = dao.LineItemInterface.Where(BuildContainsExpression<LineItemInterface, string>(l => l.BillingGroupID, strArray));

static Expression<Func<TElement, bool>> BuildContainsExpression<TElement, TValue>(

              Expression<Func<TElement, TValue>> valueSelector, IEnumerable<TValue> values)

               {

                   if (null == valueSelector) { throw new ArgumentNullException("valueSelector"); }

                   if (null == values) { throw new ArgumentNullException("values"); }

                   ParameterExpression p = valueSelector.Parameters.Single();

                   // p => valueSelector(p) == values[0] || valueSelector(p) == ...

                   if (!values.Any())

                   {

                       return e => false;

                   }

                   var equals = values.Select(value => (Expression)Expression.Equal(valueSelector.Body, Expression.Constant(value, typeof(TValue))));

                   var body = equals.Aggregate<Expression>((accumulate, equal) => Expression.Or(accumulate, equal));

                   return Expression.Lambda<Func<TElement, bool>>(body, p);

               }

Thanks,

M

# November 11, 2009 7:39 AM

AlexJ said:

@Maria,

Can you write some pseudo code - i.e. what you want to write, imagine that the language allows it.

That will help me understand your requirement.

Alex

# November 13, 2009 7:42 AM

AlexJ said:

@Abrar

I sure there are better ways, but one option would be to do this:

(

  from l in blendedItems

  join v in blendedItemsView

  on l.LineItemID equals v.LineItemID

  where v.Status == "Blended"

  select new {l, v}

)

.Where(

  BuildContainsExpression<LineItemInterface, string>(x => x.l.BillingGroupID, strArray))

)

.Select(x => x.l);

The database should do a pretty good job of optimizing this...

Alex

# November 13, 2009 7:48 AM

Chris Lomax said:

I actually found that querying a sproc was faster and easier than limiting yourself with the where in clause in LINQ. I run into the 2100 RPC limit when showing categories on my website and ended up using SPROCS. It went from a 3 second query down to 0.1 seconds. Sometimes its better to pass off to another technology than try to do a work around in LINQ.

I have just come from L2S to entity and run into this (no contains predicate) limitation. Entity in 3.5 is driving me mad at the minute, there are loads of cool things you can do in L2S that you cant do in entity but they are not going to be developing the L2S framework any further in .NET 4.

Microsoft, sort you head out and stop developing multiple technologies for the same purpose. Im sick of them doing "quick fixes" until they have plans to release a bigger project or idea (vista)

# November 20, 2009 2:27 AM

AlexJ said:

Chris,

I understand your frustration with the difference between EF and L2S. Thankfully most of those differences will be gone soon when .NET 4 comes out.

As for running into the RPC limit, wouldn't you run into the same problem in L2S too?

Alex

# November 20, 2009 3:06 AM
Leave a Comment

(required) 

(required) 

(optional)

(required) 

  
Enter Code Here: Required

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Page view tracker