Neal Stephenson has suggested that while any definition will be simplistic there is today a general cultural difference between literary and genre fiction. On the one hand literary authors nowadays are frequently supported by patronage, with employment at a university or a similar institution, and with the continuation of such positions determined not by book sales but by critical acclaim by other established literary authors and critics. On the other hand, he suggests, genre fiction writers tend to support themselves by book sales.  However, in an interview, John Updike lamented that "the category of 'literary fiction' has sprung up recently to torment people like me who just set out to write books, and if anybody wanted to read them, terrific, the more the merrier... I'm a genre writer of a sort. I write literary fiction, which is like spy fiction or chick lit".  Likewise, on The Charlie Rose Show , he argued that this term, when applied to his work, greatly limited him and his expectations of what might come of his writing, so he does not really like it. He suggested that all his works are literary, simply because "they are written in words". 
"Before I decided to work with Mark, I submitted my book to literary agents but didn’t get any interest. After I revised my query letter and book proposal, I got several agencies interested and decided to sign with Fine Print Lit, a top literary agency in New York. They got publishers bidding against each other and I ended up signing a contract with Thomas Nelson (an imprint of Harper Collins) for what I’ve been told by several people is a very large advance. What cloud is higher than 9?" [Click here to see all Mark Malatesta reviews]
Ah, I see. Well, I’d subscribe to and run a search in their deals database for the textbook publishers you’re targeting. That should help you turn up a few agents, but this is still not an area where you’ll find much agent activity on record. Another option is to simply ask the textbook authors you know about or can reach if they’d let you know the name of their agent. (Don’t ask for a referral—just ask who represents them.) Sometimes authors will thank their agents in the acknowledgments, so that’s one way to figure it out without asking.