Strictly speaking, "literary" means only "of or having to do with literature." The question then becomes, What constitutes "literature?"
Any body of written work can be called "literature" -- you can reasonably refer to the "literature" on just about any subject. However, we tend to use the word to refer to elevated, serious, written material, most often fiction, produced for an educated, informed audience by professional authors dedicated to the craft of writing well. (That's a very narrow definition, but I think it addresses the question you're asking.) This means that the word "literary" is most likely to be applied to a written work that meets that definition.
So, for example, a book by Joyce Carol Oates will be called "literary," but a book by Danielle Steele will not be. They are both professional writers and both are naturally concerned with selling books, but Steele's books are directed to a popular mass market and Oates' are not; the quality and type of writing in their books is quite different.
I hope that makes sense.
https://www.quora.com/What-does-literary-fiction-mean-today
