Sri Henri Ponicaré has some thoughts on the matter:
"... care of the beautiful leads us to the same selection as care for the useful"
"economy of effort ... is a source of beauty as well as a practical advantage"
"... what we must aim at is not so much to ascertain resemblances and differences, as to discover similarities hidden under apparent discrepancies"
"... when a rule has been established, we have first to look for the cases in which the rule stands the best chance of being found in fault"
"... facts which occur frequently appear to us simple just because we are accustomed to them"
Oh, and my favorite:
"Trying to make science contain nature is like trying to make the part contain the whole"