Rationalizing magic

I've been thinking a lot lately about how I write stories, and how my methods might be causing me some difficulty.

As I've mentioned before, ideas are the easiest for me. However, they come in bits and fragments, rather than whole story ideas.  For example, I imagined a girl tripping and falling through shadow into another dimension. Marbles that hold bits of memories and music. A society where the young people never see summer, because of a plague that happens only then. And an area so contaminated that no magic can exist there.

The problem I have is making those bits of ideas work in the context of a whole story... and a logical worldview.  I can get so frustrated with the rationalizing process that my interest in the story wanes.

Let's take the example of the place so contaminated that no magic works there. I liked the idea of there being sites around the world where no magic can exist. I've been reading about several case studies - Fukushima is always in the news. And about Japan's Minamata Disease, caused by years of chemical factories spilling tons and tons of organic methylmercury into the ocean, leaving thousands of people with nerve damage and developmental disorders (note: do not google Minamata Disease, especially pictures, if you have a weak stomach or want to be able to sleep  tonight.  I'm referring to descriptions of effects on children and a few horrifying photos).  Fukushima, of course, is always in the news.  And, of course, Chernobyl (last year, I read Voices from Chernobyl, made up entirely interviews with survivors. It's one of the most amazing and heartbreaking works I've ever read).

An abandoned grand piano in Pripyat. Photo attribution Timm Suess, from Wikimedia Commons.

So it all seems great - poison, toxins and radiation disrupt magic. But how? The scientist in me wants to know, the reader in me hates when things don't make sense.  Never mind having no idea how toxic metals might interact with magic, I don't know enough about physics to explain how radioactive iodine and other particles might interfere with whatever energy defines magic. In the current short story I'm working on, I don't go into it too much, except to state that magic comes from the energy of all living things, so anything that disrupts or corrupts that flow of energy corrupts magic.

So how about you? As a reader, how much logic do you expect from magic systems? Do odd magic systems and ideas bother you in a novel or short story if they're not properly explained?

And how do you, as writers, handle ideas that seem cool but don't make immediate logical sense? How do you explain them without getting frustrated?