Ok, I have some serious problems with this scheme. From what I've read and re-read, I'm going to break this down for you. Anything I am not 100% sure on, I'll mark in this color to make it stand out. If I 've made a mistake, feel free to correct me.
I'll agree with you on the survival of the fittest (when a GREMLIN model lis killed...) and while it applies to Sommer, the same rule does not apply to the Whisperer (since this isn't an ability of the Whisperer).
When the totem is summoned, it loses 3 health and it must be within 6" of Sommer Plus it's slowed. So it's already down 1 action point. Since Larva is an (all) abiltiy, you are not able to use Swarm the same turn you are summoned. My reasoning is this. On page 44, right collum fourth bullet, it states basically that you can preform an (all) action, however you're not allowed to preform a (0) action. IF this is the case, then you cannot cast since you've already spent an action activating.
Now, here's another point to remember. Even if you were to do this, at the start of the draw phase, you're required to discard down to your maximum hand size and SHUFFLE. Basically, that guarnteed Joker you were looking for at the top of your deck? No longer there. And since Somer is required to have one less control card that means you are already losing that edge you had before.
Like I said, if you see a flaw in what I said, feel free to let me know.