Midichlorians were invented over a decade after Return of the Jedi unmasked Vader, so it's a huge retcon, that can't be emphasized enough.
Tell me though, if one's connection to the Force is based on the physical, how is it that people like Obi-Wan and Yoda can manifest themselves as spirits on different planets and talk to people, without a body?
One might as well argue that a smaller body or one with limbs removed would have a greater concentration of midichlorians in the body that remained.
Otherwise, if it's simple BULK, then every Jedi or Sith should be packing away doughnuts... and Yoda should be a major weakling in the Force.
Midichlorians simply raise more questions than they answer about the Force and Jedi "power levels." All we know from the movies is that they are apparently sentient, and "communicate" the "will of the Force" to a person. Everybody has them, but apparently only trained people are able to "quiet themselves" to hear them "speaking." The only person we get a "count" for (with no comparison other than that it's higher than "even Master Yoda" and that "no Jedi" has a count that high) is Anakin, and a vague one at that ("off the chart," "over 20,000"), when he was a 9 year old.
Some folks SPECULATE (without confirmation whether they were right or wrong) that Anakin was "conceived by the midichlorians" (whose, Shmi Skywalkers?). Then there's speculation, which is highly suspect because it's wrapped up in the lies that Palpatine is feeding Anakin in the "water opera" scene in Episode III (I'm leaving aside what the novelization implies, since this was based on a less-than-final script draft and thus differs from the movie), that some Sith Lord "Darth Plagueis the Wise" could influence the midichlorians to create life.
So I guess if you took all that into account we're supposed to imagine that this Sith Lord waved his hand in front of some random slave and said "You will become pregnant!" and she did. Anyway, just wanted to touch on that rather vague, goofy plot point. It's probably better that Lucas didn't have the characters sit down and try to explain it further.