I don't remember playing video games before I was five years old (my earliest memories are watching a cocktail version of Asteroids Deluxe at a restaurant, with the game display in our table... or else playing "KRULL" and having my dad explain the controls to me, then dying three times in rapid succession). But then again kids don't have many memories before that age anyhow.
Still, I think the philosophy that very young children shouldn't watch TV (for fear of it harming their attention spans), I'd say the same thing about playing video games. Even "educational" games might be a problem there, and the same sorts of things they would teach can be done better by one on one with a living breathing human person (kids that age need to learn how to positively interact with others).
If I had kids, that's how I'd teach them anyway.