Yeah, I can understand the difficulty of keeping track of everything. There's so many characters and still so many to be introduced and it can get a bit confusing, even for someone who's read the books, like me. And to think some of the characters introduced will only get important in season 2 (or possibly 3 and beyond). But it's important they're introduced now, so you won't be saying "who the hell is this?!" when that character starts to be really involved in the Game of Thrones. It's part of the difficulty of creating a TV show based on a series of books that's not finished yet.
Oh, and I think there's some Aemon backstory coming up next episode, or the one after that, I'm not sure. But I agree, I sometimes wonder if people will get the same satisfaction of seeing the story unfold as me if some characters, like Ser Barristan Selmy, are underlit.