Who is Scrooge talking about when he says "Poor boy!"

Scrooge is referring to the image of himself as a young boy, alone in the boarding school for Christmas.  The Ghost of Christmas Past has taken him back into Scrooge's childhood so that he might see where his loathing of Christmas originated.  Scrooge's father was apparently a mean man and young Ebeneezer resented being sent away.  When his sister, Fan, arrives to bring him home for Christmas, she tells him that their father "is much kinder now".  This suggests that there might have been some abusive tendencies on the part of Scrooge's father.