Feeling insecure in public, is it a serious problem? What can I do to help?

I'm a student, I live in an apartment close to the school I attend. For 4 years, I've been walking to and from school from the back door of the apartment building and purposely go to school an hour early and go home at least an hour and half after school is out to avoid being seen by people (especially those who go to my school). I know that this has a lot to do with how I feel ashamed that I have a low self esteem and I don't want people to know that my family isn't rich, but is this too extreme? I'm petrified of being seen walking anywhere to and from from my apt and I would try to do everything I can to avoid it. Besides that, I also feel extremely uncomfortable when I'm walking on the streets and it feels like everyone is observing me from their car and it feels like I'm in constant danger. I'm not sure if a lot of people feel the same way. It feels like I may have a problem, but I don't know what it is. Can anyone help me?