What data can be intercepted from my IP address, if someone has the address?

I, recently, found out that packets(I don't know what a packet is. I'm pc illiterate), and other data, can be intercepted, via means of my IP address. If this is so, isn't it easy to gain someones IP address, making security difficult to maintain? How do I guard against invasion of privacy?