Floating point values are stored as pairs of integers (x,y) that represent values x*2^y. This has a few obvious limitations.
You know how you can't fully write down the value 1/3? If you're limited to, say, three digits then you either write down 3.33*10^-1 or 3.34*10^-1, neither of which are the exact value. Floating point representations have the exact same problem, but with other values. For example, 0.1 is a repeating decimal in binary (0.00011).
Another problem is that they can only store a limited number of significant digits. double can represent 10^100 and 10^-100, but it can't represent 10^100+10^-100 since that would require over 600 bits (200/log10(2)). So if you try to perform an operation between two numbers that are too apart in magnitude, the computer will have to do some approximation.