Your reading is correct.
To give a IBU of the beer, you need the correct amount of hops, AA%, boil gravity, volume of the boil and time.
Any decent homebrew book will give you the formula. Also, all the brewing software makes these calcs a snack.
Say your scales are 4 gms off. If you weigh out 17 gms of 12%AA POR instead of 13 gms, your beer will be way off. If you weigh out 43 gms of 4%AA Goldings instead of 39 gms your bitterness calcs will be out but not as much.
Bitterness can be affected by many things. Hops AA% degrades during storage. Different hops degrade quicker than others. Yeast selection. Brewing techniques. The beer balance of bitterness to maltiness.
Most people cannot detect a change of 3-5 IBU in a beer.
As a starting point, brew middle of the range IBU beers, around 25, so if your setup is too far out, you still will brew a decent drop rather than a sweet brew or a bitter brew.
Then, as your brewing skills improve and your beers show up the IBU's you are achieving, then start experimenting with the IBU.