We discuss the role of physical fluctuations in physical random bit generators focusing on the practical limits of generation rates. First we present a brief review of existing random bit generators, their mechanisms and performance characteristics. Then we discuss in detail random bit generation by chaotic lasers. It has recently been shown that bit sequences which passed statistical tests of randomness could be generated at fast rates up to 1.7 Gbps using chaotic semiconductor lasers. This is faster than any previous report of physical random sequence generation passing the same tests. In this paper it was reported that the chaotic state of the lasers gave better performance than the steady state noise. We show that one reason for the relatively good performance of the chaotic state versus the steady state is due to the drift of the threshold in the analog-to-digital converting (AD) device. Accordingly, we conclude that it is possible to generate random bit sequences by the steady state in the following alternative ways: (a) introducing another stage of broadband amplifier to amplify the steady state noise, (b) improving the stability of the threshold of the AD device by an order of magnitude. These results show that the chaotic laser is an efficient and robust mechanism for use in physical random bit generation in the sense that it did not require either (a) or (b). Moreover, the relative efficiency can be measured as the minimum amplification required to meet the randomness criteria, and the robustness can be measured as the maximum threshold variation which can be tolerated.