To maintain a Just-In-Time (JIT) production system, two kinds of kanbans, that is, a production-ordering and a withdrawal kanbans are used as tools to control the production and withdrawal quantities in each process. The numbers of kanbans used in each process decide the performance of the JIT production system. In a previous paper, the authors considered the JIT production system with a stochastic demand, and proposed an algorithm for finding optimal numbers of two kinds of kanbans that minimized an expected average cost per period. This paper deals with the optimal control problem of the same production system which does not use kanbans. A problem of determining an optimal ordering and production policy that minimizes the expected average cost per period is formulated into a Markov decision process (MDP). Numerical comparisons between optimal policies computed by the MDP and kanban controls by optimal numbers of kanbans show that the JIT production system is sub-optimal.