Computational Thinking skills are basic and important to manipulate computers. Currently, several systems exist to provide an effective way to learn programming that use computers, smartphones, tablets, or programmable robots. Although studies have reported improved programming skills and motivation to learn programming using an on-screen application or a programmable robot, the benefits of these tools have not been directly compared. To resolve this issue, especially with regard to motivation to learn programming and impression of programming, we conducted a large scale comparative experiment involving 236 middle and high school students to evaluate the effects of a game based educational application and programmable robots on learning programming. We then compared the effects of a game based educational application with and without programmable robots on learning programming. We found that employing programmable robots on learning programming did not always give an improvement to all students.