It is important for human symbiotic robots working near humans to have the adaptability to reliably follow force from humans while maintaining task performance despite unexpected disturbances. Thus, in the current study we propose a coordination control method of concurrently accomplishing task execution and human following even when Physical InterFerence and contACT (PIFACT) occur with humans. First, functional requirements for the control method are specified from the viewpoints of motion-phase transition capability, time management characteristic of respective motion phase, and forms of task-performable human-following motion. Next, a control system architecture satisfying the requirements is presented. In addition, we describe a method of quantitatively representing a rule of task to process PIFACT adapting motions that allow achieving both human following and task performance according to the attributes of imposed tasks. Finally, experiments were carried out in which PIFACT was induced between humans and a full-size anthropomorphic robot equipped with the control architecture. We evaluated the results in terms of the comparison of variations of hand orientation and position while following humans during PIFACT among conditions where tasks with diverse rules were imposed on the robot. Evaluation of experiments demonstrates the proposed control architecture is useful for coordinating task execution and human following necessary for elevating human symbiotic robots.