Purpose - The objective of this work was to demonstrate a novel approach to human machine interaction that seamlessly uses teleoperation and automation in a complex environment. Design/methodology/approach - This work leverages developments in the area of operational software Operational Software Components for Advanced Robotics (OSCAR), decision making, human-machine interface, and motion planning. This demonstration uses a 17 degrees-of-freedom (DOF) dual arm robot that is equipped with modern tool changers, crash protectors, force-torque sensors and electrical and pneumatic power at the tools. Four different end-effector tools are also provided. These are electric grippers, electric rotary saw, electric drill, and a pneumatic spray gun. The system can be used both in teleoperation and automation mode. In teleoperation mode, the user has a choice of five different input devices. These are computer keyboard, spaceball and spacemouse, RSI manual controller and kraft force feedback controller. Automation is performed using a novel graphical user interface with 3D graphics used for previewing and verifying manipulator motion. Automation tasks that are demonstrated include automatic grasping, sawing, drilling, spray painting, point-to-point motion, and teaching. The controller for the dual arm system is developed using OSCAR and supports a variety of decision-making algorithms and obstacle avoidance. The integration of this controller with the input devices and human machine interface is done using a novel protocol that is based on Extensible Markup Language (XML) for maximum reuse and distributed integration. This protocol is further based on a well-defined and scalable XML schema that can be easily extended as controller functionality is changed and/or additional input devices are added. Findings - It is necessary to combine automation with teleoperation to reduce worker fatigue and also provide higher value robotic functions. This is possible as most remote tasks can be broken down into structured and unstructured components. On the integration front, we see XML-based integration providing a loosely coupled system that can make interoperability between various robot systems possible. For end-effector tooling, it is better to have special purpose tools that can be switched out versus the use of a general purpose tool such as a robotic hand. Research limitations/implications - This research was done in a laboratory environment, and as such, its application in the field will require partnering with a commercial entity. Force-feedback on manual controllers during teleoperation was not very effective. in fact, providing visual queues to the operator about the forces were a better guide to the operator. Practical implications - The software for this work provides obstacle avoidance capability. The obstacle avoidance is based on a known world model that is derived from a CAD environment. In reality, this model will have to be sensed in real-time, and decoded into a geometric model. Significant work in this area needs to be done. Originality/value - The software developed for this work was based on the OSCAR software framework. This is a unique framework that at its core uses performance criteria to control the behavior of the robot during teleoperation and automation. The value of this work is that it shows as completely feasible the control of a 17 DOF dual arm system using the latest integration technologies (such as XML), integrated simulation, multiple tools and multiple input devices. It also shows that all these choices can be provided to an operator through a single user interface.
