Learning Human-aware Robot Policies for Adaptive Assistance

Abstract

Developing robots that can assist humans efficiently, safely, and adaptively is crucial for real-world applications such as healthcare. While previous work often assumes a centralized system for co-optimizing human-robot interactions, we argue that real-world scenarios are much more complicated, as humans have individual preferences regarding how tasks are performed. Robots typically lack direct access to these implicit preferences. However, to provide effective assistance, robots must still be able to recognize and adapt to the individual needs and preferences of different users. To address these challenges, we propose a novel framework in which robots infer human intentions and reason about human utilities through interaction. Our approach features two critical modules: the anticipation module is a motion predictor which captures spatial-temporal relationship between the robot agent and user agent, which contributes in predicting human behavior; the utility module infers the underlying human utility functions through progressive task demonstration sampling. Extensive experiments across various robot types and assistive tasks demonstrate that the proposed framework not only enhances task success and efficiency but also significantly improves user satisfaction, paving the way for more personalized and adaptive assistive robotic systems.

Framework

Video Presentation

Visualizations

Adapting to different preference settings

Algorithm
Task Success
Human Reward
Task Reward
Preference Reward

PPO
✅
120.179
130.528
-10.349

TD3
✅
101.410
113.699
-12.290

Ours
✅
125.812
131.516
-5.704

In setting 1, where the human prefers gentler delivering forces, our method produces slower and gentler delivery, whereas the baseline methods fails to do so, as the spoon is moving too fast when approaching the mouth.

Adapting to different preference settings

Algorithm
Task Success
Human Reward
Task Reward
Preference Reward

PPO
❌
27.352
63.985
-36.63

TD3
❌
25.812
76.012
-50.20

Ours
✅
111.094
111.799
-0.705

In setting 4, where the human prefers the robot to avoid spilling food, our learnt policy effectively recognizes and adapts to this preference, while the baseline methods fail to do so. In baseline policy rollouts, food are spilled midway, making both the preference reward and the task reward sub-optimal.

Adapting to different tasks

Algorithm
Task Success
Human Reward
Task Reward
Preference Reward

PPO
❌
61.458
77.989
-16.531

TD3
❌
-29.015
-16.674
-12.341

Ours
✅
126.759
132.344
-5.585

In the bed bathing environment, a person lies on a bed in a random resting position, while a robot uses a washcloth to wipe off uniformly distributed water markers on the person's right arm. Our method efficiently and effectively completes this task. In contrast, with the baseline methods, the effective contact between the robot end-effector and the arm is significantly reduced. In the TD3 baseline, the end-effector even moves beneath the arm, making task completion impossible.

Adapting to different tasks

Algorithm
Task Success
Human Reward
Task Reward
Preference Reward

PPO
✅
332.545
463.184
-130.639

TD3
❌
-140.366
-135.457
-4.910*

Ours
✅
415.830
459.211
-43.382

In the drinking environment, where a robot holds a cup of small particles representing water and helps a person drink this water, our method also completes the task efficiently and effectively. In contrast, the PPO baseline methods produces less effective coordination policies. In the TD3 baseline, the robot end-effector even gets stuck mid-way. A * marks incomplete preference reward due to task failure.

Adapting to different robot types

Algorithm
Task Success
Human Reward
Task Reward
Preference Reward

PPO
✅
35.697
111.438
-75.741

TD3
✅
56.295
102.891
-46.596

Ours
✅
106.825
113.533
-6.708

With the Jaco robot, our method yields a clean delivery, while the baseline methods produce suboptimal policies, in which the spoon hit other facial parts.

Adapting to different robot types

Algorithm
Task Success
Human Reward
Task Reward
Preference Reward

PPO
❌
-8.321
35.253
-43.575

TD3
❌
-82.014
-77.510
-4.505*

Ours
✅
98.491
103.681
-5.191

With the PR2 robot, our method also yields a clean delivery and avoided spilling food. In contrast, in the PPO baseline, the food spilled out almost entirely. In the TD3 baseline, the spoon stuck mid-way, making task success impossible.

Adapting to different robot types

Algorithm
Task Success
Human Reward
Task Reward
Preference Reward

PPO
✅
-8.571
103.787
-112.358

TD3
✅
105.182
114.434
-9.253

Ours
✅
124.308
132.242
-7.935

With the Baxter robot, our method yields superior performance in terms of delivery cleanness and gentleness. In contrast, in the PPO baseline, the spoon hit other facial parts; and in the TD3 baseline, the delivery was less gentler, making the preference reward suboptimal.

BibTeX

@misc{qin2024learninghumanawarerobotpolicies,
        title={Learning Human-Aware Robot Policies for Adaptive Assistance}, 
        author={Jason Qin and Shikun Ban and Wentao Zhu and Yizhou Wang and Dimitris Samaras},
        year={2024},
        eprint={2412.11913},
        archivePrefix={arXiv},
        primaryClass={cs.RO},
        url={https://arxiv.org/abs/2412.11913}}