How Should Agents Read Demonstrations? Hierarchical Structure Beats Flat Action Logs

This study evaluates the effectiveness of hierarchical organization of action logs in Programming by Demonstration (PbD) for LLM agents. By grouping actions into labeled subgoals, the researchers found that this structure improved task completion rates in ambiguous natural-language tasks from 76.7% to 90.7%, while flat action logs showed minimal improvement. The findings suggest that for practitioners, implementing a hierarchical approach in PbD pipelines can significantly enhance the quality of procedural knowledge conveyed to LLMs, particularly in scenarios where task descriptions are vague.