WEB-1K: WorldEngine Bimanual Manipulation Dataset and Benchmark for Real-World Post-training

WorldEngine AI

Core Contributors Shiduo Zhang1  Yuchun Feng1  Haonan Chang1  Alex Qiu1  Kevin Zhou1  Patrick Liao1  Jingru Zhao1  Di Huang1  Yue Wang1,2

1WorldEngine AI  2University of Southern California

WEB-1K captures large-scale, high-fidelity bimanual manipulation demonstrations across 90 real-world tasks.

Dataset Statistics

Tasks
90
Diverse real-world bimanual manipulation scenarios
Episodes
50,266
Expert human demonstrations collected in the wild
Frames
199M+
High-fidelity 60 fps frames across all recordings
Videos
150K+
Synchronized multi-camera streams per episode
Total Duration
923 hrs
Of real-world bimanual robot interaction
Top Actions
Top Objects

Setup

Robot setup

1. Material List

1.1 Frame

Image Item Size Qty
Hexagon bolt 5x10Hexagon bolt5×1077
Hexagon bolt 5x8Hexagon bolt5×814
T shim 5mmT shim5mm83
Screw 8mmScrew8mm18
Hexagon bolt 3x6Hexagon bolt3×62
Hexagon bolt 3x20Hexagon bolt3×202
Single black beam 120x2cmSingle black beam120×2cm8
Double black beam 100x4cmDouble black beam100×4cm2
Short black beam 40x2cmShort black beam40×2cm1
Fixed support beam T-ShapeFixed support beamT-Shape2
Fixed support beam L-ShapeFixed support beamL-Shape14
Fixed support beam RectangleFixed support beamRectangle6
Diagonal white beam 30x2cmDiagonal white beam30×2cm4
Foam corner protectorFoam corner protector2
Plastic corner protectorPlastic corner protector13

1.2 Robotic Arms & Gello

Image Item Size Qty
Fixed support cameraFixed support — camera2
Fixed support robot armFixed support — robot arm4
Robot armRobot arm2
GelloGello2
Hexagon boltHexagon bolt8
T shimT shim8
Hexagon bolt 3x8Hexagon bolt3×86
Hexagon bolt 3x6Hexagon bolt3×64

1.3 Computers

Image Item Qty
CaseCase1
MonitorMonitor1
KeyboardKeyboard1
MouseMouse1
USB Foot SwitchUSB Foot Switch1
Floor-standing monitorFloor-standing monitor1

2. Installation Steps

2.1 Workflat & Frame

  1. Assemble three 120×2cm bars into a U-shape around the table frame.Frame U-shape step
  2. Attach three pieces of 120×2cm wood to the bottom of the two 120×2cm pieces, with a distance of 26cm between the table and the middle piece, and 15cm from the middle piece to the outermost one.Bottom beams step
  3. Use a 100×4cm piece positioned vertically at a distance of 51cm from the end of the horizontal beam. Secure it using an L-bracket and a diagonal brace.Vertical beam L-bracket step
  4. Install the top horizontal beam, secure it using two diagonal braces, and add another horizontal beam below the diagonal braces.Top horizontal beam step
  5. Install the vertical beam between the two horizontal beams, securing it with two T-shaped horizontal beam spacers.Vertical beam T-spacers step
  6. Install the camera at the bottom position of the 40×2cm beam.Camera mount step

2.2 Robotic Arms & Gello

  1. Use clamps to secure the two robot arms, with a distance of 25cm from the edge of the table.Clamp robot arms step
  2. Attach the camera to the top of the robot arm using a camera mount.Camera mount step
  3. Secure the gello to the two outer metal bars, with a distance of 5cm from the edge of the table.Gello mount step

2.3 Computers & Accessories

  1. Connect the monitor, keyboard, mouse, and foot switch to the computer using the USB hub.
  2. Use USB 2.0 for the gello and robot arm. Use USB 10G for the top camera. One robot arm camera uses a USB 5GB port, while the other uses a Type-C port through an adapter.USB wiring step

SKU List

# Name Link Qty
1USB-Cable-001amazon.com ↗1
2USB-Cable-002amazon.com ↗1
3Tri-Cable-001amazon.com ↗1
4Double-Cable-001amazon.com ↗1
5Socket-001amazon.com ↗1
6Socket-002amazon.com ↗1
7Checkerboardamazon.com ↗1
8Red Checkersamazon.com ↗12
9Black Checkersamazon.com ↗12
10Box-001amazon.com ↗1
11Tape-002amazon.com ↗1
12Shape-Sorter-001amazon.com ↗1
13Cloth-001amazon.com ↗1
14Basket-001amazon.com ↗2
15Fake Bananaamazon.com ↗1
16Fake Strawberriesamazon.com ↗1
17Fake Cherriesamazon.com ↗1
18Fake Asparagusamazon.com ↗1
19Fake Multi-color Bell Peppersamazon.com ↗1
20Ping-pong-ball-001amazon.com ↗1
21Ping-pong-paddle-001amazon.com ↗1
22Bread-001amazon.com ↗2
23Egg-001amazon.com ↗1
24Lettuce-001amazon.com ↗1
25Plate-001amazon.com ↗4
26Pipe-001amazon.com ↗1
27Sock-001amazon.com ↗1
28Towel-001amazon.com ↗1
29Bag-001amazon.com ↗1
30Lipstickamazon.com ↗
31Face Wipesamazon.com ↗
32Hand Sanitizeramazon.com ↗
33Hanoi Tower Gameamazon.com ↗1
34Dish RackN/A1
35Multi-color plastic bowlsN/A4
36White platesN/A5+
37Puck Lightsamazon.com ↗5
38Tennis Ballamazon.com ↗6
39Tennis Ball Tubeamazon.com ↗2
40White 3D Printed BoxN/A1
41Green 3D Printed BoxN/A1
42Shuttlecockamazon.com ↗6
43Shuttlecock Tubeamazon.com ↗1
44Logitech G305 Computer Mouseamazon.com ↗1
45Vacuumamazon.com ↗1
46Toy Cake (with candles and numbers)amazon.com ↗
47AAA Batteriesamazon.com ↗16
48Screws, nuts, washersN/AInfinite
49Variety toolsN/AInfinite
50Retractable Knifeamazon.com ↗
51Kitchen Utensils Organizeramazon.com ↗
52Multi-colored Gel Pensamazon.com ↗
53Black Gel Pensamazon.com ↗
54Mechanical Boxamazon.com ↗
55White Erasersamazon.com ↗
56Pink Erasersamazon.com ↗
57Clear Packing Tapeamazon.com ↗
58NESCAFÉ Taster's Choice House Blendwalmart.com ↗
59Magnetic Structure Piecesamazon.com ↗
60Toy Robot with plastic screws and screwdriveramazon.com ↗
61Water Bottleamazon.com ↗
62Large foam multi-colored rolling diceamazon.com ↗
63Jewlery Hangeramazon.com ↗
64Toy Shoes (ARRIVING SOON)N/A
65Toy Vehicles (ARRIVING SOON)N/A
66Rings, Bracelets, Necklacesamazon.com ↗
67Baby Clothes (ARRIVING SOON)N/A
68Large clothes hangersN/A
69Small clothes hangersamazon.com ↗
70Blackboardamazon.com ↗
71Dry-erase MarkersN/A
72Breakroom Supplies Organizeramazon.com ↗
73Jenga Game (ARRIVING SOON)N/A

Task Cards

accessory-return
255 episodes · 2,024,952 frames · 765 videos · 60 fps
Task name
accessory-return
Episodes
255
Frames
2,024,952
Videos
765
FPS
60
Task count
1
Task Description

Hang the accessory on the jewelry stand.

Subtasks
  1. Grab a ring and place it on the base platform of the hanger.
  2. Grab a bracelet and hang it on the hanger.
  3. Grab a necklace and hang it on the hanger.
Sample videos
Storyboard
storyboard montage
arrange-shoes-in-pairs
757 episodes · 2,459,849 frames · 2271 videos · 60 fps
Task name
arrange-shoes-in-pairs
Episodes
757
Frames
2,459,849
Videos
2271
FPS
60
Task count
1
Task Description

Arrange all shoes into matching pairs.

Subtasks
  1. Grab the yellow left shoe with the closest arm and put it into the final position.
  2. Grab the yellow right shoe with the closest arm and put it into the final position.
  3. Grab the red left shoe with the closest arm and put it into the final position.
  4. Grab the red right shoe with the closest arm and put it into the final position.
  5. Grab the yellow left with the closest arm and put it horizontally in the center of the table.
  6. Grab the yellow right with the closest arm and put it horizontally in the center of the table.
  7. Grab the red left shoe with the closest arm and put it horizontally in the center of the table.
  8. Grab the red right shoe with the closest arm and put it horizontally in the center of the table.
  9. Grab the shoe at the center of the table with the appropriate arm and place it at the center of the table.
Sample videos
Storyboard
storyboard montage
baby-clothes-sorting
252 episodes · 2,082,400 frames · 756 videos · 60 fps
Task name
baby-clothes-sorting
Episodes
252
Frames
2,082,400
Videos
756
FPS
60
Task count
1
Task Description

Sort baby clothes into organized piles.

Subtasks
  1. Grab an article of clothing and raise it up into the air.
  2. Identify the clothing type, if it is pants, place it at the right center of the table, if it is a shirt, place it at the left center of the table.
  3. Pass the article of clothing to the other gripper.
Sample videos
Storyboard
storyboard montage
badminton-clean-up
354 episodes · 2,389,954 frames · 1062 videos · 60 fps
Task name
badminton-clean-up
Episodes
354
Frames
2,389,954
Videos
1062
FPS
60
Task count
1
Task Description

Put the badminton birdies into the tube.

Subtasks
  1. Pick up the empty birdie tube and place it upright on the table at the center.
  2. Pick up the pointed nose cap and place it on the birdie tube.
  3. Push down on the pointed nose cap to fully seal the cap on the birdie tube.
  4. Grab the birdie tube and flip it upside down to the empty side faces upwards.
  5. Grab a birdie and place it into the birdie tube, rubber nose pointing downward.
  6. Grab the flat nose cap and place it on the opening of the birdie tube.
  7. Press down on the flat nose cap to fully seal the cap on the birdie tube.
Sample videos
Storyboard
storyboard montage
bag-packing-with-zipper-closure
463 episodes · 2,069,954 frames · 1389 videos · 60 fps
Task name
bag-packing-with-zipper-closure
Episodes
463
Frames
2,069,954
Videos
1389
FPS
60
Task count
1
Task Description

Pack items into a bag and close the zipper.

Subtasks
  1. Pick up the travel bag with the left arm.
  2. Grab the zipper with the right arm and open the bag all the way.
  3. Grab the top opening of the bag to hold it open.
  4. Grab an item and place it into the opened bag.
  5. Grab the filled bag with the right arm.
  6. Grab the zipper with the left arm and close the bag all the way.
  7. Place the bag back onto the table.
Sample videos
Storyboard
storyboard montage
bagging-groceries
194 episodes · 2,069,643 frames · 582 videos · 60 fps
Task name
bagging-groceries
Episodes
194
Frames
2,069,643
Videos
582
FPS
60
Task count
1
Task Description

Bag groceries into shopping bags.

Subtasks
  1. Grab a plastic bag with one arm.
  2. Grab a [pepper/strawberry/asparagus] and place it into the corresponding plastic bag while the other arm holds it.
  3. Place the plastic bag onto the table.
Sample videos
Storyboard
storyboard montage
balance-scale
679 episodes · 2,373,761 frames · 2037 videos · 60 fps
Task name
balance-scale
Episodes
679
Frames
2,373,761
Videos
2037
FPS
60
Task count
1
Task Description

Balance a scale by putting weights.

Subtasks
  1. Grab a single weight and place it on the right side.
  2. Grab a single weight and place it on the left side.
Sample videos
Storyboard
storyboard montage
bowl-pouring
3361 episodes · 3,322,744 frames · 10083 videos · 60 fps
Task name
bowl-pouring
Episodes
3361
Frames
3,322,744
Videos
10083
FPS
60
Task count
1
Task Description

Pour items from one bowl into another.

Subtasks
  1. Grab a bowl filled with items.
  2. Pour the bowl of items into another bowl.
  3. Place the bowl back onto the table.
Sample videos
Storyboard
storyboard montage
bracelet-return
277 episodes · 2,352,006 frames · 831 videos · 60 fps
Task name
bracelet-return
Episodes
277
Frames
2,352,006
Videos
831
FPS
60
Task count
1
Task Description

Return the bracelets to the hanger.

Subtasks
  1. Grab a bracelet and hang it on the hanger.
Sample videos
Storyboard
storyboard montage
breakroom-essentials-organization
332 episodes · 2,080,992 frames · 996 videos · 60 fps
Task name
breakroom-essentials-organization
Episodes
332
Frames
2,080,992
Videos
996
FPS
60
Task count
1
Task Description

Organize coffee pods, cups, and straws into the organizer.

Subtasks
  1. Grab a coffee pod and place it into the organizer.
  2. Grab a cup and place it into the organizer, opening facing downwards.
  3. Grab a straw and place it into the organizer.
Sample videos
Storyboard
storyboard montage
brick-stacking
1115 episodes · 2,289,582 frames · 3345 videos · 60 fps
Task name
brick-stacking
Episodes
1115
Frames
2,289,582
Videos
3345
FPS
60
Task count
1
Task Description

Stack lego bricks into pairs.

Subtasks
  1. Put the single lego brick in the center of the table
  2. Stack one lego brick on another single lego.
  3. Stack one lego brick on the stacked lego.
  4. Move the stacked lego brick to the center of the table.
Sample videos
Storyboard
storyboard montage
bussing-table
744 episodes · 2,360,495 frames · 2232 videos · 60 fps
Task name
bussing-table
Episodes
744
Frames
2,360,495
Videos
2232
FPS
60
Task count
1
Task Description

Organize plates, bowls, spoons, and chopsticks from a dining table.

Subtasks
  1. Grab a plate and place it at the center of the table.
  2. Grab a large bowl and stack it on top of the plate.
  3. Grab a small bowl and place it in the large bowl.
  4. Grab a spoon and put it into the small bowl.
  5. Grab a chopstick and place it flat on the bowls horizontally.
Sample videos
Storyboard
storyboard montage
cable-assembling
294 episodes · 1,567,051 frames · 882 videos · 60 fps
Task name
cable-assembling
Episodes
294
Frames
1,567,051
Videos
882
FPS
60
Task count
1
Task Description

Plug cable connectors into a power strip.

Subtasks
  1. Move the power strip to the center of table.
  2. Insert the USB cable into its socket.
  3. Insert the Tri-cable into its socket.
  4. Insert the Double-cable into its socket.
Sample videos
Storyboard
storyboard montage
clean-up-board-game
125 episodes · 2,129,175 frames · 375 videos · 60 fps
Task name
clean-up-board-game
Episodes
125
Frames
2,129,175
Videos
375
FPS
60
Task count
1
Task Description

Remove chess pieces from chess board.

Subtasks
  1. Grab a chess piece on the chess board and place it on the table.
Sample videos
Storyboard
storyboard montage
clothes-rack-return
448 episodes · 2,163,331 frames · 1344 videos · 60 fps
Task name
clothes-rack-return
Episodes
448
Frames
2,163,331
Videos
1344
FPS
60
Task count
1
Task Description

Hang clothes on the rack.

Subtasks
  1. Grab an article of clothing with one arm.
  2. Grab a clothes hanger with one arm.
  3. Use the other arm to also grab the hanger.
  4. Adjust the hanger's position in gripper(s).
  5. Hang one side of the clothing onto the clothes hanger.
  6. Hang the other side of the clothing onto the clothes hanger.
  7. Adjust the clothing position on the clothes hanger.
  8. Place the article of clothing onto the clothes rack.
  9. Move the clothes rack to the center of the table.
Sample videos
Storyboard
storyboard montage
coffee-pods-organization
328 episodes · 2,103,625 frames · 984 videos · 60 fps
Task name
coffee-pods-organization
Episodes
328
Frames
2,103,625
Videos
984
FPS
60
Task count
1
Task Description

Organize coffee pods into the coffee pod holder.

Subtasks
  1. Grab a coffee pod and place it into the coffee pod holder.
  2. Grab a coffee pod and pass it off to the other arm to re-orient grip.
  3. Place the pod on the pod holder.
  4. Grab a coffee pod and re-orient its orientation on the table.
Sample videos
Storyboard
storyboard montage
coffee-pods-retrieval
276 episodes · 2,117,036 frames · 828 videos · 60 fps
Task name
coffee-pods-retrieval
Episodes
276
Frames
2,117,036
Videos
828
FPS
60
Task count
1
Task Description

Retrieve all coffee pods from the coffee pod holder.

Subtasks
  1. Grab a coffee pod and place it onto the table with one arm holding the coffee pod holder.
  2. Grab the coffee pod and re-orient its orientation on the table.
  3. Rotate the coffee pod holder.
Sample videos
Storyboard
storyboard montage
color-sorting-items
379 episodes · 2,173,690 frames · 1137 videos · 60 fps
Task name
color-sorting-items
Episodes
379
Frames
2,173,690
Videos
1137
FPS
60
Task count
1
Task Description

Sort items into plates with the same color.

Subtasks
  1. Grab the red [lego/pepper/dice/toy/ball/ring] and place it on the red plate.
  2. Grab the yellow [lego/pepper/dice/toy/ball/ring] and place it on the yellow plate.
Sample videos
Storyboard
storyboard montage
connect-pipe-segments
605 episodes · 2,041,545 frames · 1815 videos · 60 fps
Task name
connect-pipe-segments
Episodes
605
Frames
2,041,545
Videos
1815
FPS
60
Task count
1
Task Description

Connect pipe segments together into a complete unit.

Subtasks
  1. Grab a blue plus shaped pipe segment from the pile and place it on the table.
  2. Grab a blue connector pipe segment from the pile and place it on the table.
  3. Grab a red plus shaped pipe segment from the pile and place it on the table.
  4. Grab a red connector pipe segment from the pile and place it on the table.
  5. Connect the blue connector with the blue plus-shaped pipe segment to form a cross shape.
  6. Connect the red connector with the red plus-shaped pipe segment to form a cross shape.
Sample videos
Storyboard
storyboard montage
cooking-utensils-organization
508 episodes · 2,047,764 frames · 1524 videos · 60 fps
Task name
cooking-utensils-organization
Episodes
508
Frames
2,047,764
Videos
1524
FPS
60
Task count
1
Task Description

Organize cooking utensils neatly into the container.

Subtasks
  1. Move the cooking utensils cup to the center of the table.
  2. Grab the [whisk/spatula/slotted spoon/spoon] and place it into the cooking utensils cup.
Sample videos
Storyboard
storyboard montage
decorate-a-cake_toy_version
513 episodes · 2,032,303 frames · 1539 videos · 60 fps
Task name
decorate-a-cake_toy_version
Episodes
513
Frames
2,032,303
Videos
1539
FPS
60
Task count
1
Task Description

Decorate a toy cake with candles.

Subtasks
  1. Pick up a candle.
  2. Re-orient the candle position in gripper.
  3. Place the candle into the candle slot on the cake.
  4. Hand over the candle to another gripper.
  5. Pick up the number 2 candle.
  6. Place the number 2 candle into the center slot of the cake.
Sample videos
Storyboard
storyboard montage
desk-rack-organization
334 episodes · 2,073,417 frames · 1002 videos · 60 fps
Task name
desk-rack-organization
Episodes
334
Frames
2,073,417
Videos
1002
FPS
60
Task count
1
Task Description

Organize items on a desk rack.

Subtasks
  1. Grad [eraser/marker/pen/book/marker holder/small clip] and place it into the [bottom/middle/top] layer of the desk rack organizer.
Sample videos
Storyboard
storyboard montage
dice-roll-block
366 episodes · 2,028,017 frames · 1098 videos · 60 fps
Task name
dice-roll-block
Episodes
366
Frames
2,028,017
Videos
1098
FPS
60
Task count
1
Task Description

Roll dice and grab the corresponding number of blocks

Subtasks
  1. Grab the [blue/yellow/red] foam dice with the closest arm and roll it.
  2. Read the rolled dice number, the number is [1/2/3/4/5/6] and grab the corresponding number of [blue/yellow/red] blocks and place them into the corresponding clear cup.
  3. Place the rolled dice to the opposite side of its initial position.
  4. Grab the dice and roll it again if the previous dice roll did not land fully flat on a dice face.
Sample videos
Storyboard
storyboard montage
dinnerware-organization
478 episodes · 2,042,079 frames · 1434 videos · 60 fps
Task name
dinnerware-organization
Episodes
478
Frames
2,042,079
Videos
1434
FPS
60
Task count
1
Task Description

Sort dinnerware plates and bowls by color.

Subtasks
  1. Grab a [blue/pink] plate and place it at the [left/right] center of the table.
  2. Grab a [blue/pink] large bowl and place it at the center of the table.
  3. Grab a [blue/pink] large bowl and place it on the [blue/pink] plate.
  4. Grab a [blue/pink] small bowl and place it at the center of the table.
  5. Grab a [blue/pink] small bowl and place it in the [blue/pink] large bowl.
Sample videos
Storyboard
storyboard montage
dry-asparagus
651 episodes · 2,882,990 frames · 1953 videos · 60 fps
Task name
dry-asparagus
Episodes
651
Frames
2,882,990
Videos
1953
FPS
60
Task count
1
Task Description

Dry fresh asparagus with the paper towel and put them on the plate.

Subtasks
  1. Grab an asparagus stalk and place it onto the towel, aligned horizontally.
  2. Grab the far left and far right corners of the towel with the left and right grippers and fold the towel over the asparagus.
  3. Using a gripper, lightly pat down on the towel to dry the asparagus.
  4. Use the left and right grippers to grip the corners of the top fold of the towel and unfold it.
  5. Grab a dried asparagus stalk and place it onto the plate.
Sample videos
Storyboard
storyboard montage
fill-piggy-bank
336 episodes · 2,071,088 frames · 1008 videos · 60 fps
Task name
fill-piggy-bank
Episodes
336
Frames
2,071,088
Videos
1008
FPS
60
Task count
1
Task Description

Insert coins into a piggy bank through its opening.

Subtasks
  1. Move the piggy bank to the center of table with the face of the piggy bank facing the left.
  2. Grab a coin and pass it off to the other hand so that the gripper holds the coin by the face.
  3. Place the coin into the piggy bank.
Sample videos
Storyboard
storyboard montage
grocery-bag-folding
192 episodes · 2,092,307 frames · 576 videos · 60 fps
Task name
grocery-bag-folding
Episodes
192
Frames
2,092,307
Videos
576
FPS
60
Task count
1
Task Description

Fold grocery bags and stack them into compact bundles.

Subtasks
  1. Grab two bags together and place them at the left side of the table.
  2. Grab a single grocery bag and orient it to be flat, with hand straps facing the right side.
  3. Fold the straps towards the left side.
  4. Grab the far left and far right corners of the bag with both arms and fold the bag in half.
  5. Grab the right side of the bag with the closest arm and fold the bag in half.
  6. Place the folded bag to the right side of the table.
  7. Grab the folded bag from the right and place it on top of the bag in the center of the table.
Sample videos
Storyboard
storyboard montage
hang-baseball-hats
773 episodes · 2,032,108 frames · 2319 videos · 60 fps
Task name
hang-baseball-hats
Episodes
773
Frames
2,032,108
Videos
2319
FPS
60
Task count
1
Task Description

Hang baseball hats on a rack.

Subtasks
  1. Move the hanger to the center of the table.
  2. Grab the [gray/black/red] baseball hat and hang it on the hanger.
Sample videos
Storyboard
storyboard montage
hanoi-tower-ring-placement
537 episodes · 2,321,862 frames · 1611 videos · 60 fps
Task name
hanoi-tower-ring-placement
Episodes
537
Frames
2,321,862
Videos
1611
FPS
60
Task count
2
Task Description

Solve the Tower of Hanoi puzzle by moving rings.

Subtasks
  1. Move the base to the center of the table.
  2. Place the large red ring on the leftmost peg of the base.
  3. Place the small red ring on the leftmost peg of the base on top of the large red ring.
  4. Place the large green ring on the center peg of the base.
  5. Place the small green ring on the center peg of the base on top of the large green ring.
  6. Place the large blue ring on the rightmost peg of the base.
  7. Place the small blue ring on the rightmost peg of the base on top of the large blue ring.
  8. Align the base to the center of the table.
Sample videos
Storyboard
storyboard montage
hardware-sorting
285 episodes · 2,090,181 frames · 855 videos · 60 fps
Task name
hardware-sorting
Episodes
285
Frames
2,090,181
Videos
855
FPS
60
Task count
1
Task Description

Sort fastening nuts and screws onto the plates.

Subtasks
  1. Pick up a screw and place it onto the screw plate.
  2. Pick up a fastening nut and place it on the nut plate.
Sample videos
Storyboard
storyboard montage
insert-mouse-battery
831 episodes · 2,085,520 frames · 2493 videos · 60 fps
Task name
insert-mouse-battery
Episodes
831
Frames
2,085,520
Videos
2493
FPS
60
Task count
1
Task Description

Insert a battery into a mouse.

Subtasks
  1. Grab the wireless computer mouse and place it at the center of the table, with the battery slot facing the right.
  2. Grab the AA battery and place it into the computer mouse, making sure the polarity is correct.
  3. Push the battery all the way into the battery slot.
Sample videos
Storyboard
storyboard montage
jenga-game-preparation
676 episodes · 2,044,098 frames · 2028 videos · 60 fps
Task name
jenga-game-preparation
Episodes
676
Frames
2,044,098
Videos
2028
FPS
60
Task count
1
Task Description

Set up a Jenga tower.

Subtasks
  1. Grab a jenga block by the skinny side and stack it horizontally on top of the existing jenga blocks.
  2. Grab a jenga block by the long side and place it vertically aligned at the front center of the table.
  3. Adjust the orientation of the jenga block with a gripper.
  4. Grab a jenga block and move it closer to you.
Sample videos
Storyboard
storyboard montage
jewlery-rings-return
883 episodes · 2,232,450 frames · 2649 videos · 60 fps
Task name
jewlery-rings-return
Episodes
883
Frames
2,232,450
Videos
2649
FPS
60
Task count
1
Task Description

Return jewelry rings on the base platform of the hanger.

Subtasks
  1. Grab a ring and place it on the base platform of the hanger.
Sample videos
Storyboard
storyboard montage
lift-a-pot
1642 episodes · 3,079,969 frames · 4926 videos · 60 fps
Task name
lift-a-pot
Episodes
1642
Frames
3,079,969
Videos
4926
FPS
60
Task count
1
Task Description

Lift a pot and put it on the rack.

Subtasks
  1. Move the pot to the center of the table.
  2. Move the rack to the center of the table.
  3. Grab the pot with both arms and place it on the rack.
Sample videos
Storyboard
storyboard montage
load-dishware-rack
398 episodes · 2,049,694 frames · 1194 videos · 60 fps
Task name
load-dishware-rack
Episodes
398
Frames
2,049,694
Videos
1194
FPS
60
Task count
1
Task Description

Load dishes into a dish rack.

Subtasks
  1. Move the dish rack to the center of the table.
  2. Pick up a plate and place it into the plate slots in the dish rack.
  3. Drag the plate to the center of the table.
  4. Pick up a bowl and place it into the bowl section of the dish rack.
Sample videos
Storyboard
storyboard montage
magnetic-structure
674 episodes · 2,081,234 frames · 2022 videos · 60 fps
Task name
magnetic-structure
Episodes
674
Frames
2,081,234
Videos
2022
FPS
60
Task count
1
Task Description

Rotate and move the assembled pieces until they fully match the reference model structure.

Subtasks
Variant 1
  1. Grab the red equilateral triangle magnetic pieces to the proper position reference the given model structure.
  2. Grab the green equilateral triangle magnetic pieces to the proper position reference the given model structure.
  3. Grab the yellow equilateral triangle magnetic pieces to the proper position reference the given model structure.
  4. Grab the green isosceles triangle magnetic pieces to the proper position reference the given model structure.
Variant 2
  1. Grab the purple isosceles triangle magnetic pieces to the proper position reference the given model structure.
  2. Grab the yellow square magnetic pieces to the proper position reference the given model structure.
  3. Grab the green square magnetic pieces to the proper position reference the given model structure.
  4. Grab the red square magnetic pieces to the proper position reference the given model structure.
  5. Grab the purple square magnetic pieces to the proper position reference the given model structure.
Variant 3
  1. Grab the yellow isosceles triangle magnetic pieces to the proper position reference the given model structure.
  2. Grab the red isosceles triangle magnetic pieces to the proper position reference the given model structure.
  3. Grab the red square magnetic pieces to the proper position reference the given model structure.
  4. Grab the green square magnetic pieces to the proper position reference the given model structure.
  5. Grab the blue square magnetic pieces to the proper position reference the given model structure.
Variant videos (cam_high)
ep-005 — Isosceles triangles: green + red + yellow
ep-000 — Mixed: purple isosceles + green/red/yellow squares + rotate
ep-016 — Yellow isosceles + red isosceles + blue/green/red squares + rotate
Storyboard
storyboard montage
mechanical-lock-puzzle
272 episodes · 1,387,669 frames · 816 videos · 60 fps
Task name
mechanical-lock-puzzle
Episodes
272
Frames
1,387,669
Videos
816
FPS
60
Task count
1
Task Description

Open the mechanical tool box and remove the divider.

Subtasks
  1. Align the toolbox to the center of the table with the lock facing you.
  2. Push the right lock handle into an upwards position.
  3. Press the bottom of the right lock handle to release the right lock.
  4. Press the released right lock handle so that it is fully flush to the front face of the toolbox.
  5. Push the left lock handle into an upwards position.
  6. Press the bottom of the left lock handle to release the left lock.
  7. Press the released left lock handle so that it is fully flush to the front face of the toolbox.
  8. Re-orient the toolbox with the locks facing the right.
  9. Grasp the lid of the toolbox and open the toolbox.
  10. Pick the divider inside the toolbox and place on the table.
Sample videos
Storyboard
storyboard montage
mining-balls
467 episodes · 2,020,994 frames · 1401 videos · 60 fps
Task name
mining-balls
Episodes
467
Frames
2,020,994
Videos
1401
FPS
60
Task count
1
Task Description

Sort balls from a pile into containers labelled with color.

Subtasks
  1. Grab a green ball from the box of balls and place it into the green ball container.
  2. Grab an orange ball from the box of balls and place it into the orange ball container.
  3. Grab a ball from the table and place it back into the box if it isn't green or orange.
  4. Grab a ball from the table and place it into its respective container if it is green or orange.
  5. Grab a green ball wrongly placed in the orange container and put it into the green container.
  6. Grab an orange ball wrongly placed in the green container and put it into the orange container.
  7. Grab a non-green ball wrongly placed in the green container and place it back into the box of balls.
  8. Grab a non-orange ball wrongly placed in the orange container and place it back into the box of balls.
Sample videos
Storyboard
storyboard montage
multiple-dice-ordering
650 episodes · 2,101,734 frames · 1950 videos · 60 fps
Task name
multiple-dice-ordering
Episodes
650
Frames
2,101,734
Videos
1950
FPS
60
Task count
1
Task Description

Roll and arrange dice by their numeric values in an ascending order.

Subtasks
  1. Grab the [red/yellow/purple] dice and place it on the plate.
  2. Grab the [red/yellow/purple] dice and place it at the center of the table.
  3. Grab the [red/yellow/purple] dice from the center of the table and place it on the plate.
  4. Pour the dice from the plate to roll all the dice.
  5. Place the plate back onto the table.
  6. Grab [red/yellow/purple] number [1/2/3/4/5/6] dice and place it into its correct position based off ascending order from left to right.
Sample videos
Storyboard
storyboard montage
office-desk-rack-setup
566 episodes · 2,052,654 frames · 1698 videos · 60 fps
Task name
office-desk-rack-setup
Episodes
566
Frames
2,052,654
Videos
1698
FPS
60
Task count
1
Task Description

Set up and organize an office desk rack.

Subtasks
  1. Grab a desk rack level and place it at the center of the table.
  2. Grab a desk rack level and stack it onto the existing desk rack assembly.
  3. Place a book on the top level of the desk rack.
Sample videos
Storyboard
storyboard montage
open-lock-box
638 episodes · 2,042,401 frames · 1914 videos · 60 fps
Task name
open-lock-box
Episodes
638
Frames
2,042,401
Videos
1914
FPS
60
Task count
1
Task Description

Open a locked box with a key.

Subtasks
  1. Move the lock box to the center of the table.
  2. Grab the key with the closest arm.
  3. Pass the key to the other arm to re-orient grip.
  4. Unlock the lock box using the key.
  5. Open the lid of the lock box.
  6. Place the key back down on the table.
Sample videos
Storyboard
storyboard montage
open-sealed-box-and-take-something-out
591 episodes · 3,635,829 frames · 1773 videos · 60 fps
Task name
open-sealed-box-and-take-something-out
Episodes
591
Frames
3,635,829
Videos
1773
FPS
60
Task count
1
Task Description

Open a sealed box and remove items.

Subtasks
  1. Grab a box cutter.
  2. Re-orient grip on box cutter.
  3. Cut open box tape seal with box cutter.
  4. Place the box cutter back on the table.
  5. Open the box flaps.
  6. Take out an item from the box and place it on the table.
Sample videos
Storyboard
storyboard montage
organize-office-supplies
1338 episodes · 3,348,052 frames · 4014 videos · 60 fps
Task name
organize-office-supplies
Episodes
1338
Frames
3,348,052
Videos
4014
FPS
60
Task count
1
Task Description

Organize a pen, eraser, or binder clip into the office supply organizer.

Subtasks
  1. Grab a pen and place it in the office supply organizer.
  2. Grab an eraser and place it in the office supply organizer.
  3. Grab a binder clip and place it in the office supply organizer.
  4. Move the office supply organizer to the center of the table.
Sample videos
Storyboard
storyboard montage
pack-bento-box
1052 episodes · 2,914,044 frames · 3156 videos · 60 fps
Task name
pack-bento-box
Episodes
1052
Frames
2,914,044
Videos
3156
FPS
60
Task count
1
Task Description

Pack food items into a bento box.

Subtasks
  1. Move the bento box to the center of the table aligned horizontally.
  2. Place the corn into the bento box.
  3. Place the cherry into the bento box.
  4. Place the fried chicken leg into the bento box.
  5. Grab the bento box lid and place it onto the bento box.
  6. Press down on the lid to seal it onto the bento box.
Sample videos
Storyboard
storyboard montage
pack-lunch-boxes
347 episodes · 2,144,986 frames · 1041 videos · 60 fps
Task name
pack-lunch-boxes
Episodes
347
Frames
2,144,986
Videos
1041
FPS
60
Task count
1
Task Description

Pack a lunch box with food items.

Subtasks
  1. Align the empty lunch box to the center of the table.
  2. Pick up a slice of bread and place it in the lunch box.
  3. Pick up a slice of lettuce and place it in the lunch box.
  4. Pick up a piece of steak and place it in the lunch box.
  5. Pick up the fried egg and place it in the lunch box.
  6. Pick up the fork and place it in the lunch box.
  7. Pick up the kinfe and place it in the lunch box.
  8. Close the lid.
  9. Re-align the sealed lunch box.
Sample videos
Storyboard
storyboard montage
pack-sports-bag
482 episodes · 2,273,439 frames · 1446 videos · 60 fps
Task name
pack-sports-bag
Episodes
482
Frames
2,273,439
Videos
1446
FPS
60
Task count
1
Task Description

Pack a sports bag with a tennis ball, badminton birdie, ping pong ball and ping pong paddle.

Subtasks
  1. Move a cloth bag to the center of the table with the opening facing you.
  2. Grab the [badminton/ping pong ball/tennis ball/ping pong paddle] and place it into the cloth bag.
  3. Grab the bag on both sides and lift it up a little to let the items settle to the bottom of the bag.
  4. Place the bag back onto the table.
Sample videos
Storyboard
storyboard montage
pack-suitcase-for-trip
286 episodes · 2,076,266 frames · 858 videos · 60 fps
Task name
pack-suitcase-for-trip
Episodes
286
Frames
2,076,266
Videos
858
FPS
60
Task count
1
Task Description

Pack items into suitcase.

Subtasks
  1. Grab the suitcase and move it to the center of the table.
  2. Grab an item with the closest arm and place it into one half of the suitcase.
  3. Fold the empty side of the suitcase over the other half.
  4. Grab the main zipper of the suitcase and zip it to the maximum extent
  5. Pick up the bag and position it with the handles facing upward.
  6. Adjust the bag to a proper position.
Sample videos
Storyboard
storyboard montage
paddle-bouncing-a-ping-pong-ball
1360 episodes · 1,903,303 frames · 4080 videos · 60 fps
Task name
paddle-bouncing-a-ping-pong-ball
Episodes
1360
Frames
1,903,303
Videos
4080
FPS
60
Task count
1
Task Description

Bounce a ping pong ball with a paddle into the basket.

Subtasks
  1. Grasp the ping pong paddle with right arm.
  2. Use left arm to pick up a ping pong ball and drop the ball onto the paddle to bounce it into the basket.
  3. Put down the ping pong ball onto the table.
Sample videos
Storyboard
storyboard montage
paper-towel-setup
652 episodes · 2,060,288 frames · 1956 videos · 60 fps
Task name
paper-towel-setup
Episodes
652
Frames
2,060,288
Videos
1956
FPS
60
Task count
1
Task Description

Set up a paper towel holder.

Subtasks
  1. Grab the paper towel holder and place it at the front center of the table.
  2. Grab the paper towel roll and place it on the paper towel holder.
  3. Grab a piece of paper towel while the other arm holds the paper towel roll.
  4. Place the paper towel on the table.
Sample videos
Storyboard
storyboard montage
place-books-into-shelves
450 episodes · 3,063,446 frames · 1350 videos · 60 fps
Task name
place-books-into-shelves
Episodes
450
Frames
3,063,446
Videos
1350
FPS
60
Task count
1
Task Description

Place books vertically onto shelves.

Subtasks
  1. Grab a book by the thin side with another arm holding, and place it on the shelf.
Sample videos
Storyboard
storyboard montage
pour-instant-coffee
416 episodes · 2,058,813 frames · 1248 videos · 60 fps
Task name
pour-instant-coffee
Episodes
416
Frames
2,058,813
Videos
1248
FPS
60
Task count
1
Task Description

Pour instant coffee in a cup.

Subtasks
  1. Grab the instant coffee jar with one arm.
  2. Grab the lid of the jar with another arm while holding it with the other arm.
  3. Twist the jar lid counter-clockwise to open it and place it on the table.
  4. Let go of the jar.
  5. Grab the cup and place it at the center of the table.
  6. Grab the instant coffee jar and place it at the center of the table.
  7. Place the coffee jar back onto the table.
  8. Pour some instant coffee into the cup.
Sample videos
Storyboard
storyboard montage
prepare-new-shoes
402 episodes · 2,050,461 frames · 1206 videos · 60 fps
Task name
prepare-new-shoes
Episodes
402
Frames
2,050,461
Videos
1206
FPS
60
Task count
1
Task Description

Unpack new shoes for wearing.

Subtasks
  1. Move the shoe box to the center of the table and orient it so that the lid opens to the left.
  2. Open the shoe box lid.
  3. Remove the wrapping paper covering the new shoes.
  4. Pick up the left shoe from the box and place it at the front-left side of the box, with the toe facing away from the operator.
  5. Pick up the right shoe from the box and place it at the front-right side of the box, with the toe facing away from the operator.
  6. Remove the stuffing from a shoe and place it into the shoe box.
  7. Close the lid of the shoe box.
  8. Adjust the position of the shoebox to avoid interfering with the operation.
Sample videos
Storyboard
storyboard montage
prepare-sandwich
619 episodes · 2,255,759 frames · 1857 videos · 60 fps
Task name
prepare-sandwich
Episodes
619
Frames
2,255,759
Videos
1857
FPS
60
Task count
1
Task Description

Prepare a sandwich by assembling ingredients.

Subtasks
  1. Move the empty plate to the middle of the table.
  2. Pick up a slice of bread and put it onto the plate.
  3. Pick up the piece of green lettuce and place it on the sandwich.
  4. Pick up the fried egg and place it on the sandwich.
  5. Pick up a slice of bread and place it on the sandwich to complete it.
Sample videos
Storyboard
storyboard montage
push-buttons-in-sequence
467 episodes · 2,055,250 frames · 1401 videos · 60 fps
Task name
push-buttons-in-sequence
Episodes
467
Frames
2,055,250
Videos
1401
FPS
60
Task count
1
Task Description

Push buttons into a sequence.

Subtasks
  1. Push the button labeled number 1 to turn it on.
  2. Push the button labeled number 1 to turn it off.
  3. Push the button labeled number 2 to turn it on.
  4. Push the button labeled number 2 to turn it off.
  5. Push the button labeled number 3 to turn it on.
  6. Push the button labeled number 3 to turn it off.
  7. Push the button labeled number 4 to turn it on.
  8. Push the button labeled number 4 to turn it off.
  9. Push the button labeled number 5 to turn it on.
  10. Push the button labeled number 5 to turn it off.
  11. Pick up the button labeled number 1 and place it at the left front center of the table.
  12. Pick up the button labeled number 2 and place it to the right of the button labeled number 1.
  13. Pick up the button labeled number 3 and place it to the right of the button labeled number 2.
  14. Pick up the button labeled number 4 and place it to the right of the button labeled number 3.
  15. Pick up the button labeled number 5 and place it to the right of the button labeled number 4.
Sample videos
Storyboard
storyboard montage
restaurant-takeout
340 episodes · 2,041,874 frames · 1020 videos · 60 fps
Task name
restaurant-takeout
Episodes
340
Frames
2,041,874
Videos
1020
FPS
60
Task count
1
Task Description

Take out a food box.

Subtasks
  1. Grab the plastic bag and move it to the center of the table.
  2. Untie the plastic bag with both arms.
  3. Take out a food container and place it in front of the plastic bag.
  4. Adjust the plastic bag’s handles so that the contents are fully exposed.
Sample videos
Storyboard
storyboard montage
return-necklaces
534 episodes · 2,029,963 frames · 1602 videos · 60 fps
Task name
return-necklaces
Episodes
534
Frames
2,029,963
Videos
1602
FPS
60
Task count
1
Task Description

Return necklaces to the hanger.

Subtasks
  1. Grab a necklace and hang it on the hanger.
Sample videos
Storyboard
storyboard montage
return-new-shoes
425 episodes · 2,078,090 frames · 1275 videos · 60 fps
Task name
return-new-shoes
Episodes
425
Frames
2,078,090
Videos
1275
FPS
60
Task count
1
Task Description

Return shoes to the shoe box.

Subtasks
  1. Grab a shoe stuffing and place it into the left shoe.
  2. Grab a shoe stuffing and place it into the right shoe.
  3. Open the shoe box lid.
  4. Open up the paper wrapping.
  5. Place the left shoe back to the left of box as the way they were placed by the manufacturer.
  6. Place the right shoe back to the right of box as the way they were placed by the manufacturer.
  7. Close the lid of the shoe box.
Sample videos
Storyboard
storyboard montage
roll-dice-write-on-blackboard
808 episodes · 2,079,112 frames · 2424 videos · 60 fps
Task name
roll-dice-write-on-blackboard
Episodes
808
Frames
2,079,112
Videos
2424
FPS
60
Task count
1
Task Description

Roll dice and check the result on a blackboard.

Subtasks
  1. Move the blackboard to the center of the table.
  2. Grab the dice with the closest arm and roll it.
  3. Grab the marker with the closest arm.
  4. Pass the marker off to the other arm to re-orient grip.
  5. Draw a check mark next to the number [1/2/3/4/5/6] that was rolled.
  6. Place the back on the table.
Sample videos
Storyboard
storyboard montage
scoop-with-containers
856 episodes · 2,184,913 frames · 2568 videos · 60 fps
Task name
scoop-with-containers
Episodes
856
Frames
2,184,913
Videos
2568
FPS
60
Task count
1
Task Description

Scoop coffee beans into containers using a utensil.

Subtasks
  1. Grab the cup.
  2. Scoop some coffee beans from the bean container.
  3. Pour the coffee beans in the cup into the bowl.
  4. Place the cup back onto the table.
Sample videos
Storyboard
storyboard montage
seal-water-bottle-cap
379 episodes · 2,036,650 frames · 1137 videos · 60 fps
Task name
seal-water-bottle-cap
Episodes
379
Frames
2,036,650
Videos
1137
FPS
60
Task count
1
Task Description

Seal a water bottle by closing its cap.

Subtasks
  1. Grab the water bottle and move it to the center of the table, keeping it standing up vertically.
  2. Pick up the water bottle cap with the closest arm and place it onto the water bottle.
  3. Grab the water bottle with the left arm.
  4. Grab the water bottle cap with the closest arm.
  5. Pass the water bottle cap to the other arm for grip re-positioning.
  6. Place the water bottle cap onto the water bottle.
  7. Twist the right arm clockwise and left arm counter-clockwise to tighten the water bottle cap.
  8. Twist the right arm clockwise while the left-arm holds the bottle to tighten the water bottle cap.
Sample videos
Storyboard
storyboard montage
separate-sports-equipment
445 episodes · 2,110,713 frames · 1335 videos · 60 fps
Task name
separate-sports-equipment
Episodes
445
Frames
2,110,713
Videos
1335
FPS
60
Task count
1
Task Description

Sort sports equipment to their plates.

Subtasks
  1. Grab a tennis ball and place it on the corresponding sports equipment (middle) plate.
  2. Grab a ping-pong ball and place it on the corresponding sports equipment (right) plate.
  3. Grab a golf ball and place it on the corresponding sports equipment (left) plate.
  4. Grab a ping-pong paddle and place it on the corresponding sports equipment (right) plate.
Sample videos
Storyboard
storyboard montage
set-up-board-game
88 episodes · 2,067,798 frames · 264 videos · 60 fps
Task name
set-up-board-game
Episodes
88
Frames
2,067,798
Videos
264
FPS
60
Task count
1
Task Description

Set up chess pieces on a chess board.

Subtasks
  1. Grab the white king and place it at the proper position on the white side according to international chess rules.
  2. Grab the white queen and place it at the proper position on the white side according to international chess rules.
  3. Grab a white rook and place it at the proper position on the white side according to international chess rules.
  4. Grab a white bishop and place it at the proper position on the white side according to international chess rules.
  5. Grab a white knight and place it at the proper position on the white side according to international chess rules.
  6. Grab a white pawn and place it at the proper position on the white side according to international chess rules.
  7. Grab the black king and place it at the proper position on the black side according to international chess rules.
  8. Grab the black queen and place it at the proper position on the black side according to international chess rules.
  9. Grab a black rook and place it at the proper position on the black side according to international chess rules.
  10. Grab a black bishop and place it at the proper position on the black side according to international chess rules.
  11. Grab a black knight and place it at the proper position on the black side according to international chess rules.
  12. Grab a black pawn and place it at the proper position on the black side according to international chess rules.
Sample videos
Storyboard
storyboard montage
set-up-checkerboard
230 episodes · 1,860,954 frames · 690 videos · 60 fps
Task name
set-up-checkerboard
Episodes
230
Frames
1,860,954
Videos
690
FPS
60
Task count
1
Task Description

Set up a checkerboard with game pieces.

Subtasks
Variant 1 (no matching data found in dataset)
  1. Place a red checker onto a black square on the left side of the checkerboard as it should be for a checkers game setup.
  2. Place a black checker onto a black square on the right side of the checkerboard as it should be for a checkers game setup.
Variant 2
  1. If no pieces are present on board, grab a white background piece and place it on leftmost or rightmost side of board on a dark square.
  2. If no pieces are present on board, grab a dark background piece and place it on leftmost or rightmost side of board on a white square.
  3. Grab a white/brown background piece, determine which side it should belong to based off the pieces already on the board, and place it where it belongs.
  4. If pieces are already randomly placed on board in incorrect positions, grab a white/brown background piece and place on left side of board in correct position.
  5. If pieces are already randomly placed on board in incorrect positions, grab a white/brown background piece and place on right side of board in correct position.
Variant videos (cam_high)
ep-000 — Empty board: white piece first → determine-side placements
ep-001 — Empty board: dark piece first → determine-side placements
ep-005 — Incorrect positions only: fix white + brown pieces
ep-016 — Mixed: determine-side + fix incorrectly placed pieces
ep-030 — Determine-side only: grab white + brown, place by side
ep-172 — cam_high
ep-229 — cam_high
Storyboard
storyboard montage
shape-sorter
158 episodes · 2,189,587 frames · 474 videos · 60 fps
Task name
shape-sorter
Episodes
158
Frames
2,189,587
Videos
474
FPS
60
Task count
1
Task Description

Sort shapes into their matching slots.

Subtasks
  1. Place the square in the square slot.
  2. Place the circle in the circle slot.
  3. Place the triangle in the triangle slot.
  4. Place the trapezoid in the trapezoid slot.
  5. Place the heart in the heart slot.
  6. Place the rhombus in the rhombus slot.
  7. Place the oval in the oval slot.
  8. Place the pentagon in the pentagon slot.
  9. Place the fan-shaped piece in the fan-shaped slot.
  10. Place the star in the star slot.
  11. Place the semicircle in the semicircle slot.
  12. Place the rectangle in the rectangle slot.
Sample videos
Storyboard
storyboard montage
shopping-basket-loading
638 episodes · 2,147,789 frames · 1914 videos · 60 fps
Task name
shopping-basket-loading
Episodes
638
Frames
2,147,789
Videos
1914
FPS
60
Task count
1
Task Description

Load items into a shopping basket.

Subtasks
  1. Move the shopping basket to the center of the table.
  2. Grab an item and place it into the shopping basket.
Sample videos
Storyboard
storyboard montage
simple-puzzle
476 episodes · 2,258,479 frames · 1428 videos · 60 fps
Task name
simple-puzzle
Episodes
476
Frames
2,258,479
Videos
1428
FPS
60
Task count
1
Task Description

Solve a simple jigsaw puzzle.

Subtasks
Variant 1 — Butterfly puzzle
  1. Grab the butterfly body piece and place it upright in the center of the puzzle board.
  2. Grab the purple upper-left wing piece by its and place it onto the upper-left area of the puzzle board.
  3. Grab the light green upper-right wing piece and place it onto the upper-right area of the puzzle board.
  4. Grab the dark green lower-left wing piece and place it onto the lower-left area of the puzzle board.
  5. Grab the red lower-right wing piece and place it onto the lower-right area of the puzzle board.
Variant 2 — Bird puzzle
  1. Grab the eye piece and place it onto the top of the puzzle board.
  2. Grab the red left wing piece and place it on the left side of the puzzle board.
  3. Grab the red right wing piece and place it on the right side of the puzzle board.
  4. Grab the blue body piece and place it on the bottom of the puzzle board.
Variant videos (cam_high)
ep-004 — Butterfly puzzle: body + purple UL + light green UR + dark green LL + red right wing
ep-000 — Bird puzzle: eye + blue body + red left wing + red right wing
Storyboard
storyboard montage
sock-matching-and-folding
416 episodes · 2,021,623 frames · 1248 videos · 60 fps
Task name
sock-matching-and-folding
Episodes
416
Frames
2,021,623
Videos
1248
FPS
60
Task count
1
Task Description

Match socks into pairs and fold them.

Subtasks
  1. Move the pile of socks away from center if it is at the center.
  2. Grab a random colored sock from the sock pile and bring it to the center of the table, with the toes pointing to the left and bottom of sock facing you.
  3. Grab a sock colored the same as the sock at the center of the table and stack them together, aligning them together.
  4. Grab the toe side of the sock and fold it over to fold the socks in half.
Sample videos
Storyboard
storyboard montage
sort-fruits-and-vegetables
334 episodes · 2,189,696 frames · 1002 videos · 60 fps
Task name
sort-fruits-and-vegetables
Episodes
334
Frames
2,189,696
Videos
1002
FPS
60
Task count
1
Task Description

Sort fruits and vegetables into categories by type.

Subtasks
  1. Pick up a yellow bell pepper and place it in the "VEGGIE" basket.
  2. Pick up a red bell pepper and place it in the "VEGGIE" basket.
  3. Pick up a green bell pepper and place it in the "VEGGIE" basket.
  4. Pick up a cherry and place it in the "FRUIT" basket.
  5. Pick up a lettuce leaf and place it in the "VEGGIE" basket.
  6. Pick up a strawberry and place it in the "FRUIT" basket.
  7. Pick up an asparagus stalk and place it in the "VEGGIE" basket.
  8. Pick up a banana and place it in the "FRUIT" basket.
Sample videos
Storyboard
storyboard montage
sort-kitchen-utensils
373 episodes · 2,068,008 frames · 1119 videos · 60 fps
Task name
sort-kitchen-utensils
Episodes
373
Frames
2,068,008
Videos
1119
FPS
60
Task count
1
Task Description

Organize spoons, forks, and knives into the container.

Subtasks
  1. Move the organizer and align it at center of the table.
  2. Pick up one spoon and place it in the right slot of the organizer.
  3. Pick up one fork and place it in the middle slot of the organizer.
  4. Pick up one knife and place it in the right slot of the organizer.
Sample videos
Storyboard
storyboard montage
sort-lego-by-color
281 episodes · 2,082,936 frames · 843 videos · 60 fps
Task name
sort-lego-by-color
Episodes
281
Frames
2,082,936
Videos
843
FPS
60
Task count
1
Task Description

Organize lego pieces by color.

Subtasks
  1. Grab a yellow lego brick with the closest arm and place it into the yellow pile of lego bricks.
  2. Grab a green lego brick with the closest arm and place it into the green pile of lego bricks.
  3. Grab a red lego brick with the closest arm and place it into the red pile of lego bricks.
  4. Grab a blue lego brick with the closest arm and place it into the blue pile of lego bricks.
Sample videos
Storyboard
storyboard montage
sort-toy-vehicles
350 episodes · 2,078,964 frames · 1050 videos · 60 fps
Task name
sort-toy-vehicles
Episodes
350
Frames
2,078,964
Videos
1050
FPS
60
Task count
1
Task Description

Sort toy vehicles by type.

Subtasks
  1. Grab a [green / white / black / red] [sedan / pickup truck] and place it into the corresponding group of similar vehicle types (lined up in a horizontal line).
  2. Grab a [green / white / black / red] [sedan / pickup truck] and place it at the center of the table.
  3. Grab a [green / white / black / red] [sedan / pickup truck] from the center of the table and place it into the corresponding group of similar vehicle types (lined up in a horizontal line).
Sample videos
Storyboard
storyboard montage
stack-3d-prints
567 episodes · 2,059,113 frames · 1701 videos · 60 fps
Task name
stack-3d-prints
Episodes
567
Frames
2,059,113
Videos
1701
FPS
60
Task count
1
Task Description

Stack 3D printed objects vertically.

Subtasks
  1. Grab a green cube and place it at the center of the table.
  2. Grab a white cube and stack it on top of the green cube.
  3. Grab a green rectangular prism and stack it on the white cube horizontally.
  4. Grab a white rectangular prism and stack it on the green rectangular prism horizontally.
  5. Grab a green triangular prism and stack it on the white rectangular prism to complete the tower.
Sample videos
Storyboard
storyboard montage
stack-nesting-cups
640 episodes · 2,737,743 frames · 1920 videos · 60 fps
Task name
stack-nesting-cups
Episodes
640
Frames
2,737,743
Videos
1920
FPS
60
Task count
1
Task Description

Stack cups into nested tower.

Subtasks
  1. Pick up a cup and place it upside down.
  2. Pick up a cup and place it at a leftward position for the first level of a tower.
  3. Pick up a cup and place it at the middle position for the first level of the tower.
  4. Pick up a cup and place it on the right position for the first level of the tower.
  5. Pick up a cup and place it on the right position for the second level of the tower.
  6. Pick up a cup and place it on the right position for the second level of the tower.
  7. Pick up a cup and place it at the top of the tower.
Sample videos
Storyboard
storyboard montage
tableware-stacking
432 episodes · 2,047,306 frames · 1296 videos · 60 fps
Task name
tableware-stacking
Episodes
432
Frames
2,047,306
Videos
1296
FPS
60
Task count
1
Task Description

Stack tableware plates and bowls.

Subtasks
  1. Grab a [green/blue/pink] plate and place it at the center of the table.
  2. Grab a [green/blue/pink] plate and stack it on the stack of plates.
  3. Grab a [green/blue/pink] large bowl and stack it on the stack of plates.
  4. Grab a [green/blue/pink] large bowl and add it to the stack of plates and bowls.
  5. Grab a [green/blue/pink] small bowl and add it to the stack of plates and bowls.
Sample videos
Storyboard
storyboard montage
tennis-ball-cleanup
774 episodes · 2,846,209 frames · 2322 videos · 60 fps
Task name
tennis-ball-cleanup
Episodes
774
Frames
2,846,209
Videos
2322
FPS
60
Task count
1
Task Description

Clean up and collect tennis balls into their tube.

Subtasks
  1. Pick up an empty tennis ball tube and place it on the table standing upright.
  2. Grab a tennis ball and place it into the tennis ball tube.
  3. Pick up the full tennis ball tube and move it to the back center of the table.
Sample videos
Storyboard
storyboard montage
tennis-preparation
400 episodes · 1,859,536 frames · 1200 videos · 60 fps
Task name
tennis-preparation
Episodes
400
Frames
1,859,536
Videos
1200
FPS
60
Task count
1
Task Description

Arrange tennis balls onto a tray.

Subtasks
  1. Move the box to the center of the table and align it horizontally.
  2. Pick up a tube of tennis balls.
  3. Pass the tube of tennis balls to the right arm.
  4. Pour a tennis ball out and grab it with the left arm.
  5. Place the tennis ball fully into the box, adjust the ball if needed.
  6. Place the empty tennis ball tube to the left of the table vertically standing up.
Sample videos
Storyboard
storyboard montage
tool-collection-organization
681 episodes · 2,141,386 frames · 2043 videos · 60 fps
Task name
tool-collection-organization
Episodes
681
Frames
2,141,386
Videos
2043
FPS
60
Task count
1
Task Description

Organize tools in a toolbox.

Subtasks
  1. Using the closest arm, grab the toolbox and move it to the front center of the table.
  2. Grab the tape, and place it into the tool box.
  3. Grab the plier, and place it into the tool box.
  4. Grab the screw, and place it into the tool box.
  5. Grab the tape measure, and place it into the tool box.
  6. Grab the screwdriver, and place it into the tool box.
Sample videos
Storyboard
storyboard montage
towel-folding-and-stacking
589 episodes · 3,512,924 frames · 1767 videos · 60 fps
Task name
towel-folding-and-stacking
Episodes
589
Frames
3,512,924
Videos
1767
FPS
60
Task count
1
Task Description

Fold towels and stack them neatly.

Subtasks
  1. Move the pile of towels to the left side of the table.
  2. Grab a single towel and place it at the center of the table.
  3. Open the towel into a fully open and flat position.
  4. Using both arms, grab the far left and right corners of the towel and fold them toward you in half.
  5. Using the right arm, grab the right side of the folded towel and fold it over to the left in half again.
  6. Grab the folded towel and move it to the right side of the table.
  7. Grab the folded towel on the right and stack it on the folded towel in the center of the table.
Sample videos
Storyboard
storyboard montage
tower-of-hanoi-game
1004 episodes · 2,143,353 frames · 3012 videos · 60 fps
Task name
tower-of-hanoi-game
Episodes
1004
Frames
2,143,353
Videos
3012
FPS
60
Task count
1
Task Description

Play the Tower of Hanoi puzzle game.

Subtasks
  1. Move the tower to the center of the table.
  2. Grab the small red ring and place it on the opposite peg.
  3. Grab the large red ring and place it on the center peg.
  4. Grab the small red ring and place it on top of the large red ring on the center peg.
Sample videos
Storyboard
storyboard montage
toy-robot
418 episodes · 2,089,891 frames · 1254 videos · 60 fps
Task name
toy-robot
Episodes
418
Frames
2,089,891
Videos
1254
FPS
60
Task count
1
Task Description

Disassemble a toy robot.

Subtasks
  1. Grab toy robot with closest arm and bring it to the front center of the table.
  2. Use left arm to hold the toy robot.
  3. While the left arm is holding the robot down, use the right arm to grab the loose screw and unscrew it completely.
  4. Take the loose screw with the right arm and place it onto the table.
Sample videos
Storyboard
storyboard montage
transfer-items-with-a-tray
1492 episodes · 2,630,017 frames · 4476 videos · 60 fps
Task name
transfer-items-with-a-tray
Episodes
1492
Frames
2,630,017
Videos
4476
FPS
60
Task count
1
Task Description

Transfer items using a tray.

Subtasks
  1. Grab a book and place it on the tray.
  2. Grab a plate and place it on the tray.
  3. Move the tray to the center of the table.
Sample videos
Storyboard
storyboard montage
transfer-storage-items
329 episodes · 2,082,125 frames · 987 videos · 60 fps
Task name
transfer-storage-items
Episodes
329
Frames
2,082,125
Videos
987
FPS
60
Task count
1
Task Description

Transfer items between storage containers.

Subtasks
  1. Move the large container filled with items to the back center of the table.
  2. Move the smaller container to the front center of the table.
  3. Grab an item from the large container and place it into the smaller container.
Sample videos
Storyboard
storyboard montage
trash-sorting
468 episodes · 2,048,305 frames · 1404 videos · 60 fps
Task name
trash-sorting
Episodes
468
Frames
2,048,305
Videos
1404
FPS
60
Task count
1
Task Description

Sort trash into recycling and trash bins.

Subtasks
  1. Grab a [banana/battery/usb cable/ ] from the table and place it into the trash bin.
  2. Grab a [plastic cup/towel/lego/napkin] from the table and place it into the recycling bin.
  3. Grab a [] and place it at the center of the table.
Sample videos
Storyboard
storyboard montage
unload-dish-rack
432 episodes · 2,075,868 frames · 1296 videos · 60 fps
Task name
unload-dish-rack
Episodes
432
Frames
2,075,868
Videos
1296
FPS
60
Task count
1
Task Description

Unload dishes from a dishwasher rack.

Subtasks
  1. Grab a plate from the dish rack and place it at the right front center of the table to create a stack of plates.
  2. Grab a bowl from the dish rack and place it at the left front center of the table to create a stack of bowls.
  3. Adjust the plate to properly stack it.
  4. Adjust the bowl to properly stack it.
Sample videos
Storyboard
storyboard montage
untie-shoelaces
236 episodes · 2,060,131 frames · 708 videos · 60 fps
Task name
untie-shoelaces
Episodes
236
Frames
2,060,131
Videos
708
FPS
60
Task count
1
Task Description

Untie shoelaces on shoes.

Subtasks
  1. Move the [left/right] shoe to the center of the table with the toes facing you, vertically aligned.
  2. Using both arms, grab the loose ends of each shoelace.
  3. Pull the shoelaces apart to untie them.
  4. Untie the final knot.
Sample videos
Storyboard
storyboard montage
usb-cable-disassembling
323 episodes · 2,073,771 frames · 969 videos · 60 fps
Task name
usb-cable-disassembling
Episodes
323
Frames
2,073,771
Videos
969
FPS
60
Task count
1
Task Description

Unplug cable from the socket and place it into the basket.

Subtasks
  1. Unplug Tri-cable from the socket.
  2. Place Tri-cable into basket.
Sample videos
Storyboard
storyboard montage
usb-plugging
299 episodes · 2,284,292 frames · 897 videos · 60 fps
Task name
usb-plugging
Episodes
299
Frames
2,284,292
Videos
897
FPS
60
Task count
1
Task Description

Plug USB cables into the charging block.

Subtasks
  1. Move the charging block to the center of the table.
  2. Grab the charging block with the left gripper.
  3. Grab the charging block with the right gripper and pass it off to the left gripper.
  4. With one arm, grab a USB-A cable and plug it into the charging block while holding the block with the other arm.
  5. Place the cable down to re-grasp it in the correct direction.
  6. With one arm, grab a USB-C cable and plug it into the charging block while holding the block with the other arm.
Sample videos
Storyboard
storyboard montage
vacuum-dirty-surface
770 episodes · 2,037,810 frames · 2310 videos · 60 fps
Task name
vacuum-dirty-surface
Episodes
770
Frames
2,037,810
Videos
2310
FPS
60
Task count
1
Task Description

Vacuum the coffee beans from a surface.

Subtasks
  1. Move the vacuum to the center of the table.
  2. Turn on the hand-held vacuum by pressing the power button until it lights up.
  3. Pick up the vacuum by its handle.
  4. Vacuum all the coffee beans from the table.
  5. Place the vacuum back on the table.
  6. Turn off the vacuum by pressing the power button until the lights turn off.
Sample videos
Storyboard
storyboard montage
whiteboard-supplies-organization
429 episodes · 2,048,608 frames · 1287 videos · 60 fps
Task name
whiteboard-supplies-organization
Episodes
429
Frames
2,048,608
Videos
1287
FPS
60
Task count
1
Task Description

Organize whiteboard markers and erasers.

Subtasks
  1. Move the organizer to the center of the table.
  2. Grab the [black/blue/red] marker and place it into the organizer.
  3. Grab the eraser and place it into the organizer.
  4. Grab the eraser and place it at the center of the table.
  5. Grab the eraser from the center of the table and place it in the organizer.
Sample videos
Storyboard
storyboard montage
wrap-a-toy-for-storage
482 episodes · 2,122,378 frames · 1446 videos · 60 fps
Task name
wrap-a-toy-for-storage
Episodes
482
Frames
2,122,378
Videos
1446
FPS
60
Task count
1
Task Description

Wrap a toy in a towel for storage.

Subtasks
  1. Grab the toy with the left arm and pass it off to the right arm.
  2. With the right arm place the toy into the center of the towel.
  3. Fold the towel corners over the toy.
  4. Pick up the folded towel and drop it into the basket.
Sample videos
Storyboard
storyboard montage