ThreeDWorld (TDW)

A Platform for Interactive Multi-Modal Physical Simulation

ThreeDWorld (TDW) is a platform for interactive multi-modal physical simulation. With TDW, users can simulate high-fidelity sensory data and physical interactions between mobile agents and objects in a wide variety of rich 3D environments.

Getting Started (Read this first!)

Code of Conduct

Changelog

Website

C# Code

License

How to upgrade from TDW v1.7 to v1.8

API

Commands and Output Data

| Document | Description | | ----------------------------------------------------------- | ------------------------------------------------------------ | | Command API | API for every command a controller can send to the build. | | Command API Guide | Overview of how to send commands to the build. | | Output Data | API for all output data a controller can receive from the build. |

High-Level APIs

| API | Description | | ------------------------------------------------------------ | ---------------------------------------------- | | Image dataset | Generate 1.3M photorealistic images. | | tdw_physics | Generate physics datasets. | | Magnebot | High-level Magnebot robotics API. | | Transport Challenge | A higher-level API that uses the Magnebot API. |

Python

tdw

module

Frontend

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | |

tdw

tdw

Controller

Controller

Controller

Backend

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | Build | Helper functions for downloading the build. | | PyPi | Helper functions for checking the version of the

tdw

Audio and Video

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------- | | Impact Sounds | Generate impact sounds at runtime using PyImpact. | | PyImpact | PyImpact API. | | Audio/Video Recording | Record audio, video, or audio+video. | | Remote rendering | How to render using xpra. |

Avatars (Agents)

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | Observation Data | Different means of determining what an agent is observing. | | Depth Maps | How to use depth maps. | | Avatar Movement | Different techniques for moving agents. | | Magnebot | High-level Magnebot API. |

Benchmarks and Speed

| Document | Description | | ------------------------------------------------------------ | -------------------------- | | Benchmarks | Performance benchmarks. | | Performance Optimizations | Increase simulation speed. |

Examples

| Document | Description | | ------------------------------------------------------------ | ---------------------------------------------------------- | | Example Controllers | Index of all example controllers in the repo. | | Use Cases (see below) | The "use cases" section showcases "advanced" usage of TDW. |

Misc.

| Document | Description | | ------------------------------------------------------------ | ------------------------------------ | | VR | VR in TDW. | | Humanoids | Add "humanoids" and play animations. |

Physics (PhysX and Flex)

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | Physics | Common physics problems and solutions. | | NVIDIA Flex | Add soft bodies, cloth, and fluids to TDW. | | FluidTypes | Access different NVIDIA Flex fluid types. | | Physics Determinism | Benchmark of PhysX physics determinism. | | tdw_physics | Generate a physics dataset. | | Rube Goldberg (demo) | Demo of complex physical interactions between objects, with PyImpact generation of impact sounds, set in a photorealistic scene. |

Releases

| Document | Description | | --- | --- | | C# code | Access to C# backend source code | | Releases | Release versioning in TDW. | | Freezing your code | "Freeze" your controller into a compiled executable. | | v1.6 to v1.7 | How to upgrade from TDW v1.6 to TDW v1.7 |

Remote Server

| Document | Description | | ------------------------------------------------------- | ----------------------------------------------------------- | | Docker | Create a Docker container for TDW. | | Remote rendering | How to render using xpra. | | BinaryManager | Manage multiple instances of TDW builds on a remote server. | | bash scripts | Useful bash scripts for Linux. |

Rendering and Photorealism

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | Asset Bundle Librarians | Overview of what asset bundles are, how to add objects, scenes, materials, HDRI skyboxes, and humanoids, and how to access each asset bundle's metadata. | | Model Screenshotter | Generate images of every model in TDW. | | Material Screenshotter | Generate images of every material in TDW. | | Materials, textures, and colors | Defines materials, textures, and colors. | | Depth of Field | Prevent blurry images and increase realism. | | Depth Maps | How to use depth maps. | | Remote rendering | How to render using xpra. | | Observation Data | Different means of determining what an agent is observing. |

Robotics

| Document | Description | | ------------------------------------------------------------ | --------------------------------------------- | | Robots | Overview of robotics and the Magnebot in TDW. | | Robot Librarian | API for accessing robot metadata. | | Magnebot | High-level Magnebot API. |

Scene Setup

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | Scene Setup | Overview of how to set up a scene. | | Asset Bundle Librarians | Overview of what asset bundles are, how to add objects, scenes, materials, HDRI skyboxes, and humanoids, and how to access each asset bundle's metadata. | | Model Librarian | Overview of how to add objects and access metadata. | | Rotation | Different means of rotating objects and agents in a scene. | | Scene Reset | How to reset a scene. | | FloorplanController | Child class of

Controller

TDW and 3D Objects

| Document | Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | Model Librarian | Overview of how to add objects and access metadata. | | Model Screenshotter | Generate images of every model in TDW. | | Non-free models | Access the TDW "full model library". | | Local 3D models | Add your own objects to TDW. | | ShapeNet models | Convert ShapeNET models into TDW objects. | | Composite Objects
Creating Composite Objects | Use and create "composite objects". | | AssetBundleCreator | API for the

AssetBundleCreator

Troubleshooting TDW

| Document | Description | | ------------------------------------------------------------ | ----------------------------------------------------- | | Debug TDW | Several strategies for debugging errors in your code. | | DebugController | API for the

DebugController

Use Cases

| Use Case | Description | | ------------------------------------------------------------ | ------------------------------------------ | | Image dataset | Generate 1.3M photorealistic images. | | IntPhys (demo) | Demo of how to simulate IntPhys in TDW. | | Humanoid videos | Generate a dataset of humanoid animations. | | tdw_sound20k | Generate an audio dataset. | | tdw_physics | Generate a physics dataset. | | Rube Goldberg (demo) | Demo of complex physical interactions between objects, with PyImpact generation of impact sounds, set in a photorealistic scene. | | Magnebot | High-level Magnebot API. |