Python API Reference#

The Python-level entry point is the lychsim.api.LychSim class. It composes three mixins – CameraCommandsMixin, ObjectCommandsMixin, and DataCommandsMixin – that group the command surface by what it acts on (the camera, an object, or simulation data). Beyond the wrapper, lychsim.core provides serializable bounding-box and scene-graph types, and lychsim.utils collects camera-projection, color, and image helpers.

from lychsim.api import LychSim

sim = LychSim(server_name="localhost", port=9000, width=1920, height=1080)
rgb    = sim.get_cam_lit(cam_id=0)
annots = sim.get_obj_annots()
sim.close()

The LychSim Class#

The user-facing wrapper. Constructing it opens a socket connection to the running UnrealCV / LychSim plugin and resizes camera 0 to the requested film size; close() tears the connection back down.

class lychsim.api.LychSim(server_name: str = 'localhost', port: int = 9000, width: int = 640, height: int = 480)#

Bases: CameraCommandsMixin, DataCommandsMixin, ObjectCommandsMixin

High-level Python wrapper for a running LychSim / Unreal Engine instance.

Composes CameraCommandsMixin, ObjectCommandsMixin, and DataCommandsMixin, so every camera / object / simulation command is available as a method on this single class.

Constructing an instance opens a socket connection to the UnrealCV + LychSim plugin running inside UE; close() tears it down. The usual pattern is:

sim = LychSim(server_name="localhost", port=9000, width=1920, height=1080)
try:
    rgb = sim.get_cam_lit(cam_id=0)
finally:
    sim.close()

close() → None#: Disconnect from the LychSim server and stop the receive thread.

get_status() → str#: Return the C++ status string (server / client info, configuration).

post_init() → None#

Run after __init__() to prime per-instance state.

Sets up the warmup-camera bookkeeping and resizes camera 0 to width x height via lych cam set_film_size 0 .... If you are capturing from a different camera and care about camera 0’s existing resolution, leave the constructor’s defaults at the running instance’s actual size.

print_status() → str#: Print the C++ status string (server / client info, configuration).

Camera Commands#

Image capture (lit / segmentation / element segmentation / normal / depth / point map / per-object z-buffer), camera pose queries and updates, intrinsics, and per-camera annotations. All methods operate on a single camera identified by cam_id (0 is the default).

class lychsim.api.wrapper.camera_mixin.CameraCommandsMixin#

Mixin for camera-related commands.

clear_annot_comps() → dict#

Clear cached annotation components on the C++ side.

Returns:: Raw JSON envelope with status.

get_cam_annots(cam_id: int) → dict#

Get full per-camera annotations (pose, intrinsics, film size).

Parameters:: cam_id – Camera ID.
Returns:: Raw JSON envelope {"status", "outputs": [...]} with the requested camera under outputs[0]. Fields include location, rotation, fov, c2w, width, height. Convenience wrappers (get_cam_loc etc.) return individual fields.

get_cam_c2w(cam_id: int) → ndarray#

Get camera-to-world transformation matrix. :param cam_id: Camera ID. :type cam_id: int

Returns:: Camera-to-world transformation matrix.
Return type:: np.ndarray

get_cam_depth(cam_id: int) → ndarray#

Capture a depth map from the given camera.

Parameters:: cam_id – Camera ID.
Returns:: np.ndarray of shape (H, W) with depth in centimeters (Unreal world units). Out-of-range / sky pixels read 65500+.

get_cam_ele_seg(cam_id: int) → Image#

Capture an element-level segmentation map from the given camera.

Like get_cam_seg but at sub-actor granularity (e.g. per primitive component within a Blueprint actor).

Parameters:: cam_id – Camera ID.
Returns:: PIL Image of the element segmentation map (RGB).

get_cam_fov(cam_id: int) → float#

Get camera horizontal field of view in degrees. :param cam_id: Camera ID. :type cam_id: int

Returns:: Horizontal field of view in degrees.
Return type:: float

get_cam_lit(cam_id: int, warmup: int = 0, experimental=False) → Image#

Capture a lit RGB image from the given camera.

Parameters:

cam_id – Camera ID.
warmup – Number of extra render frames to issue before the capture. Useful when the scene was just modified and TAA / streaming still need a few frames to settle. Default 0.
experimental – If True, request the experimental rendering path (-experimental). Default False.

Returns:

PIL Image of the camera view (RGB, decoded from PNG).

get_cam_loc(cam_id: int) → list#

Get camera location in world space. :param cam_id: Camera ID. :type cam_id: int

Returns:: Location as [x, y, z] in world space.
Return type:: list

get_cam_normal(cam_id: int) → Image#

Capture a world-space surface normal map from the given camera.

Each pixel encodes the surface normal at that hit point in the RGB channels (alpha, if present, is dropped).

Parameters:: cam_id – Camera ID.
Returns:: PIL Image of the normal map (RGB).

get_cam_pointmap(cam_id: int, space: str = 'all') → dict[str, ndarray]#

Get point map in requested spaces.

Parameters:

cam_id – Camera ID.
space – One of {“camera”, “world”, “opencv”, “all”}; “all” returns all three concatenated.

Returns:

Dictionary with keys for each requested space. Each value is a float32 array shaped (H, W, 3).

get_cam_rot(cam_id: int) → list#

Get camera rotation in world space. :param cam_id: Camera ID. :type cam_id: int

Returns:: Rotation as [pitch, yaw, roll] in degrees.
Return type:: list

get_cam_seg(cam_id: int) → Image#

Capture an instance segmentation map from the given camera.

Each actor in the scene is painted with a unique color (see get_obj_annots’s color field for the per-object color).

Parameters:: cam_id – Camera ID.
Returns:: PIL Image of the segmentation map (RGB, decoded from PNG).

get_cam_zbuffer(cam_id: int, obj_ids=None) → ndarray#

Capture per-object depth (z-buffer) maps from the given camera.

Returns a stack of 2D depth maps – one per requested object – useful for occlusion analysis (compare the z-buffer to the depth map to find pixels where the object is occluded by something nearer the camera).

Parameters:

cam_id – Camera ID.
obj_ids – Object ID, list of object IDs, or None / empty list for every object in the scene.

Returns:

np.ndarray of shape (N, H, W) – one slice per object – with depth in centimeters. Pixels not covered by the object read 65500+.

is_cam_pose_invalid(cam_id: int, loc: list | ndarray, radius_cm: float = 5.0) → bool#

Check whether a camera location is in invalid space by overlap testing. :param cam_id: Camera ID. :type cam_id: int :param loc: Location as [x, y, z] in world space. :type loc: list | np.ndarray :param radius_cm: Safety sphere radius in centimeters. Default is 5.0. :type radius_cm: float

Returns:: True if pose is invalid (collision/inside), else False.
Return type:: bool

set_cam_loc(cam_id: int, loc: list | ndarray) → dict#

Set camera location in world space. :param cam_id: Camera ID. :type cam_id: int :param loc: Location as [x, y, z]. :type loc: list | np.ndarray

Returns:: Response envelope with status (“ok” or “error”) and outputs (empty on success). Includes error message on failure.
Return type:: dict

set_cam_rot(cam_id: int, rot: list | ndarray) → dict#

Set camera rotation in world space. :param cam_id: Camera ID. :type cam_id: int :param rot: Rotation as [pitch, yaw, roll] in degrees. :type rot: list | np.ndarray

Returns:: Response envelope with status (“ok” or “error”) and outputs (empty on success). Includes error message on failure.
Return type:: dict

warmup_cam(cam_id: int, num_steps: int = 10) → None#

Issue num_steps no-op render frames to let the camera settle.

Useful when temporal anti-aliasing or texture streaming would otherwise produce a blurry / under-streamed first capture. Cheaper than scattering warmup= arguments across multiple calls.

Parameters:

cam_id – Camera ID.
num_steps – Number of render frames to issue. Default 10.

Object Commands#

Scene queries and manipulation: list / spawn / delete actors, query bounding boxes (AABB / OBB / bounds), update locations and rotations, fetch per-object semantic and pose annotations, render per-object masks. Object IDs match the Unreal actor labels seen in get_obj_list().

class lychsim.api.wrapper.object_mixin.ObjectCommandsMixin#

Mixin for object-related commands.

add_obj(obj_id: str, obj_path: str, loc: list | ndarray, rot: list | ndarray, scale: float = 1.0, skip_if_colliding: bool = False, adjust_if_possible: bool = False, lock_rotation: bool = False) → None#

Spawn a new actor in the running scene.

Parameters:

obj_id – ID to assign to the new actor (must be unique).
obj_path – Unreal asset content path to spawn, e.g. /Game/HousePropsFurniture/Blueprints/BP_Toilet.BP_Toilet.
loc – Spawn location [x, y, z] in cm.
rot – Spawn rotation [pitch, yaw, roll] in degrees.
scale – Uniform scale factor. Default 1.0.
skip_if_colliding – If True, abort the spawn when the actor would overlap an existing one. Mutually exclusive with adjust_if_possible.
adjust_if_possible – If True, nudge the spawn location to resolve collisions instead of aborting.
lock_rotation – If True, freeze the actor’s rotation against physics so it doesn’t tip / spin under gravity.

Returns:

Raw JSON envelope describing the spawn outcome.

Raises:

AssertionError – If skip_if_colliding and adjust_if_possible are both True.

adjust_light(obj_id: str, intensity: float = None, rot: list[float] = None, color: list[int] = None, temp: int = None) → bool#

del_obj(obj_id: str) → None#

Delete an actor from the running scene.

Parameters:: obj_id – Object ID of the actor to remove.

export_meshes(root_path: str | None = None, output_dir: str | None = None, max_count: int | None = None, recursive: bool = True, glb_only: bool = False, keep_extras: bool = False)#

Export static meshes under the given content path to glTF and glb files.

Parameters:

root_path – Content path to scan, e.g. /Game/AIUE5_vol8_01/Mesh.
output_dir – Destination directory for exported files.
max_count – Limit number of meshes to export (useful for testing).
recursive – Whether to search subfolders.
glb_only – If True, only export .glb and skip .gltf.
keep_extras – If False, will remove ancillary files (bin/png) after export when gltf is produced.

get_mesh_extent(obj_id: str | list[str]) → dict#

Get an actor’s static-mesh extent (half-sizes from the asset).

This reads from the underlying static-mesh asset rather than the spawned actor bounds, so it is independent of pose and ignores any blueprint-time scaling. Useful for sizing-driven sampling decisions before placement.

Parameters:: obj_id – Object ID, or list of object IDs.
Returns:: Raw JSON envelope {"status", "outputs": [...]} where each output carries the asset-space half-extents.
Raises:: ValueError – If the C++ response is not parseable JSON.

get_obj_aabb(obj_id: str = None)#

Get the world-space axis-aligned bounding box for one or all objects.

Backed by FActorController::GetAxisAlignedBoundingBox(), which typically reflects the root / collision component only – and so understates Blueprint composites with non-colliding visual meshes. For visual alignment (e.g. computing spawn offsets), prefer get_obj_obb(), which aggregates every primitive component.

Parameters:: obj_id – Object ID, or None for every object in the scene.
Returns:: Raw JSON envelope {"status", "outputs": [...]} where each output carries aabb, obb, bounds, bounds_tight and pose / scale / color metadata.

get_obj_annots(obj_id: str | list[str] = None) → dict#

Get the full annotation record for one or more objects.

This is the workhorse query for scene state – it returns everything the C++ side knows about each requested actor.

Parameters:: obj_id – A single object ID, a list of object IDs, or None to query every object in the scene.
Returns:: Raw JSON envelope {"status", "outputs": [...]} where each output entry contains object_id, status, guid, aabb / obb / bounds / bounds_tight (each with center and extent; OBB also has rotation), location (cm), rotation (deg), scale, and color (the 4-channel segmentation color used by get_cam_seg()). Locations are in Unreal world coordinates (centimeters, left-handed, Z-up); rotations are [pitch, yaw, roll] in degrees.
Raises:: ValueError – If the C++ response is not parseable JSON.

get_obj_list() → list[str]#

List every actor ID currently in the scene.

Returns:: List of object IDs (Unreal actor labels), one per actor.

get_obj_loc(obj_id: str | list[str] = None) → dict#

Get object world locations.

Parameters:

obj_id – A single object ID, a list of object IDs, or None to query every object in the scene.

Returns:

The raw JSON from the C++ handler:

{
    "status": "ok" | "partial" | "none",
    "error":  "<readable message>",   # only when status != "ok"
    "outputs": [
        {"object_id": str,
         "status": "ok" | "not_found",
         "location": [x, y, z]},      # omitted on not_found
        ...
    ]
}

Top-level status is "ok" when every requested object resolved, "none" when none did, and "partial" for a mix. Locations are in Unreal world coordinates (centimeters, left-handed, Z-up).

get_obj_mask(cam_id: int, obj_id: str | list[str] = None) → tuple[list[str], ndarray]#

Get object mask(s) from a specific camera.

Parameters:

cam_id – Camera ID.
obj_id – If specified, get the mask for this object. If a list of object IDs is provided, get the masks for these objects. If None, get the mask for all visible objects.

Returns:

A list of object masks.

get_obj_obb(obj_id: str) → tuple[ndarray, ndarray]#

Get the world-space oriented bounding box for an object.

Backed by Actor->GetActorBounds(false, ...), which aggregates all registered primitive components (including non-colliding visual meshes). This is the right bbox flavor for visual alignment such as bottom-center spawn-offset computation.

Parameters:: obj_id – Object ID.
Returns:: the first is shape (3,) in world cm; the second packs the half-sizes followed by the rotation as returned by the C++ handler.
Return type:: Tuple (center, extent_rotation) of np.ndarray

get_obj_rot(obj_id: str | list[str] = None) → dict#

Get object world rotations.

Parameters:

obj_id – A single object ID, a list of object IDs, or None to query every object in the scene.

Returns:

The raw JSON from the C++ handler, using the same shape as get_obj_loc() but with a "rotation": [pitch, yaw, roll] field (degrees) instead of "location":

{
    "status": "ok" | "partial" | "none",
    "error":  "<readable message>",   # only when status != "ok"
    "outputs": [
        {"object_id": str,
         "status": "ok" | "not_found",
         "rotation": [pitch, yaw, roll]},  # omitted on not_found
        ...
    ]
}

list_selected() → dict#

Return the actors currently selected in the Unreal editor.

Useful when authoring in the editor and you want to round-trip a selection into Python.

Returns:: Raw JSON envelope with selected object IDs under outputs.

update_obj(obj_id, loc: list | ndarray = None, rot: list | ndarray = None) → None#

Update an existing actor’s location and/or rotation in place.

Parameters:

obj_id – Object ID of an actor already in the scene.
loc – New world-space [x, y, z] in cm. If None, the current location is preserved.
rot – New [pitch, yaw, roll] in degrees. If None, the current rotation is preserved.

Raises:

ValueError – If both loc and rot are None.

Data Commands#

Simulation-state control (pause / resume) and debug-line drawing. Use pause() before reading scene state when physics or animations would otherwise drift between requests.

class lychsim.api.wrapper.data_mixin.DataCommandsMixin#

Mixin for data-related commands.

clear_debug_lines() → None#: Clear all debug lines.

draw_debug_line(object_ids: list[str], color='green') → None#

Draw a debug line connecting the center of a list of objects.

Parameters:: object_ids – List of object IDs.

draw_debug_line_pts(points: ndarray | list[list[float]], color='green', thickness=2.0) → None#

Draw a debug line connecting a list of points.

Parameters:: points – List of points, each point is a list of 3 floats.

pause() → dict#

Pause the simulation – freezes physics, animations, and actor ticks.

Calls lych data pause. Useful for taking deterministic snapshots before reading camera/object state. Pair with resume().

resume() → dict#: Resume a previously paused simulation. Calls lych data unpause.

Bounding Boxes#

Serializable axis-aligned and oriented bounding-box types, plus a corner-construction helper. These are produced by get_obj_aabb() / get_obj_obb() and consumed by downstream collision / sampling code.

class lychsim.core.AABB(center: ndarray = None, extent: ndarray = None, translation: ndarray = None)#

Axis-aligned bounding box represented by a center, half-extents, and a translation.

The box is defined in a local frame by center +/- extent, then shifted into world space by translation. Use AABB.corners to materialize the eight world-space corner points.

property corners#: Returns the corners of the AABB.

to_dict()#: Returns a dictionary representation of the AABB.

class lychsim.core.OBB(center: ndarray = None, extent: ndarray = None, rotation: ndarray = None, translation: ndarray = None)#

Oriented bounding box: an AABB plus a 3x3 rotation matrix.

The box is defined in a local frame by center +/- extent, rotated by rotation, then translated into world space by translation. OBB.corners returns the eight world-space corners with the rotation applied.

property corners#: Returns the corners of the OBB.

to_dict()#: Returns a dictionary representation of the AABB.

lychsim.core.bbox.get_corners(min_pt, max_pt)#

Return the eight corners of a 3D box defined by min / max points.

Parameters:

min_pt – (x_min, y_min, z_min) – the box’s minimum corner.
max_pt – (x_max, y_max, z_max) – the box’s maximum corner.

Returns:

np.ndarray of shape (8, 3) listing the corners. The first four are the bottom face (z = z_min), going counter-clockwise from (x_min, y_min, z_min); the last four are the top face, in matching order.

Scene Graph#

A scene is a hierarchy of SemanticScene -> SemanticLevel -> SemanticRegion -> Object. Each level holds a list of regions and aggregates objects from its children; serialization is supported via to_dict / from_dict, and SemanticScene additionally loads from .npz via SemanticScene.from_npz().

class lychsim.core.Object(name: str = None, uid: str = None, obb: OBB = None)#

A scene-graph leaf node: a named, uniquely-identified object with an oriented bounding box.

These are the leaves under SemanticRegion. The obb field carries the world-space pose and extent.

to_dict()#: Returns a dictionary representation of the Object.

class lychsim.core.SemanticRegion(name: str, uid: str = None, objects: list[Object] = None, polygon: Polygon = None, obb: OBB = None)#

A named region within a SemanticLevel – e.g. "kitchen" or "sidewalk_segment_3".

A region carries a 2D footprint (polygon), a 3D bounding box (obb), and a list of Object instances placed inside it.

get_all_objects()#: Returns the list of objects in the region.

to_dict()#: Returns a dictionary representation of the SemanticRegion.

class lychsim.core.SemanticLevel(name: str, uid: str = None, regions: list[SemanticRegion] = None)#

A floor / story within a SemanticScene.

A level is a flat collection of SemanticRegion instances; it aggregates the objects in each region for convenience.

get_all_objects()#: Returns the list of objects in the level.

get_all_regions()#: Returns the list of regions in the level.

to_dict()#: Returns a dictionary representation of the SemanticLevel.

class lychsim.core.SemanticScene(name: str, uid: str = None, levels: list[SemanticLevel] = None)#

Top-level scene graph: a hierarchy of levels, regions, and objects.

The hierarchy is SemanticScene -> SemanticLevel -> SemanticRegion -> Object. Use from_npz() to load a serialized scene from disk and get_all_objects() / get_all_regions() to flatten it.

get_all_objects()#: Returns the list of objects in the scene.

get_all_regions()#: Returns the list of regions in the scene.

to_dict()#: Returns a dictionary representation of the SemanticScene.

Camera Projection#

3D bounding-box corner / edge construction, world <-> camera transforms, and intrinsics-based projection helpers used to back-project depth maps and project bounding boxes into image space.

lychsim.utils.camera_projection_utils.get_bbox3d(center, extent)#

Get 3D bounding box corners and edges.

Parameters:

center (np.ndarray) – Center of the bounding box (3,).
extent (np.ndarray) – Extent of the bounding box (3,).

Returns:

3D bounding box corners (8, 3). edges (np.ndarray): 3D bounding box edges (12, 2).

Return type:

corners (np.ndarray)

lychsim.utils.camera_projection_utils.project_3d_to_2d(corners, c2w, fov, W, H)#

Project 3D points to 2D image plane.

Parameters:

corners (np.ndarray) – 3D points (N, 3).
c2w (np.ndarray) – Camera-to-world transformation matrix (4, 4).
fov (float) – Horizontal field of view in degrees.
W (int) – Image width.
H (int) – Image height.

Returns:

2D projected points (N, 2). in_front (np.ndarray): Boolean mask indicating if points are in front of the camera (N,).

Return type:

uv (np.ndarray)

lychsim.utils.camera_projection_utils.rot_mat(rotation)#

Convert rotation from (XZY) Euler angles in degrees to rotation matrix.

Parameters:: rotation (list or np.ndarray) – Rotation angles in degrees [rx, ry, rz].
Returns:: Rotation matrix (3, 3).
Return type:: np.ndarray

Colors#

Pre-baked categorical colormaps in three formats (COLORMAPS_INT, COLORMAPS_FLOAT, COLORMAPS_HEX) for visualizing object masks and bounding boxes.

Image Helpers#

lychsim.utils.general.rgbd2rgb(image, bg='white')#