RGB2Normal i
Press spheres, take lines through each contact, and fit the dominant colour (PCA) against the surface-normal angle. A higher R² means colour maps to shape more linearly.
IJRR · arXiv 2504.14739
A Modularized Design Approach for GelSight Family of Vision-based Tactile Sensors
Design a tactile sensor in simulation, then build it.
GelSight sensors see touch through an optical system: a coated elastomer, coloured lights and a camera. OptiSense Studio breaks that optical system into modules, gives them parameters, and scores each design with physically based rendering. This page lets you change the geometry, the coating and the lights yourself and see the rendered tactile image.
SIM · GelSight Mini · 648×486
The framework
Five modules, four objectives, one loop
Every GelSight-like sensor is decomposed into the same optical modules. Each module has a few parameters: a 27-point cage for shape, a BSDF for material, and position and type for lights and camera. A design is rendered with indenters pressed into it, then scored. You can tune it by hand (forward design) or let a search choose (inverse design: grid search or CMA-ES).
- Soft elastomerPDMS gel that deforms on contact · RoughDielectric
- Support structureClear epoxy / acrylic that pipes the light · RoughDielectric
- Opaque coatingPaint on the sensing surface · RoughConductor, specularity ∈ [0,1]
- LightsCalibrated LEDs (IES / area models), grouped by colour
- CameraPerspective model with height, width and FoV, plus a saturation model
NormDiff i
Add camera noise of 30 % of the RGB value and measure how much the normal estimate gets confused. Closer to 0 means more robust.
AOAP i
As-orthographic-as-possible. Trace camera rays through mirrors and refractions. They should hit the gel head-on, which means low distortion.
2D-to-3D warping i
Each pixel's footprint on the sensing surface should stay square and of equal size. The score is the mean area over its standard deviation.
Design Lab
Four sensors, four design problems, and one of your own
Each case study changes one part of the optical design (shape, coating or lights) and scores the result with the objectives the paper uses for that sensor. Hover a metric name for its definition.
Case I · Curved GelSight Mini: coating optimization
RGB2NormNormDiffFig. 9
Bend the flat GelSight Mini gel into a cylinder or a dome by moving cage points, then find the coating that still reads shape well.
Case II · GelBelt: light placement and coating
RGB2NormNormDiffFig. 10–11
A new sensor: a flat sensing window between two rollers. The belt traps a thin layer of air over the acrylic, so light sent into the acrylic is lost to total internal reflection. Move the lights until the gel is evenly lit, then pick the coating.
Case III · GelSight360: light colours and component shape
RGB2NormNormDiff2D-to-3D warpingFig. 12 · Table 1
An omnidirectional fingertip. LEDs embedded in a resin shell light the gel from inside. Choose the colours of the four LED arms, then reshape the whole tip and the inner resin with cages.
Case IV · GelSight Svelte: mirror shape optimization
AOAPFig. 13–14
A human-finger-shaped sensor. One camera sees the whole finger through two mirrors. The large back mirror (M1) smears objects that touch near the base. CMA-ES reshapes M1's 81 cage parameters to maximize AOAP. The left side of the image always shows the original mirror; drag the divider to compare.
Original M1Optimized M1
Build your own sensor
A GelSight-style stack built from OptiSense modules: camera, acrylic support, silicone elastomer and a coated sensing surface, lit by three LED bars. Pick a stop for every parameter to see the tactile image of that design.
drag to orbit
Why these objectives?
Good objective scores carry over to downstream tasks
The paper trains small random-forest models directly on simulated images: one estimates how far a plane is tilted against the gel, the other estimates a cylinder's radius. Both get worse as the coating gets more specular, the same trend RGB2Norm and NormDiff predict. The images below are part of the datasets those models were trained on. Fig. 15–16
Tilt a plane against the gel
θxθy
θx 0°
θy 0°
spec.
Cylindrical GelSight Mini pressed against a plane tilted up to ±10° about both axes. With low-specularity coatings the estimated angle reaches 0.5° mean absolute error.
Press cylinders of growing radius
spec.
The cylinder's x/y scale goes from 2 to 50. The model regresses the radius from the per-channel standard deviation inside the contact.
Cite
@article{agarwal2025modularized,
title={A modularized design approach for GelSight family of vision-based tactile sensors},
author={Agarwal, Arpit and Mirzaee, Mohammad Amin and Sun, Xiping and Yuan, Wenzhen},
journal={The International Journal of Robotics Research},
volume={44},
number={13},
pages={2195--2214},
year={2025},
publisher={SAGE Publications, Sage UK: London, England}
}