NeoteAI and Fudan University Release Three Technical Reports: Bringing the Sense of Touch to Embodied AI with 30,000 Hours of Tactile Data_News_上海新智具身智能科技有限公司

EN
Home News NeoteAI and Fudan University Release Three Technical Reports: Bringing the Sense of Touch to Embodied AI with 30,000 Hours of Tactile Data
2026.07.26
NeoteAI and Fudan University Release Three Technical Reports: Bringing the Sense of Touch to Embodied AI with 30,000 Hours of Tactile Data

From cooking noodles with eggs and preparing lemonade to organizing household items, robots are becoming increasingly capable of handling everyday tasks. Their capabilities extend to industrial environments, where they can perform assembly-line operations, precision component assembly, and intricate cable winding and organization. These are just some of the impressive real-world manipulation capabilities demonstrated by the N0 series of models.

But have you ever seen a robot fail at a task that looks surprisingly simple? A plug repeatedly rubbing against the edge of a socket without going in. A plastic bottle collapsing under excessive gripping force. A neatly folded towel falling apart the moment the robot releases it.

As embodied AI moves toward real-world deployment, such failures are far from uncommon: the robot's vision system may perceive everything correctly, yet the task fails at the very moment of physical interaction.

These failures point to an increasingly recognized challenge in robotics: what determines whether a robot can operate reliably in the physical world is not merely the accuracy of visual perception, but also the tactile feedback it receives at the moment of contact.

Cameras can identify spatial positions, but they cannot directly determine whether contact has occurred, whether an object is pressing against an edge, whether the gripping force is excessive, or whether an object is slipping. The absence of this feedback is a major obstacle preventing robots from successfully navigating the critical "last centimeter" of physical interaction.

新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图2)

Recently, NeoteAI Tech Co., Ltd. (NeoteAI), in collaboration with the Institute of Trustworthy Embodied Artificial Intelligence (TEAI) at Fudan University, officially released three technical reports in the N0 series, elevating tactile sensing from an auxiliary modality to a foundational component of embodied AI.Together, these three reports outline a comprehensive technical roadmap: one tactile data foundation supporting two complementary approaches to embodied AI modeling.

Of particular significance to the broader research community, the project includes plans to make its data and models open source, enabling researchers and developers to access and build upon the work.

Explore the complete research projects: https://research.neoteai.com



新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图1)-Foundation:

Unifying the “Languages” of Touch with 30,000 Hours of Interaction Data

The large-scale adoption of tactile data has long been constrained by incompatibilities between the data formats produced by different tactile sensing devices.




Gel deformation images, capacitive sensing matrices, and six-axis force vectors generated by different sensors are like distinct dialects without a shared grammar, making it difficult to integrate them within a unified model.

To address this challenge, NeoteAI developed the NeoData dataset, designed to overcome these barriers through large-scale, multimodal tactile data collection.

The dataset includes:


  • Over 30,000 hours of visual-tactile interaction and manipulation data.

  • Approximately 1.4 million manipulation episodes and 3.3 billion timesteps.

  • 8 billion RGB frames and 10 billion tactile image frames.

  • Data collected using six robotic embodiments, including Franka, Piper, and UR5e.

  • 450 real-world long-horizon tasks, such as clothing folding, connector insertion and removal, and liquid pouring, collected by 90 operators.

  • 5,000 hours of visual-tactile interaction data already released as open-source resources.

新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图5)


In addition, the team introduced NeoForce, a unified tactile representation model.

Regardless of the underlying tactile sensor hardware, NeoForce learns transferable, temporally structured tactile representations that can support downstream embodied AI policies.

Where is pressure being applied? How much force is involved? Is there any lateral pulling or shear force?

By translating heterogeneous tactile signals into a shared representation, NeoForce effectively establishes a common language for touch, addressing a key challenge in cross-device tactile data integration.



新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图1)-VTLA:

Empowering Lightweight VLA Models with Predictive Tactile Intelligence

With the data foundation established, the team's next challenge was to determine how tactile sensing could be meaningfully integrated into Vision-Language-Action (VLA) architectures.

In existing multimodal fusion approaches, simply concatenating tactile images with RGB visual signals often presents significant challenges.

Tactile signals tend to produce high-frequency responses only during physical contact and remain largely inactive at other times. As a result, they can easily be overwhelmed by the massive number of visual tokens.

More importantly, even when a model successfully extracts tactile features from the current moment, those features still represent feedback from actions that have already occurred.

This reactive, "rearview-mirror" approach to perception leaves robots constantly one step behind during dynamic physical interactions.


新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图8)

新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图1)-VTLA introduces a different approach: instead of relying solely on delayed tactile feedback, it predicts how tactile states will evolve over the next 50 steps.

The model encodes current tactile observations into compact latent tokens and combines them with visual context to anticipate changes in contact conditions, including potential slippage and force variations. This allows the action module to adjust its behavior proactively.

Experimental results across different tasks demonstrate significant improvements:

  • Plug insertion: N0-VTLA achieves an 85% success rate, compared with 60% for a comparable vision-only approach.

  • Key removal: N0-VTLA reaches a 99% success rate, compared with 35% for the vision-only approach.

Beyond predictive tactile modeling, N0-VTLA also addresses a fundamental limitation of conventional imitation learning: its dependence on successful expert demonstrations.

The model enables robots to improve by learning from failures.

By incorporating tactile information, post-deployment data, and human-in-the-loop feedback, the team trained a task progress evaluation model capable of identifying different stages of task execution, including complex behaviors such as regression and recovery.

Combined with offline reinforcement learning, this approach enables substantial improvements in long-horizon manipulation tasks.

These results demonstrate how robots can learn from unsuccessful attempts and continuously improve their manipulation policies, moving beyond imitation toward experience-driven evolution.


新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图12)

新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图1)-TWAM:

Reconstructing “Tactile Imagination” Within World Models

If N0-VTLA equips robots with a predictive sensing system, N0-TWAM takes a step further by rethinking the robot's cognitive architecture.

It integrates tactile perception directly into a world model, enabling robots to mentally simulate not only what an action will look like, but also what it will feel like, before physically executing it.

Most existing world models focus primarily on generating future visual observations. However, when applied to real-world robotic manipulation, they face substantial challenges in maintaining physical consistency.

A generated video may show a robot successfully grasping a cup, but is the grip actually secure?

The plug may appear perfectly aligned with the socket, but will it become jammed during insertion?

These critical contact-related details are often missing from visual-only predictions.


新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图19)

N0-TWAM addresses this limitation by jointly predicting three interconnected sequences: future video, tactile observations, and robotic actions.

The model adopts a mixture-of-experts architecture with three asynchronous expert networks dedicated to vision, touch, and action.

The video expert is initialized from a pretrained model, while the tactile and action experts use narrower architectures to improve computational efficiency.

The resulting model contains 7.2 billion parameters, approximately half the size of an equivalent full-width architecture.

N0-TWAM demonstrates significant advantages in both simulated and real-robot evaluations.

  • Simulation: An average success rate of 84.5%, compared with 36% achieved by the strongest world-model baseline.

  • Real-world robotic evaluation: An average success rate of 46.3% across eight tasks, compared with 21.9% for LingBot-VA.

Ablation studies further confirm the importance of tactile information. Removing either future tactile prediction or current tactile conditioning results in substantial performance degradation.

These findings establish tactile sensing as an essential component of world models for contact-rich robotic manipulation.


新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图21)



Touch: The Next Scaling Law for Embodied AI?

Taken together, the three N0 technical reports convey a clear technological vision from NeoteAI.

N0-Foundation demonstrates that tactile data has scaling potential similar to that of visual data.

N0-VTLA shows that tactile information can be effectively integrated into existing VLA architectures, enabling proactive manipulation and continuous policy improvement.

N0-TWAM establishes tactile sensing as a central component of world-model design, placing touch alongside vision in the prediction and planning of physical interactions.

Together, these advances point toward a fundamental shift in robotic intelligence: from vision-centric perception to integrated vision-tactile intelligence.

Rather than relying solely on what they see, robots are beginning to explore the physical world through active touch, much like infants learning about their surroundings. They can use tactile feedback to test assumptions, understand physical interactions, and plan subsequent actions.

Seeing a cup does not mean being able to hold it securely. Recognizing a socket does not guarantee successful plug insertion.

These seemingly simple facts about the physical world are now being incorporated into the fundamental architecture of robotic intelligence.

Of course, achieving general-purpose embodied intelligence with intuitive physical understanding remains a long-term challenge.

The cost of collecting real-world interaction data, generalization across different robotic embodiments, and real-time inference efficiency are still major engineering obstacles that must be addressed.

Nevertheless, the N0 series represents a meaningful step forward in elevating tactile sensing from a specialized research topic to a foundational infrastructure for embodied AI.


新智具身联合复旦大学发布三份技术报告:用3万小时触觉数据补齐具身智能「手感」(图27)


项目链接:https://research.neoteai.com

公司官网:https://www.neoteai.com