Problem Statement: Tiny Mile, a “last-mile delivery” company, faced challenges in consistently operating their delivery robots in real-world environments due to inaccurate object detection, inefficient path planning, and dynamic obstacle avoidance, leading to delays and errors in the delivery process. This was due to a lack of a continuous feedback loop about the model’s performance, and no insights into the model’s potential blind spots.

‍

Solution Statement: Tiny Mile partnered with Manot and Activeloop to reduce retraining costs by 32%, increase model accuracy by 19.5% and achieve a 10x improvement in the time to production. By establishing a continuous feedback loop, regularly identifying blind spots and using actionable insights to drive model performance, Tiny Mile's robots overcome unexpected scenarios and adapt to real-world dynamic changes, leading to a more reliable and safe delivery process. 

‍

The Promise and Pitfalls of Last Mile Delivery

Tiny Mile, operating in both Miami and Charlotte, is on the forefront of an innovative industry.  Rapid advancements in machine learning algorithms, computer vision modeling, and sophisticated high-resolution cameras allow the company’s small pink robots (known as “Geoffrey”) to navigate complex city streets and deliver packages directly to customers’ doors.

Tiny Mile has recently partnered with Manot and Activeloop to monitor, observe, and explain the computer vision models in production as well as increase the efficiency of the data pipeline for model retraining, enabling Geoffrey to deliver packages more reliably.

‍

Problem

• Delivery robots depend on sophisticated computer vision models based on large datasets to automate their navigation; consistent operation over time is a challenging task.  
• Tiny Mile trains its models using millions of miles worth of images, but 50% of the time these models encounter unexpected scenarios (e.g., odd lighting, shadows, etc.) not addressed in the underlying data.
• Moreover, objects can be partially occluded by other objects or shadows, making them difficult to detect. 
• Consequently, a lengthy feedback loop is required to identify blind spots and fine tune the model for prompt redeployment. It is vital to proactively identify potential model failures, with a data curation pipeline that mitigates model performance degradation, which the development team can utilize for model refinement and redeployment purposes.

‍

Solution

• Manot automatically assesses Tiny Mile's computer vision model -- trained to segment for elements such as walls, sidewalks, roads, etc. -- in multiple ways: diagnosing dynamic “real world” challenges, identifying outliers, and predicting unexpected scenarios.
• When potential issues are detected, such as shadows that prevent the correct detection of the ground, thus posing a risk to the robot's safe navigation, the corresponding images are automatically uploaded to Activeloop's Deep Lake. This triggers alert notifications, which are then sent to relevant teams for timely intervention and problem resolution. 
• The data is labeled and exported back to Deep Lake, which helps fill gaps in the model's data, improving its accuracy. The model is then fine-tuned using the new data.
• This automated pipeline creates a real-time feedback loop for Tiny Mile to achieve continuous improvement.
• Deep Lake's visualization and query functions allowed for the real-time curation of the dataset. Thanks to the version control, the team was able to fully track the changes with data lineage. With Activeloop's real-time data streaming to ML models while retraining,  the team was able to maintain a unified view of the continuously updated dataset, and train their models efficiently while utilizing GPUs to the fullest extent.
• Geoffrey can be deployed confidently, knowing that model limitations will be captured and remediated, leading to enhanced efficiency.

Here is what Ignacio Tartavull, the CEO of Tiny Mile had to say about the solution Manot and Activeloop delivered for his company:

"Teaming up with Manot and Activeloop has been pivotal for our growth. Their cutting-edge solutions have enabled us to streamline ML model training and cut down costs, opening new frontiers in last-mile delivery. With their profound knowledge in developing robust computer vision models, we're excited about our continued partnership, with a shared vision of revolutionizing the delivery world."

‍

Results

Tiny Mile achieved remarkable results through its partnership with Manot and Activeloop. The collaboration brought about significant enhancements in the performance and dependability of Tiny Mile's delivery robots. The notable accomplishments include:

• Cost savings: The automated pipeline led to a 32% reduction in retraining costs, streamlining model retraining and redeployment expenses.
• Enhanced model accuracy: By leveraging our solution, Tiny Mile witnessed a 19.5% increase in model accuracy, enabling better predictions of potential false positives and false negatives, resulting in safer and more reliable robot navigation.
• Faster time to production: The continuous feedback loop facilitated quicker insights gathering from production to development, resulting in a remarkable 10x improvement in the time it took to bring new features into operation, leading to a more efficient delivery process.

‍

Tiny Mile's partnership with Manot and Activeloop has enabled the company to overcome the challenges associated with deploying delivery robots in the real world. By establishing a continuous feedback loop and using actionable insights to drive model performance, Tiny Mile's robots have become more reliable and safe, resulting in a more efficient delivery process.

‍

As the use of delivery robots becomes more widespread, the need for proactive monitoring, observation and refining models will only increase. The successful implementation of Tiny Mile's automated feedback loop can serve as a model for other companies looking to improve the reliability of their robots in the real world. 

‍