LLM Visual Assistant video analytics

Introduction

vCloud.ai's Visual Assistant feature harnesses the power of Large Language Model (LLM)-based video analytics, allowing users to create custom video analytics detectors directly from the Cluebase Video Management System (VMS) interface. Visual Assistant does not require an internet connection, enabling seamless operation in various environments. This manual aims to provide an overview of how the Visual Assistant works, its capabilities, and best practices for maximizing its effectiveness.

Tutorial video

<iframe width="560" height="315" src="https://www.youtube.com/embed/ebYZQ7ZrWmM?si=nVrY4D-Mrnunxmey" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

Quick Setup for Video Analytics Detectors

One of the standout features of the Visual Assistant is its ability to enable users to create video analytics detectors in less than a minute. This rapid deployment is facilitated through an intuitive interface within the Cluebase VMS, allowing users to tailor their video surveillance needs without requiring extensive technical expertise or coding knowledge.

Simply navigate to the Visual Assistant feature, follow the prompts, and within moments, you can set up a detector that meets your specific requirements.

Generalization of LLM Video Analytics

It is essential to understand that LLM video analytics operates on a generalized knowledge base rather than being specifically trained for particular objects or behaviors. While this broad understanding allows it to perform well in many scenarios, there may be instances where accuracy can vary. Users should approach the Visual Assistant with this context in mind. While it can deliver impressive results in many situations, it is not guaranteed to achieve high accuracy across all use cases.

Complementary Role to Custom Development

While the Visual Assistant offers a powerful tool for quick video analytics setup, it is not a complete replacement for custom video analytics solutions that vCloud.ai can develop for clients. Custom solutions are meticulously designed to address specific use cases, providing higher accuracy and reliability for targeted applications.

Absence of camera installation requirements

Since the system does not rely on detecting specific objects, there are no stringent requirements for camera placement or orientation. Users can install cameras in various locations and only real-time testing and fine-tuning can ensure the video detector achieves acceptable accuracy. Yet we recommend that the objects and behaviors to be detected are positioned not far from IP camera, and that the camera has a high resolution.

Importance of Prompting

The prompt provided to the LLM is a critical element that significantly influences the accuracy of video analytics outcomes. A well-crafted prompt can enhance the model's effectiveness in identifying specific objects or behaviors.

For instance, simply instructing the model to “detect gun” may not yield satisfactory results. A more effective prompt would be “detect gun and return yes only if absolutely sure it is a gun.” This level of specificity helps guide the model toward more accurate assessments.

Users should invest time in developing precise prompts that clearly outline their expectations and requirements. The quality of input directly correlates with the reliability of results produced by the Visual Assistant.

Visual Assistant set up

Please follow this link (YouTube link) to learn how to create and enable Visual Assistant custom detector.

Conclusion

vCloud.ai's Visual Assistant feature offers an innovative approach to video analytics by leveraging LLM technology. With its quick setup capabilities, flexibility in installation, and importance placed on effective prompting, users can harness its potential while understanding its limitations. By following best practices outlined in this manual, users can maximize their experience with the Visual Assistant and enhance their video surveillance strategies.