The video analytics market has grown faster than its standards. There are providers with a decade of proven deployments and providers who just completed their first real installation. The gap between them is not visible from a sales deck or a demo video. It becomes visible when you try to scale - and that's too late. This guide gives you the evaluation framework to tell the difference before you commit.
Why Evaluation Is Hard
Video analytics is particularly difficult to evaluate because:
- !
Demo conditions are controlled. A provider can show you a perfectly lit, unobstructed entrance in their test environment. Your real stores have low-light zones, reflective floors, and partially blocked camera angles.
- !
Accuracy claims are easy to make and hard to verify without a structured test. Providers frequently cite 95-99% accuracy without specifying the conditions, methodology, or what errors they're excluding.
- !
Integration complexity is invisible until implementation. A system that works standalone may fail when connected to your POS, ERP, or network infrastructure.
- !
AI is a rapidly moving field. A provider who was technically leading two years ago may be falling behind - or may have built on a foundation that doesn't scale.
The Evaluation Framework
Organize your provider evaluation across five dimensions. Ask every provider the same questions and compare answers systematically.
1. Accuracy and reliability
What is your measured accuracy, and how was it measured?
Accuracy without methodology is marketing. You need to know: what environment, what camera conditions, how many counting points, what time period, and who validated the count.
Buena señal
Provides a validation methodology (e.g., manual recount comparison over X hours), specifies conditions, and offers to reproduce the test in your environment.
Señal de alerta
Cites a single percentage without specifying conditions, or claims 100% accuracy.
How does your system perform in challenging conditions - low light, partial occlusion, crowded environments?
Most real stores have at least one of these conditions. A system that only works in optimal conditions is not a production system.
Buena señal
Acknowledges performance variation across conditions and provides specific accuracy ranges for different environments.
Señal de alerta
Claims consistent performance regardless of environment, or cannot answer the question.
What is your uptime SLA? How do you handle failures?
A counter that goes offline during peak hours may miss 30% of your traffic. Understanding failure modes and recovery time is critical.
Buena señal
Provides an SLA (e.g., 99.5% uptime), describes failure detection and alerting, and has a defined recovery process.
Señal de alerta
No defined SLA, or SLA is provided but with no description of how it's enforced.
2. Scalability and architecture
What is your largest deployment? How many stores, how many counting points?
A provider who has deployed 5 locations operates very differently from one who manages 500. Ask for reference clients at a scale comparable to your target.
Buena señal
Can name specific reference clients (or offer to connect you with them) at scale comparable to your planned deployment.
Señal de alerta
References only small pilots or is vague about their largest deployment.
Does each location require a local server? What is the infrastructure requirement per store?
Server-per-store architectures multiply infrastructure cost and maintenance complexity. Centralized architectures are more scalable - but require reliable connectivity.
Buena señal
Explains the architecture clearly, including what is required on-premise vs. cloud, and has a defined connectivity requirement.
Señal de alerta
Requires significant on-premise hardware at each location with no centralized alternative.
How long does it take to add a new location?
Onboarding time per store directly affects how quickly you can scale. If it takes 4 weeks per store and you have 100 stores, that's 2 years just to roll out.
Buena señal
Has a documented onboarding process with a defined timeline (e.g., 3-5 business days per location after camera access is confirmed).
Señal de alerta
Cannot provide a consistent timeline or describes onboarding as heavily manual.
3. Integration and data access
Do you provide a REST API? What data is available via API and at what latency?
Your analytics system needs to connect to your other data infrastructure - POS, BI tools, ERP, custom dashboards. A closed system that only shows data in its own interface limits your ability to act on the data.
Buena señal
Provides documented REST API with authentication, rate limits, and a data dictionary. Offers real-time or near-real-time data availability.
Señal de alerta
Data is only accessible through their dashboard, or API access is available only on an enterprise plan with significant uplift.
Have you integrated with [your specific tools]? What does the integration process look like?
Every integration has specific requirements. Knowing whether the provider has integrated with your POS or BI tool before - and how they handled it - is more informative than general API claims.
Buena señal
Has documented integrations with major platforms and can describe the technical steps involved. Provides a dedicated integration support contact.
Señal de alerta
Claims integrations are easy without specifics, or routes all integration questions to a separate sales process.
4. Privacy and compliance
Does your system collect or store personal data? Does it use facial recognition?
GDPR and equivalent regulations impose strict requirements on personal data collection. Facial recognition triggers additional compliance obligations. You need a clear answer.
Buena señal
Clearly states that no personal data is collected, no biometric processing occurs, and anonymous tokens are discarded after session close. Provides GDPR compliance documentation.
Señal de alerta
Vague about what data is collected, or conflates 'anonymized' with 'pseudonymized' without being able to explain the distinction.
Where is data stored? Who has access to it?
Data residency requirements vary by jurisdiction. For chains operating in Europe, data stored outside the EU may create compliance issues.
Buena señal
Can specify data residency options, has a defined data access policy, and provides a Data Processing Agreement (DPA) as standard.
Señal de alerta
Cannot specify where data is stored, or DPA is only available upon request after contract.
5. Support and track record
Can you provide references in my industry at comparable scale?
A reference is the most valuable validation you can get. Speak to someone who has operated the system for at least 12 months and ask them directly about failure modes, accuracy in production, and support responsiveness.
Buena señal
Provides 2-3 references willing to take a call, ideally in retail, mall, or airport contexts.
Señal de alerta
Provides case studies but resists connecting you with actual clients, or references are only from very small deployments.
What does your support model look like? What is the escalation path for a production issue?
When a counter fails during Black Friday, you need to know exactly who to call, what the response time commitment is, and whether that person can actually resolve the issue.
Buena señal
Defined SLA with response time tiers (e.g., critical: 1-hour response, major: 4-hour response), named support contact, and escalation path to technical team.
Señal de alerta
Support is via a general ticketing system with no defined response times or escalation path.
The Pilot Process: How to Structure a Proof of Concept
Before a full deployment, run a structured pilot. Here is a recommended framework:
Baseline period
2 weeksActividades
Deploy the system at 2-3 representative locations. Define success metrics upfront: target accuracy, uptime, integration delivery date.
Hito de cierre
System operational, baseline data flowing, accuracy validation protocol agreed.
Validation
1 weekActividades
Conduct a manual recount at each location for a 48-hour period. Compare system counts to manual counts hour-by-hour. Document discrepancies and their causes.
Hito de cierre
Accuracy report with methodology. Provider reviews and responds to discrepancies.
Integration test
1 weekActividades
Connect the API to your data destination (BI tool, dashboard, or data warehouse). Test real-time data flow, error handling, and latency under normal operating conditions.
Hito de cierre
Data flowing reliably to your infrastructure. No manual intervention required.
Decision
1 dayActividades
Review pilot results against success criteria. Negotiate final contract terms based on observed performance, not claims.
Hito de cierre
Clear go / no-go decision with documented evidence.
Red Flags That Warrant Caution
These patterns appear in providers who are not ready for production deployment:
Refuses to do an accuracy validation test in your actual environment.
Claims the same accuracy across all environments and conditions.
Cannot name a reference client at comparable scale in your industry.
Requires you to replace all existing cameras with their hardware.
Has no documented SLA or refuses to put performance commitments in writing.
Cannot explain how they handle GDPR compliance or what data they store.
Integration timeline is 'flexible' with no defined deliverable.
Offers a pilot but insists on controlling the validation methodology.
Green Flags That Signal Maturity
These are signs of a provider ready for enterprise deployment:
Proactively offers to run an accuracy test in your environment before the pilot begins.
Acknowledges accuracy variation across conditions and gives honest ranges.
Has deployed at scale (100+ locations) and can connect you with a reference client.
Works with your existing cameras - any brand, any model.
Provides a documented API with a sandbox environment for integration testing.
Delivers a Data Processing Agreement as standard, not upon request.
Has a named customer success manager assigned at contract signing.
Publishes a roadmap and communicates product updates proactively.
What to Negotiate in the Contract
Before signing, make sure these items are explicitly defined:
- 1
Accuracy commitment: minimum accuracy threshold (e.g., 95%) with a defined measurement methodology.
- 2
Uptime SLA: defined availability percentage and remedies if the SLA is breached.
- 3
Response time SLA: tiered by severity with defined escalation path.
- 4
Data ownership: confirmation that all data generated belongs to you, not the provider.
- 5
Exit clause: clear data export format and timeline if you choose to terminate.
- 6
Integration delivery: defined date and scope for API integration deliverables.
- 7
Price lock: protection against significant price increases during the initial contract period.
Conclusion
Choosing a video analytics provider is a longer-term commitment than it first appears. The system will be embedded in your operations, connected to your data infrastructure, and relied on for business decisions. Getting the evaluation right upfront - with structured questions, a rigorous pilot, and clear contract terms - is far less expensive than discovering the limitations of a poorly matched provider 12 months into a multi-store deployment.
Want to see how KSI Vision performs against this evaluation framework?
Request a pilot
