A model looks at a scene, explains a risk and suggests what to do next. Suddenly, years of work on detectors, rules, annotations and integrations look like an awkward staircase built just before someone invented the lift. Can we throw the staircase away?
This is the uncomfortable question behind multimodal large language models, or MLLMs. They change what video analytics can do. But confusing a better model with a complete product is an expensive mistake.
In my view, the breakthrough is in interpretation and context. The engineering challenge is to turn it into a predictable, affordable operating system for the business. That is where the platform matters.
A game changer or a brain upgrade?
One camp says the old world is dead: replace everything. The other says this is just another fashionable layer. Both make the problem sound simpler than it is.
The real question is not whether an MLLM is smarter than a detector. It is what changes in the product, architecture and cost of ownership. A model that gives a compelling explanation in a demo is not the same as a system you can afford to run across hundreds of cameras, integrate with operations and measure consistently.
Separate the technological breakthrough from the engineering economics. Otherwise, the first impressive answer will lead straight to the next expensive pilot.
From objects to the meaning of a scene
Classical computer vision turns pixels into signals: an object, a bounding box, a class, a confidence score. Tracking, zones and rules turn those signals into events. It is an effective, relatively inexpensive foundation for continuous monitoring.
A vision-language model, or VLM, connects visual inputs with language. An MLLM can work across multiple modalities, depending on the model. These models add a different level of interpretation: objects, relationships, situations and an explanation.
“A person and a barrier are visible” becomes “the person is behind the barrier, inside the restricted area.” “An object is present” becomes “the object is blocking the exit.” The answer changes from a list of detections to an interpretation that a person can assess.
This opens up archive analysis, audits, incident explanations and recommendations. It also introduces new ways to be wrong. A convincing explanation is still something the system must verify.
Video is time, not a pile of images
A single frame cannot tell the whole story. Video introduces three different questions:
- Event: did someone enter, did a vehicle pass, did an object fall?
- Change: did an obstruction appear, did a barrier disappear, did the area become dirtier?
- State: has the exit been blocked for twenty minutes, or the gate left open beyond the permitted interval?
We can assemble time from tracking, buffers, thresholds and repeated detections. It works, but an expanding collection of exceptions becomes fragile. A new angle or a changed scene can turn the rules into noise.
VLMs and MLLMs offer another way to examine a sequence: ask what changed, whether a condition persists and why it matters. They still need the right frames, timestamps and evidence. The model does not invent a reliable timeline for us.
The underestimated opportunity is the archive. Too often, it is a graveyard of terabytes waiting for someone to investigate an incident manually. Now we can build a different interface to it: “When did this passage become blocked?”, “What changed in this area over the week?”, “Is there evidence that cleaning was completed?”
That turns stored video into a source of analysis. It does not imply that a large model should watch every stream, every second.
The economics: do not ask an expensive expert to watch everything
Video consumes frames, context and computation. More frames, longer clips and repeated analysis all have a price. At the scale of hundreds of cameras, treating an MLLM as the continuous first-line detector can destroy the economics of the project.
A practical design samples a frame at a suitable interval or submits a short clip when something relevant happens. It sets limits on context, output size and the time allowed for a response. The interval follows the task: a slowly accumulating pile of waste and a fast-moving vehicle are different problems.
This makes persistent conditions and retrospective analysis particularly attractive. An exit blocked for twenty minutes does not require a large model to inspect every millisecond. Nor does a review of whether an area was cleaned during a shift.
Detectors remain the inexpensive, fast sensors that monitor the site. The MLLM is the expert called when interpretation is worth paying for.
The practical pattern: detector → trigger → MLLM
In industrial deployments, I expect the winning architecture to be the one that can count money. Start with continuous detection. Add an explicit reason to call the more expensive model.
- A detector raises an alert. The MLLM examines a frame or clip, checks the context and returns a supported conclusion.
- The detector is uncertain. A difficult angle or ambiguous object goes to a second level of review instead of simply lowering a threshold and flooding the operator.
- Signals conflict. Different modules disagree; the model receives their results and the scene to help resolve the contradiction.
- A condition persists. Repeated observations establish that something has lasted long enough to matter. The MLLM checks what the condition means.
The output can confirm or reject the event, explain the observation or refer an unresolved case for human review. Detectors provide coverage and scale. The second layer provides context. The customer pays for useful incidents and a manageable process, not for the number of neural networks in the diagram.
Where context earns its place
If you treat an MLLM as a detector on steroids, you will be disappointed by its cost, latency and variability. Its strength is the relationship between objects and the situation around them.
Is the barrier actually separating the worker from the hazard? Is a stored item blocking an escape route? Is a bent posture part of normal work? Does the equipment’s position create a dangerous arrangement? These are the questions that otherwise become dozens of manually maintained exceptions.
An explanation can connect the observation to an applicable instruction and make an incident record useful on its own: the frame or clip, the conclusion, the reason and the relevant requirement.
It can also help filter false alerts caused by missing context. But the reduction has to be measured. An elegant explanation that incorrectly dismisses a real incident is no victory.
Persistent conditions that can make economic sense
Cleanliness and infrastructure
Accumulating waste, growing obstructions, visible spills, snow or ice left uncleared, open access covers and deteriorating site conditions can be checked over minutes or hours. Compare states and keep evidence of the duration, rather than asking a large model to catch the exact instant a scrap of rubbish arrives.
The Class A office facility monitoring case shows the operational setting: cleaning, visible conditions and a response assigned to the responsible team.
Warehouses and logistics
Blocked routes, overloaded storage areas, unsafe stacking, equipment left in the wrong place and persistent queues matter because they affect access, order and capacity. “A pallet exists” is a detection. “This passage has been unusable throughout the shift” is an operational finding.
Safety and site access
A gate left open too long, a person remaining in a hazardous zone, or a vehicle occupying a restricted area can justify a contextual check. The timing and escalation logic remain explicit parts of the scenario.
Maintenance and audits
Did the work happen? Did the condition improve afterwards? Which areas repeatedly deteriorate? These questions make video useful for work verification and planning. They also give managers something more useful than another folder of screenshots.
Related applications are covered in Smart City, construction site monitoring and workplace safety.
Turn intelligent prose into an operating component
The fastest way to bury an MLLM project is to leave it in “chat with the video” mode. A demo produces impressive paragraphs. An integration needs stable fields, defined criteria and something you can test.
Free text is a poor production interface. Today the model uses one expression; tomorrow it uses another. An engineer reads certainty into a phrase, while an operator reads doubt. The audit team asks for acceptance criteria, and someone opens a screenshot.
Give the model a strict role and a structured output. A scenario can define fields such as:
- event_type: the agreed event classification;
- severity and confidence: values interpreted under the scenario’s policy;
- evidence: the relevant frames, timestamps or clip references;
- reason_codes: a small, controlled vocabulary for the conclusion;
- recommended_action: a proposed next step for the workflow.
Validate the output against a JSON schema. If validation fails, use a bounded retry, a defined fallback or human review. A schema makes the interface testable; it does not make a wrong visual conclusion true.
Keep the inputs, model version, prompt template, validated output and timing. Use fixed templates and controlled generation settings. You are building a component with an audit trail, not hiring an eloquent narrator.
Human review is also a learning loop
“Is this waste or stored material?” “Is this a violation or an authorised operation?” Disputes are part of real deployment. Uncertain results, conflicting signals and cases with high consequences need a defined review path.
Each confirmed or rejected event can become a training example: the input, necessary context and correct structured answer. That is useful data, not simply a cost to be hidden in an operator’s workload.
RAG is not a substitute for training
The temptation is predictable: add retrieval-augmented generation, feed the model procedures and a few reference images, and assume the problem is solved.
Text retrieval can supply definitions, thresholds and instructions. Visual examples are trickier. A model may transfer details from an example into the target scene or confuse the reference with the image it is supposed to inspect. More context can produce a more confident wrong answer.
Keep a clear distinction between evidence being analysed and supporting material. Validate visual retrieval against real cases. Do not assume a collection of similar pictures is a cure for systematic recognition errors.
My practical rule is simple: a rare logical mistake may be addressed through the prompt, schema or validation. A recurring visual mistake needs data and, where appropriate, fine-tuning.
Your cameras, lighting, protective clothing, equipment and site conditions may form a specific domain. Repeatedly persuading a general model with a longer prompt is not the same as making it reliable on that domain.
The improvement loop is familiar: collect real cases, prepare input-and-answer examples, fine-tune a defined scenario, deploy a version, check regressions and monitor disagreements. Adapters or LoRA may be sufficient for a particular task. The method still has to prove itself on held-out examples.
Version the model and scenario, preserve rollback and compare outcomes. Otherwise, you will not know whether the new model is better or merely different. The MLLM and RAG technical guide provides further context for implementation planning.
Why new models do not have to destroy old investments
The fear that everything you built will become worthless is usually an architecture problem. If the system consists of fragile scripts and isolated detectors, any major change really can trigger another rewrite, another integration project and another pilot.
An MLLM is a powerful computing module. It is not, by itself, the product. With a platform, the engineering response should be calm: another inference type, another block in the pipeline.
The expensive assets are already there: camera connections, archives, event histories, processes, integrations, scenarios and operating knowledge. A better model does not make those obsolete. It gives them another use.
If a new model burns through all your previous investment, you may have invested in disconnected features rather than a system.
A reference architecture that can survive the next wave
The following is my vendor-neutral view of what a mature platform should provide. Choosing a model comes after deciding how these parts will work together.
1. Media access and ingestion
Connect cameras, recorders, thermal sources, audio and archives through a consistent media layer. Normalise access and permissions. Analytics teams should not have to rebuild camera integrations for every new model.
2. An archive with meaning attached
Store video references alongside event timelines, camera and zone metadata, structured model results and the relevant operating context. Evidence clips need to remain connected to the findings they support. That is the foundation for search, comparison and audits.
3. Inference orchestration
Route frames and clips to detectors, OCR and other models. Schedule computing resources across edge devices, site servers and central infrastructure. Expensive inference should run because the scenario needs it, not because someone connected every stream to it.
4. Contextual interpretation
The MLLM layer receives the frame or clip, detector results and scene metadata. It checks relationships and changes, explains its conclusion and returns a structured result. Temporal checks need evidence over time, not an unsupported statement about a single image.
5. Scenarios people can maintain
Provide reusable blocks for detectors, conditions, timers, persistent states, MLLM verification, notifications and incident creation. Version the scenarios and test them on recorded data.
If the operating logic lives only in the heads of two engineers, it does not scale. When those people leave, the system’s memory leaves with them.
6. Incidents and integrations
An incident needs a life: created, confirmed or disputed, assigned, remediated, closed and reported. Ownership, response targets and evidence matter as much as detection.
Connect that lifecycle to service desks, enterprise systems, reporting and the channels used by the people responsible. The business needs a consistent picture, not a meeting full of screenshots from unrelated modules.
7. Model lifecycle management
Maintain model versions, compatibility with scenarios, deployment policies, regression checks, quality monitoring and rollback. Support replacing one model without rewriting every downstream integration. That is a practical way to protect investment from the next technology wave.
ViewLogic Platform 4.0 and the deployment overview are useful starting points for discussing how a platform architecture fits a particular site.
The business value is video plus enterprise context
Detecting an object is rarely where the money ends up. The value is in a decision: why an incident keeps recurring, what should change and who needs to act.
Combine video evidence with event records, shifts, zone information, equipment or access logs and applicable procedures. That combination supports three useful product directions:
- Archive audits: find blocked exits over a month, review storage conditions or compare the state of an area before and after work.
- Recurring-pattern analysis: identify the zones, shifts and operations associated with repeated incidents, then investigate the causes.
- Risk-reduction recommendations: use the evidence to review routes, barriers, storage, cleaning schedules or training.
The model’s suggestion is the beginning of a management decision. The value is realised when someone changes the process and measures what happened.
The devil’s advocate: what can still go wrong?
A model can be confidently wrong. Use schemas, validation, regression examples and human review where the consequences or ambiguity require it. Do not give free prose the authority of an industrial control signal.
Cost and latency can erase the benefit. Trigger the model selectively, sample at a justified interval and limit clips, context and output. Optimisation and quantisation help; they do not remove the need for a computing budget.
RAG can introduce errors of its own. Use retrieved procedures as context and evaluate reference images carefully. Recurring visual errors belong in the data and training loop. RAG is not a replacement for training; used that way, it becomes a crutch that often breaks first.
A plausible answer is not a business metric. Measure false positives and false negatives at the incident level, time spent reviewing events, alert noise and the relevant operational outcome. Track response times, downtime, losses or risk exposure where the project can establish a meaningful measure.
If you cannot show a measurable effect on money, risk or response time, you still have a demonstration.
A revolution in capability needs disciplined engineering
MLLMs change interpretation, context, archive analysis and the kinds of questions we can ask of video. Detectors remain useful as fast, economical stream sensors. Put the two together deliberately.
The platform keeps the data, scenarios, evidence, integrations and model lifecycle under control. It costs money and requires discipline. The alternative is paying for another rewrite every time a more impressive model arrives.
I would rather build a system in which a new technology becomes another usable component. Do you want to chase a revolution every year, or build a platform that can survive the next one and turn it into value?
Continue reading
- AI cameras or server-based video analytics?
- The dog that wanted to be a tiger: why recognising a pattern is not understanding a scene.
- A detector is not a safety process: connect observations to action.