copyd-nn

The model is SSCD, a network Meta trained specifically for copy detection. Most vision models learn to answer what is this a picture of. This one learned something narrower and more useful here. It saw millions of examples of how copies get altered in the wild, from crops and mirrors to recolouring, re-encoding and pasted-on logos, and it was rewarded for giving the altered version the same answer as the original.

A frame goes in at 224 × 224 pixels. A stack of convolutions reduces it to a grid of features, and a pooling step called GeM squeezes that grid into a single list of 512 numbers. Scaling the list to unit length is what lets two frames be compared with one multiply-and-add, the cosine similarity behind every score on this site.

One detail matters more than the rest. SSCD's training includes a term that actively spreads unrelated descriptors apart, so the gap between "this is a copy" and "this merely rhymes" stays wide, and stays roughly the same width everywhere in the space. That consistency is what lets one global threshold work across an entire library instead of being retuned per video.

The numbers travel one way only. Nothing reconstructs the picture from them, so a fingerprint gives away nothing about what it describes.

One descriptor per frame is a summary, and summaries dilute. Someone takes a small crop of your video and drops it into the corner of theirs. The frame they publish is mostly their content, so its 512 numbers describe their frame, and the match to yours is lost.

So each frame is described several times over, once for the whole picture and once for each of a few windows inside it. A partial copy can then match window to window rather than needing the whole frames to agree. The windows are concentric because the edits that dilute a global descriptor tend to be concentric too: a centre crop, a letterbox, a picture-in-picture sitting in the middle.

The two sides get deliberately uneven treatment. A reference is stored forever and every window multiplies the size of the index, so a reference gets three, permanently. A query is a passing visitor whose cost disappears the moment it is answered, so a query can afford far more: switch on every window capability and it is described fourteen ways, each also embedded mirrored, for twenty-eight views. That asymmetry buys the recall of a fine-grained search at the storage cost of a coarse one — and because it is transient, it is yours to spend or not, per search (chapter 07). The default is the whole frame alone.

An earlier version also cut each frame into quadrants. It cost twice as much and found everything the nested windows already found, so it went. There is a real ceiling here as well. Every extra window is another chance for something unrelated to look similar, which is why the grid stops where it does.

Nested windows are a guess about where an edit put the content, and real edits are under no obligation to cooperate. Put two videos side by side and the split runs down the middle, where no centre crop reaches. A centre crop of that frame captures half of each picture and resembles both sources equally badly. Letterbox a clip and the whole-frame descriptor spends much of itself describing black bars.

So before searching, the query gets measured. Two things give an edit away.

Flat borders. Scanning inward from each edge, rows and columns with almost no variation are padding: black bars, a plain backdrop, the surround of an inset. What survives the scan is the real content box. This single test catches letterboxing, pillarboxing, and a picture-in-picture sitting anywhere in the frame, including the corners that concentric windows can never reach.

Seams. A composite has a boundary. Sometimes that boundary is a visible divider, which shows up as a narrow band of flat columns. Sometimes the two pictures simply abut, and the giveaway is neighbouring columns that stop resembling each other far more abruptly than ordinary footage ever manages.

A layout only counts once it recurs across the clip. Editing persists for a whole shot at minimum, while a moment that merely looks like an edit passes. That distinction is what keeps a cut to black or a dark sky over a dark sea from being read as letterboxing. It also keeps every batch the same shape, which is what lets all the windows of all the frames go through the network in one pass.

Found windows join the ladder, capped at four, and anything that duplicates a window already there gets dropped. The result raises recall and lowers false positives at the same time, because a region that really is one picture makes better evidence than a guessed one straddling two.

Watching every frame of a video would mean thousands of descriptions per minute for footage that barely changes, so the sampler takes a couple of frames per second and moves on. That works well, and it carries one weakness worth being honest about. A fixed grid is predictable, and anything predictable can be aimed at.

Blacking out a single frame every second costs one line in a video editor and passes a viewer completely, since it is one frame in twenty-five and gone before the eye registers it. It lands squarely on a regular sampling grid. A sampler that keeps whatever the clock hands it then spends a large share of the clip describing black rectangles, and those descriptions carry no identity, so the evidence for a real match quietly thins out. This is a published attack with measured results behind it.

The counter is small. When the frame the clock points at carries nothing, meaning near-uniform pixels of any colour, the sampler steps sideways and takes the next frame that carries something. The timestamp moves by a few hundredths of a second, well inside the tolerance the timeline fit already allows, and what gets described is a frame that means something.

Stepping is bounded, and the bound is the interesting part. When a clip genuinely fades to black for several seconds, skipping past all of it would leave a hole in the timeline exactly where the query really was, and a fit that sees an unexplained gap is worse off than one that sees darkness. So after a few tries in a row the sampler takes what it is given. On clean video it behaves exactly like the fixed grid.

A similarity of 0.6 sounds like a fact, and it hides one. Some frames sit close to everything: a title card, a fade, a stretch of anonymous studio footage, a talking head on a plain backdrop. Their 0.6 means very little. A frame with unusual composition scoring 0.6 against one reference and nothing else means a great deal. A single global threshold reads both as the same number, and thresholds are how verdicts get made here.

So every query frame answers a second question: how well do you score against the collection in general? A small, fixed sample of the library stands in for material at large, and a frame's similarity to its nearest few of those becomes its background level, a measure of how generic it is. Discounting that level from its scores leaves the part that actually distinguishes one reference from another, and that is what the evidence gets weighed on.

Two decisions keep this honest. The discount applies to the query side only. Textbook recipes discount both sides, which works when the background sample comes from somewhere unrelated to the collection being searched. This library exists to hold things that resemble each other, so on the stored side the same arithmetic penalises a reference for resembling its own near-duplicates, which is precisely backwards.

The discount is also centred, so an ordinary frame stays where it was while an unusually generic one loses ground and an unusually distinctive one gains. Every score therefore stays on the familiar scale you read on a result page, instead of the whole system sliding down by some arbitrary amount. The effect is higher recall at no cost in false positives.

A locked-off shot of someone talking for thirty seconds gives a descriptor the same 512 numbers hundreds of times over. Storing all of them inflates the index and adds nothing anyone could ever find.

So a sampled frame earns its place only once it has drifted far enough from the last one kept, meaning its similarity to that stored frame has fallen below a threshold. A static shot collapses to a single entry however long it runs. A fast-cut sequence keeps almost everything, because almost everything in it is new. The rate follows the footage rather than a number someone guessed.

What makes this pleasant is that it needs no scene-detection step. The descriptors already know when the picture changed, since measuring change is exactly what they do. Shot boundaries fall out of the same comparison that does everything else.

One guard keeps it safe. However still a shot is, a frame goes in at least every couple of seconds. Timeline alignment needs timestamps spread along the video, and a shot that produced a single entry in four minutes would leave the fit nothing to anchor to.

Think of this as a search optimisation first and a storage one second. Every row left out is a row never compared, on every query, forever.

Most queries are easy. A re-upload, a re-encode, a resize, a speed change or a recolour leaves the whole frame still looking like the whole frame, and one descriptor per frame settles it. That is the baseline every search runs, and on its own it answers in about a second per minute of footage.

The hard cases are the ones where the copy occupies only part of the picture: a deep crop, a corner inset, a split screen, a phone recording of a screen across the room. Those need the frame described in windows rather than whole, and each family of window is a capability you switch on: crops adds four concentric windows, insets a nine-cell grid, splits whatever the pixels' own geometry argues for, mirrored a flip of everything selected.

The cost is honest and stated on each one, because it is multiplicative: every window is another pass of the neural network over every sampled frame. All four together is roughly thirty descriptors per frame instead of one.

So the trade is yours rather than the engine's guess. Screening a stream of mostly-clean uploads wants the baseline; hunting a known leak through re-cuts and camrips wants everything, and the vectors are cached per query so the second look never re-embeds what the first already did.

Fitting a line through matching moments is the core idea, and left alone it is far too eager. Give it enough candidate pairs and some line always fits something. Most of the care in this system goes into refusing the fits that mean nothing.

A line has to touch real reference material. Mapping a long query onto two seconds of a reference satisfies a line beautifully while involving almost none of the reference. A run must land on several distinct reference moments to count.

One vote per query frame. A single frame can produce dozens of candidates, and inside a static shot most of them are near-duplicate reference moments scattered across the tolerance band. Counting all of them lets one frame outvote a hundred others and tilts the line. Each query frame casts one vote, its best.

Speed has a supported range, and its edges are a tell. When a fit reports a speed sitting exactly on the edge of what is supported, the constraint chose that number rather than the evidence. The honest answer lay outside the range, which puts the content outside what counts as a copy here, so those fits get rejected instead of reported at the boundary.

Some agreement carries no information. Two videos that share a black tail or a standard title card agree perfectly and constantly, and a line drawn through that field passes every other test. The giveaway shows up across the library, where content-free frames match many different references equally well. Every run therefore carries a measure of how much of its support actually points at one moment, and runs built on ambience are held back from a strong verdict.

Repeats are one fact, not twenty. A rolling broadcast that loops the same segment all day produces many runs against the same stretch of reference at different query times. Those get grouped and reported as a single recurring match with its period, in place of twenty near-identical rows.

Every stored row is one moment of one window of one file. The descriptor leaves the network as 512 full-precision numbers, which comes to 2 KB per row, and that is more than the job requires.

On disk, half precision. Rows go down as 16-bit floats, halving the fingerprint. The values are already unit vectors in a small range, and all that follows is a dot product, so the loss stays far below anything that could change which row wins. The precision the network runs at is a separate question entirely, and there the same shortcut is genuinely destructive.

Small libraries: no index at all. Below a few hundred thousand rows, a query multiplies against every stored row on the GPU. Brute force is embarrassingly parallel and perfectly exact. It also stays affordable much further up than you might expect here, because this design keeps roughly three vectors per frame where a keypoint-based system would keep a thousand.

Mid-size: a proximity graph. Once a full scan stops being interactive, an HNSW graph takes over. Think of a navigable network where search walks from node to neighbouring node, always downhill toward the query. The settings deliberately trade a sliver of accuracy for build speed. A moderate number of links per node (M = 16) builds several times faster than a denser graph and gives up a fraction of a percent of recall, and build cost is what dominates the operational story. The search-time effort knob (ef = 256) stays generous, since that one only costs you when you use it.

Archive scale: quantization. Past a few hundred thousand rows, search moves to FAISS IVF-PQ, which does two things at once. IVF carves the space into cells, roughly four times the square root of the row count, and a query visits only a fraction of them, so lookup cost follows cells probed rather than archive size. PQ, product quantization, chops each vector into 64 short pieces and replaces every piece with the nearest entry from a small learned codebook, storing a one-byte code in its place. A 2 KB vector becomes 64 bytes, about thirty times smaller, and that is what turns thousands of hours into something that fits in memory.

Compression only ever nominates. Compressed codes are good at saying which rows deserve a look and poor at ranking them, so they hand over a shortlist several times longer than the answer. That shortlist gets re-scored against the exact vectors, memory-mapped from disk so only the shortlisted rows are ever touched. The ranking you read is the exact ranking, and the approximation merely decided who got to compete for it.

One structural oddity is worth naming. FAISS runs in a separate process, because it and the neural network library each bring their own parallel-computing runtime and loading both into one process is unstable. The split turns out to suit the work anyway. Describing frames is GPU work, searching is CPU work, and the two exchange nothing but arrays of numbers.

Modern GPUs run half-precision arithmetic far faster than full precision, which makes describing frames in 16-bit an obvious win. Taking that win the obvious way destroys this model completely.

The pooling step is why. GeM raises every activation to a power before averaging, and in 16-bit those raised values run off the top of what the format can represent. Everything saturates, and every descriptor collapses onto the same vector. What makes the failure vicious is how quiet it is. Nothing crashes, no warning appears, the vectors still have unit length, and unrelated frames simply start scoring a perfect match against each other.

Autocast solves this by applying a policy per operation. Convolutions and matrix multiplies carry the bulk of the work and tolerate low precision happily, so they run in 16-bit. Reductions stay in full precision, including the pooling step that would otherwise break. The dangerous operation never sees the narrow format at all, and the descriptors come out matching their full-precision counterparts to five decimal places.

The gain shows up mainly in queries rather than ingest, for a reason worth knowing. Pulling frames out of a video file takes longer than describing them, so ingest runs at the speed of the video decoder. A query describes many windows per decoded frame, and that is where faster arithmetic earns its keep.

A start-up self-check embeds a handful of structured noise images and fails loudly if unrelated ones score alike. Should the collapse ever return by some other route, it will announce itself.