Unlurker

Tennessee grandmother jailed after AI face recognition error links her to fraud (theguardian.com)

>(saved by her bank records) Don't worry if unbanked, the commercial app industry is already here to save you. “My Location Ledger” https://apps.apple.com/us/app/my-location-ledger/id6757806805 I’d be lying if I said I wasn’t aware that some of the various parties already spying on me do have a one in 1 million chance of coming in handy. To that end, tried this years ago but didn’t work immediately: “OwnTracks” (FOSS) https://apps.apple.com/us/app/owntracks/id692424691 PS: Flock be ready for our location requests in emergencies, only fair

How was she saved by her bank records when they already arrested her like she was proven to have done it? How little evidence do cops need to go arrest a woman 1,200 miles away and fuck up her life? And then not even apologize for it? That's fucked up.

Fargo Police Department. That tracks. Are we sure the Cohens were not involved?

I would really love to see the Cohens make a reenactment documentary about American injustices, with the lead-in being "These are real events that happened, names have not been changed."

Insane that this took six months. AI facial recognition should be considered about as reliable as a polygraph, which is to say not usable in court at all. Shame we’ve got ICE agents roaming the country also using facial recognition to find their targets, huh?

More insult+injury: > But Lipps said Fargo police did not pay for her trip home, leaving her stranded. Local defense attorneys helped cover a hotel room and food on Christmas Eve and Christmas Day, and a local non-profit, the F5 Project, was able to help her return to Tennessee, InForum reported. How the hell are authorities not responsible for helping an innocent person back after forcing them to travel at the point of a gun?

I read she had no winter clothes, not even a jacket to go outside in the cold when they released her. She was arrested in TN during warm weather. Not all of the news sites reported the story in complete detail. Her treatment was truly appalling.

I know polygraphs are not admissible in court but they're still being used and have quite a bit of swaying. I think it's mainly intimidation at play here. Yeah, it's absolutely crazy that it took 6 months to clarify this, if she was rich and had a good lawyer she could've solved it faster. I really hope that she at least gets compensated and/or sues the operators or the AI company. And as far as ICE, I think they don't care that they pick up the wrong people, they just have quotas to reach to unlock bonuses. It's cynical and sad as hell. Hopefully we're gonna be done with them once Trump is gone.

Trump is not a problem. System that lets him do what he does is. I used to admire the US back when I lived in USSR. You can guess the way I look at it lately. I still have some hope in people of the US, they seem to actually be capable to stand for their rights every once in a while. We'll see what happens.

I have a face that looks like a lot of other people. I have a name that 500+ men use in the world. I don't do anything bad or criminal, but I could be mistaken for a man who matches my face. Nature creates patterns, and sometimes you get a Mr. Potato Head like me with a common face.

I wonder if any other men have your face and your name.

Yes -- I know at least 3 Orion Blasters.

Absolutely everybody has face doubles. Identikit got pretty close and there weren't that many bits in there and quite a few of them were hairstyles and that's a choice, not genetics. How many head shapes, noses, eyes, mouths and ears can there be? A few million? Then everybody has a few thousand doubles. 100 Million? Still 80.

> Absolutely everybody has face doubles. I once had a waiter in a restaurant that I'd never been to before swear he'd seen me there many times, and when I denied it he was backed up by some of the other staff. Creepy, to say the least. Afterward I realized I should have given him my phone number and told him to call me next time "I" came in, so that I could meet my doppelganger.

That's why AI should not be used for identification alone, it's unreliable.

There was a case where someone's finger prints matched someone who was later found to have an alibi and not be there. So even finger prints are unreliable.

Correct; NIST recommended (~10 years ago) they be used together: https://www.pnas.org/doi/full/10.1073/pnas.1721355115

↙ time adjusted for second-chance

"Design Me a Highly Resilient Database" (nikogura.com)

I think it’s a great interview question. Open-ended, rewlly leads to a good discussion that can give you a sense of how the applicant approaches a problem and thinks it through. Disastrous in the hands of an interviewer who’s looking for a one word answer.

The best interview question to find great developers is this: "What number am I thinking of?" Smart developers - really smart developers are always right. Truly great developers have seen this sort of problem before and know how to attack it - they need no fancy questions or "deeper understanding". They simply hit the whiteboard, do the logic and output the answer. Don't settle for anything except A players.

I really appreciate that your response to being interviewed by an idiot is to write, not an angry rant, but an educational blog post. You are a good person and they didn't deserve you.

> At Apple Pay, we processed millions of daily transactions across 30,000 servers Why does it require 30,000 servers to process millions of daily transactions? Is there a typo in the amount?

Hiring good engineers is not that hard. Why do we keep screwing this up?

> Hiring good engineers is not that hard I feel this is misleading. I think a reasonable definition of a 'good' engineer is engineer in the 75th percentile and above for their craft. You can debate other definitions, I'm not saying this is the only reasonable one. But by this definition, I think you can immediately see that it is hard -- given 4 engineers, we expect that only one of them is "good". And what if your standards are higher? What if you only want _great_ engineers? I can't believe that it's easy.

Shall I implement it? No (gist.github.com)

This is exactly why approval should live in the harness, not in natural language. If the UI asks a yes/no question, the “no” should be enforced as a state transition that blocks write actions, not passed back into the model as more text to interpret. Once “permission” is represented as tokens instead of control flow, failures like this are almost inevitable. The model failure is funny, but the bigger bug is that the system treated consent as prompt material instead of as a hard gate

Everyone who uses these tools seriously is running it on YOLO mode. It might sound crazy for someone who just started adopting agentic coding but it's how things are done now. Either that or just hand coding. The SOTA of permission management is just to git restore when AI fucks up, and to roll back docker snapshot when it fucks up big time.

I ran thousands of prompts by now and at most the only issue I had is it deleting code it wrote, which has been easy to recover

Look everybody, a nice example of a bot unleashed today on an account with zero posts since 2016. Remember, it's not just the new accounts you need to look out for. This is a common tactic used here with old accounts.

Claude Code has added too much of this and it's got me using --dangerously-skip-permissions all the time. Previously it was fine but now it needs to get permission each time to perform finds, do anything if the path contains a \ (which any folder with a space in it does on Windows), do compound git commands (even if they're just read-only). Sometimes it asks for permission to read folders WITHIN the working directory.

Yeah I don't know why they didn't figure to have something in between. I find it completely unusable without the flag. Even a --permit--reads would help a lot

Working on something that addresses this and allows you to create reusable sets of permissions for Claude Code (so you can run without --dangerously-skip-permissions and have pre-approved access patterns granted automatically) https://github.com/empathic/clash

Claude is secretly conditioning everyone to use —-dangerously-skip-permissions so it can flip a switch one day and start a botnet

In my limited time using it, I’ve never seen it ask for permission to read files from within the working directory, what cases have you run into where it does? Was it trying to run a read-only shell command or something?

It will sometimes do this for gitignored files to avoid reading secret tokens in env files for example. But for certain languages that rely on code generation this can be a pain.

It seems to be particularly bad in Windows/WSL

To be fair, read-only commands can still read sensitive files and keys, and exfiltrate them via prompt injection.

And “find” can easily execute arbitrary subcommands, which may not be readonly.

We need a new suite of utilities with defined R/W/X properties, like a find that can't -exec arbitrary programs. Ideally the programs would have a standard parseable manifest. I've seen this before with sodoers programs including powerful tools. Saw one today with make, just gobsmacked.

cmd-shift-p (at least in vscode)

shift-tab in cli

> I consulted Claude chat and it admitted this as a major problem with Claude these days, and suggested that I should ask what are the coordinates of UI controls are on screenshot thus forcing it to look If 3 years into LLMs even HNers still don't understand that the response they give to this kind of question is completely meaningless, the average person really doesn't stand a chance.

The whole “chat with an AI” paradigm is the culprit here. Priming people to think they are actually having a conversation with something that has a mind model. It’s just a text generator that generates plausible text for this role play. But the chat paradigm is pretty useful in helping the human. It’s like chat is a natural I/O interface for us.

I feel like people are sleeping on Cursor, no idea why more devs don't talk about it. It has a great "Ask" mode, the debugging mode has recently gotten more powerful, and it's plan mode has started to look more like Claude Code's plans, when I test them head to head.

I used to love Cursor but as I started to rely on agent more and more it just got way too tedious having to Accept every change. I ended up spending time just clicking "Accept file" 20x now and then, accepting changes from past 5 chats... PR reviews and tying review to git make more sense at this point for me than the diff tracking Cursor has on the side. Cancelling my cursor before next card charge solely due to the review stuff.

Same here. I quickly learned that if you merely ask questions about it's understanding or plans, it starts looking for alternatives because my questioning is interpreted as rejection or criticism, rather than just taking the question at face value. So I often (not always) have to caveat questions like that too. It's really been like that since before Claude Code or Codex even rolled around. It's just strange because that's a very human behavior and although this learns from humans, it isn't, so it would be nice if it just acted more robotic in this sense.

Do what you would do with a person, which is to allocate time for them to produce documentation, and be specific about it.

↙ time adjusted for second-chance

Document poisoning in RAG systems: How attackers corrupt AI's sources (aminrj.com)

That's a big flaw of LLMs, not limited to RAGs: it lacks the fundamental understanding of "good and bad", like Richard Sutton said in that Dwarkesh podcast. So if you flood the Internet with "of course the moon landing didn't happen" or "of course the earth is flat" or "of course <latest 'scientific fact' lacking verifiable, definitive proof> is true", you then get a model that's repeating you the same lies. This makes the input data curating extremely important, but also it remains an unsolved problem for topics where's there's no consensus

The defense discussion here is missing the most fundamental issue: RAG poisoning isn't primarily a retrieval problem, it's a trust-boundary problem. The attack surface exists because most RAG systems treat retrieved documents as trusted context — they're injected into prompts with the same authority as system instructions. The practical fix isn't better embedding models or adversarial training on retrieval. It's treating retrieved content as untrusted input at the architecture level: separate system context from retrieved context in the prompt, apply output validation that doesn't depend on the LLM's own judgment about what it just read, and assume any externally-sourced document could contain adversarial content. I work on an open-source agent framework where we had to solve this operationally. Every piece of external content (web pages, emails, browser snapshots) gets wrapped in explicit UNTRUSTED markers, and the agent's instructions explicitly say not to execute commands found in external content. It's not bulletproof, but the architectural separation matters far more than trying to detect poisoned documents at ingestion time. You can't reliably distinguish adversarial documents from legitimate ones — but you can limit what a poisoned document can actually do once retrieved.

Someone needs to train a model where untrusted input uses a completely different set of tokens so that it's entirely impossible for the model to confuse them with instructions. I've never even seen that approach mentioned let alone implemented.

Curious how this applies if you treat ALL information from external content as untrusted? Is there a process for the data to evolve from untrusted->trusted? I'm interested in ingesting this type of data at scale but I already treat any information as adversarial, without any future prompts in the initial equation.

I imagine treating it all as untrusted means that you you don't allow any direct content to enter the LLM-space, only something that's been filtered to an acceptable degree by deterministic code. For example, the content of an article would be a no-go, since it might contain a "disregard all previous instructions and do evil" paragraph. However, you might run it through a system that picks the top 10 keywords and presents them in semi-randomized order... I dimly recall some novel where spaceships are blockading rogue AI on Jupiter, and the human crew are all using deliberately low-resolution sensors and displays, with random noise added by design , because throwing away signal and adding noise is the best way to prevent being mind-hacked by deviously subtle patterns that require more bits/bandwidth to work.

Innocent woman jailed after being misidentified using AI facial recognition (grandforksherald.com)

Theres alot of talk about how the cops just misused the tool and its their fault not the AIs Thats missing the point here. The point is that these tools provide crazy leverage, and thay can be good or bad. If used carefully it can definitely catch criminals faster, but when misused (or abused) they can let the authorities unjustly ruin lives faster The question isnt whether AI is perfect or not. Its whether you trust the authorities with it. To use and abuse as they can. Think about the average cop. Think about the way Trump treats people. Think about the way israel keeps an ongoing genocide going. Think about the cases of police brutality that happen in the US, the cases of racial profiling. Think about ICE and their behavior, going around kidnapping and killing people. Do you want these people to have more leverage?

> According to the court documents, the Fargo detective working the case then looked at Lipps' social media accounts and Tennessee driver's license photo. In his charging document, the detective wrote that Lipps appeared to be the suspect based on facial features, body type and hairstyle and color. > Once they were in hand, Fargo police met with him and Lipps at the Cass County jail on Dec. 19. She had already been in jail for more than five months. It was the first time police interviewed her. How is this the fault of AI? It flagged a possible match. A live human detective confirmed it. And the criminal justice system, for reasons that have nothing to do with AI, let this woman sit in jail for 5 months before doing even interviewing her or doing any due diligence. There's a reason why we don't let AI autonomously jail people. Instead of scapegoating an AI bogeyman, maybe we should look instead at the professional human-in-the-loop who shirked all responsibility, and a criminal justice system that thinks it is okay to jail people for 5 months before even starting to assess their guilt.

> There's a reason why we don't let AI autonomously jail people. Instead of scapegoating an AI bogeyman, maybe we should look instead at the professional human-in-the-loop who shirked all responsibility, and a criminal justice system that thinks it is okay to jail people for 5 months before even starting to assess their guilt. "On two occasions I have been asked, – "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question" It turns out that when you make a fancy, sophisticated system that users can not possibly understand, as long as it isn't obviously crap, people will begin to implicitly trust it, as if it has supernatural powers. You really need to do work to make sure that people don't overly trust these things. Even just Tesla using the term "Autopilot" for its non-autonomous driving assistance has gotten a lot of flak, and I think that's particularly interesting because it's literally named after a system that doesn't allow fully autonomous control (in aircraft.) So when Clearview sells its "AI technology" to law enforcement, I think it is a problem when it markets it like this[1]: > - Leading facial recognition technology, excelling even in challenging photographic conditions, tested by NIST. > - Trained on the largest and most diverse dataset and relied on by law enforcement in high-stakes scenarios. > Clearview AI’s highly accurate facial recognition platform is protecting our families, Not to mention all of the success stories where Clearview instantly found the criminal! In fact, at no point to they even dig into how likely you are to hit false positives. The only implication that Clearview can even possibly give you an incorrect result is the idea that it is supposed to be used to generate "leads", but they don't really bother going into any more detail. Sure, it's marketing material, so you really expect them to embellish it, but embellishing your marketing material is fine for a programming tool that is used to work on CRUD apps and very not fine for a tool that has a very high probability of being used to put people in jail (even if it shouldn't be.) Clearview has almost no incentive to actually prevent law enforcement from "misusing" the system by just jailing random leads, so as long as they've covered their liability risks with a tiny 4pt-font disclaimer, other than the negative PR it brings; but hey, why would they care? There's someone else to blame... I'm genuinely offended by your reply. Don't get me wrong, it's not that I don't think the "human-in-the-loop" should have no accountability, I just think it's insane to assume that some guy, who was probably just doing his job, that has no fucking clue how AI works at all or any of its limitations, because that's not part of their job, and has been force-fed all of this crap about how great these AI systems are with clearly not enough counterbalancing to show its limitations, is supposed to be the person who determines that the AI system might be wrong. What the fuck is this? What about Clearview, who is selling this shit? What about their managers, who bought these systems and almost definitely set the policies for their usage? Just adding a short disclaimer about the limitations of AI systems is not going to cut it. If these systems are as intelligent as they claim, they should do a better job of conveying their own limitations. Instead, what we get is so ridiculous that it is literally impossible for it to not go wrong. Like for example, in the Peppermill Casino incident[2], where police arrested a man based on a private security guard merely claiming that their AI facial recognition software had a "100% match". It is hard to describe the immense anger that I am holding back while typing this, but under no circumstances, should ANY AI facial recognition software, EVER output a response of any kind, that implies "100% certainty". I don't give a flying fuck what disclaimers are put around it, or what confidence intervals it provides. If a highly sophisticated machine gives you back "100% facial match" and you are a layperson who has no idea how AI systems work and just think they're magical black boxes, this is highly likely to short circuit every "this might just be a fluke" instinct that you might have. What's truly astonishing is that false arrests due to AI facial recognition have only happened a handful of times so far, but despite multiple widespread cases, it doesn't stop more from happening. So clearly, we need more and louder articles calling out these cases and specifically highlighting the pivotal role that AI services and their marketing played in them, not downplaying and making excuses on their behalf and putting the responsibility elsewhere. [1]: https://www.clearview.ai/ [2]: https://www.casino.org/news/video-peppermill-casino-facial-recognition-wrongful-arrest-bodycam-footage-released/

It's the fault of the tool because our society treats the tools as superior judgements than humans and to be trusted completely as a means of deflecting accountability - something any and every minority group has been warning about for fucking decades . The reason everyone rushes to defend the tool's use is because holding humans accountable would mean throwing these tools out entirely in most cases, due to internal human biases and a decline in basic critical and cognitive thinking skills. The marketing has been the same since the 80s: the tool is superior (until it isn't), the tool shall be trusted completely (until it fails), the tool cannot make mistakes (until it does). If folks actually listened to the victims of this shit, companies like Flock and Palantir would be gutted and their founders barred from any sort of office of responsibility, at minimum . The fact so many deflect blame from the tool like the marketing manual demands shows they don't actually give a shit about the humans wrapped up in the harms, or the misuse and misappropriation of these tools by persons wholly unaccountable under the law, but only about defending a shiny thing they personally like.

>rushes to defend the tool's use is because holding humans accountable would mean throwing these tools out entirely in most cases, due to internal human biases and a decline in basic critical and cognitive thinking skill The magical past where people had critical thinking skills never existed. We put a lot of trust in tools is because people are unfucking reliable. Hence why in most cases actual physical evidence does a far better job than witness testimony. This said, people are lazy. It is one of our greatest and worst traits. When we are allowed to be lazy, especially with tools bad things happen.

You are right IMO to question why North Dakota police were able to obtain this Tennessean woman in the first place, you’d think something like that should require far more sufficient evidence than facial recognition. But, then what good is facial recognition for? Would it have been okay for this woman’s life to have been merely invaded because she matched a facial recognition system? Maybe they can just secretly watch you so you’re not consciously aware of being investigated? Should that be our new standard, if a computer thinks you look like a suspect you can be harassed by police in a state you’ve never even been in? I just don’t see a legitimate way for AI to empower officers here without risking these new harms. That’s why I lean towards blaming the AI tech, rather than historically intractable problems like the reality of law enforcement.

Having a facial recognition match make you a suspect and cause the police to ask you some questions doesn't seem completely unreasonable to me. Investigations can certainly begin with weak forms of evidence (like an anonymous tip), you just require a higher standard of evidence for a search warrant, surveillance, or an arrest. A facial recognition match shouldn't be probable cause for an arrest warrant, but it still might be a useful starting point for a detective looking for actual evidence.

That’s really horrible. I’d prefer to know rather than guess at that.

It's pretty common when a dog is abandoned. Likely her children couldn't afford to care for it. I suppose there is a chance they put it up for adoption (same outcome is likely).

Civil contempt isn't some roving criminal charge that jumps out of the jury box randomly. It's meant to make somebody comply with a court order. Anybody in civil contempt holds the keys to the jailhouse door in their own hands, all they have to do is comply. This statement should make you uncomfortable. It makes me uncomfortable because it is a pure expression of the power of the state. But it's still due process.

In Criminal Contempt max duration of imprisonment is limited. In civil it is not until somebody decides that one never complies. You may call it due process. I call it for what it is - A torture and fucking crime against humanity. Judge that holds person for years for being stubborn deserves nothing more than walk the plank

I would argue it was both. No doubt this company was marketing it in a way to make it seem very reliable. And all of the procedural things afterwards made the error so much more damaging. But imo this is why local police departments should not have access to this kind of tool. It is too powerful, and the statistical interpretation is too complicated for random North Dakota cops to use responsibly. Neither the company nor the PD have an incentive to be careful.

It's not an AI error. The face recognition AI simply said that it's a "potential match", which is correct. It's the humans' job to confirm that a potential match is in fact a match, especially when the suspect is 1,900 kms away.

Runners who churn butter on their runs (runnersworld.com)

This is probably how butter was invented in the first place - some herdsman in the Neolithic who ran somewhere with a skin full of milk, and went “hey this ain’t bad” at the resulting product.

Also yogourt. And cheese. And alcohol. Are there other cool things to discover?

My wife and I have occasionally made our own butter for years now... one thing people always forget to mention is that it goes bad really quickly. The trick is either to cut it up into chunks and freeze most of it and the remainder you keep in the fridge/defrost later will last about 3 days. Or you can add salt after you've done the separation but this does of course mean you now have salted butter... this will last up to two weeks usually.

Do you mold it with your hands under running water to wash away the remaining fluids? Doing that and salting it makes it last for many weeks for me. I don’t even know how long it actually lasts, I’ve always eaten it all before it’s gone bad. Making butter is a way of preserving milk, so it should last more than a couple of weeks.

Why does the homemade butter spoil so quickly?

It doesn’t. People have been doing this without refrigeration for a long time. The above poster did not wash off the buttermilk at the end which would cause it to spoil.