Unlurker|Replay|About
AI benchmarks are a bad joke – and LLM makers are the ones laughing (theregister.com)
•
AI works, the joke is on everyone in denial
•
Any argument to back up your claim?
•
The fact that I could type this exchange into an LLM and get a list of reasonable arguments.
•
lol
•
I've been getting flagged by high-on-their-own-supply AI boosters for identifying that LLM benchmarks have been obvious bullshit for at least the last year and a half. What changed to make "the inevitable AI bubble" the dominant narrative in last week or so?
•
Companies are talking about needing trillions of dollars is why.
•
Definitely one of the weaker areas in the current LLM boom. Comparing models, or even different versions of the same model, is a pseudo-scientific mess. I'm still using https://lmarena.ai/leaderboard . Perhaps there is something better and someone will pipe up to tell me about it. But we use LLMs at work and have unexplainable variations between them. And when we get a prompt working reliably on one model, we often have trouble porting it to another LLM - even straight "version upgrades" such as from GPT-4 to -5. Your prompt and your model become highly coupled quite easily. I dunno what to do about it and am tending to just pick Gemini as a result.
•
Ratings on LMArena are too easily gamed. Even professional human evaluators are quite vulnerable to sycophancy and overconfident-and-wrong answers. And LMArena evaluators aren't professionals. A lot of the sycophancy mess that seeps from this generation of LLM stems from reckless tuning based on human feedback. Tuning for good LMArena performance has similar effects - and not at all by a coincidence.
•
I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mess. Product testing (with traditional A/B tests) are kind of the best bet since you can measure what you care about _directly_ and at scale. I would say there is of course “benchmarketing” but generally people do sincerely want to make good benchmarks it’s just hard or impossible. For many of these problems we’re hitting capabilities where we don’t even have a decent paradigm to use,
•
A/B testing is radioactive too. It's indirectly optimizing for user feedback - less stupid than directly optimizing for user feedback, but still quite dangerous. Human raters are exploitable, and you never know whether the B has a genuine performance advantage over A, or just found a meat exploit by an accident. It's what fucked OpenAI over with 4o, and fucked over many other labs in more subtle ways.
Things I've Heard Boomers Say That I Agree with 100% (wildingout.substack.com)
•
Someone called me a boomer a few months ago and I was deeply offended. I’m gen x. The person was gen z. I was like dude do you not know what a boomer is? They had no idea that baby boomers were the generation born after world war 2. To them “boomer” was a slur applied to anyone older than them. I agree that actual boomers tend toward some unfortunate beliefs and characteristics, but “boomer” has become an ageist slur. As a 50+ year old in tech maybe I’m overly sensitive to this?
•
Other than that the author wont survive one minute in China (I doubts they are interested to visit anyway), a lot of the reasons/rationales seem pretty weak. I also hate QR codes as menus, but not because printing a menu is easy, it is the human interaction that suffers. I can spend days without communicating with humans here… which is quite distopic. Although enjoyable every now and then (after a 80 hour work week for example…)
•
What does China have to do with anything in the post? Neither you nor the author would survive a trip to North Korea either, I don't see this as a valuable comment here - it's a complete non sequitur.
•
I still am a fan of QR code menus but the west never executed on it well enough. In China I just order from the menu on my phone. No waiting for front of house. If I do have questions it’s easy to buzz front of house and they will come over. It’s simply China is ahead some areas of tech, cashless, QR codes everywhere, everything is an app.
•
Why even go to a restaurant in person anyway, when food delivery exists? The reason is to be social. QR menus neglect this.
•
LED headlights are fantastic. They work everyday. They never get burnt out. In the spirit of boomers, if they're blinding you, go complain about the lack of inspections in your state. I love LED lights.
•
Agreed, I love my (stock and factory aimed) LED headlights. Haven’t had an issue in 5 years where I would’ve replaced multiple lamps in an older style headlight, which was the case with the Subaru I owned before. LED headlight retrofit kits are probably what should be illegal, that’s where you get the poorly aimed and overly bright headlights.
•
I STILL don’t get this obsession with “participation trophies.” I’ve never seen one, I’ve never met anyone who says they’ve awarded one to someone, and I’ve never met anyone who received one. I’ve seen very minor “door prizes” that say, thanks for attending this event, etc. But this “participation trophy” canard has coasted for 30+ years now.
•
I always have to think about "Eighty percent of success is showing up" when I read about participation trophies. I think it's a good idea to stimulate participation. "Winning" is something you only do if you participate a lot.
•
A similar thread that you should have seen is grade inflation. Most people get A's and don't learn that much, teachers are punished for giving bad grades, a lot of people graduate without much added knowledge or skill. I would prefer no grades, but telling so many people they're doing top notch work when they aren't is a problem.
•
I take it you don't work in the corporate world. I see folks get "participation trophies" all the time, they come in different forms.
•
It’s a sports thing. If you go to some youth sports league, it is common that every kid will get a medal or trophy regardless of which team in the league won or lost. But it also exists for adults. Go to the NYC marathon? Everyone gets a medal. I’ve participated in a lot of organized bicycle rides. The rides aren’t even competitive like the marathon is. They are not races. But at the finish line everyone gets a medal regardless of what distance they rode, or how quickly. The harsh truth about the participation trophies is that boomers complain about them the most, but they are the ones responsible for them! I’m a millennial. I remember being in a youth basketball league in middle school. Our team did not win. At the final day, every kid on every team got a tiny trophy. I was very confused by this at the time. I expected only the best team to get anything. But who was running that league and decided to hand out those trophies?! Our boomer parents!
•
"Participation trophy" has become a modern aphorism in many situations where people are rewarded for simply showing up.
•
> Go fellate yourself with a cactus. I don’t think this makes sense.
•
The writing is insufferable.
With a QR menu, if I can also order and pay from them, then they are far superior to having to try to flag someone down. If not its less useful, but it's useful for multiple languages. In all cases it should be a website, not a "download our app" There are no adverts on netflix, and I have a lot of subscriptions, but its still less (pro-rata) than I paid for satelite tv with adverts in the 00s.
•
With a QR menu the restaurant will sell all information they can get from your phone to ad vendors. The only place I have have see QR menus are at national restaurant chains. If a restaurant does not have a printed menu, I leave and go elsewhere. Luckily were I live, QR Menus are very rare.
•
> should be a website Yes, and please make the website one that actually works on mobile. In some restaurants, I've seen the QR code go to a full-page PDF version of the menu. Like, thanks, but I can't read that on a mobile device.
•
There’s something about the inclusion of tech in the ordering of food that takes away from the experience. It turns it into something where the focus is on fast calories per dollar. It might make sense at your regular lunch spot, but not at a place you take a date or meet your friends.
•
So you pay before you get your food? Some boomers might not accept that ;)
•
> There are no adverts on netflix The base ($7.99) Netflix plan does indeed have ads.
Why hasn't there been a new major sports league? (humaninvariant.substack.com)
•
> Fans only want to watch the very best players. Without TV money, new leagues can’t match salaries, and cannot attract top talent. Is there enough "top talent" to fund a new league? Take American football: there are more players in high school, than college, because the game gets faster and better; the same goes from college to the NFL. The game changes, too; so. success in college doe not mean success in the NFL. There have been 88 Heisman Trophy winners; but, only 10 have made the NFL Hall of Fame. (The only two-time Heisman Trpohy winner is not one of those ten.) Did (DO) the XFL and AAF have "top talent"? These were players who were not good to get drafted by the NFL. Sure, some players have from the XFL to NFL; but, that makes the XFL more like a minor league or developmental league -- a notch lower.
•
Can we please stop pretending that WWE (World Wrestling Entertainment) is a sport? Clue: It’s got “entertainment” right there in the name.
•
All sports are entertainment. No one watches the Jets expecting them to win.
•
I assume you’re being facetious, but just to be clear: WWE is scripted, while the Jets have a fair chance to win any given game.
•
But you can bet on WWE games: https://www.sbo.net/wwe/
•
Within kayfabe, WWE is a sport. Outside of kayfabe, it's not. :)
•
Medical dramas are not medicine, courtroom dramas are not court, wrestling dramas are not wrestling.
•
Indian Premier League (Cricket) is not a US league. Some reasons mentioned in the post apply to IPL. And IMO some don't (nerdy billionaires). There's even an acquired.fm episode on it. https://www.acquired.fm/episodes/indian-premier-league-cricket "So successful in fact that it is the fastest growing major sports league in the world, growing 20x in value since 2008 to be worth more than $16 billion today."
•
Is there any market in the world where rival leagues of the same sport compete in the same geographic location? Maybe there aren’t new leagues because Americans don’t care enough about new sports? Further, it’s the teams, not the leagues that make money, which I didn’t pick up in a brief skim of the article. Why would any team want to join an rival league? League monopoly seems like the natural fit, and let the competition happen between teams.
•
Both the NFL (football) and MLB (baseball) formed from competing leagues in the US.
•
Right, but they did merge, and I’d argue it’s because it’s natural for a sports league to be a monopoly. Fans wanna see the best players and teams compete with each other.
•
The popularity of the English Premier League in the US is a factor not considered anywhere in the piece.
•
Looks like they missed Major League Cricket (MLC) founded in 2023
•
The bit about getting the government to invest with a goal that isn't financial ROI reminds me of the Olympics during the cold war. All that investment to "prove" the dominance of one economic system or another.
•
The scourge of unfettered capitalism. Everything boils down to "how much money does it make" Small-time capitalism is good. But beyond a certain size, it falls apart and can even become quasi-governmental. When is somebody going to blend the best part of all political isms to get closer to the best possible form of society
•
Best for who though? There just is no one size that fits all. Personally I would be happy in a more equal socialistic society but I recognize that others prefer the individuality and excitement of playing the game of chance in our capitalist system.
Are they just going to gloss over Twitch and video gaming like that? I'm fairly sure it qualifies if you don't specifically exclude their online only influence.
•
ESports is a sizeable chunk of the article?
•
And yet it's not a "new major sports league" ?
•
"Esports" is not a league. That would be like saying "sports" is a league. There are leagues around some games (like the ones mentioned in the article). There are also events with "league" in the name that are not really leagues (like ESL Pro League). In any case, none of them are financially successful in the US.
•
Not by $$$, which is the main focus of the article. In the second table, LoL esports is explicitly highlighted as a success by mindshare, but not profitability. And below that: > LoL Esports: loses hundreds of millions of dollars annually, exists solely as a marketing mechanism to get people to play the actual game
•
There's a weird perception that's floating out there claiming that video gaming cannot be "a sport", just because it's more mental than physical (although even that varies per game, for example high-level Guitar Hero players require extreme endurance and stamina). I also would wager a non-zero percentage of people with such mindset would also deny that chess is a sport, unless they grandfather it in due to the sheer age of chessdom. It doesn't help that there is a partially overlapping mindset that believes that video games are for kids, something to grow out of. Since the vast majority of competitive gamers at the extreme top end are young adults, that theory is incorrect, but those who happen to be in the center of that Venn diagram would probably also deny that successful Twitch streamers have a "real job", etc.
•
This doesn't strike me as that weird. Playing a guitar also takes extreme stamina, for instance, but it takes a great deal more technical skill to hit the rhythmn and fingering. (Or at least if that isn't the case, that's certainly the perception.) Many guitarists also write their own music, and the truly impressive ones improvise. Many sports require full-body coordination. I would also consider, say, a skilled ballet dancer much more obviously impressive and intriguing than being the best video gamer in the world. The only reason one might not consider it a sport is that it's not as competition-oriented (and I may be completely wrong in that; I can't feign deep knowledge of ballet). I don't think it's disparaging to not consider video games a sport, either, it just seems like a category error. Much of the skill is simply not visible to people not intimately familiar with the game, the mechanics, and what might be difficult about it. I would put it in the same category as, say, live coding, or watching an expert artisan at work. I also can't fathom, say, the skill it takes to operate a crane as a world-class expert, or ice carving, or fixing a mechanical watch, or blowing glass. Frankly, I also don't consider most streaming to be a "real" job. But that's probably related to the fact that I don't respect most content that emerges. But you could say the same thing about podcasting, or being an opinion columnist, or a pundit/talking head. This doesn't mean it doesn't take skill or effort; I just think it's not producing much of value.
My friends and I accidentally faked the Ryzen 7 9700X3D leaks (old.reddit.com)
Journalism is thrash and journalists are happy to lie about anything if they have the smallest amount of plausible deniability.
•
> Journalism is thrash and journalists are happy to lie about anything if they have the smallest amount of plausible deniability. Funny, if you see the NYT lawsuit against OpenAI, they act like they are still doing serious journalism.
•
Let me try one: Software is trash and programmers are happy to lie about anything if they have the smallest amount of plausible deniability. See how you sound?
Tech journalism was always infamously bad.
•
If a tech journalist actually understood their subject matter at a competent level, they wouldn't be a journo and go out and get in job in that subject matter dramatically increasing their salary. Those that can't do, teach. Those that can't teach, become journalists???
Well it’s not so far fetched if the friends are people you studied with and have common interests with, but don’t currently work together with. I have good friends that love to discuss highly technical topics over a beer or whiskey.
•
“My friends and I share interests but they can’t all talk about the relative pros/cons of full frame vs. cropped sensors in digital cinema with me” is kind of the framing here if that makes sense.
↙ time adjusted for second-chance
Dark mode by local sunlight (2021) (ctnicholas.dev)
•
Syncing dark mode or f.lux or OS night mode to local sunlight is nice if you live near the equator, but for folks living in the far north, it actually makes seasonal changes so much worse. Let the user decide the schedule, please.
Please use `prefers-color-scheme` to respect users' environments and default preferences instead. It's also worth noting that, for some people, this is more than just a cosmetic choice.
•
From TFA: >Automatically setting the theme is a nice touch for most, but accessibility comes first, and if the user has a preference, it's always right to respect their choice. >You can check for users' colour scheme preferences in CSS & JavaScript with the following snippets:
•
I'm glad it's mentioned, but I was trying to say "never do the thing TFA recommends" in a less harsh way, and inadvertently dulled my point.
This should be an OS feature and apps should just use the system theme.
•
Both iOS and Android allow you to set dark mode by some schedule, and that's conveyed by their respective browsers to websites.
 Top