•••••••••••••••••••••••••••••••••••••
> the problem is that AI will soon be better at explaining itself than humans (I assume you mean explain itself to humans here.) I keep seeing people saying this as fact. It might, but it might not, and there are various reasons one might be skeptical that it will, using current techniques. Not least among these is that explaining something well to a human requires a degree of empathy and sophisticated theory-of-mind that LLMs have not yet exhibited yet*, and that is an area where humans are uniquely advantaged by being humans. Someone will almost certainly reply by just stating as fact that they will achieve this goal, or pointing out that past predictions that LLMs wouldn't be able to do particular things have failed, and that's all very will. I certainly wouldn't be rash enough to claim it's impossible, but there are still models of all this available that are consistent with all the successes so far but don't predict "LLMs can do anything and will do everything". We'll find out in due course. * My idle speculation is that this is why LLMs still suck at using the "it's not X it's Y" construct. I doubt they actually use it any more frequently than humans when writing on the same topics, but they use it wrong and it's jarring. This particular rhetorical flourish works when there's some actual reason to think that X is an idea that might actually be present or forming in the mind of someone reading the passage (or in some other mind that they in turn might be imagining, eg. a fictional character), and that explitly calling it out and negating it will have a particular dramatic effect. LLMs probably use it with about the expected frequency, but they use it for that reason (frequency) and then come up with something to put in the X and Y slots. The result often jars because the X it uses just isn't something relevant to the reader's thought process and/or the Y isn't a convincing, persuasive or relevant counterpoint to it. The result is that they stick out and seem more frequent than they are.
•••••••••••••••••••
> From what I know about the academic community (mostly from Twitter and LinkedIn, which I assume are representative), there seems to be near-unanimous agreement that the purpose of math and science research is to gather a large quantity of potentially true factoids in a repository that nobody will actually ever read, in a ritual called “advancing humanity’s[3] knowledge”. That these results are well-written, understandable, pleasant to read or look at, or even completely correct is a secondary concern — in fact, in the pursuit of advancing humanity’s knowledge as quickly as possible, many academics will recoil in anger at the suggestion that it is worth spending any time making results understandable to humans. Am I the only who thinks this is rhetorical? Modern mathematics is so far removed from the base average knowledge (lets just assume high school instead of going to axioms) that the only people who understand most (if not all new) mathematical achievement are the mathematicians themselves, heck not even all mathematicians, it needs to be spread and developed from "hot spots"? Your math results are not "well-written, understandable, pleasant to read or look at" unless you literally operate with two or three levels within the proof paper compared to the average person who probably starts at an order of magnitude levels higher. To the average person, asking a mathematician to explain something is the same as asking the AI/LLM. Both are asking an oracle to dumb it down to their level of abstraction. To the person who are not career mathematicians, both LLMs and mathematicians do the same thing, pump out garbled output that they themselves understand, and to mathematicians, the difference is one probably explains using axioms rather than theorems.
••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
Yes. In that sense it is equivalent to a compression method that uses K-means clustering to do the partitioning. But because there are many more bits of knowledge out there than there are bits in the models you end up with this amorphous smear. I'm sure some of those bits carry more information about some of the concepts but in general - and that's the way it is explained in just about all the papers I've read that I can still comprehend - every weight contributes to the output in a dense model. In an MOE things get more complicated but there within every 'MOE' column the same principle applies. I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights , you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood. It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion it is difficult to give that a lot of weight right now. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. But I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
•••••••••••••••••••••••••
 Top