• 0 Posts
  • 1.47K Comments
Joined 1 year ago
cake
Cake day: February 10th, 2025

help-circle




  • Their resources are focusing on the science and practical applications of machine learning and neural networks, robotics for example.

    In the US our resources are allocated towards fleecing investors with a flashy use-case that doesn’t have much practical use.

    All of these tech companies, who are themselves heavily invested in the LLM bubble, are trying to push LLMs into everything. On top of that they’re using these artificially inflated adoption statistics to project future demand and then using that modeling to justify massive spending on hardware and datacenters, creating all kinds of secondary effects from the collapse of the consumer computing industry to local resource issues around power and water.

    This is all resulting in massive negative sentiment on the topic of ‘AI’ (see: any social media post where AI is mentioned). Secondary effects of this are things like lower rates students entering the CS/ML field. After all, why try for a CS degree if everyone around you is saying that AI will replace all programmers or go into a field that is massively unpopular?

    There’s also the opportunity cost of not putting resources towards other use-cases, like medicine and robotics leading to those fields falling farther behind other countries.

    China has plenty of their own problems, but on this topic they at least have adults in the room making rational decisions. The US’s AI strategy is being decided by whatever sociopaths happen to have won the stock market lottery and the executives operating the companies who can only see as far as their next quarter’s bonuses.










  • Here is the paper: https://ai-project-website.github.io/AI-assistance-reduces-persistence/

    No the test is not training, that’s a weird thing to claim.

    The control group solved 12 questions manually and then the 3 test questions manually. The AI grouped solved 0 questions manually and the 3 test questions manually. One group had 12 more manual math tasks to prepare for the manual math test the other group had 0 and also had to context switch.

    The AI-assisted group was dealt a context switch, which results in a pretty severe performance loss. A context switch causes performance loss of around 40% according to this paper, which was peer-reviewed and published and is also the most cited paper on the topic, in the APA: https://www.apa.org/pubs/journals/releases/xhp274763.pdf

    The AI-assisted group also did not have 12 questions to adjust to the new context, like the control group did. If they wanted to wipe out the context switching performance loss they should have kept asking questions to see if, after 12 questions, the AI-assisted group had a similar performance.

    The switch is what is tested, and you disregard that 2 other tests have shown similar results.

    No, they did not switch what was tested. Here is an image from the actual paper.

    They were given 12 tasks with one group using AI and another doing mental math and then 3 tasks doing mental math. One group had 12 more tasks worth of preparation than the other.

    Nothing, not even the article in theOP, says that they did math and swapped to reading to test.

    They did 3 different experiments, in each experiment they gave 12 tasks and then disabled the AI for one group and gave 3 more tasks as a test. At no point did they ask 12 math questions and then finish with 3 reading questions or vice versa. They did 2 experiments using math tasks and 1 experiment using reading comprehension tasks.

    So one group had 15 math tasks and one group had 12 ‘how to ask an AI’ tasks and then 3 math questions.

    They also did not control for context switching losses, which is a well documented (see the APA paper) effect. The proper control would be to continue asking questions so the AI group also had 12 math tasks before the test.

    There’s a reason that this is published on arXiv and not in a peer-reviewed journal. Designing a poor quality experiment doesn’t tell you anything useful even if you do multiple different versions of the same experiment.

    This paper demonstrates a lack of a proper control group, specifically a failure to control for context switching performance loss.






  • To add to this, we already know that context switching causes a loss in performance.

    A person who’s thinking about how to solve a problem one way and then has to suddenly think about solving it in another way will perform worse.

    https://medium.com/@codewithmunyao/the-hidden-cost-of-context-switching-why-your-most-productive-hours-are-disappearing-43c5b501de19

    The Neuroscience Behind the Pain

    Context switching isn’t just annoying — it’s neurologically expensive. When you shift from debugging a race condition to answering emails, your brain doesn’t simply “change tabs.” It goes through a complex process:

    -Memory consolidation: Storing your current mental model

    -Attention disengagement: Breaking focus from the current task

    -Cognitive reloading: Building a new mental model for the next task

    -Re-engagement: Getting back into flow

    Research from Carnegie Mellon shows that even brief interruptions can increase task completion time by up to 23%. For complex cognitive work like programming, this cost multiplies dramatically.

    Here’s another article from CMU discussing the same thing: https://www.sei.cmu.edu/blog/addressing-the-detrimental-effects-of-context-switching-with-devops/

    What this study shows is that a person who is faced with an unexpected context switch performs worse on a task than a user who has spent the last 12 questions performing the task the same way.

    This exact problem would happen if you replaced AI with a calculator, or made a person swap from using paper to doing mental math. The problem here is context switching, not AI.

    The way to ensure that the problem is AI and not the context switch, would be to continue the quest and see if the first group reverts back to baseline after 12 questions. 12 questions is how long the control group had to become acclimated to the task before their last context swap at the start of the test.

    Also, of note, this is a paper on arXiv it is not published so it has not gone through a peer-review process which would certainly catch the failure to set a proper control group.