And no - not every development task can be done reliably on a local setup, I know that - but a huge amount of the chatbot usage that people do could easily be run locally on a modest setup, dispersing the energy usage and heat disspation to a point where it does not become a localized problem any longer.

There is still a major issue with training on illegally obtained data, but twisting LLM answers to fit a particular political world view would at least be much more difficult. You could either use an abliterated version of the model (although that comes with its own can of worms, security wise) or simply use another model - there is a huge amount of very capable models for just about every possible scenario, and with the latest Gemma and Qwen models, most people will be able to find something that suits their need and hardware.

I run models with everything from 2B to 31B active parameters on a humble Radeon780M iGPU on my Ryzen7-based PC (granted, with 64 GB of system RAM), and many will generate answers faster than I can read them. What we do need, though, is a way to make the setup procedure a lot easier and with much better help for new users. We already have local systems like AnythingLLM and LocalAI that takes care of the infrastructure for you, but what model to download and use for what scenario is as much of a learning curve as the initial setup.

Browsing models on Hugging Face is intimidating at first, and model descriptions are way, way, WAY more technical than most users have the knowledge to appreciate, and if AI in general are to be much more distributed and local, that needs to change.

For example, prompting AI to write a program that does something is bad. Even if prompting it to do smaller things like write a function that does something small is not ideal.

However, asking AI what command is ideal for performing a task then the programmer reads the docs on that command and implements it, is safe and helpful.

In any case the programmer should not take the AI as a source of truth. The AI provided program or function will most likely be buggy and/or insecure. An AI provided command recommendation can still be complete bullshit.