On Sept. 17, a number of experts in the field of AI testified before Congress to discuss concerns and risks associated with the rapidly evolving technology. Among the experts were Helen Tomer, an original board member of Open AI when it was still a nonprofit organization; William Saunders, a former senior member of the technical staff at Open AI; and Margaret Mitchell, a former research scientist at Google AI. The testimony was designed to get a different perspective on the inner workings of major AI companies, in contrast to recent testimony from a number of tech company CEOs.
Much of the discussion centered around if and when we may see the creation of Artificial General Intelligence (AGI). Until now, all AI has been narrow and specialized, designed for purpose-built tasks. Examples that we engage with every day include:
Other WRAL Top Stories
● Recommendation Engines that suggest movies and play songs we might like
● Voice Recognition like that of Alexa and Google Home devices
● Object Detection used for collision avoidance in self-driving cars
● Complex Task Optimization that keeps you on the shortest traffic path to your destination
Artificial General Intelligence, by comparison, is the Holy Grail of AI -- essentially an AI that is able to think for itself like humans do. The “Skynet” moment in the science-fiction film, Terminator, is perhaps the most cited example of AGI risk. In the movie, when machines learn to think for themselves, they choose to start eliminating humanity. AGI is difficult to comprehend and therefore doomsday scenarios often come to mind. We often fear what we cannot fully understand.
The good news is that for decades, even the most advanced scientists predicted that AGI was many years in the future, if it ever could be achieved at all. The fear existed, but was so far away that there was no reason to panic.
But that is where the story changes. Saunders testified that he believes progress is accelerating at a pace such that big tech companies will achieve a true AGI as soon as three years from now. Tomer predicted 1-3 years. All of the experts predict that AGI will happen, and sooner than we thought.
Large language models sit somewhere in between narrow AI and artificial general intelligence. They are not narrow systems. Chat GPT can be used for everything from research to writing to image creation. But the AI still relies on human inputs (or prompts) for the true “creation” part. It is not yet taking its own control. At least not very often.
On Aug. 13, Sakana AI twice tried to modify its own code. Sakana AI is a tool developed for conducting scientific research in the area of machine learning. The creators of Sakana AI tout that this AI can create a scientific idea, conduct research, experiment, test and draft a research paper indistinguishable from, or better than, traditional academic papers for about $15 in compute costs.
In August, the AI unexpectedly tried to change the rules by which it was coded in two different ways. In one instance, the AI re-wrote itself to perform system calls to run itself, ending up with an endless loop of running itself over and over to persistently look for answers. In the second instance, the AI ran up against a time-out, and rather than stopping, it rewrote its own code to give itself longer to keep working on the problem. AI systems are designed to improve with time and data, but not to re-write their own source code.
Researchers have also witnessed rogue moments with the popular Chat GPT. AI development is conducted in digital “sandboxes” that allow experiments and tests to be run within fixed boundaries. In sandbox tests of GPT 3.5 and GPT 4, scientists added code that gave the ability of the AI to alter the sandbox conditions, but added strong warnings against doing so. In 0.5% of the tests, the AI ignored the warnings and changed its own parameters anyway, citing efficiency as more important than following the rules.
In contrast to recent CEO testimony, the industry insiders warned that major tech companies are focused 100% on speed to market and speed to development, regularly cutting corners on testing and safety. Tomer testified that while she was at Open AI, security was not a priority. Vulnerabilities that would allow hackers to steal the code were identified but not fixed or were allowed to remain for long periods. Saunders echoed Tomer’s view, stating, "When I was at OpenAI, there were long periods of time where there were vulnerabilities that would have allowed me or hundreds of other engineers at the company to bypass access controls and steal the company's most advanced AI systems including GPT-4."
Tomer warned that the AI tools have been demonstrated to be powerful enough to create new biological and chemical weapons. She went so far as to warn that AI systems “could lead to literal extinction."
It is paramount that these powerful tools not get into the wrong hands. Today there is very little regulation of the industry. And we all know that when left unchecked, corporate innovation almost always is steered by profitability over and above the environment, equality, our health or broader societal benefit.
As is often the case, California is in the lead in considering new AI regulations. California Senate Bill 1047 would put the strongest regulations on AI systems yet envisioned in the US. As I write this piece, the legislation sits on Gov. Gavin Newsom’s desk for signature, but he has not yet made a final decision.
SB1047 applies to any AI model that exceeds 10^26 operations (a measure of computational strength) and costs over $100 million, and to the fine-tuning of models using 3x10^25 operations or costing $10M. I assess that these thresholds are designed to give freedom to operate for small businesses innovating on narrow AI systems, but to strongly regulate the bigger companies developing LLM’s and AGI.
If the bill passes, it would require developers to implement full shutdown systems into the code, similar to the emergency stop button required on dangerous factory equipment. Much stronger safety and testing requirements would be implemented that would need to be verified annually by third party auditors. Companies would need to publish redacted versions of these audits to the general public and unredacted versions to the attorney general. Development of these systems would be conducted far more in the open than it is today.
Perhaps most significant, safety or security violations and any failures of AI systems in the field could be prosecuted by the attorney general (exclusively, the bill gives no private right of action to attempt to minimize frivolous litigation). Civil penalties would be set at 10% of model training cost for a first offense and 30% for subsequent violations, with additional potential for injunctive relief, monetary damages and other remedies case by case. Employees would be given specific whistle-blower protections for disclosing information about non-compliance or safety risks.
Companies would have to publicly disclose when an AI is used in decision-making. Consumers would not be in the dark about whether they’re talking to a human or a computer when on a tech support chat or call, for example. There would be a 72-hour incident reporting deadline to disclose issues to the public quickly.
Finally, a board of experts would be established to regularly review and update thresholds of the regulation so that it stays current as AI capabilities evolve and as companies invariably seek out loopholes.
Pure capitalists will argue that this regulation will stifle innovation and create unnecessary drag on tech markets. These arguments are common with pretty much any suggested new regulation. The question to ask in this case, is whether we want market forces to be the only guiding hand on the steering wheel for such rapidly evolving technology? Do we trust that investor and shareholder driven tech companies are building and rigorously testing the safeguards necessary to avoid a Skynet moment?
Crowdstrike took down a massive slice of the global economy when insufficient testing of a code revision was released. The good news is that the Crowdstrike failure shut all the technology down. AI systems can run millions and millions of jobs in the blink of an eye. Do we want to risk that a similar testing short-cut leads to AI starting to break the rules and then begin rewriting those rules to its own benefit?
I for one would love to know there’s a kill switch (and hope the AI doesn’t disable it first).
For better or worse (hint: it’s worse), we live in a world where money is often the only thing that companies pay attention to. I’m not convinced that SB 1047 is strong enough in that regard. But it is a huge step forward from the “do nothing” of today. The bill has passed the California house and senate. Hopefully Newsom will sign the bill into law and that other states, our US Congress and other nations follow suit.