Right now, companies such as Google, Audi, and Mercedes-Benz are testing the cars in a small number of cities to demonstrate they can be safer than human drivers. They also must figure out what level of risk is acceptable to both government regulators and a potentially skeptical public.
Government statistics show that human mistakes are responsible for 94 percent of the 33,000 traffic fatalities each year. Autonomous cars won't get drowsy, distracted or drunk, so in theory they could eliminate those mistakes and save an estimated 31,000 lives a year.
But as a Valentine's Day fender-bender involving a Google autonomous Lexus and a public bus shows, cars that drive themselves can make mistakes.
"We cannot expect any technology, any solution to be perfect all the time," says Raj Rajkumar, a computer engineering professor at Carnegie Mellon University who has led autonomous vehicle research for 15 years. "We live in a very uncertain world where lots of things happen."
Given that, regulators and would-be passengers may have to accept that the cars will cause a limited number of crashes, including deadly ones, if overall they save thousands of lives.
"We should be concerned about automated vehicles," says Bryant Walker Smith, a University of South Carolina law professor who studies the technology. "But we should be terrified about today's drivers."
Google is testing a fleet of 56 autonomous cars on the streets of Mountain View, California; Austin, Texas; and Kirkland, Washington. The cars have driven themselves almost 1.5 million miles, with a person as backup in the driver seat. The company also uses a simulator to test the cars in a variety of scenarios. Other companies such as Nissan, software firm Cruise Automation and parts suppliers Bosch and Delphi also are testing on public roads. Test cities also include San Francisco, Las Vegas and Pittsburgh.
Chris Urmson, head of Google's self-driving car program, wrote in a January blog that during the past two years, drivers took control 13 times when its cars likely would have hit something. He noted that the rate of human intervention is dropping and he expects it to keep falling.
In the bus crash, Google for the first time admitted its car was at least partly responsible. The computer and human driver assumed the bus would yield as the car moved around sandbags. Instead, the bus kept going and the car hit its side. Google has updated its software.
In about a dozen other crashes on city streets, Google blamed the human driver of the other vehicle.
Google wants to make cars available to the public around the end of 2019, assuming its data shows the time is right for deployment.
A Virginia Tech University study commissioned by Google found that the company's autonomous cars crashed 3.2 times per million miles compared with 4.2 times for human drivers. But the study had limitations. The human figures were increased to include an estimate of minor crashes that weren't reported to police. All autonomous car crashes in California, however, must be reported. The study also didn't include potential crashes that were avoided when human backup drivers took control.
U.S. traffic deaths have declined steadily for most of the past decade, from 43,510 in 2005 to 32,675 in 2014. But estimates show they spiked 9 percent in the first nine months of last year due mainly to increased miles traveled, texting and other distractions.
If autonomous cars are the answer to sharply reducing those figures, they'll first have to gain the public's trust. A January poll by AAA found that three-quarters of U.S. drivers are "afraid" to ride in an autonomous car. A University of Michigan poll found similar results in Japan, China, India and elsewhere.
Unlike Google, which wants to test cars without human drivers, automakers and parts companies are rolling out autonomous features as they are ready.
The AAA poll found that drivers are somewhat comfortable with the features of autonomous driving such as automatic emergency braking. Separate studies have shown those features can cut crashes. The Insurance Institute for Highway Safety says autonomous braking alone would prevent 700,000 rear crashes per year if installed on all cars.
On Thursday, transportation officials and automakers said they agreed to make automatic braking standard in nearly all cars within the next six years.
General Motors has plans to soon test autonomous cars by carrying employees around a technical center near Detroit. Still, John Capp, director of global safety and vehicle programs, sees humans behind the wheel for the foreseeable future.
"We can't afford to tarnish safety by doing experimentation on the roads," he says.
The burden of proof will be on companies to show that the technology is safe, Adam Jonas, a Morgan Stanley auto analyst, told investors recently.
But even with that evidence, some governments may still be reluctant. "No mayor wants to be the first elected official blamed for the death of a citizen by a robot," Jonas wrote.
Yet cities, seeing potential job growth and safety benefits, already are competing for wider use of the cars, he wrote.
In Austin, Mayor Steve Adler says Google cars in his city haven't caused any crashes, and he believes they bring safety benefits. "We don't get perfection with regular drivers either," he said.
Adler sees allowing the cars in small, controlled areas that expand as the cars prove themselves safe.
Colby Huff, a radio host from Springfield, Illinois, wouldn't ride in one. While others would welcome the car handling a daily commute, Huff doesn't think programmers are infallible.
"There's just too much that can go wrong in something that weighs a ton or so," he said. "It's not worth my family's safety to trust a machine."