The tremor struck around one mile from East Foothills in the US state at around 10.30am local time (7pm BST), according to the US Geological Survey.
The quake lasted for several seconds and shakes were were felt in parts of the East Bay and South Bay.
Its epicentre was six miles from San Jose – a city surrounded by the hills of Silicon Valley, a major technology hub that's home to Google and Facebook.
One terrified resident tweeted: "Omfg we just had a mini earthquake in San Jose and I'm freaking out."
Another posted: "Just felt an earthquake in San Jose. Good thing I'm in a 10 storey apartment building!"
A third wrote: "I just felt an earthquake while sitting in my office in San Jose. It was short and not too powerful but still a little scary!"
San Jose is situated precariously close to the San Andreas fault line, an 800-mile fissure that runs almost the length of California.
Scientists agree that large swathes of southern California – including Los Angeles and San Diego – are long overdue a “Big One” earthquake of magnitude 7 or more.
Although scientists have been studying earthquakes for decades, there is still no reliable method of accurately predicting the timing of a major tremor.
The biggest ever earthquake to strike the region in the 20th century was in 1906 off the coast of northern California.
As a result more than 3,000 people died and 80% of San Francisco was destroyed.
