Dave Mackay scored Saints' first Scottish Premier League goal of the season to give the Perth men their first win at Parkhead in 13 years.
Peter Enckelman saved Kris Commons' spot-kick in the third minute to enrage the Hoops boss, whose mood worsened when Mackay's deflected shot on the hour mark sealed a miserable day for the Hoops, who hit the woodwork through Commons and substitute Paddy McCourt during an afternoon of missed chances.
Lennon, who confirmed reported interested in Manchester City defender Wayne Bridge, was raging afterwards.
"I am angry, really angry," he said. "We threw it away. We missed clear-cut chances, were complacent and wasteful in front of goal
"The longer the games goes on and it's nil-nil, you give the opposition encouragement.
"I'm fed up with it and fed up with us missing penalties, that would have given us a real good start.
"In the final third again we were awful. It was immature. That is one way to describe it
"Paddy gets in and he has the whole goal to aim at and he hits the post and it bounces out to James Forrest and he panics and hits the defender.
"So there is a bit of that in our play at the minute."
The Irishman continued: "My back four gets the blame when a team scores and they don't deserve that.
"My front six were bang average and I am more angry with them than anyone else.
"We hit the post a couple of times but we missed simple chances.
"We might not have deserved the result but we got it and if you don't take your chance that's what happens in football."
Lennon did not reserve his criticism for his players and indulged in some introspection.
He said: "Maybe I need to look at myself as well. I am not immune to criticism obviously.
"If there is any comes my way I will take it."
Lennon was perplexed by Commons' sideways run-up to take his penalty.
"I don't understand why he shaped up to take it that way because when he practises in training he has a totally different run-up," he said.
"He has taken two steps to the right and I don't know why and he's slipped when he has take it.
"It was a pretty poor penalty to be fair.
"That's two we have missed this season. I don't know how many we have missed, it might be six out of nine, it is nowhere near good enough.
"When you get a penalty I expect us to be one-nil up."
Lennon, who revealed Charlie Mulgrew came off at half-time with a hamstring injury, hopes to have Kelvin Wilson, Beram Kayal and Gary Hooper back for the second leg of their Europa League play-off against Sion in Switzerland on Thursday night.
"I am looking for a huge reaction from everybody, they need to toughen up mentally," he said, before refuting speculation that McCourt has asked for a transfer.
He said: "Paddy hasn't asked for a transfer so I don't know where that has come from.
"It's unsettling for him and its people making trouble for him.
"We have made an inquiry for Wayne Bridge but it hasn't gone further than that."
Saints boss Derek McInnes was determined to savour the result and put it down to the experience he had in his side.
The former Rangers and Dundee United midfielder said: "We have to enjoy today.
"I have left here with my tail between my legs often enough as a player and as a manager in the past.
"I think we had a more experienced team than Celtic and it is not often you can say that when you play against the Old Firm.
"I thought the penalty was harsh but we are lucky to have Peter Enckelman, he has been a different class for us."
Source: Team Talk
Source: Team Talk