The city side went down 24-20 and collected a welcome bonus point, but they had many opportunities to make it an even better afternoon.
James Stuffins, Nic Langdon and Rob Jacobs scored Borough’s tries, the latter after a fantastic 70-metre run.
Borough head coach Shane Manning said: “We only had ourselves to blame. With our scrum and lineout misfiring we couldn’t turn our field position into more points.
“The lads were very frustrated at the end because they know we really should have won the game.
Peterborough United should change their winning League One team!
Darren Ferguson says Peterborough United lacked the ambition to stay in the Championship and he's staying patient while waiting for the right job opportunity to come along
Peterborough United's young stars shone to help secure a rare EFL Cup victory
Peterborough United's cross between CMS and Jack Marriott has seen a weight lifted off his shoulders
Supercomputer predicts likely outcome for Plymouth Argyle v Peterborough United, Derby County v Barnsley, Sheffield Wednesday v Charlton Athletic, Bristol Rovers v Oxford United and every other League One fixture
“But there were a lot of positives. We never stopped trying against a very good Kettering side and our work rate was top notch. We have a lot to work on, but we are in a good place after a performance like that. Our backline played really well and all our debutants did themselves proud.
“We will regroup and get ourselves ready for this Saturday.”
Borough host Lutterworth on Saturday (September 11, 3pm) when they will hope to convert pressure into points more regularly.
They pinned Kettering back before Stuffins dotted down for the opening score of the match. The try was converted by Byron Van Uden who then added a penalty give Borough a 10-0 lead.
But the home side dominated either side of the break to move 17-10 ahead.
Borough, thanks to some great work at the breakdown by Josh Myles, Zac McClure and Michael Hall, managed to come back into the game and after some great attacking rugby from the backline, winger Nic Langton squeezed over in the corner for 17-15.
Unfortunately another converted try saw Kettering move far enough clear to make some individual brilliance from Jacobs irrelevant apart from the bonus point.