It was probably time to start looking for something else, he said.
Around that time, his wife, Karen, got a call from the owners of what is now the Lititz Family Cupboard Restaurant & Buffet.
They wanted to know if she was interested in managing a gift store attached to another of their businesses, he said.
The Family Cupboard in Warwick Township was not up for sale.
But that conversation led to the Fishers buying the Lancaster County restaurant, which they've run since then, with varying degrees of help from their four children along the way.
Over the years, they've branched out from the core business, most recently and evidently with two locations of the Tomato Pie Café.
One started in Lititz in 2010, and a second one opened in Swatara Township, Dauphin County, last fall.
The idea to open the first Tomato Pie Café grew out of Karen Fisher's creative side, which she previously indulged years ago through her at-home business of making — of all things — fake food.
The products were craft-store items such as realistic-looking apple pies to put in kitchens as decorations, she said.
The Cupboard specializes in Pennsylvania Dutch comfort food, and she had long sought a place to get creative with the menu.
And where'd the name come from?
Tomato Pie is a specialty in the South. Karen Fisher's recipe for it found a place eventually as the name of the café.
"It just sounded unique," she said.
The breadth of the success over the years has come as a surprise, Karen Fisher said, since the family started out with no capital, no budget plan and no formal training. Passion and hard work can go a long way, she said.
Today, Karen Fisher is president and CEO of the cafés, while Chris Fisher is president and CEO of the Cupboard, which also has a retail and wholesale bakery and a catering business.
It was definitely a challenge starting out, Chris Fisher said. But his philosophy is that any business is about how you manage people. Hire workers who know more than you do and manage from there, he said.
Part of how they came to run the business was an extension of how the family ran at home: Everyone chipped in to help get the work done.
Of course, that can cause relationship strains along the way, such as when someone needs to be chosen from among the children to cover for a member of the wait staff who calls off, Karen Fisher said.
In the end, there are more advantages than disadvantages to working with family, and overall it is a beautiful thing, she said.
The most important thing to remember as a mother or father is to make sure you wear your parent hat with your children more than you wear the boss hat, Karen Fisher said.
Established trust is another advantage to working with family, said their daughter, Emily Fisher, who is now creative and marketing director.
It's also important to ensure that the entire relationship doesn't turn into work, Emily Fisher said. Make sure there are experiences together that don't involve the business or talking about it outside work hours.
Several years ago, the Fishers expanded the Cupboard with an on-site bakery that came to supply many of the cafes and coffee shops in the area.
Even the construction work for the renovation was a family affair.
"I remember doing the drywall, I remember doing all of that with (my dad)," Emily Fisher said.
She and her brother, Christopher, work full time in the business.
He handles the finances and until recently managed the Tomato Pie Café in Lititz, Karen Fisher said.
Even the two sons who have jobs outside the family business help out as information technology and general repair support, she said.
Emily Fisher was 13 years old when her family decided to buy the Cupboard, and she has worked pretty much every job in the industry: waitressing, bussing tables, delivering wholesale goods from the bakery — you name it.
At first, her parents' move surprised the rest of the family, she said. But it's definitely worked out for the best.
"They went for it, and I'm glad they did," Emily Fisher said. "They love what they do."