As with most things…it depends.
How many years do you have left on your mortgage?
What interest rate are you paying vs the current rate?
What is your motivation for doing either?

If you are paying 6% and the current rate is 4% then it probably makes sense to refinance. However, if you have a 4% note…that is pretty cheap money. It may make more sense to use your money to pay off something with a higher interest rate.

Keep in mind if you don't have a mortgage many of your tax deductions go away.

Everyone's situation is different.
Talk with a trusted mortgage advisor to see what is the best option for Refinance your Mortgage.