What does Scripture tell us?
“Be ye not unequally yoked together.”

This applies first to our religious or ecclesiastical connections. How many Christians are members of so-called “churches,” where much is going on which they know is at direct variance with the Word of God either the teaching from the pulpit, the worldly attractions used to draw the ungodly, and the worldly methods employed to finance it or the constant receiving into its membership of those who give no evidence of having been born again. Believers in Christ who remain in such “churches” are dishonoring their Lord. Should they answer: “Practically all the churches are the same, and were we to resign, what could we do? We must go somewhere on Sundays,” such language would show they are putting their own interests before the glory of Christ. Better stay at home and read God’s Word, than fellowship that which His Word condemns.
~ A.W. Pink

It is difficult to be without a church. It is far worse to affirm what is called a church but is in essence, a synagogue of Satan. A true Church is given to the exercising of Church discipline. One of the three marks of a true Church. A true believer may discipline a false church by leaving or not attending its services.

Take a look at what the Belgic Confession teaches:

Belgic Confession
Article 29. The Marks of the True Church.
We believe that we ought to discern diligently and very carefully from the Word of God what is the true church, for all sects which are in the world today claim for themselves the name of church.

We are not speaking here of the hypocrites, who are mixed in the church along with the good and yet are not part of the church, although they are outwardly in it. We are speaking of the body and the communion of the true church which must be distinguished from all sects that call themselves the church.

The true church is to be recognized by the following marks:
1. It practices the pure preaching of the gospel.
2. It maintains the pure administration of the sacraments as Christ instituted them.
3. It exercises church discipline for correcting and punishing sins.

In short, it governs itself according to the pure Word of God, rejecting all things contrary to it and regarding Jesus Christ as the only Head. Hereby the true church can certainly be known and no one has the right to separate from it.

Those who are of the church may be recognized by the marks of Christians. They believe in Jesus Christ the only Saviour, flee from sin and pursue righteousness, love the true God and their neighbour without turning to the right or left, and crucify their flesh and its works. Although great weakness remains in them, they fight against it by the Spirit all the days of their life. They appeal constantly to the blood, suffering, death, and obedience of Jesus Christ, in whom they have forgiveness of their sins through faith in Him.

The false church assigns more authority to itself and its ordinances than to the Word of God. It does not want to submit itself to the yoke of Christ. It does not administer the sacraments as Christ commanded in His Word, but adds to them and subtracts from them as it pleases. It bases itself more on men than on Jesus Christ. It persecutes those who live holy lives according to the Word of God and who rebuke the false church for its sins, greed, and idolatries.

These two churches are easily recognized and distinguished from each other.
Rev 2:9; Rom 9:6; Gal 1:8; 1 Tim 3:15; Acts 19:3-5; 1 Cor 11:20-29; Mt 18:15-17; 1 Cor 5:4-5, 13; 2 Thes 3:6, 14; Tit 3:10; Jn 8:47, 17:20; Acts 17:11; Eph 2:20; Col 1:23; 1 Tim 6:3; 1 Thes 5:21; 1 Tim 6:20; Rev 2:6; Jn 10:14; Eph 5:23; Col 1:18 Jn 1:12; 1 Jn 4:2 10Rom 6:2; Php 3:12; 1 Jn 4:19-21; Gal 5:24; Rom 7:15; Gal 5:17; Rom 7:24-25; 1 Jn 1:7-9; Acts 4:17-18; 2 Tim 4:3-4; 2 Jn 9; Jn 16:2

Titus 3:10,11 A factious man after a first and second admonition refuse; 11 knowing that such a one is perverted, and sinneth, being self-condemned.

Mt 18:15-17 And if thy brother sin against thee, go, show him his fault between thee and him alone: if he hear thee, thou hast gained thy brother. 16 But if he hear thee not, take with thee one or two more, that at the mouth of two witnesses or three every word may be established. 17 And if he refuse to hear them, tell it unto the church: and if he refuse to hear the church also, let him be unto thee as the Gentile and the publican.

1 Cor 5:4-5 in the name of our Lord Jesus, ye being gathered together, and my spirit, with the power of our Lord Jesus, 5 to deliver such a one unto Satan for the destruction of the flesh, that the spirit may be saved in the day of the Lord Jesus.

1 Cor 5:13 But them that are without God judgeth. Put away the wicked man from among yourselves.

2 Thes 3:6 Now we command you, brethren, in the name of our Lord Jesus Christ, that ye withdraw yourselves from every brother that walketh disorderly, and not after the tradition which they received of us.

2 Thes 3:14,15 And if any man obeyeth not our word by this epistle, note that man, that ye have no company with him, to the end that he may be ashamed. 15 And yet count him not as an enemy, but admonish him as a brother.

The Chestnut Mare