I fear that these members are trying, in one way or another, to put the church back under the Old Testament law.
The fact is that Paul, as a Jew, continued to observe the Jewish holidays (Acts 20:16; 1 Corinthians 16:8). This was a voluntary matter on Paul's part. He was, of course, a Jew, and so doing so would be meaningful to him. For the sake of his Jewish brethren he made a point of demonstrating that, as a Jew, he continued to observe some of the Jewish rituals (Acts 21:17-26f.). (In point of fact he was trying to demonstrate that he was not teaching Jewish converts to Christ that they had to totally abandon Jewish rituals.)
Early on in the history of the church the question arose as to whether or not the Gentiles would be placed under the law. The Jerusalem Council was called to settle this issue, and the matter is recorded in Acts 15. Here is their authoritative decree concerning what Gentile converts to Christianity were obligated to do:
23 They sent this letter with them: From the apostles and elders, your brothers, to the Gentile brothers and sisters in Antioch, Syria, and Cilicia, greetings! 24 Since we have heard that some have gone out from among us with no orders from us and have confused you, upsetting your minds by what they said, 25 we have unanimously decided to choose men to send to you along with our dear friends Barnabas and Paul, 26 who have risked their lives for the name of our Lord Jesus Christ. 27 Therefore we are sending Judas and Silas who will tell you these things themselves in person. 28 For it seemed best to the Holy Spirit and to us not to place any greater burden on you than these necessary rules: 29 that you abstain from meat that has been sacrificed to idols and from blood and from what has been strangled and from sexual immorality. If you keep yourselves from doing these things, you will do well. Farewell (Acts 15:23-29, NET Bible).
As you can see, there is nothing here at all requiring Gentile believers to observe Jewish feasts. If this were a requirement, this would have been the place to say so. As your pastor pointed out from Romans 14, this is certainly a matter of personal freedom, and not a matter of law-keeping.
Paul speaks quite clearly on this matter of Gentiles not being put under the law. Galatians is Paul's strongest statement on the matter. But we also read this in Colossians chapter 2:
8 Be careful not to allow anyone to captivate you through an empty, deceitful philosophy that is according to human traditions and the elemental spirits of the world, and not according to Christ. 9 For in him all the fullness of deity lives in bodily form 10 and you have been filled in him, who is the head over every ruler and authority. 11 In him you also were circumcised-not, however, with a circumcision performed by human hands, but by the removal of the fleshy body, that is, through the circumcision done by Christ. 12 Having been buried with him in baptism, you also have been raised with him through your faith in the power of God who raised him from the dead. 13 And even though you were dead in your transgressions and in the uncircumcision of your flesh, he nevertheless made you alive with him, having forgiven all your transgressions. 14 He has destroyed what was against us, a certificate of indebtedness expressed in decrees opposed to us. He has taken it away by nailing it to the cross. 15 Disarming the rulers and authorities, he has made a public disgrace of them, triumphing over them by the cross. 16 Therefore do not let anyone judge you with respect to food or drink, or in the matter of a feast, new moon, or Sabbath days 17 that are only the shadow of the things to come, but the reality is Christ. 18 Let no one who delights in humility and the worship of angels pass judgment on you. That person goes on at great lengths about what he has supposedly seen, but he is puffed up with empty notions by his fleshly mind. 19 He has not held fast to the head from whom the whole body, supported and knit together through its ligaments and sinews, grows with a growth that is from God (Colossians 2:8-19, NET Bible).
One can hardly overlook the fact that frequently the false teaching which is addressed in the New Testament has a Jewish flavor. We have already seen the insistence of the Jews that Gentile converts must be circumcised and keep the law (Acts 15:1ff.). Romans 14-15 speak about personal liberties, and the issue there seems to be largely matter of Jewish practices or prohibitions. In 2 Corinthians chapter 11 Paul finally identifies his adversaries in Corinth as deceivers who are "servants of Satan" and "false apostles." He also makes it clear that they are Jews (11:22).
In Philippians chapter 3 Paul warns the saints in Philippi about Jewish legalizers. In 1 Timothy chapter 1 Paul warns Timothy and the Ephesian saints about those would-be teachers of the law, who were most surely Jews (1:3-11). In 1 Timothy 4 Paul warns about the "doctrines of demons," which involves the prohibition against eating certain foods (4:3). In Titus 1:14 Paul warns about "Jewish myths." I think one must say that the Judaisers are still with us today, and we must beware of falling into their error.
Do certain holidays have a pagan origin, like Christmas or Easter? Perhaps so, and we certainly need to avoid any pagan elements in these holidays (Halloween is perhaps the one that is still most obviously pagan). But we should also take into account that Christian rituals like circumcision and baptism may have pre-existed the Christian practice and symbolism. Since these rituals have been redefined by Scripture we have no qualms observing them.
Whether or not one chooses to observe Christmas or Easter is a matter of personal liberty. But these liberties must not be argued about or imposed on others (Romans 14:1, 4, 22). If a person chooses to observe a Jewish feast that is all well and good, but once again this must not be imposed on others.
The kind of thing you have described seems to happen with a fair degree of frequency. I would stand fast with your pastor and I would warn your brothers about being put back under the law.