The problems you are facing are, unfortunately, a common occurrence in the body of Christ. Typically, in my experience over the years, pastors are rarely balanced in their ministry. Either they are seen as great pastors in the sense of being “people persons” who are friendly, available to their people, and visit, but often are weak in the pulpit, or you have the opposite. If they are good students of the Bible and strong in their teaching, then people feel their “pastoral skills” are lacking. Of course, occasionally there are those who are strong in both areas or weak in both as well. Unfortunately, because of the many false expectations people have of the pastorate, churches expect them to be everything to all people (able to leap tall buildings, etc.).
Part of the problem stems from, at least in my understanding of the New Testament, a wrong theology of the pastorate and from a failure of churches and its leadership to delegate the work to others according to gifts. There needs to be a definite job description for the pastor that is realistic with his skills and burden and the needs of the church. Part of the problem is that pastors can become so stretched and fragmented in their responsibilities and the expectations of the church that they do not have the time needed to study and prepare. Solid study and preparation time is an absolute necessity. Of course, some pastors do not like to study and prefer the busy routine of meetings, visitation, counseling, etc. Here is where the leadership needs to sit down and evaluate all this with their pastor. Find out where and how he is spending his time and try to discover if he is really committed to study and even feels gifted in this area. He may be out of his gift. This can really damage a church and the pastor himself.
You mentioned the complaints mainly center around not being fed, not relevant, and not challenging. Wow! In our existential, experience and emotionally-oriented society, one that is so self-centered and oriented to self-help books and preaching, etc., we must be very careful regarding such claims. We live in what may be the most anti-intellectual period in the history of Western civilization. Today, what we are seeing is revival of emotionalism, but not the true knowledge of God. The church today is more guided by feelings than by biblical convictions. People today value enthusiasm more than informed faith and commitment. Emotions are not wrong, but they must be guided by a truly biblically-oriented faith. Sermons need to target people’s thinking as much as their wills and feelings. Certainly we do not want a church absent of emotion and feelings, but such must be guided by sound thinking.
The primary concern of many today is: “What does this passage say to me?” But such a question is only legitimate if first we ask, “What does this passage say? What did it say and mean to its original hearers”? And then, once we are sure of that, “What does it say to us.?” We often hear it said in church that we don’t want a discussion to get too theological, we want to keep it practical, as though good practice did not require careful thought to direct it. Remember, it was Jesus who said, “And He said to him, “‘YOU SHALL LOVE THE LORD YOUR GOD WITH ALL YOUR HEART, AND WITH ALL YOUR SOUL, AND WITH ALL YOUR MIND.’ Note the final focus, “with all your mind. There is an excellent book entitled, Love Your God With All Your Mind, by J. P. Moreland, that deals with this issue.
So, as the leadership, you need to sit down with the pastor and evaluate his schedule, the use of time, his gifts, the true nature of his preaching, the negative comments (are they legitimate, partly or totally?), the job description (is it biblical?), the delegation of responsibilities, and his willingness and that of the congregation to make adjustments.