Needless to say, there are many divorced pastors serving in churches today -- in evangelical, Bible believing churches. The fact that divorced pastors still serve as pastors does not prove the matter biblically.
My assumption is that the primary biblical texts are going to be those which deal directly with divorce (especially Matthew 5:27-32; 19:3-12; 1 Corinthians 7:10-16) and those which set down the qualifications for elders (1 Timothy 3:1-7 and Titus 1:5-9). I believe that a pastor must certainly meet the qualifications for an elder.
An unbiblical divorce would surely disqualify a man from serving as an elder. Folks disagree about what constitutes a biblical divorce, but I would say that it would either require marital infidelity on the part of her husband (Matthew 5:31-32; 19:9) or a denial of faith on the part of the wife (1 Corinthians 7:12-16).
One would probably want to inquire further as to why the wife divorced her husband. Did the wife divorce as a Christian, or as an unbeliever? Does she still profess to be a Christian? Did she make accusations against her husband, and were they answered? Was the divorce sought on biblical grounds?
In both elder texts an elder must be "the husband of one wife" (literally he must be a "one woman/wife man") -- 1 Timothy 3:2; Titus 1:6. Thus, if this pastor remarries, he would, in the opinion of many (including myself), no longer be considered the husband of one wife. But what if he chooses to remain single? Several issues should be dealt with. First, Paul says that an elder must "manage his own household well":
4 He must manage his own household well and keep his children in control without losing his dignity. 5 But if someone does not know how to manage his own household, how will he care for the church of God? (1 Timothy 3:4-5)
Does this divorce cast a shadow over the pastor's leadership in his home? If so, it also casts a shadow over his leadership in the church.
Paul also insists that an elder must have a good reputation with those outside the church (1 Timothy 3:7). In some communities a divorced pastor would not have the respect which is required for his office. That is a matter the pastor and the governing board of the church will have to decide.
In addition to this, whatever decision is reached, you must recognize that some Christians and churches will have reached other conclusions, so that his ministry will be handicapped.