The issue here is just what this expression ("one woman man") means. The most liberal interpretation would be understood to mean "one woman at a time" -- not a polygamist. A few think it means that an elder must be married once in a lifetime (he cannot re-marry even if his wife were to die). Many would think (I among them) that divorce and remarriage would disqualify a man from being an elder.
There are several further matters pertaining to how this relates to you and to your question:
First, are the qualifications for elders the same thing as qualifications for pastors? I would be inclined to think so.
Second, does all this apply to a man who has been married only once, but who is married to a divorced woman? I don't know that the New Testament texts directly apply. One must then ask whether the qualifications for a priest in the Old Testament apply. They certainly deal with the circumstance you have described, but this is a part of the law which does not directly apply to Christians today, though it may apply in principle. That is a matter you will need to decide.
Third, one would probably want to inquire as to whether your wife's divorce was biblical. Folks disagree about what constitutes a biblical divorce, but I would say that it would either require marital infidelity on the part of her previous husband (Matthew 5:31-32; 19:9) or desertion/divorce on the part of her unbelieving husband (1 Corinthians 7:12-16).
There is one final factor, in my opinion. Did she divorce as a Christian, or as an unbeliever? If the divorce took place when she was an unbeliever, then some (including myself) would reason that this sin (even if the divorce was not based on biblical grounds) is a part of the past that out Lord's shed blood has erased.
These are matters that you will need to wrestle with, and come to a conclusion for which you will give account to our Lord.
In addition to this, whatever decision you reach, recognize that some churches will have reached other conclusions, so that ministering there as a pastor will not be possible.