Chillingsworth makes a passage to Europe to taunt Hester. Dimmesdale makes his speech to the townspeople. Hester looks on. Chillingsworth approaches her and states that Hester and Pearl can go on the ship by themselves and he will make arrangements for Dimmesdale. The whole crowd is staring at them. Dimmesdale has surmised that "resistance is futile." (per Spark Notes) and Chillingsworth will haunt and taunt them for the rest of their lives.
I agree Chillingsworth is the most evil character. Spark Notes refers to the "dominant order" and Dimmesdale gives in to this theory.
The plot thickens and as readers we probably wonder what, if any, more attempts to leave they will make. It is a bold act for them.