I used var returnValue = showModalDialog('url') to open a modal dialog. It seems however it can't submit a form back to the opening window, which makes some sense as that is in a frozen state expecting the result of the modal window.
So setting window name for the target of the form also won't work
<form target="windowName"...
luckly there is a hack/workaround. First create a new seprate page having frameset
<frameset >
<frame src='<%=Request.QueryString["url"]%>' id="shimpage" name="shimpage"
/>
</frameset >
...
(shim.aspx)
Then use use this newly created page (shim.aspx) to load other page
onclick="javascript:showModalDialog('shim.aspx?url=targetpage.aspx');"
(pagetoloadfrom.aspx)
Hence even tho window is in frozen state, I am able to submit the form using frame in same window. Thou this solution is not elegent, still its serves the purpose.