Place the rectangle with corners $(0,0),(18,0),(18,12),(0,12)$, and let the square have vertex $A=(a,0)$ on the bottom side, with neighbors $B=(b,12)$ on the top side and $D=(18,d)$ on the right side.
- $AB=13$ gives $(b-a)^2+144=169$, so $b-a=-5$ ($B$ is to the left of $A$, as in the figure).
- $AB\perp AD$ gives $(b-a)(18-a)+12d=0$, so $12d=5(18-a)$; and $AD=13$ gives $(18-a)^2+d^2=169$. Hence $(18-a)^2\left(1+\frac{25}{144}\right)=169$, $(18-a)^2=144$, $a=6$, $d=5$, $b=1$.
- So $A=(6,0)$, $B=(1,12)$, $D=(18,5)$, and the fourth vertex is $C=B+D-A=(13,17)$, outside the rectangle.
The edge $DC$ crosses the rectangle's top side at $X=(181/12,12)$. The overlap is quadrilateral $A DXB$. By the shoelace formula, its area is
\[
\frac12\left|30+\frac{1687}{12}+169-72\right|=\boxed{\frac{3211}{24}}.
\]