Why we devide by the maximum:
To calculate |R1 JOIN_(A=B) R2|:
For every possible value a of A, we have the size of R1 Select A=a times the size of R2 select B=a.
That sums up to V(R1,A) * (B(R1)/V(R1,A) * (B(R2)/V(R2,B) = (B(R1) * (B(R2)) / V(R2,B)
But we could also numerate on B and get (B(R1) * (B(R2)) / V(R1,A),
So the option that will give us the minimum number of IOs of this two is to take:
(B(R1) * (B(R2)) / Max(V(R1,A), V(R2,B))
I don't know about the second question you asked - for that I need to see the question, probably has something to do with indexing.