I would like to show an entire square shape (image, matrix, ...) in a browser at maximal size in any size window. How can I do this? I cannot find the controls for this in Responsive Layout Maker. Does this (or any) Coffeecup software accommodate this? Max width is easy, but max height remains elusive to me.

How about using a 1:1 image and set the max-width to none?

You could use viewport units to make it keep its height (good for background items):
width: 100vw;
height: 100vh;
I don't know if you can do this within RLM, you might have to adjust your code after export.