While similar to the Metrics By Speed chart, the Size pane instead focuses on the actual file size. Having large file sizes not only slows down the loading of your page but significantly impacts user experience. Slower sites will almost always cause users to bail for faster loading ones.
A simple way to dramatically speed up your website is to start by removing larger files. If these files are mandatory, most images can decrease in file size by compressing them to make them smaller. Similarly, HTML and CSS files can also be compressed by removing unneeded characters such as spaces and line breaks..
You may also notice that the graphs may look exactly the same, and that is Ok. If you are serving the same content to all viewports, the file sizes for your content is the same, so the download speeds would roughly be the same as well. Where you would see a difference here is if your website is responsive and you are serving different content on different viewports.