It is established in Genesis and the rest of the Old Testament that God's children are those who love him, adore him, and keep his commandments. Seth was created in Adams image and likeness, therefore he is also created in the image and likeness of God, as we all are. Also, Seth and his family were the ones to invoke God, meaning adore Him. This is mentioned to teach us that God is the Father of those who adore Him, not just our Creator. The "sons of God' referred to the lineage of Seth, not angels since they have no flesh. Even Jesus himself says this when he presents himself to the Apostles after the Resurrection. The "daughters of man" referred to the lineage of Cain. Both families lived differently, Cain's pursued war and glory, while Seth's adored God, though they too had the seed of sin that we all inherit from Adam and Eve. The sons of Seth were also tempted into polygamy, as Lamech the first man to commit polygamy in the Bible and the one who also wrote the song about vengeance. Through these marriages sin became institutionalized and, as can be seen throughout the Old Testament, the rupture of the conjugal alliance brings a condemning judgement of God, as it is explained in chapter 6.In hebrew, the words "the heroes of days gone by" literally means "the men of shem" or "the men of name". These men were evil tyrants that searched for their own glory. As the descendants of Seth and Cain married, the world became dominated by the lineage of Lamech, in other words by unjust and violent men. All of this is why they were called giants, or as you said baby giants.Sorry if it wasn't anything more fantastical or anticlimactic. The way it's written is similar to the greek and norse myths, which is why one tends to think of the story going somewhere and the author just forgot his line of thinking. I was also fascinated with this mysterious chapter, but after reading a book about studying the Bible I finally got the answer. In fact, most of what I wrote was actually taken from the book. The english version of the book is titled "Understanding The Scriptures: A Complete Course On Bible Study", it's from the Didache Series.
It may be settled as far as you are concerned but for others it is not. I know the story and the 'sons of Seth' vs 'sons of angels' debate has been going on for a very long time. The point is, I would prefer more of an explanation on that rather than decorating details. Just my preference.