Hilbert first mentioned ideal elements in his 1898–99 lectures on geometry. He described them as important, fruitful, and of frequent occurrence in mathematics, pointing to the examples of negative, irrational, imaginary, ideal and transfinite numbers. In geometry, he had in mind the examples of points, lines, and planes at infinity, whose introduction gives geometry a certain completeness, by making theorems such as those of Pappus and Desargues universally valid. In this article I will discuss how Hilbert transformed our view of the Pappus and Desargues theorems by showing that they express the underlying algebraic structure of projective geometry. I will compare this result with another of Hilbert's great contributions, his calculus of ends. By studying the ideal elements of the hyperbolic plane, Hilbert similarly extracted algebraic structure from the axioms of hyperbolic geometry. Hilbert's treatments of projective and hyperbolic geometry have another important common element: construction of real numbers. To achieve this, Hilbert has to add an axiom of continuity to the geometry axioms, but he evidently wants to show that the real numbers can be put on a geometric foundation.