Augmented Reality Based on Apollo Lunar Imagery: Searching for Stars and Adding Virtual Objects

In recent years, we have developed 3D models of the first three Apollo landing sites using an extensive photogrammetric analysis of numerous photos (Pustynski & Jones, 2014; Pustynski, 2022, 2021a, 2021b; Pugal & Pustynski, 2022). Our high-precision models include camera coordinates and rotations. Using these data, we developed an algorithm and computer code to calculate positions of stars and planets in Apollo photos. Although we were unable to detect even the brightest stars such as Canopus due to inadequate exposure settings, we successfully recreated views of the starry sky and constellations in some iconic lunar images. Our algorithm has potential for a wide range of applications. By inserting virtual objects into the 3D space of the landing sites, it is possible to create augmented reality based on the lunar environment. These visualizations may be used for scientific and educational purposes. In addition to drawing constellations and coordinate grids, we generated terrain modeling contour lines and demonstrated a realistic visualization of free fall under lunar gravity.

A detailed description of the process for creating a photogrammetric model of an Apollo landing site can be found in Pustynski and Jones (2014) and Pustynski (2021a). The resulting model includes coordinates and rotations of camera stations. However, since lunar photos do not contain explicit references to the local horizontal plane and cardinal points, the initial coordinate frame of the model is arbitrary and linked to hardware elements or adjacent rocks. In our first Apollo 11 landing site model (Pustynski & Jones, 2014), an approximate method was used to determine the local horizontal frame, which assumed the Solar Wind Composition Experiment (SWC) pole to be vertical and cardinal directions to be set according to orbital photos. However, this method does not provide the required accuracy for our purpose.

Our method ensures that the transformed coordinates of the celestial bodies in the model closely match their actual horizontal coordinates, resulting in a transformed coordinate frame that closely approximates the true horizontal frame. We achieved high accuracy for the Apollo 11 and Apollo 14 missions, with $S$ values in Equation 1 approximately equal to 0.1° and 0.2°, respectively.

The Sun can be used in all models, despite its center being difficult to determine due to glare. However, its direction can be deduced by analyzing the shadows cast by tall pointed objects such as the flagpole or the SWC pole (Pustynski, 2021a). Earth is present in photos from the Apollo 11 and Apollo 14 missions, and its center can be easily identified as the center of the circumscribed circle. For the Apollo 14 mission, we also used Venus as a third celestial body. Unfortunately, Apollo 12 photos allowed only for the use of the Sun. In this case, we used a modified version of our algorithm. The mean plane of rocks scattered at different distances from the Lunar Module (LM) was identified with the local horizontal plane. While this approach may have resulted in slightly larger deviations of the transformed coordinates from the true horizontal frame, the error is still sufficiently small (Pustynski, 2021a).

Since the precise determination of camera orientation is crucial to our study, we conducted independent checks to validate the model coordinate frame.

The Apollo 11 landing site has a northeast tilt of several tenths of degree at a scale of several kilometers. This tilt can be deduced from Digital Terrain Models (DTM) based on orbital photos (Arizona State University, 2012). We discovered a similar tilt in the photogrammetric model by analyzing the distribution of rocks around the LM. The mean plane of 28 rocks scattered at distances from tens to hundreds of meters around the LM was found to have a tilt of approximately 1.1° with an azimuth of about 70° from north. This value falls within the expected accuracy margin.

The azimuth of the LRRR station relative to the center of the LM can be independently determined from DTMs (Wagner et al., 2017), as shown in Figure 4d. The same azimuth measured in the transformed coordinates differs from the DTM-based value by less than 0.2°. This confirms that the transformed coordinates are accurately aligned with the cardinal directions.

Our next task is to project a distant point-like object with the known altitude $α$ and azimuth $γ$ onto the image plane of a specific photograph. We assume that the HDC lens is free of distortions, which is a reasonable first-order assumption based on the parameters of the Zeiss Biogon lens (see Carl Zeiss, n.d.). We utilized the same assumption when constructing our models of the landing sites.

When analyzing digital scans, it is necessary to multiply the linear coordinates $(ximage,yimage)$ by the pixel-to-millimeter scale of the scan. This scale can be determined by using the réseau grid again (the distance between two adjacent crosses is 10 mm in the film). It is important to measure this scale for each photograph individually, as it can vary from scan to scan.

There are several factors to consider when working with film scans. Firstly, it is important to note that the horizontal and vertical pixel scales of the available scans may differ slightly. However, we found that this difference is small enough to use the average scale. Secondly, some scans may be slightly rotated, meaning that the entire réseau is rotated by a small amount. This rotation should be taken into account when the angle $δ'$ is calculated. In the Apollo 11 landing site model, the camera roll was measured from the middle horizontal line of the crosses, and the rotation of the scan was added as correction to the camera roll. For the Apollo 12 and Apollo 14 models, the roll is measured from the horizontal line of scans downloaded from the Lunar and Planetary Institute website (Lunar and Planetary Institute, n.d.), so no additional correction was needed. However, scans from a different source may require roll correction. Thirdly, other deformations of the scanned film (revealed as varying distances between réseau crosses) are small enough to be ignored. Finally, it is problematic to work with photos taken from the LM windows with the Hasselblad Electric Camera (HEC), as it lacked the réseau plate. In these scans, principal point coordinates, scale, and roll of the scan can only be estimated roughly, so we prefer not to use HEC photos.

We developed a computer code that calculates coordinates of 3D point projections onto the image plane of photos taken from different camera stations. The code also draws these projections graphically and performs other tasks (Pustynski, 2023). The produced digital images are provided with a réseau grid to match them to scans of original photographs. We used the Asymptote Vector Graphic Language¹ to write the code. In the following sections, we present several combined images.

We have carried out multiple checks of our algorithm and code. First of all, we used model benchmarks (rocks and elements of hardware) as test objects. The direction vector of such a benchmark for the specific camera is $oc→=B;→-C;→$⁠, where $B;→=(xB,yB,zB)$ is the benchmark position vector and $C;→=(xC,yC,zC)$ is the camera position vector. We calculated pixel coordinates of a number of benchmarks in different scans and found a very good agreement between the calculated coordinates and actual images of benchmark objects.

With our algorithm, we can create virtual objects in the 3D space of the landing site's surroundings by predefining coordinates of points in this space. Views of such an object can be generated and added to photographs taken from different camera stations. In a photogrammetric software package, this virtual object should be indistinguishable from real objects, as it obeys the same rules of perspective. Lines of sight from different cameras should intersect at points on the object, and coordinates of these points can be determined with the instruments provided by the package. To verify our algorithm, we created virtual objects in the 3D space and determined coordinates of their points with ImageModeler 2009 (the software used to compile the photogrammetric models).

Based on these results, we conclude that our projecting algorithm is functioning properly.

To find a celestial body position in a lunar photograph, its horizontal coordinates at the time of shooting are needed. Fortunately, the shooting times of many lunar photographs can be determined with an accuracy of about 1 min from voice transcripts and TV recordings (Jones & Glover, 2018). The celestial bodies' horizontal coordinates depend on the selenographic coordinates of the landing site. JPL Horizons (JPL SSD Group, 2023) is a tool providing horizontal coordinates of the solar system bodies at any point on the lunar surface, including the Apollo landing sites, at any time during the Apollo expeditions. We use solar and planetary ephemerides from this source to accurately calculate the positions of celestial bodies. Additionally, we use distances to the Sun and Earth to reproduce their angular sizes correctly. However, JPL Horizons does not provide horizontal coordinates of objects outside the solar system. To find the horizontal coordinates of stars, we transform their equatorial coordinates using two solar system bodies as references. The Sun and any planet, or any pair of planets, can be used as references to perform this transformation.

Now we proceed with two rotations: (1) The vector $BBh;;→$ is rotated by an angle $α$ around the vector $B2h;;→$ in the direction depicted in Figure 8. This rotation takes place in the horizontal frame. The resulting vector aligns with $BO;→$⁠, allowing us to deduce the horizontal coordinates of $BOh;;→$⁠. It is not possible to determine them directly from the cross product of vectors $B2;→$ and $O;→$ since initially the horizontal coordinates of $O;→$ are unknown. (2) Finally, the vector $B2h;;→$ is rotated by the angle $β$ around the vector $BOh;;→$ (obtained after the first rotation). This rotation occurs in the direction indicated in Figure 8, and the resulting vector aligns with $O;→$⁠. As these rotations are performed within the horizontal frame, we obtain the horizontal coordinates of the object $Oh;;→$ after the second rotation.

It is important to note that the direction of the first rotation depends on whether the object is located above or below the plane defined by the reference objects' vectors (as in Figure 8). To ensure the correct direction, we multiply $α$ by the parameter $k= sgn (B2;→·[ BB ;→× BO ;→])$⁠. If the object is above the plane, the cross product in the brackets is parallel to $B2;→$⁠, the dot product is positive, and the rotation by the angle $α$ is performed clockwise. If the object is below the plane, the dot product is negative, and the rotation is performed counterclockwise.

To find horizontal coordinates of stars at any photograph, we use ephemerides of the Sun and planets calculated with JPL Horizons and tabulated at 1-min intervals. Two reference objects are needed. Theoretically, higher accuracy is provided when the angle $∠B1B2$ is close to $±90$°, but practically, the error is very small even when this angle is much larger or smaller.

To validate the accuracy of our coordinate transformation, we used the digital planetarium software Stellarium² to calculate the horizontal coordinates of some stars at the landing sites. We found that the discrepancies between the coordinates provided by Stellarium and those calculated through our transformation were less than one minute of arc, which is significantly smaller than the model's uncertainty. Any discrepancies observed can likely be attributed to differences in the algorithms used by JPL Horizons and Stellarium.

In Figure 9c, we have marked the actual locations of Venus in 9 photographs from the Apollo 14 mission using small stars. Venus is depicted relative to its predicted location (the center of the target). For the compilation of Figure 9c, we intentionally adjusted the horizontal frame in the Apollo 14 landing site model. While our basic horizontal frame is aligned using the Sun, Earth, and Venus as described in Section 2, for Figure 9c, we aligned the horizontal frame using only the Sun and Earth. By doing so, the coordinate system becomes independent of Venus, allowing the deviation of the actual locations of Venus from its predicted position in this figure to better represent the deviation we may expect for an arbitrary star. The average distance between the actual and predicted locations is approximately 0.27 mm (equivalent to 0.25°), which can be attributed to the uncertainty of the coordinate frame. The dispersion of the actual locations is primarily caused by the uncertainty in the orientations of individual camera stations within the model.

Based on our analysis, we can conclude that the error in the calculated coordinates of objects in the image plane is reasonably small. This level of accuracy enables us to effectively search for stars in Apollo photographs. The precision of the calculated coordinates provides confidence in the reliability of our method for identifying celestial objects in the images.

Due to the astronauts' primary focus on capturing images of the lunar surface and expedition-related equipment, only a small portion of the lunar sky is visible in Apollo photographs. Typically, the visible lunar horizon divides a photograph into two parts, with the sky occupying the upper half or even less of the image. With the 52-mm linear size of the frame edge and the camera focal length of 61.1 mm, the angular half-size of the image covers only $arctan522·61.1≈23$°, which corresponds to a strip $≈20$% of the celestial sphere. Consequently, many bright celestial bodies did not appear in the photographs, either due to their position below or too high above the horizon. It is not surprising, therefore, that Sirius, the brightest star aside from the Sun, was not inside the frame of any of the 123 photographs taken by the Apollo 11 crew during their extravehicular activity (EVA). Unfortunately, Sirius was also not inside the frame of the nearly 300 of Apollo 12 photographs included in the model (Pustynski, 2021b), or in the 65 of Apollo 14 photographs in the model (Pugal & Pustynski, 2022). Venus appeared in nine Apollo 14 EVA photographs taken by Alan Sheppard, who raised his camera to capture images of Earth and Venus high above the LM. Mercury, being very close to the Sun, was always hidden in the solar glare, while other bright planets were simply outside the frame of the photographs.

Canopus was the most obvious star to search for in the lunar photographs. Other bright stars, such as $α$ Centauri, Arcturus, and Vega, are 1.5–2 times dimmer. We began by examining all the Apollo 11 photographs that could potentially include Canopus, as listed in Table 1. The predicted coordinates of Canopus are given relative to the principal point of the photograph, which is the center of the central cross. We excluded photos taken from the 4 o'clock panorama station, as Canopus was hidden behind the LM. In the lower portion of the table, we added some, but not all, Apollo 12 expedition photographs that could potentially feature Canopus.

The accuracy of coordinates in Table 1 can be estimated based on the uncertainty of the model, which is about 0.3° or better. The linear uncertainty of the coordinates is approximately $61.1·tan0.3∘≈0.3$ mm near the center of the image and slightly larger near the edge. A conservative value $ΔX=ΔY=0.4$ mm may be adopted, which is approximately 0.8% of the edge length (see for comparison Figure 9c), where the distance between the actual and predicted location of Venus never exceeds 0.4 mm). For the $3900×3900$ px scans from LPI, this is equivalent to 30 px. However, we believe that the actual error is smaller in most cases. The uncertainty of the shooting time is negligible because the horizontal coordinates of stars change at a maximum rate of $≈0.5$°/h, so a timing error of 5 min results in a coordinate error less than 0.05°.

After Canopus, the next brightest star that might potentially appear in Apollo 11 photographs is Achernar. However, it is approximately three times dimmer than Canopus, and we did not identify it. We did not attempt to search for any dimmer stars. The only planet that appears in Apollo 11 photographs is Earth.

In Apollo 12 photographs, the proximity of the Sun and the resulting glare made it difficult to capture $α$ Centauri, the third brightest star, which is 1.5 times dimmer than Canopus. Other potential bright targets were Capella, Rigel, and Betelgeuse. Mercury was hidden in the solar glare, and other planets were too high or too low. We were unable to detect $α$ Centauri in the color photo AS12-47-6992, nor Capella in the black-and-white photo AS12-48-7136.

In the Apollo 14 mission, the brightest star that may have potentially been featured in analyzed photos was Vega. We attempted to locate it in the color photograph AS14-66-9301 without success. As previously noted, Earth and Venus are clearly distinguishable in Apollo 14 photos. Other bright planets were not in suitable positions to be captured.

As¹⁸¹⁸ we have seen in Section 3.1, it is possible to insert virtual objects into¹⁹ the 3D-space of the Apollo landing sites and to add their images to lunar photographs. When we realized that stars cannot²⁰ be revealed in available scans, we got an idea to “restore” starry sky views in some of the iconic photographs. To recreate these views, we used the Yale Bright Star Catalog (Hoffleit & Jaschek, 1991), as it is a mostly complete list of stars with apparent magnitudes up to $mV=7$⁠. The catalog contains coordinates of stars, their names and catalog numbers, apparent magnitudes, spectral classes, and other data. Our code parses the catalog, chooses stars brighter than a predefined threshold, transforms their equatorial coordinates to the horizontal frame, and projects them as small disks onto the selected lunar images. The size of the disk and its color depend on the apparent magnitude and spectral class of the star (brighter stars are larger, dimmer stars are darker, and colder stars are redder). We also used data on constellation boundaries from Barbier (2022) to draw the stars in the corresponding context. Asterisms are drawn according to the O. Hlad sky culture in Stellarium. Images of the Sun and planets are also added where necessary, and the size of the Sun and Earth images corresponds to their actual distance at the instances the photographs were taken.

When we began this project, we hoped to identify at least the brightest stars and planets in Apollo photographs, based on the presence of Venus in all Apollo 14 photos. Our expectation was that we would be able to detect these stars as a group of several brighter pixels, given their brightness in comparison to Venus (which was 15 times brighter than Sirius and nearly 30 times brighter than Canopus, and it is represented by a group of several tens of pixels in the Apollo 14 photos). We used accurate values of camera rotations from the photogrammetric models of the Apollo 11–14 landing sites to calculate expected coordinates of the brightest stars in the image plane. However, Sirius and bright planets were outside the frame of the studied photographs, and Canopus was not revealed. Dimmer stars like $α$ Centauri, Capella, and Vega were not detected either.

We believe that our failure cannot be attributed to inaccuracy of the models and the algorithm. Our accuracy tests show that the determined coordinates are accurate to at least 0.4 mm in the image plane of the original film, and the stars should indeed be inside the circles shown in Figures 13–14. We suspect that the stars were not revealed due to insufficient exposure of the photographs or available scans. Many high-resolution scans in (NASA/JSC/ASU) are quite dark, and their dynamic range in sky areas is very small. It is possible that if these areas were scanned with brighter light settings or if the original films were studied with a microscope, there may still be a chance to find the brightest stars. We suggest using the coordinates from Table 1 to search for stars in Apollo 11 and Apollo 12 mission photographs. Apollo 14 photographs included in the model do not contain stars as bright as Sirius and Canopus inside the image frame. Accurate photogrammetric models of other landing sites are required to search for stars in photographs from the later missions.

Below are some potential applications of augmented reality based on lunar photos:

• Virtual objects can be inserted into the landing site's 3D space, and their views from various camera stations can be simulated. As an example, we provide in Pustynski (2023) a couple of photographs featuring a virtual stadiometer set at the ground near an astronaut figure, which can be used to easily estimate the astronaut's height. If accurate models of the terrain are available, shadows of these objects can also be modeled, as shown in Figures 7 and 22. To achieve more natural visualization, the actual position of the Sun and reflectance of the terrain may be considered for more precise modeling of shades and colors.

• Animations can be created to demonstrate the motion of virtual objects as they are viewed from different camera stations, for instance, animations of rocks launched and moving in lunar gravity, or the LM during landing and launch, etc. These animations can be interactive, allowing the user to define coordinates of the launch point, initial velocity vector and rotations, object dimensions, etc.

• Views from virtual camera stations can be simulated, where no actual photographs were taken. To generate these views, highly detailed photogrammetric models of artifacts and terrain should be constructed, with textures either modeled or partially obtained from Apollo photographs. An example of such digital model can be found in Le Mouélic et al. (2020).

• A virtual planetarium can be developed to display positions and movements of celestial bodies as viewed from various camera stations. To accomplish this, the illumination of the terrain and hardware must be modeled for different solar elevations (to simplify this task, actual objects may be replaced by 3D models).

The computer code used to compile graphics for Apollo photographs (e.g., stars, constellations, contour lines) can be found in Pustynski (2023), which includes a comprehensive guide.

This study was conducted as part of the author's research work in Tallinn University of Technology (Taltech). The author would like to thank Siim Pugal for his assistance in compiling the photogrammetric model of the Apollo 14 landing site. The author expresses sincere gratitude to the anonymous reviewers whose remarks and valuable advice have contributed to the improvement of this manuscript.

The data supporting the findings of this study, including camera coordinates, rotations, and the computer code used for visualizations, are openly accessible in the Mendeley Data repository (Pustynski,²²2023).