I’m sharing my workflow for processing and analysing spectra obtained using the ASD Field Spec Pro, partly as a resource and partly to see whether others have refinements or suggestions for improving the protocols. I’m specifically using Python rather than any proprietary software to keep it all open source, transparent and to keep control over every stage of the processing..
Working with .asd files
By default the files are saved as a filetype with the extension .asd which can be read by the ASD software ‘ViewSpec’. The software does allow the user to export the files as ascii using the “export as ascii” option in the dropdown menus. My procedure is to use this option to resave the files as .asd.txt. I usually keep the metadata by selecting the header and footer options; however I deselect the option to output the x-axis because it is common to all the files and easier to add once later on. I choose to delimit th data using a comma to enable the use of Pandas ‘read_csv’ function later.
To process and analyse the files I generally use the Pandas package in Python 3. To read the files into Pandas I first rename the files using a batch rename command in the Linux terminal:
Then I open a Python editor – my preference is to use the Spyder IDE that comes as standard with an Anaconda distribution. The pandas read_csv function can then be used to read the .txt files into a dataframe. Put this in a loop to add all the files as separate columns in the dataframe…
If you chose to add any header information to the file exported from ViewSpec, you can ignore it by skipping the appropriate number of rows in the read_csv keyword argument ‘skiprows’.
Usually each acquisition comprises numerous individual replicate spectra. I usually have 20 replicates as a minimum and then average them for each sample site. Each individual replicate has its own filename with a dequentially increasing number (site1…00001, site1….00002, site1…00003 etc). My way of averaging these is to cut the extension and ID number from the end of the filenames, so that the replicates from each sample site are identically named. Then the pandas function ‘groupby’ can be used to identify all the columns with equal names and replace them with a single column containing the mean of all the replicates.
filenames = 
for file in filelist:
file = str(file)
file = file[:-10]
#rename dataframe columns according to filenames
filenames = np.transpose(filenames)
DF.columns = [filenames]
# Average spectra from each site
DF2 = DF.transpose()
DF2 = DF2.groupby(by=DF2.index, axis=0).apply(lambda g: g.mean() if isinstance(g.iloc[0,0],numbers.Number) else g.iloc)
DF = DF2.transpose()
Then I plot the dataset to check for any errors or anomalies, and then save the dataframe as one master file organised by sample location
During a long field season I sometimes forget to change the date in the ASD software for the first few acquisitions and then realise I have a few hundred files to rename to reflect the actual date. This is a total pain, so here is a Linux terminal command to batch rename the ASD files to correct the data at the beginning of the filename.
e.g. to rename all files in folder from 24_7_2016 accidentally saved with the previous day’s date, run the following command…
rename “s/23_7/24_7/g” ** -v
Interpolating over noisy data and artefacts
On ice and snow there are known wavelengths that are particularly susceptible to noise due to water vapour absorption (e.g. near 1800 nm) and there may also be noise at the upper and lower extremes of the spectra range measured by the spectrometer. Also, where a randomising filter has not been used to collect spectra, there can be a step feature present in the data at the crossover point between the internal arrays of the spectrometer (especially 1000 nm). This is due to the spatial arrangement of fibres inside the fibre optic bundle. Each fibre has specific wavelengths that it measures, meaning if the surface is not uniform certain wavelengths are over sampled and others undersampled for different areas of the ice surface. The step feature is usually corrected by raising the NIR (>1000) section to meet the VIS section (see Painter, 2011). The noise in the spectrum is usually removed and replaced with interpolated values. I do this in Pandas using the following code…
for i in DF.columns:
# calculate correction factor (raises NIR to meet VIS – see Painter 2011)
The ubiquitous smartphone contains millions of times more computing power than was used to send the Apollo spacecraft to the moon. Increasingly, scientists are repurposing some of that processing power to create low-cost, convenient scientific instruments. In doing so, these measurements are edging closer to being feasible for citizen scientists and under-funded professionals, democratizing robust scientific observations. In our new paper in the journal ‘Sensors’, led by Andrew McGonigle (University of Sheffield) we review the development of smartphone spectrometery.
Abstract: McGonigle et al. 2018: Smartphone Spectrometers
Smartphones are playing an increasing role in the sciences, owing to the ubiquitous proliferation of these devices, their relatively low cost, increasing processing power and their suitability for integrated data acquisition and processing in a ‘lab in a phone’ capacity. There is furthermore the potential to deploy these units as nodes within Internet of Things architectures, enabling massive networked data capture. Hitherto, considerable attention has been focused on imaging applications of these devices. However, within just the last few years, another possibility has emerged: to use smartphones as a means of capturing spectra, mostly by coupling various classes of fore-optics to these units with data capture achieved using the smartphone camera. These highly novel approaches have the potential to become widely adopted across a broad range of scientific e.g., biomedical, chemical and agricultural application areas. In this review, we detail the exciting recent development of smartphone spectrometer hardware, in addition to covering applications to which these units have been deployed, hitherto. The paper also points forward to the potentially highly influential impacts that such units could have on the sciences in the coming decades
I recently published an article in French pop-sci magazine La Recherche about the wondrous microbial ecosystems on glaciers and ice sheets (here for French speakers). For those English speakers who do not subscribe to la Recherche, here is a translation.
Also, I strongly recommend the excellent translator who worked on this article with me – contact me if you need translation services and I can link you up.
The microbes accelerating glacier melting
Our planet is getting warmer and losing its ice. Mountain glaciers are disappearing and the great Greenland and Antarctic ice sheets are shrinking. These masses of ice are giant coolers for the planet and they reflect energy from the Sun back out into space, meaning the smaller they become, the more the planet warms. Surprisingly, the process of melting the vast glaciers and ice sheets is accelerated by microscopic life.
Glacier and ice sheet melting depends upon more than just temperature. Most of the energy driving melt comes from sunlight that hits the ice surface. Dirtier, darker ice absorbs more solar energy than clean, bright ice meaning more energy is available to drive melting. On the Greenland Ice Sheet in particular, the ice becomes very dark in the summer, with large areas reflecting just 20-30% of the sunlight hitting them. This is not a new phenomenon – in fact it was noticed by explorers during the great polar expeditions of the late 1800s. Intrigued, they examined samples of ice under their microscopes. The dark colour of the ice was not simply due to dust as they expected – astonishingly, the ice was stained by life (Nordenskjold, 1875). The ice surface is a patchwork of greys, reds and purples coloured by the collective effect of countless microscopic organisms, with potential knock-on effects for Earth’s climate (Uetake et al., 2010; Takeuchi et al., 2006; Yallop et al., 2012; Cook et al., 2017).
Microbes on Ice
When explorer Adolf E Nordenskjold arrived on the Greenland Ice Sheet in 1870 he immediately noticed the dark grey-purple colour of the ice. His colleague, a biologist called Berggren, examined the ice under the microscope and discovered a rich variety of microbial life. The importance of their discovery was clear to them – this life darkens the ice and increases its melt rate. Nordenskjold even suggested that the microbial life was the “greatest enemy of the mass of ice” and an accelerator of deglaciation at the global scale (Nordenskjold, 1875)!
Until recently, Nordenskjold’s observations of life on ice have remained obscure footnotes in the history of Polar exploration; however, as climate science has become increasingly urgent in the twenty-first century, Nordenskjold’s work has gained new significance. Contemporary scientists have confirmed the presence of a microbial ecosystem growing on the surface of the Greenland Ice Sheet and elsewhere and are now attempting to quantify their ice-darkening effect. Although it is an extreme environment where temperatures are low and nutrients scarce, there is abundant sunlight and liquid water to support photosynthesis, meaning microalgae can grow on the ice surface (Uetake et al., 2010; Yallop et al., 2012). The days are long in the Arctic in summer, with the sun staying above the horizon for twenty-four hours per day for part of the season, exposing the algae to intense and prolonged solar energy. This powers photosynthesis but over time the exposure stresses the ice algae, causing them to produce biological sunscreen molecules to protect their delicate photosynthetic machinery. These ‘carotenoids’ colour their cells very dark purple and enhance the biological darkening of the ice surface.
At the same time, the ice surface is peppered with holes that are often cylindrical but can have complex and irregular shapes (Cook et al., 2015). These holes range from centimeters to meters in diameter and depth and contain mixtures of biological and nonbiological material bundled up into small balls that sit on the hole floors. Nordenskjold first noticed these holes on the Greenland Ice Sheet and named them ‘cryoconite holes’, from the Greek for ‘holes with frozen dust’. These holes are the most biodiverse microbial habitat on Earth’s ice. They form when dust and debris becomes tangled up by long, thread-like cyanobacteria. The cyanobacteria are photosynthetic and as they grow they exude polymers that act as biological glues, binding the bundles of material together into stable granules. This biological bundling and binding of material creates a microhabitat for other microbes, especially those that can feed on molecules produced by the photosynthesizing cyanobacteria. As the granules grow they become heavier, meaning they settle on the ice surface. The biological material makes them especially dark, so the ice underneath melts quickly, causing holes to form in the ice surface with the granules sitting on the hole floor. The holes provide protection from the weather and intense sunlight and also prevent the microbes from being washed away. The cyanobacteria therefore sculpt the ice surface and engineer a comfortable, stable habitat where diverse microbial life can thrive in this extreme environment.
Cryoconite holes are more than icy buckets that hold microbial life. They are more like microbial mini-cities on ice, with each connected to many others by meltwater flowing between ice crystals just under the ice surface. Cryoconite microbes engage in engineering and construction, production, consumption, competition, predation, growth, reproduction, death, decay, immigration and emigration. There is both import and export of nutrients, waste and other biological material. At the same time, the hole itself changes its shape and size in response to changing environmental conditions with the emergent effect of maintaining the light intensity at the hole floor, promoting photosynthesis (Cook et al., 2010). Algal blooms and cryoconite are crucial components of the wider Arctic ecosystem, acting as stores of carbon (which they draw down from the atmosphere and fix into organic molecules), nutrients and biomass which can all be delivered to soils, rivers and oceans as glaciers melt (Stibal et al., 2012). Truly, these are widely interconnected complex adaptive systems created biologically on Earth’s ice.
The Cutting Edge of Life on Ice
While life on ice has been known for many years, most of the literature on the subject has been produced during the twenty-first century. Modern molecular biological techniques have enabled scientists to catalogue the species present in cryoconite and algal blooms, and modern instruments can measure their darkening effect. However, there are several major gaps in our understanding of life on ice. To quantify their effect on ice darkening worldwide, we need a reliable method to map icy microbes at the scale of entire ice sheets. From a biological perspective, we know which organisms live in algal blooms and cryoconite so we must now concentrate on determining how they function and what ecosystem services they might provide that could impact human society.
To estimate the total coverage of life on ice, we must detect it without actually being present to take samples. It is relatively easy to take samples and analyse them in a laboratory to tell if life is present, but doing the same from the air is a different problem. In addition to biological darkening, soots and mineral dusts colour the ice. Also, as the ice melts the crystals change shape and melt water can fill the spaces between them, which in itself changes the way the ice absorbs and reflects solar energy. Disentangling the biological signal from these other darkening processes has proven to be challenging.
However, because the darkening of ice by living cells is due to biological molecules that absorb light at specific wavelengths, we may be able to use the spectrum of reflected light to identify them. Chlorophyll, for example, absorbs red and blue light much more effectively than it absorbs green light (which is why we see leaves as green). For other biological molecules, the peak absorption will be at slightly different wavelengths, and non-biological materials will have their own absorption patterns too. However, while identifying ‘signature spectra’ is simple when only one material is present, it is much more difficult when several species with different light absorbing properties are mixed with non-biological materials. All of the light absorbers can be scattered unevenly and mixed vertically within the volume of ice which can itself be a complex aggregate of variously sized ice crystals and liquid water. The reflected light is a tangle of signals that can be hard to unpick.
At our laboratory at the University of Sheffield, we are working on a purpose-built drone which will fly back and forth over a patch of the Greenland Ice Sheet taking images at specific wavelengths of light. By analysing these images we hope to be able to produce a map of life on ice. Using the drone means we can follow the flight on foot and take ground samples to examine in the laboratory, enabling us to link the drone images to actual concentrations of different light absorbers on the ground. The wavelengths imaged by the drone match up with those measured by several Earth observation satellites, meaning that achieving life-detection using a drone should then enable the same from space.
As well as knowing where the life is, we also need a deeper understanding of how it functions. Recognition of ice surfaces as microbial habitats came at the same time as an explosion in accessible and affordable techniques in field molecular microbial ecology, meaning several groups have used high-throughput sequencing of marker-genes to identify the particular microbes present within cryoconite communities (e.g. Cameron et al., 2012; Edwards et al., 2014; Stibal et al., 2014, 2015). Environmental genomic techniques have also been used to investigate the total genetic composition of cryoconite communities (Edwards et al., 2013). To date, these have been snapshot studies, but in the very near future great insights into the functioning of cryoconite microbes will come from rapid metagenomic, metabolomic and metatranscriptomic studies. It has been suggested that ice surface microbes might be good targets for bioprospecting. Since they are able to thrive in conditions of low temperature, high light and low nutrients, they may well utilize survival strategies that we can exploit, either by extracting novel genes and biomolecules, or by observing and gaining ecological knowledge. Cryoconite has been suggested to be a potential source of antifreeze proteins, novel antibiotics and cold-active enzymes. The shape, illumination conditions and flushing with flowing meltwater make cryoconite holes natural analogs to industrial bioreactors which are commonly used to synthesise valuable biomolecules (Cook et al., 2015).
Deep insights will come from combining the expertise of microbial ecologists with glaciologists and physicists who, together, will link processes operating at the molecular level with changes in ice surface colour and patterns of melt, which suggests insights into the ecology of ice surfaces might one day be obtainable from the sky or from space. While this is some way off, great insights could be gained from a shift towards a holistic understanding of the ice surface as a ‘living landscape’.
We are working hard to achieve remote detection of life on ice for the purposes of mapping biological ice darkening from satellites and improving our ability to predict future ice melt. However, there is another potential outcome from this work… what if instead of looking down from space at our own planet, we turn the sensors around and start looking out?
The Greenland Ice Sheet is, in many ways, a good place for developing life detection technologies that can be applied to the search for life on other icy planets and moons. Take, for example, Europa. A recently funded NASA project will examine this icy moon of Jupiter for signs of life because of its potentially habitable icy shell and subsurface ocean. On Europa, the icy surface is sunlit and seeded with possibly mineral-rich snow that forms when liquid water in its subsurface oceans escapes via huge geysers (Hand et al., 2017). There is therefore a potentially dusty ice surface illuminated by sunlight that could support photosynthesis, just like the Greenland Ice Sheet (although the solar energy flux and temperature is lower on Europa and photosynthesis is highly unlikely). Any life detection technology that works on the Greenland Ice Sheet will have to overcome the challenges of ice optics, interference by mineral dusts and uncertain biological pigment composition, which would also be the main challenges for remote detection of life on the surface of other icy planets and moons. The frontiers of glacier biology on Earth may therefore intersect with the cutting edge search for extraterrestrial life.
While many people think of Arctic and Antarctic ice as lifeless places, there is in fact abundant microbial activity on Earth’s glaciers and ice sheets. But more surprising is the huge impacts of these tiny organisms. By changing the colour of the ice surface, microbes are potentially enhancing the rate at which glaciers and ice sheets are shrinking, but we cannot yet build them into our climate models. The research priority now is mapping these ecosystems from space because this will enable us to estimate their impact on ice melt worldwide and improve our melt forecasts. The same technologies that will enable us to detect life on Earth may eventually be useful tools for searching for icy life elsewhere in the universe. There is also much to be learned about way these microbes function that can educate us about the limits of life in extreme environments. The true sharp edge of glacier biology research involves understanding how these microbes are able to sense, survive and drive environmental change. The study of life on Earth’s ice is deeply interdisciplinary and ultimately it requires us to recognize – as Nordenskjold did – the intricate bridges joining the very big and the very small.
Cameron K, Hodson A J, Osborn M (2012) Carbon and nitrogen biogeochemical cycling potentials of supraglacial cryoconite communities. Polar Biology, 35: 1375-1393
Cook J, Hodson A, Telling J, Anesio A, Irvine-Fynn T, Bellas C (2010) The mass-area relationship within cryoconite holes and its implications for primary production. Annals of Glaciology, 51 (56): 106-110
Cook, J.M., Edwards, A., Irvine-Fynn, T.D.I., Takeuchi, N. 2015. Cryoconite: Dark biological secret of the Cryosphere. Progress in Physical Geography, 40 (1): 66 -111, doi: 10.1177/0309133315616574Cook et al., 2017
Edwards A, Pachebat J A, Swain M, Hegarty M, Hodson A, Irvine-Fynn T D L, Rassner S M, Sattler B (2013) A metagenomic snapshot of taxonomic and functional diversity in an alpine glacier cryoconite ecosystem. Environmental Research Letters, 8 (035003): 11pp
Edwards A, Mur L, Girdwood S, Anesio A, Stibal M, Rassner S, Hell K, Pachebat J, Post B, Bussell J, Cameron S, Griffith G, Hodson A (2014) Coupled cryoconite ecosystem structure-function relationships are revealed by comparing bacterial communities in Alpine and Arctic glaciers. FEMS Microbial Ecology, 89 (2): 222-237
Hand, K.P., Murray, A.E., Garvin, J.B., Brinckerhoff, W.B., Christner, B.C., Edgett, K.S., Ehlmann, B.L., German, C.R., Hayes, A.G., Hoehler, T.M., Horst, S.M., Lunine, J.I., Nealson, H.H., Paranicas, C., Schmidt, B.E., Smith, D.E., Rhoden, A.R., Russell, M.J., Templeton, A.S., Willis, P.A., Yingst, R.A., Phillips, C.B., Cable, M.L., Craft, K.L., Hofmann, A.E., Nordheim, T.A., Pappalardo, R.P., and the Project Engineering Team (2017). NASA, Report of the Europa Lander Science Definition team. Posted Feb 2017. https://solarsystem.nasa.gov/docs/Europa_Lander_SDT_Report_2016.pdf
Stibal M, Sabacka M, Zarsky J (2012a) Biological processes on glacier and ice sheet surfaces. Nature 1554 Geoscience, 5: 771-774
Stibal M, Schostag M, Cameron K A, Hansen L H, Chandler D M, Wadham J L, Jacobsen C S (2014) Different 1558 bulk and active microbial communities in cryoconite from the margin and interior of the Greenland ice 1559 sheet. Environmental Microbiology Reports, DOI: 10.1111/1758-2229.12246
Stibal, M., Schostag, M., Cameron, K. A., Hansen, L. H., Chandler, D. M., Wadham, J. L. and Jacobsen, C. S. (2015), Different bulk and active bacterial communities in cryoconite from the margin and interior of the Greenland ice sheet. Environmental Microbiology Reports, 7: 293–300. doi:10.1111/1758-2229.12246
Takeuchi, N., Dial, R., Kohshima, S., Segawa, T., Uetake, J., 2006. Spatial distribution and abundance of red snow algae on 35 the Harding Icefield, Alaska derived from a satellite image. Geophysical Research Letters, 33, L21502, doi:10.1029/2006GL027819
Uetake, J., Naganuma, T., Hebsgaard, M. B., and Kanda, H. 2010. Communities of algae and cyanobacteria on glaciers in west Greenland. Polar Sci. 4, 71–80. doi: 10.1016/j.polar.2010.03.002
Yallop, M.L., Anesio, A.J., Perkins, R.G., Cook, J., Telling, J., Fagan, D., MacFarlane, J., Stibal, M., Barker, G., Bellas, C., 25 Hodson, A., Tranter, M., Wadham, J., Roberts, N.W. 2012. Photophysiology and albedo-changing potential of the ice-algal community on the surface of the Greenland ice sheet, ISME Journal, 6: 2302 – 2313
Camping on the ice sheet in September/October was a new experience – I’d never seen darkness on the ice before! The lack of light pollution and cloud-free skies made for a truly spectacular display of the Northern Lights. It was -25C and 35 knot winds pretty much constantly, so it was a constant battle between wanting to get into a tent and warm up and not wanting to miss a second of watching the aurora dancing over the milky way, with passing satellites and the occasional shooting star.
While on a personal level this was an incredible treat, it also presented some pretty major challenges for working with drones on the ice. The aurora knocked out the radio communications linking our drones to their controllers, meaning they could only be controlled over local wifi, reducing their range from a few hundred metres to about 30!
In our new paper we report on some novel tech that uses the sensor in a smartphone for ultraviolet spectroscopy. It is low cost and based entirely on off-the-shelf components plus a 3-D printed case. The system was designed with volcanology in mind – specifically the detection of atmospheric sulphur dioxide, but may also have applications for supraglacial spectroscopy. As far as we know this is the first nanometer resolution UV spectrometer based on smartphone sensor technology and the framework can be easily adapted to cover other wavelengths.
This follows on from a Raspberry-Pi based UV camera reported in Sensors last year which was recently adapted to sense in the visible and near-infra-red wavelengths for use on ice. The plan now is to compare the images from the Pi-cam system to those made using an off-the-shelf multispectral imaging camera that detects the same wavelengths. A report of testing this camera system for detecting volcanic gases is available at Tom Pering’s blog here.
Raspberry-Pi and smartphone based spectroscopy could make obtaining high-spectral resolution data a real possibility for hobbyists and scientists lacking sufficient funds to purchase an expensive field spectrometer. The system is also small and light and therefore more convenient for some field applications than the heavy and cumbersome field specs available commercially and can easily be mounted to a UAV.
A new paper, led by Johnny Ryan, shows that a consumer grade digital camera mounted to a drone can be used to estimate the albedo of ice surfaces with an accuracy of +/- 5%. This is important because albedo measurements are fundamental to predicting melt, but satellite albedo data is limited in its spatial and temporal resolution and ground measurements can only be for small areas. Methods employing UAV technology can therefore bridge the gap between these two scales of measurement. The work demonstrates that this is achievable using a relatively simple workflow and low cost equipment.
The full workflow is detailed in the paper, involving processing, correcting and calibrating raw digital images using a white reference target, and upward and downward shortwave radiation measurements from broadband silicon pyranometers. The method was applied on the SW Greenland Ice Sheet, providing albedo maps over 280 km2 at a ground resolution of 20 cm.
This study shows that albedo mapping from UAVs can provide useful data and as drone technology advances it will likely provide a low cost, convenient method for distinguishing surface contaminants and informing energy balance models.
After testing the UAV performance in Svalbard in March, I realised the original ‘tripod’ landing assembly was not going to cut it for work in the Arctic. To prevent damage from landing in cryoconite holes and to spread the drone’s weight when landing on snow, I have added some ski’s modified from off-the-shelf landing gear for RC helicopters. This also has the added advantage that if one attachment point fails, the UAV is still landable, which is not the case for the tripod design.
As well as the ski’s, I have now added the Red-Edge camera’s down-welling light sensor to the top of the casing. This will automatically correct the images for changes in the ambient light field in each wavelength.