Research update: road-testing our bat call detection software
Last week we published our latest Bat Detective research update, explaining how we’ve used the data you’ve labelled to improve our algorithms for detecting bat calls in audio data. The next step is to assess how well they perform in real-world bat survey situations – and the best way to do this is to put them into practice on new data. So following last week’s update, this post will focus on examples of where we’ve road-tested our new tools this year.
The first was a two-week garden bat survey we carried out over two weeks in July, in a row of suburban gardens in southeast England. We were testing our algorithms on data collected by two different bat detector types, both of which were deployed outside to collect data autonomously over multiple nights. Alongside some specialised full-spectrum ultrasonic detectors, which are commonly used in acoustic wildlife monitoring research and citizen science projects including the Norfolk Bat Survey, we were also also using some brand new, low-cost acoustic sensors developed by our collaborators in Oxford for a variety of biodiversity monitoring applications (including citizen science). This offered a great chance to test our call detection tools on data from two different types of acoustic sensor.
The videos below show examples of how the algorithms are used on the UK garden survey data. The first shows the distinctive calls of a common pipistrelle passing by the detector, and the second are calls from a noctule, the UK’s biggest bat. Each red line on the spectrogram shows where the algorithms predict a bat call – so you can see in these cases they’re performing well, successfully detecting every echolocation call in these recordings. For each predicted call, the algorithms also calculate a probability of detection – a measure of how likely the sound is to be a bat call, with reference to our training data from Bat Detective. This enables us to set a threshold for call detection – by setting a high probability threshold, this then makes it possible to only detect sounds that are very likely to be a bat call. As we discussed in the last post, this can help to reduce the risk of false positive detections (i.e. falsely thinking there’s a bat call, when actually there isn’t one).
Once the calls have been detected, the next step in the analysis is to classify those calls to species – what bat is that? Doing this requires a second set of algorithms, trained to distinguish between the calls of different species. Our group are currently working on developing these, and incorporating them into our tools, in order to more fully automate the analysis process (and therefore make it faster and more reliable for researchers monitoring bats).
As well as the UK data we collected in July, we’re also now using our call detection tools to analyse recordings collected during a huge bat survey on the Atlantic island of Madeira. There are thought to be three bat species on Madeira, including the endemic Madeira pipistrelle (Pipistrellus maderensis), which is listed as vulnerable on the IUCN Red List. However, little is known about the distribution and abundance of bats on the island, and its relatively small size makes it an ideal study system for an island-wide bat survey.
So a member of our research group (in collaboration with M-ITI in Madeira), has been busy out in the field throughout August and September, deploying full-spectrum bat detectors in locations across the island, which were selected to provide a randomised sample of its full range of habitats and altitudes. A map of the sample sites is shown below, with each blue marker showing where bat detectors have been placed. Now the data have all been collected, we’re starting to use our automated tools to detect bat calls in all the recordings. From there we can start to ask questions about the distribution of bats on the island, and to assess what habitats and locations might be particularly important for conservation.
This is a great example of how the tools we’ve developed using the Bat Detective data can now be applied to understand bat ecology and assist in conservation efforts. Without tools like these, the sheer quantity of audio data collected during a summer-long survey at this level of detail – which clocks up to hundreds of hours of survey-time in total – would be almost impossible to analyse by hand. Keep an eye on the Bat Detective blog in the coming months, as we’ll keep you informed on the last few steps in developing our bat call detector tools for open-source release, as well as letting you know about this and other test-case projects.
Research update: developing new bat monitoring tools with Bat Detective data
Hello from the Bat Detective team! It’s been a busy summer at Bat Detective HQ after an amazing spring with British Science Week and the World Tour. So we’ve been a bit quiet on the blog in the last couple of months while we’ve been working on updating our automated tools and road-testing them on new data. We’re now coming close to having our results ready for scientific publication, and we’ve also had the chance to put our new software tools into practice on analysing some brand new bat survey data. So over the next two blog posts, we’ll be updating you on our progress, explaining where the Bat Detective project is at right now, and showing how we’ve been using all the data you’ve helped us to label.
In this first post we’ll discuss how we’ve used Bat Detective data to improve our automated bat call detection tools, and highlight some of the challenges we’ve encountered along the way. In the next post, we’ll show some examples of where we’ve been testing out our software tools on bat survey data from the UK and Madeira – keep an eye on the blog for that one very soon.
The team have also been up to a few other things in the last few months. Bat Detective’s Rory Gibb (me) gave a project update talk at the Zooniverse’s first ever ecology workshop, where there was some fascinating discussion about how citizen scientists can become increasingly involved in some of the major challenges facing ecology and conservation in future. We’ve also just had a big article on Bat Detective and its sister project iBats published in the latest citizen science-themed issue of Environmental SCIENTIST – the article will be available to read online in the near future, so we’ll share it here when that happens.
Training machines to recognise bat calls: why and how?
In our last research update post a year ago, we explained how advances in machine learning technology have enabled us to train algorithms to automatically recognise bat calls in ultrasonic survey recordings. This is important because newer bat detectors can be deployed in the field for weeks or months, collecting so much audio data that it’s almost impossible to analyse them manually. By making it possible for bat researchers to quickly and reliably find where bat echolocation calls are in these recordings, automated tools are creating exciting new opportunities to study bat ecology, behaviour and conservation at much larger scales than ever before.
Machine learning involves training computer algorithms to automatically recognise bat echolocation calls in recordings, by showing the computer thousands of examples of what they look and sound like. In our last research update we showed how training the algorithms on increasingly large amounts of data from Bat Detective improves their performance. For that reason, and also to include a greater diversity of bat sounds from around the globe, we’ve asked for your help in labelling our World Tour data over the last year. And thanks also to the efforts put in by volunteers during British Science Week, we’ve now got thousands of new bat call annotations to incorporate into our detector tools – so one of our current challenges is exploring the best ways to use all of these new data.
We’ve now got the detector algorithms up and running, and we’re currently testing them out to assess how well they perform. The figure below shows an example of the detector in action on a snippet of audio data from the iBats global bat monitoring programme. The recording is displayed as a spectrogram underneath, with sounds showing up as bright markings. The graph above shows the computer predicting where it thinks the bat calls are – each vertical red line shows where the computer predicts there is a bat call. The height of the red lines tell us how certain the computer is about its predictions, where higher indicates more confident. The green bars show where a human expert has confirmed that bat calls are present – so in this example, the computer has successfully recognised all the bat calls.
Are you sure that’s a bat? The problem of false positives
However, there are still some errors where the computer thinks there is a bat call, when there actually isn’t one (a ‘false positive’). This is a problem for monitoring bat populations, because too many false positives could result in researchers overestimating the true number of bats in an area, which could for example have an impact on conservation efforts. You can see a clear example of these errors in the next figure below, where the computer falsely predicts that the mechanical noises at the bottom of the spectrogram are lots of bat calls.
So to improve this, we’ve also been including non-bat sounds from Bat Detective – those insect calls and mechanical noises you’ve also helped us to find. By training the algorithms to also recognise what bat calls don’t look like, we can significantly improve their accuracy. The image below shows the difference: it’s the same audio clip, but there are now far fewer false positives (red lines).
This is a great example of the importance of testing out these tools on new data from a variety of times, places and detector types. This helps us get a better idea of where they’re under-performing, and how they can be improved before we release them as open-source tools for other researchers to use. So with that in mind, keep an eye on the Bat Detective blog next week for our next research update: we’ll be showing some examples of where we’ve road-tested them on new bat survey data recorded during this summer. We’ll also be uploading a new set of data from Russia – one of our last few World Tour stops – so stay tuned for that.
And a huge thanks again for all your efforts with labeling the data on Bat Detective, both during this year and throughout the whole project – we wouldn’t have been able to get to this stage without your input, and it’s really exciting to see the work of our community of volunteers starting to produce results.
Bat Detective arrives in Japan, our next World Tour stop
Welcome to the latest stop on our World Tour! We’re now in Japan, after spending the last two months uploading data from Australia and New Zealand to the Bat Detective site. Firstly, a massive thanks from the Bat Detective team for all your efforts in listening to and classifying our data so far this year – thanks to the amazing efforts of citizen scientists during the World Tour as well as British Science Week back in March, we’ve got a much larger dataset of labelled bat calls to train our automated algorithms with, and the results are improving.
In the coming months we have a few more World Tour stops before we reach the end of our global bat search. This month we’re in Japan, with a new set of data uploaded to the Bat Detective site that was recorded on car-driven transects during 2010 and 2011 in locations throughout Japan: you can see where the surveys were carried out on the map shown below. We hope you’ll enjoy searching for bats in Japan, and if you have any queries just let us know via the Talk section of the Bat Detective website.
For this month’s blog post, to accompany our Japan data, we’re publishing a short piece written by iBats and Bat Detective’s founder Kate Jones during her 2010 trip to Japan, during which she collected iBats audio data and hosted training workshops for the iBats monitoring program. Scroll down below the map to start reading…
“I stare slightly queasily down at the tiny but perfectly formed green-tea plantations and rice paddies in the valley far below, as the car winds down the narrow mountain roads of the Mount Fuji highlands in Japan. We stop to try to manoeuvre around an impossibly large truck loaded with locals, and I am struck by the beauty of the mountains surrounding us, lush green forested slopes and azure blue lakes matching the skies overhead.
A crazy expansion of the iBats monitoring program over the past few months has me visiting places and people that I have only imagined. ‘What are you doing in Japan?’ asked the Japanese air attendant politely as I waited for the bathroom on my 11 hr flight to Tokyo. ‘I am hoping to develop a program with local people and scientists to monitor bat populations’, I replied carefully. ‘You see, you can use changes in bat populations like a heart monitor to check the health of nature and the impact of people on the environment’. ‘Bats?’ she squealed, ‘I LOVE bats,’ and proceeded to draw me a map of Japan marked with large crosses where I should visit to see bats. Equally unexpected was finding out at dinner on our first night given by our host, Dai Fukui, that grilled eels are actually very tasty. Especially as tempura with sesame dressing. Yum.
Stuart Parsons, sitting next to me in the car, is looking even more pale than I. Obviously the sake of the previous evening is not going well with Dai’s mountain driving. David Hill on the other hand is made of sterner stuff, alternating between calmly explaining Japanese culture to us in the back and chatting easily to Dai in Japanese in the front. Yesterday was spent exploring caves in the mountains with members of the Japanese bat group (Komori no koui) and listening to horseshoe bats bubbling and warbling over our heads.
In the evening we were introduced to a whole new concept in fieldwork, a ‘mist netting barbecue’. We left the hard work to the local experts and sat around chatting to the group while some of the students brought us mist netted bats for us to look at. The endemic tube-nosed bat (Murina ussuriensis) was especially cute — David’s favorite. Dai explains that he has found this bat hibernating in little tubes it has made in the snow in winter. We drink cold sake and ponder how this bat copes with subzero temperatures. I explain the importance of monitoring to the group and how our acoustic equipment works. Stuart displays the calls in real time from the bats flittering over our heads on his brand new iPad. Whilst outwardly dismissing this gimmick, I am secretly marveling at how Stuart is among the select few in the world that can out-geek me with Apple products.
Although bats are protected in Japan, there is no formalised monitoring of their populations and little general public awareness of the important role bats play in ecosystems. This is a fact that the bat group is trying to change with their awareness-raising activities around Japan every year, culminating in a bat festival in August. The car lurches down the mountain and I see a Bat Conservation Trust sticker on the car in front in our little convoy.
‘Are you a member?’ I ask curiously of the owner, Keiko Osawa, earlier that day, during our lunch overlooking Nagashima Dam. ‘Yes,’ motioning to her husband Yushi, ‘we like getting Bat News’. Yushi is a photographer and they seem to spend most of their time travelling the world photographing fruit bats.
‘Kate San,’ asked the secretary of the bat group Akeiko Mekosa politely. ‘How many members does Bat Conservation Trust have?’. She exclaimed in surprise when I told her over 5000, and said she struggles to get their membership up to 500. The group wants to develop an iBats project here over the next year, and hopes to raise the profile of bats and start its first national monitoring program. Stuart and I are here to help this get started, and to run a workshop on iBats monitoring, volunteer management and acoustic analysis for them.
Beside me in the car, Stuart is beginning to look more normal and is checking his photos on his new iPad just to annoy me. We chat about his plans for his iBats project in New Zealand next summer (our winter). In contrast to Japan’s forty species of bats, New Zealand only has two. Stuart bristles at my dismissive tone and says that what they lack in numbers they make up in distinctiveness.
I have to agree with him for once — New Zealand is home to the short-tailed bat (Mystacina tuberculata) which spends most of its time in the moist fern-filled forests scampering on the ground hunting for fruit and insects. Although the iBats car-based acoustic monitoring would not be useful to monitor Mystacina as they are confined to deep forest, Stuart sees the potential for using iBats to monitor long-tailed bats (Chalinolobus tuberculatus). He has agreed to trial our new iBats application for the iPhone – you just attach your iPhone to an ultrasound detector and send the recording and GPS information straight to the iBats website. We did a test run in the Fijordland of New Zealand’s South Island in February and apart from us being bitten to death alternatively by sand flies and mosquitoes, it worked perfectly. Long-tailed bats happily flew over the car as we made our way along the transect through Lord Of The Rings country. Stuart is excited about using the technique to better understand the distribution of this threatened endemic species.
We head back up into the clouds with the help of the nice Japanese lady satnav to where we are staying tonight and holding the workshop. The workshop venue is a lodge in the highlands with traditional Japanese style rooms, where the bed is made every night from bedding beautifully folded and organised in wooden cupboards with ornate sliding doors. I’m especially excited about the tales of the Japanese bath houses, with their piping hot plunge pools fed from the surrounding hot springs.
The expansion and interest in the iBats project has been rather overwhelming over the last few months and has seen the team giving workshops in Hungary, Ukraine and most recently Russia, where the vodka flowed a little too easily but the welcome and enthusiasm for the project was amazing. I am overwhelmed by the generosity of the people I have met around the world and their commitment to conserve their bats in the face of conditions much more problematic than those we face. As we stop to investigate the first bat house built in Japan, one of the group asks me where next for the iBats project. Hmmmmm, what about Australia?”
Bat Detective New Zealand: our latest World Tour location
Welcome to New Zealand, the latest stop on the Bat Detective World Tour! As of today we’ve just uploaded a new set of audio data to Bat Detective, recorded along survey transects on New Zealand’s South Island. You can see the locations of the surveys on the map below, and visit the Bat Detective site now to get searching for bats.
Prior to this we’ve spent the last month hosting audio data from iBats Mexico, which was neatly timed to coincide with the publication of the latest automated bat call classifier from members of our research group – a classifier for Mexican bat species. As with our results from the algorithms we’re training with Bat Detective data, it’s another example of how advances in machine learning technology are increasingly enabling the development of tools and systems for effective acoustic monitoring of bats (as well as biodiversity more broadly). You can find out more about the Mexican classification tool and how it will assist in bat population monitoring via some great coverage in the media, including in Science and an interview with our group’s Dr. Veronica Zamora-Gutierrez and Prof. Kate Jones on the BBC.
Bats occupy a unique space in the ecology of New Zealand, since they are the country’s only endemic terrestrial mammals – before humans settled the islands, the only mammals native to New Zealand were three bat species (the greater short-tailed bat, lesser short-tailed bat and long-tailed bat) and several species of marine mammal. Since human settlement this has changed, with invasive mammalian predators (such as rats and cats) driving massive declines in the populations of endemic birds and bats. Indeed, the last sighting of the greater short-tailed bat was in 1967, and it is now believed to be extinct, while New Zealand’s other two bat species, the lesser short-tailed (pictured below) and long-tailed bat, have both experienced major declines and are priorities for conservation.
The acoustic data on Bat Detective New Zealand, recorded on South Island in 2010, are much noisier than lots of the recordings you’ll have previously heard on Bat Detective. Many clips have a great deal of background noise and static, in addition to distinctive bats and unique rattling insect calls. Although this can make it challenging to determine what sounds you’re hearing, it’s very useful to include data like these while training algorithms to automatically find bat calls – this will help improve the algorithms’ ability to detect bat echolocation calls in even the most noisy of real-world acoustic recordings. This will make them more useful for surveying bats in naturally noisy and complex acoustic environments, such as urban areas where there is lots of human-generated sound, or highly biodiverse (and therefore very loud) rainforests.
We hope you’ll enjoy helping us search for bats in our New Zealand data, and as ever if you’re struggling to figure out whether a sound is a bat, an insect, or something else, you can use the Talk page to flag it up and discuss it with us and other users.
Bat Detective World Tour begins next week!
As we announced recently, Bat Detective is about to go on a World Tour, and we’re inviting you to join us, starting this coming Monday 2nd November. The iBats monitoring programme, which provides us with our audio data, has now been running for a decade, with our volunteers collecting recordings of bat surveys in locations worldwide. However, to date Bat Detective has made only some of that recorded audio available for our citizen scientists to explore, and that has come mainly from Eastern Europe.
So starting on 2nd November, over the course of the World Tour we’ll be regularly uploading new sets of data to Bat Detective from different countries across the globe. Each country – ranging from Europe to places in Africa, the Americas and Asia – has its own selection of bat species alongside other acoustic inhabitants, so you can expect to encounter a variety of different soundscapes while searching for bat calls worldwide. Your help with classifying bat calls, insect noises and other sounds in these places will be of valuable assistance in our work towards creating automated bat detectors – read more about our research here.
On Monday we’ll begin our trip in the United Kingdom, home of the Bat Detective team and the Bat Conservation Trust. Keep an eye on the Bat Detective blog for a more in-depth post about British bats, citizen science, and what you might hear while exploring our UK data. We hope you’ll enjoy joining us in searching for bats across the globe, so stay tuned and see you next week…
The Bat Detective World Tour – join us soon!
Bat Detective has now been running for over three years, and all the input from our community of citizen scientists has been invaluable in helping us to develop machine learning algorithms for detecting bat calls in audio recordings – so thank you! As we explained in our recent post about our current research, adding more annotated data – and from a wider variety of recorded sound environments – will further improve the accuracy and reliability of our bat detector software. This will bring us closer to our goal of creating smart automated tools for monitoring global bat populations, which we hope will in turn help us to learn more about how human activities are affecting the earth’s ecosystems.
So we’re about to take Bat Detective on a World Tour, and we’re asking for your help in searching for bat calls in recordings from across the globe.
Since 2005 the amazing groups of volunteers and researchers on the iBats monitoring programme have been recording audio bat surveys in places ranging from the UK to Japan, North America to sub-Saharan Africa — each with their own distinct environmental soundscapes and unique selection of bat species. So far, however, the audio snapshots we’ve uploaded to Bat Detective have only been those from Eastern Europe. This means we still have lots of new data from all over the world in need of exploring and annotating, all of which will build into improving our automated bat detectors.
So throughout the World Tour we’ll be travelling from country to country, regularly uploading new sets of audio data from a selection of places where iBats volunteers have surveyed. We’ll begin in the UK, where the Bat Detective team are based, before jetting across the globe to search for bats in countries in Africa, North America, Australia and Asia. And as we go we’ll be adding posts to this blog, reporting on where and when the surveys were recorded, and highlighting some of the local bat species (and other curious sonic inhabitants) you can expect to encounter in each location.
Keep an eye on the Bat Detective blog for dates, news and updates as we progress through the tour. And until our travels start in a few weeks’ time, you can still help us track down bats in our current Eastern European data – visit the Bat Detective site to get searching. Thank you for your contributions over the last three years, and we hope you’ll enjoy helping us to search for bats worldwide!
Bat Detective research update – and stay tuned!
This week it’s our birthday! It’s been exactly three years since we first launched the Bat Detective project on 1st October 2012. Since then we’ve had an amazing response from our community of citizen scientist bat detectives, with over 94,000 unique audio snapshots explored by nearly 4,000 volunteers, and more than 11,000 bat calls discovered.
All the hard work you’ve put in so far has been invaluable. Using the data from Bat Detective, we’ve been developing computer algorithms that can automatically search for and detect bat calls in audio recordings with a very good success rate. To do this we’ve taken advantage of recent rapid improvements in machine learning technology for recognising complex patterns within data — such as the distinctive shapes of bat calls.
We’ve had great results so far, thanks to all the audio data the bat detective community has searched through, and all the calls you’ve identified. The majority of those have been searching calls (over 7,000), but you’ve also labelled over 2,000 each of the more rarely recorded social and feeding calls. We’ve used this annotated data to train our machine learning algorithms, by showing them thousands of examples of what bat calls look and sound like. This enables them to better tell apart the sounds we’re interested in from other background sound, such as insect calls and mechanical noise.
We’re now at the stage where we can use these algorithms to detect bat calls throughout the millions of recordings collected through the iBats monitoring project. What this means is that we’re a key step closer to developing automated software for accurately detecting and species-identifying bat calls from recorded audio — a vital move towards a global monitoring programme for bat populations. To read more in-depth summaries of the work our team members have been doing towards that goal, see our recent blog post for Methods In Ecology & Evolution.
This graph shows how well our algorithms are currently performing at finding known bat calls within a large set of audio data that we’ve already annotated. The closer the curve reaches to the top right of the graph, the better the results we’re getting — this means we’re maximising the proportion of the bat calls detected within the audio (increasing the recall) while minimising the number of non-bat sounds that are incorrectly classified as bat calls (improving the precision). When we use four times as much data from Bat Detective to train the algorithms (shown as a green line), we get a large improvement in performance compared to when we use much smaller amounts of data (shown as the blue and purple lines).
So the more data we can use to train our algorithms, the more accurate and reliable they will be. This will allow them to more successfully detect even calls recorded in challenging acoustic conditions, when there’s lots of background noise or the bats are far away from the detector — those trickier cases where they’re failing now. That’s why the ongoing help from the bat detective community is so valuable for our research. So later this month we’ll be announcing some new developments in the Bat Detective project, where you can help us search for bat calls in recordings from all around the globe — stay tuned for more information very soon!
I’ve Found a Bat … Now What?
As you know, we have a huge number of audio snippets and we want to know what’s in them! We would certainly never be able to find the bats all by ourselves — There are just that many recordings.
This is where the Bat Detective project steps in. We’re really hoping that citizen scientists will help us to locate and identify bat tweets and insect clicks that have been captured across the globe by the lovely volunteers.
Once we gather your votes on the contents of a snippet, this will highlight the recordings of interest and provide us with labels. On top of hunting for the bats, your labelled sequences of searching, feeding and social tweets will also allow us to analyse the different sounds.
As the number of your identifications grows, the more information we will have; it doesn’t even matter if a recording is given a range of different labels! By pointing out the controversial sounds, we can gain an insight into the calls that are hard to distinguish.
If the majority of your votes on a snippet’s contents agree, then this recording can be compared to other snippets that have been similarly labelled. By finding the features they have in common, and what sets them apart from other sounds, we can begin to automate our quest for interesting sounds.
To help us decide on useful identifying features as an seo company for bats, which could be wildly obvious or devilishly subtle, we can use machine learning. This involves designing a computer program that we can hand our snippets and your voted labels to, for it to then return different ways the sounds can be grouped together. Analysing these resulting clusters will then pave the way for automatically detecting the different bat tweets (and insect buzzes) within any number of snippets!
Welcome to Bat Detective!
Bat Detective is an online citizen science project where the public can help to monitor bats across Europe and track changes in the environment by listening to the weirdly wonderful ultrasonic tweets of bats.
Bat Detective project allows visitors to take part in conservation by listening out for bat tweets in recordings collected over 80,000 km of roads across Europe by thousands of volunteers from the iBats program, including bat recordings from the heart of Transylvania.
By sorting the sounds in the recordings into insect and bat calls, bat detectives will help the Bat Detective team learn how to reliably distinguish bat tweets to develop new automatic identification tools.
Bats use lots of different types of sounds, from singing to each other to find a mate, to using the echoes from their tweets to find their way around. Usually bat sounds are inaudible to humans as they are too high for us to hear, but special ‘time expansion’ ultrasonic detectors convert these sounds to a lower frequency, and visitors to Bat Detective can listen to these unique recordings and help the Bat Detective team distinguish different sounds.
One out of every five species of bats is threatened with extinction and better automatic identification tools are desperately needed to quickly process vast amounts of sound data collected by volunteers from bat monitoring programs who survey bat populations each year.
Bats are found all over the world from local parks to pristine rainforests and monitoring their population trends provides an important indicator of healthy ecosystems. Developing new tools that allow the Bat Detective team to interpret population trends from sound will allow bats’ tweets to act as a way to track environmental change.
Bat Detective has been developed by the science team at University College London and Bat Conservation Trust with the development team at Citizen Science Alliance, which runs Zooniverse.org with funding from the Alfred P. Sloan foundation.
Holy Citizen Science! Watch this space.