Flight #2
On the 11th of August we quietly launched the Spacebits 2 balloon, it was planned since our first flight to correct the problems we had and prove that our initial design worked. In case you don’t remember, althoughSpacebits 1 was a success, we had major last minute troubles with the main computer which included the telemetry and the main radio (read the story here for details if you haven’t yet).
So we gathered the team at Fernando’s house at Lagos on the 10th to get everything ready for the Spacebits 2 launch. Until then we all had been working remotely on several tasks like buying new cameras, a good Yagi antenna and new and improved Xbee radios, getting authorizations from INAC, fixing lots of software problems and small hardware components and discussing procedures and new ideas. So one day should be enough to put everything together, we thought. (mistake #1)
Celso arrived at Fernando’s place in the morning. Both Filipes arrived just after lunch from Lisbon, carrying the (huge) Helium bottle with them in the car. Time passed by, a few snooker games and swimming pool dives later and eventually we managed to put the new radio up and running for tests. We fixed the equipment and the omnidirectional antenna at Fernando’s house rooftop and started the test transmission.
Filipe Valpereiro, trapped at the roof.
The plan was: Celso, Fernando and Filipe Varela would go for a drive in the car with the Yagi antenna on the front window and the Macbook laptop and the other Xbee in the back and check on how far could they receive the packets. A few rounds around Lagos with the car and we were positively impressed with the results, we had received xml packets at all locations, but from one minute to the other it just stopped working. (mistake #2) We went home, fixed a bad solder on the Xbee receiver connector and went back to the road for take two.
It was then around 20h30, night. That time we went far to the Serra de Monchique, it’s a 46km car ride to the top of a mountain where you can see Fernando’s house at Lagos, approximately 30km if you use a straight line. When we finally got there we had a very good surprise, we were receiving the xml packets in perfect conditions, we didn’t even had to point the Yaggi to the transmitter spot with much precision, amazing we said.
So we were pretty confident that the new radio and antennas would kick ass. Plus the software bits had been ironed out too. There were still some troubles with the GPS chip of the main board, it seemed to take a few minutes to boot up, find the satellites and start working, but Filipe Varela and Valpereiro would handle and fix that through the night.
Celso checked the upper winds forecast online for the next day, it looked very good. Mild southeast winds. He tweets “there’s a good chance we’ll recover the balloon in Spain tomorrow” at the @flyspacebits account just out of fun, not really believing it.
Time flew, and suddenly It was 7am, Fernando was sleeping a few minutes at the couch, Celso went home for a 2 hours sleep and a bath, and both Filipes kept working (mistake #3).
Alright, sunshine, it was 11am, everybody was up and running. We finished the final details to close the payload box, new battery packs, a few hot glue drops here and there and strong plastic strips to hold the electronics together, cameras were ready too, INAC’s authorization papers, we load all the gear into the two cars and off we went to Castro Verde, the same location that we used for the Spacebits 1 flight. We made a pit stop at the first gas station and bought lots of water, coke and cookies for the road, it was going to be very hot that day (>35ªC).
By around 1pm we were finally at the launch spot.
Team unloading the heavy helium bottle
The day was extremely hot and there was some wind at the ground. The sky seemed clear, there were no clouds but it had a funny slightly browned color. The reason, we found later after investigating, was this: there was a huge cloud of dust traveling from the Sahara desert all the way to Portugal. There was this thin dust everywhere in the sky, it’s a rather strange phenomena we learned but not rare. We had noticed and commented amongst friends that 2 days before, when it rained, all the cars went covered with dirt, almost mud. Also reading the main Portuguese meteo forum looking for answers helped. See the impressive photo of the mega dust cloud arriving to Portugal 2 days before?
Cloud of dust approaching south of Portugal
So we unload all the gear from the cars, the usual stuff from Spacebits 1 and in 15mins we’re set. Two problems emerge: first, there’s an adaptor missing, the one that connects the Helium bottle to the balloon tube used to fill the balloon. Second, we completely forgot about the the amount of Helium calculations. (mistake #4)
The first problem was easy to fix on spot, the solution was… yes, duct tape, the best invention ever and the solution for every problem. It worked. The second problem was however a bit more difficult to solve in a hurry.
Duct tape fixes every problem in the world
Let’s go back in time to the day before the Spacebits 1 launch. On that glorious day someone joined the team to help us: our friend João Pedro. He’s also a uber-geek like us and during that day he focused solely and completely on the Helium calculations, browsing through all the other HAB projects on the Internet, reading all the theory and making a neat and nice spreadsheet where we would input the weight of the balloon and the payload and it would spit the weight of the counterweight object that we would use during the fill to get the correct amount of helium into the balloon. Except the radio & telemetry problems, the Spacebits 1 flight went so well in terms of physics that we underestimated the small details for the Spacebits 2.
Turns out that calculating the right amount of Helium is very important for any HAB flight. In fact it can completely determine the success or failure of the project. If you fill the balloon with too much Helium it will go up very fast, inflate quickier, and explode earlier than expected. If on the contrary, you fill it with less Helium, it will take a lot longer to go up, making it vulnerable to high altitude winds and very low temperatures, it wont reach the maximum altitude either, and ultimately increases the chances that you loose the HAB. Lets repeat: the right amount of Helium is very important (mistake #5).
So there we were, standing like four fouls doing the palm-face dance. Anyway, after a rather humiliating failed attempt to lift off Spacebits 2 we eventually filled up the balloon with enough Helium, turned everything on, made the basic ground tests, and the thing was launched. As soon as the balloon left ground we noticed that it was more windy than the first time causing the balloon to travel very fast horizontally.
First take. The balloon didn’t lift. Humiliating. Yes, that’s Fernando Afonso dying in laughter.
Back to the launch spot for Take 2.
We packed everything back into the two cars and the hunt began. Fernando Afonso took Filipe Valpereiro in one car with the iPad 3G and Celso, Filipe Varela and a friend took the other with the Yaggi antenna and the ground station hardware and software. Immediately, we started receiving xml packets with the telemetry data.
The onboard GPS was working great, as well as the radio, and we had full control of the payload coordinates. Everything looked great. We also found another mistake (#6), some of the units in the packets weren’t normalized. For instance, the altitude value would rotate each 6500 meters back to zero, which could easily be fixed with data post-processing but did cause us to be unaware of the exact altitude in real time.
The Yaggi antenna that we used to receive the data packets from the Balloon.
Quickly we found that the balloon was:
- Ascending very slowly (this was due to the helium miss calculations, it should have a lot more helium for our payload weight)
- It was traveling very fast (due to 1) to southeast (to Spain and Algarve).
We couldn’t keep up to the speed with our cars. Constantly calculating road directions while following the balloon wasn’t easy too (todo #1). Soon, we were into the torrid desert roads of the deep Alentejo/Algarve heading Spain and the more time passed the more distant the balloon was. At one time, we were getting packets from a distance of more than 80km. The only good news at this point was that the radio was working surprisingly well, we could easily pick the data packets in the air just by pointing the Yaggi antenna to the theoretical balloon position (todo #4), inside the car, while traveling, impressive (todo #3).
Two hours later something very stupid happened to us. I’d say it was the fail of the day. The night before we made a simple minor change to the website API by introducing a fallback condition in the code where the timestamp of the received radio packet would be the webserver’s time if the received packet wouldn’t carry a timestamp itself. Big mistake (#7 and #8).
It turned out that from time to time (specially at 80kms of distance) we got bad data packets in the air with garbled values. Also, we found later, the on-board realtime clock was (very) delayed, we forgot to set it at launch time (see mistake #4). So in practice we got a packet into the system which had a valid GPS coordinate but, because the radio timestamp was garbled, had the timestamp of the webserver, causing it to be the most recent valid coordinate for hours to the eyes of the client APIs. You know what means don’t you? Our iPhone/iPad apps, and the website itself, were telling us that the balloon had fallen in a valley near Spain (while in fact it was traveling to the sea of Spain at high speed). Here was a situation where having altitude measurements would have saved us.
We lost almost two hours due to this stupid bug until we realized something was wrong. I swear I thought I was going to die. We went into unpaved roads of all sorts, trespassing private properties (some of them advertising dangerous signs of wild life) until we finally got to the deep hot valley. Temperature was 45C! We had no sleep, were very tired, and had no proper lunch. As we climbed those torrid hills up and down, my heart started pumping to a scary and dangerous rate. Seriously dear readers, this was bad. On the other hand, “killed by a software bug” would give an excellent darwin award.
Team returning from the 45ºC hot valley, where they spent 2h looking for nothing.
Soon enough, after we fixed the API problem, we had a clear idea on where the balloon actually was. It was in the middle of the sea, near Vila Real de Santo Antonio, on the Spanish side, heading Cadiz. It had fallen in the water. By this time we were no longer receiving radio packets. Also the fallback GSM module couldn’t possibly work because we made yet another mistake: the SIM card had the roaming disabled so it wouldn’t work in Spain (mistake #9).
We drove all the way to VRSA and went straight to the coast guard office near the port. Had a chat with the captain, explained him the whole story and gave him the exact GPS location of the balloon. He smiled and told us: amazing story. If it was under Portuguese waters, we would take you on a speed boat and get the thing right now. Unfortunately it’s not, and we can’t go into Spanish waters, sorry (todo #5).
But we couldn’t give up so easily, not after what we went through. We went to the Spanish border and talked to the Police there. This time they didn’t find the story that funny, just said they couldn’t help, asked us for our contacts in case something would come up and waved us goodbye with a poker face.
We had no more options. It was night already so we called the day off and declared Spacebits officially dead. It was kind of a sad moment. In a way we were happy that we finally got the radio working and had the full telemetry log, but sad because we lost the payload due to our own new and apparently unimportant mistakes.
At the Portugal/Spain border, talking with the Spanish police.
During the ride back to Lagos, Filipe Varela was still in denial. Kept calling the fallback module’s phone number in the hope that it would ring, over and over. The day ended up with the team having a few burgers at the McDonalds. Boy, were we hungry.
Days passed by and the team was still discussing on what went wrong and what should be fixed for Spacebits 3 on the project’s mailing-list. Varela did an amazing job and normalized the received data (see mistake #6) on a spreadsheet, so we had pretty graphics and a clear notion of the Balloon’s rise and fall, now with altitude. And of course, it confirmed that it took a dive in the sea.
And then, 5 days later, when least expected something amazing happened. Celso got a phone call from Spain that went more or less like this:
- Hello, is this Celso?
- Er.. Yes?
- We have found your balloon!
- (silence)
- Hello?
- (unquotable exciting talk follows)
Unbelievable, the payload had been floating on the high sea for 5 days and finally gets to the shore, near Cadiz. Someone doing a walk at the beach finds the payload, reads the contacts and decides to call us. Two miracles here: first, I don’t usually answer unknown numbers at my cellphone. Second, we found later that our contacts could barely be read, a few more days and we couldn’t even be contacted (mistake #10).
So, after a few phone calls we agree on a 100€ reward and the finder sends our beloved balloon and payload through the mail. We got everything back by the 27th of August. The excitement in the room when the box was opened was huge. Of course the whole electronics and cameras were severely damaged and beyond repair, the batteries had exploded inside, possibly due to the salty water, and the mix of salt with acid had ruined everything.
But the big question was: are our 3 SD cards intact with data on them? (2 cards for both cameras, 1 card for the GPS and sensor’s data). While we waited for the box to arrive in the mail, we read about memory cards recovery. It seems that memory cards falling in the water is a pretty common thing these days. It turns out that memory cards are hermetically sealed by design (and tested against it during the QA process). All you need to do is to clean the contacts with alcohol (>95%), wash the cards with water to get the salt off and dry them well, and they should be ok.
All the 3 cards were OK indeed. We had the sensor’s logs intact and lots of great footage from above. Unfortunately, the vertical camera didn’t took any photos or videos for some reason, but the problem was CHDK related (todo #6).
Are we the luckiest guys or earth or what? Here’s a set of photos that speak for them selves:
Box arrives in the mail, from Spain.
Taking the mini SD card off the main board.
This Arduino Mega will have no more missions.
You can see what salt, water and battery acid did to our beloved payload box and electronics.
I don’t think this IXUS100 will work again.
Team and SAPO friends look at the data in the cards, for the first time.
Beyond repair.
And now some photos of the Spacebits 2 flight:
Flight starts. You can see the dust in the sky already.
See the cloud shades projected in the dust? That’s how dirty the sky was. Visibility to the ground was near zero.
Balloon approaches the sea.
Check the video of the flight. Remember the dust storm? You can clearly see it in the videos. Watch the shadow of the clouds in the dust. The footage isn’t near spectacular as the first flight.
Here’s the RAW data from the onboard mini SD card as well as the spreadsheet with the data post-processed and normalized.
Spacebits 3, 4 and 5 are in the making right now. We just can’t stop doing this. Expect more news within a month or so. See you soon ;)
Comments on mistakes:
#1 – It wasn’t. We would need at least two days to complete everything, which led us to do some of the things in a hurry and have no sleep (again).
#2 – Make sure you solder everything nicely before you go on the road. Bad connections and little details can be a big deal.
#3 – Having no sleep when a big day that requires concentration and physical strength is ahead, is bad, very bad. Most of team members didn’t sleep more than 2-3 hours.
#4 – Do an extensive checklist ffs and review everything on the day before (related to #1 and #3)
#5 – Helium quantities calculations are critical to the success of the flight.
#6 – Normalize the data units and test the radio payload packets with real data before launch, otherwise you nay get in trouble and/or loose important data.
#7 – Don’t change structural stuff, like the API, on the night before the launch.
#8 – Make sure you prepare unit tests for everything by simulating a real flight with real data. Repeat those tests if you change anything. This is basic software development cycle best practices stuff.
#9 – Make sure the SIM cards have roaming enabled. Also, don’t use special operator numbers to send the SMS to, just use a regular +351 prefixed number. If possible, test your module under roaming conditions too.
#10 – The contacts sheet on the payload is very important. It can save it. Make sure you use good materials and protect the sheet with plastic for the case of water.
Todo list for Spacebits 3:
- We could automate the real time road directions calculations with the iPhone/iPad app.
- Also, the client apps on-ground should send their GPS coordinates to the website for future data correlation and analysis.
- Do not touch the main radio design. It works great.
- Client apps should use the built-in compass and tge GPS readings to aid us pointing the Yaggi antenna to the optimal position.
- If you’re going to send your balloon to the sea, make sure it’s Portuguese waters so you can get a free speed boat ride adventure included.
- CHDK is great but it isn’t exactly the most stable piece of software. It’s a hack. It caused us problems before with scripts stopping after the first round of photos. We should improve this and find a way for the camera to give us feedback at the launch spot. Also, we should look at other solutions, we’re buying a Go Pro camera to experiment with.
Gallery of Photos from Spacebits 2
