11:00am 4th December 2016 :: Think: Health

In a special collaboration, Think: Health and Think: Digital Futures take a look at the future of non-invasive medical technology.

Harriet – Type 1 Diabetic
Fiona – Harriet’s mum
Professor Hung Nguyen – Director Health Technologies, University of Technology Sydney
Dr. Peter Puya Abolfath- Inventor, Exoflex
Dr. Marc Carmichael- Lecturer, School of Electrical Mechanical and Mechatronic Systems, Unviersity of Technology Sydney
Nick Barnes – Project Lead, VibroMat, CSIRO Data61

Presenters/Producers: Ellen Leabeater & Josh Nicholas
Producers: Sam King & Jake Morcom


Fiona Wilkinson: …So this is the diabetes straw.

Ellen Leabeater: Oh, so there’s a whole lot of pink balls almost…

Fiona Wilkinson: So this is the little syringe that fits into the pump… (fades down)

Ellen Leabeater: This is Fiona. Her daughter Harriet has Type 1 diabetes.

Harriet Wilkinson: I’m Harriet Wilkinson. I’m 11 years old and I have Type 1 Diabetes.

Fiona Wilkinson: I’m Fiona, I’m Harriet’s Mum.

Ellen Leabeater: So Harriet, how old were you when you found out you had diabetes.

Harriet Wilkinson: I was six years old and just about to turn seven.

Ellen Leabeater: There’s an organ in your body called the pancreas that creates a hormone called insulin. That insulin breaks down the food you eat and turns it into energy. If your pancreas stops making insulin, it can’t properly breakdown food into energy. Symptoms of diabetes include being really thirsty and really tired.

(TO HARRIET) And what happened that made you find out you had diabetes.

Harriet Wilkinson: Well, I’d been drinking lots of water and then one night I was really sick and we went to the doctors. I got checked – they checked my blood sugar and it was too high, and we went to hospital for a night and then that was that. We went home and I’ve had it ever since… not like it’s ever going to go away…

Ellen Leabeater: Nobody knows what causes Type 1 diabetes, but there is no cure and it’s not preventable. It’s different to Type 2 diabetes which is caused by lifestyle factors such as being overweight or having high blood pressure. Type 2 diabetes can be managed with healthy eating and exercise. But it’s the same idea – the pancreas struggles to make insulin. Of people with diabetes, about 15% have Type 1 and 85% have Type 2.

So, Fiona is testing Harriet’s blood, something they have to do a couple of times a day. They use a needle to prick a spot of blood on Harriet’s finger and they use what’s called a glucometer to test her blood sugar levels.

(TO HARRIET) Does it hurt anymore?

Harriet Wilkinson: No, not really.

(Glucometer beeps)

Fiona Wilkinson: And she’s 4.4.

Ellen Leabeater: What are you meant to be?

Harriet Wilkinson: 6.

[Music plays]

Ellen Leabeater: Remember, people with diabetes don’t have insulin to help them breakdown energy. To get a little bit more technical, your digestive system breaks down carbohydrates into glucose. Glucose is more easily recognised by your cells as a form of energy, and insulin helps transport this glucose energy to each cell via your bloodstream. When it reaches the cell, glucose is burned to produce energy. Without insulin, the glucose kind of just floats around in your blood. If you’re a diabetic, there are two things that can happen: you can have too much blood sugar in your blood, or not enough blood sugar. Too much blood sugar is called a hyperglycaemic attack, and usually occurs if you eat food and don’t follow it up with insulin. But the real worry is a hypoglycaemic attack, or “hypo”. This is when blood sugar drops too low.

(TO HARRIET) Harriet can you explain to me what a “hypo” is?

Harriet Wilkinson: A hypo is when my blood sugar goes below what it’s meant to be and there’s not enough glucose in my blood, so I start feeling dizzy and then eventually if we leave it too long I pass out.

Ellen Leabeater: And then what do you need to do if you’re having a hypo.

Harriet Wilkinson: I need to have something sugary, like sometimes just a banana or mostly a popper.

Ellen Leabeater: For example, if you go for a run, your body is able to use stored energy to keep your blood sugar at a healthy level while you’re exercising. For diabetics, their pancreas can’t regulate their blood glucose level, and they start to have a hypo. It’s easily managed if you test yourself and you’re below the right level – 6 as you just heard a moment ago – or if you can recognise the symptoms like mood changes, shakiness or paleness. If you don’t pick up these symptoms, it can get pretty serious. In the later stages of a hypo, you can begin to slur your speech, not be able to drink or swallow and you can lose consciousness. Remember Harriet was six going on seven when she was diagnosed with diabetes. Diabetes is difficult to manage with kids as it is, just because it’s hard for them to know the symptoms of a hypo and self-manage insulin injections and needles. But the real worry for parents of diabetic children is what happens when they go to bed.

Fiona Wilkinson: Certainly in the early years, we would get up 2-3 times a night to make sure she wasn’t going too low in her sleep. So we would check her – if it was too low, we’d have to wake her up and give her some juice or something sugary. If it was too high, we’d often have to inject her while she was asleep with more insulin to try and bring it down to a safer level.

Ellen Leabeater: So your alarm would go off 2-3 times a night to go and check Harriet?

Fiona Wilkinson: Yes, we would often set a phone alarm, and if he got up once and we knew we’d have to get up again, he’d reset it under my pillow and we’d do that. We’ve had nights we’re we’ve got up every two hours probably.

Ellen Leabeater: He being Harriet’s father and Fiona’s husband Ray.

(TO FIONA) How did that make you and Ray feel?

Fiona Wilkinson: We’ve had some very very anxious times. Not so much now, because we’re more confident with it, but in those early years, we were highly anxious.

Ellen Leabeater: And sleep-deprived.

Fiona Wilkinson: Very sleep-deprived! And very protective… just frightened I guess.

Ellen Leabeater: And are you still testing Harriet’s blood sugar at night?

Fiona Wilkinson: Yeah, we’re four years down the track now and just last night I know Ray set an alarm to check her because he wasn’t happy with her level at 10pm when we checked it before we went to bed.

Ellen Leabeater: “Dead in bed” syndrome is one of the major causes of death among Type 1 diabetics under the age of 40. It’s the sudden and unexpected death of a diabetic who otherwise appears healthy. Nobody quite knows what causes it, but night-time hypoglycaemia is one theory.

Fiona Wilkinson: I think we’re uncertain whether Harriet would wake herself up if she had a very bad hypo. And I guess at this stage, we still feel responsible for her if she’s too high for the whole night, which is why we still do some night time clicking. I think if we were confident she’d wake up before she got too low, we’d feel more relieved. Or if there was some sort of way of being alerted that she was going low, then we’d not have to do night time checks.

Ellen Leabeater: So Harriet, do you currently wake up if you have a hypo?

Harriet Wilkinson: Well, normally if I am too low I can’t fall asleep. But if I am asleep and I go low, normally I can’t really wake up.

[Music plays]

Ellen Leabeater: Hi, I’m Ellen Leabeater.

Josh Nicholas: And I’m Josh Nicholas.

Ellen Leabeater: Today, it’s a Think collaboration between Think:Health and Think:Digital Futures.

Josh Nicholas: We’re going to be exploring a couple of different stories, all to do with how non-invasive medical technologies have the potential to change lives here in Australia.

Before we go any further, you should probably know what non-invasive technology is. It’s best described in relation to it’s binary: invasive technology. That’s something that you have to have surgery for, like a pacemaker. Non-invasive is gentler on your body. It doesn’t require surgery and in some cases and in some cases you can hide it under your clothes.

Ellen Leabeater: To avoid waking up multiple times a night, Fiona and her partner Ray could use a type of invasive technology that would monitor Harriet’s blood sugar.

Fiona Wilkinson: There is a compatible continuous glucose-monitoring device. It’s an invasive way of continuously monitoring her sugar though, so she’s not particularly keen – and nor are we – on having more needles or cannulas or whatever on her skin… Or more tubing and another pager sized device for her to carry around. Other than that, the only other one I’ve heard of are the dogs. People are training dogs to pick up when a child’s blood sugar goes low, and then the dogs come and alert Mum and Dad by licking them on the face… yeah, I’ve heard of a couple of people who’ve had success with a dog, but we haven’t really thought seriously about going down that path.

Josh Nicholas: So that’s where Harriet and her family are right now. They have to be constantly vigilant, making sure her blood sugar never goes to high and certainly never goes too low. But there are also people working on just this problem.

Hung Nguyen: …And I met a doctor. She had a three-year old boy at that time and she was getting up every night, every two hours, everyday to check blood, and then I thought, there must be a better way. And so that’s why I went into how to essentially measure blood glucose without taking blood, and to measure it continuously.

Ellen Leabeater: This is Professor Hung Nguyen. He’s Assistant Deputy Vice Chancellor at the University of Technology Sydney.

Josh Nicholas: We took a walk around Professor Nguyen’s workshop at UTS. Just imagine workbench after workbench, spilling with wires and electronic parts that do god-knows what. And there are all sorts of mad inventions as well, like a robotic hospital bed that can follow you around…

Ellen Leabeater: A wheelchair equipped with artificial intelligence…

Josh Nicholas: Right! And we’ve even featured him on Think:Digital Futures before, because he’s made a robot that can play chess and tic-tac-toe.

Ellen Leabeater: So when Professor Nguyen started thinking about this problem, he started thinking about all the other kinds of devices that people use when they go to bed.

Hung Nguyen: When I started to realise the danger of hypo, I saw a device that would shock you if you snored. And I wore the snoring watch for a few nights and it was very painful!

Josh Nicholas: So the snoring watch was a bust. But eventually he hit on something that many of us have used before, just not in this way.

Hung Nguyen: My technology works on ECG.

Ellen Leabeater: What’s ECG?

Hung Nguyen: Well, heart signals… so, you have electrocardiogram – you know, the ECG that the doctor uses to measure your heart with sensors. So we have a system to measure blood glucose by looking at ECG, and from the heart signal, you can actually decipher the information to read how blood glucose is working.

Ellen Leabeater: Just so you can picture it, an electrocardiogram is where someone – usually a doctor or a nurse – will stick a bunch of electrodes to your chest. It measures the electrical activity as your heart beats.

Josh Nicholas: It’s that green monitor beside your bed at a hospital, and normally doctors are looking at the line going across the screen, searching for a pattern.

Ellen Leabeater: Are the spikes too high or too low? Is there too much space between them?

Josh Nicholas: What Professor Nguyen realised is that electrical signals can tell us other things about what your body is doing or experiencing. And maybe it could help to diagnose blood sugar levels as well.

Hung Nguyen: What’s happening is when the blood sugar starts to go low – when it gets to a certain level – you start sweating, and it’s actually causing the heart to respond. From there we looked into those rates and worked out when it’s hypo and so on…

Ellen Leabeater: From this original idea, it took 15 years for Professor Nguyen to actually make it.

Josh Nicholas: It wasn’t the mechanics that was hard. Judging by all the spare parts lying around his workshop, it probably wouldn’t have taken him that long to knock something together. Rather, he spent that time gathering a bunch of data and training an algorithm to be able to read those patterns.

Ellen Leabeater: And this is what he came up with.

Hung Nguyen: So this is the device. You put this one directly across the chest and it will communicate…

Josh Nicholas: Oh, so you put on kind of an old heart rate monitor around the chest…

Hung Nguyen: Yeah, definitely it’s something like that…

Ellen Leabeater: And you just wear this at night?

Hung Nguyen: Yeah, you wear this at night and it will tell you if your blood glucose is low, and it will alert the parents or anybody to take some sort of food.

Ellen Leabeater: So what we’re seeing here at the moment is a device that fits in the palm of your hand and measures your ECG. When your ECG levels change, the device alerts the parents’ smartphone that the child needs to be woken up and given something to eat to get their blood sugar levels back to normal. Although, it’s still a work in progress due to funding.

Josh Nicholas: He isn’t there yet. You can’t yet buy any of these devices, but soon someone like Harriet might be able to sleep a little bit easier.

[Music plays]

Ellen Leabeater: This device that this Professor at UTS has invented where it just sits on your chest and it uses your ECG to test your blood sugar and sends an alert to your smartphone if your blood sugar is low – so it’s a non-invasive technology… if you had something like that how do you think it would change your lives?

Fiona Wilkinson: It would take enormous amounts of pressure off parents and I guess, as Harriet takes on more and more responsibility as she gets older, it would also take enormous pressure off her. It would be a dream come true for us, so well done Professor!

{Music plays]

Ellen Leabeater: You’re listening to Think:Health Futures, a special collaboration between Think:Health and Think:Digital Futures.

[Music plays]

Josh Nicholas: So the next story you’re about to hear comes from Producer Sam King.

Ellen Leabeater: Sam, would you like to tell us a little bit about this story.

Sam King: Hey guys. Yes, absolutely. I started with this idea that exoskeletons are inherently cool… I kind of started with exoskeletons and tried to go somewhere from there. But in doing that I kind of came across this cool little niche that exoskeletons have made for themselves. They’re a new technology, and like any new technology, they’re still kind of finding their place in the world, and they’re finding niches in places I never would of thought. One of them is in medicine, which is when I came across the EXOFLEX, which is what you’re going to hear with Peter Abolfathi, which is like a hand therapy device.

Ellen Leabeater: So let’s just go back a step. What exactly is an exoskeleton?

Sam King: Yeah, well, it’s a wearable robot. That’s how I describe it to people.

Ellen Leabeater: So who’s going to benefit most from this?

Sam King: Well, they have a variety of uses as far as I understand. Through doing this story I kind of focused on the medical angle. The one I looked at was a therapy device. So basically it allows people who have experienced paralysis in their hands to wear this robotic glove that provides tension as they move their hands so it helps them build their muscles back up. And there was another for stroke patients to help them get movement back in their arms. So these machines focus on specific limbs, but eventually the idea is to have full body exoskeletons to maybe help people that have had full paralysis.

Ellen Leabeater: So is the idea that if these people don’t have exoskeletons, then it’s just heaps and heaps of physiotherapy?

Sam King: Well, exactly, with expensive therapists. But with stuff like the EXOFLEX, all they need is the machine and they can do it by themselves at home through an easily-programmable interface – a little tablet kind of thing.

Ellen Leabeater: And you actually got to wear one of these exoskeletons. What was it like?

Sam King: I did! I was actually lucky enough to try both of them on actually. I checked out the EXOFLEX first. It was this really weird feeling. I think Peter in the story describes it as having someone else’s hand sitting behind your own and moving your hand for you. But it’s more like having a machine that know what you’re about to do just before you actually do it, because it reacts to the tiny muscle movements in your hand. It’s just incredible- it’s really difficult to describe.

Ellen Leabeater: You’re about to hear just what it’s like. Sam, thanks very much!

Sam King: No problem at all! Thanks guys.

[Music plays]

[Whirring, metallic sounds]

Peter Abolfathi: This is the quick calibration. So, just do your maximum extension.

Sam King: Ok.

Peter Abolfathi: Perfect.

Sam King: So now the machine knows my limits?

Peter Abolfathi: Exactly.

Sam King: If you could see this machine, you’d think I was being kitted out for some high-tech sci-fi spacewalk. I’m in this sleek white boardroom in a small tech start-up in Pymble, and I’m in the process of having a robot strapped to my arm.

[Whirring sounds]

Sam King: Wow, that’s cool!

I’m wearing a robotic exoskeleton, which runs down my left forearm, curving over the top of each of my fingers. It’s called the EXOFLEX.

Sam King: How do people usually react when they do this for the first time?

Peter Abolfathi: Well, you know, we haven’t shown a huge number of people what you’re doing now. Some of these are new behaviours that we’ve implemented and some of it is a bit secretive.

Sam King: That’s Peter Abolfathi, walking us through my new augmentation. And who better to introduce us to this futuristic power fist. After all, he invented it.

Peter Abolfathi: Imagine it as a bunch of fingers behind your hand. If you’re paralysed and you’re having difficulties moving your fingers, the therapists comes from behind you, on top of your hand, and put their hand over yours and gently pushes down. But then they’ll have to have suction caps as well and do the reverse. That’s what this does. It goes behind your fingers – on what we call the dorsum of your fingers – and it’s stuck to your fingers through these Band-Aids that we’ve put on.

Sam King: I’ve got to say, it looks like something out of a Ridley Scott movie. But it’s actually a tool for rehabilitation. Exoskeletons are bleeding out of science fiction and into our lives. Lacking space flight or giant aliens to defend ourselves from, the catalyst for this is medicine. But before we dive into that, I’m going to take a moment to visualise this thing for you. It’s kind of hard to describe, so bear with me.

Start with a cast made of blue plastic and strapped to your left forearm. It’s identical to the kind of thing you’d wear if you sprained your wrist or something, only it’s covered in Velcro. Next, they attach the motors- they’re these long thin bricks that run along the arm from elbow to wrist. They’re roughly the length of a ruler and about an inch thick. They’re Velcro-ed on, side-by-side along the cast: four motors for the fingers, one special motor for the thumb. Now here is where the magic happens: at the wrist end of each motor, three thin, flat hyper-flexible pieces of metal slide out over the back of the hand and curve all the way over each finger. These strips of metal are the same size, and they’re stacked on top of each other, only a few millimetres thick and a centimetre wide.

Take a look at your hand right now: that’ll make this easier, trust me. Now imagine if each bone on each of your fingers had a magnet strapped to the top. So what you’re left with is a series of long thin motors side-by-side on your forearm with a thin strip of metal curving over each finger. And there are plastic nodes connecting the metal to each bone. Now those metal strips slide over each other individually, pushing and pulling each bone in your finger based on the algorithm being fed to the motor. So, it’s completely customisable – one motor, one finger; three motors, three fingers… you get the idea. Today, I’m wearing two: one controlling my pointer and the other controlling my index finger. With the power of maths, the exoskeleton knows exactly what you’re about to do before you do it. The feeling of having a machine you’re wearing push and pull your fingers for you is absolutely utterly indescribable.


Peter Abolfathi: So try using your fingers – you see how it reacts to you? That’s what we’re talking about when we say human-machine interface. So it’s a very novel self-sensing mechanism inside that can feel the forces from your fingers and move the fingers accordingly.

Sam King: So, I’m not pulling this metal out – the motor just knows that I’m moving my finger.

Peter Abolfathi: Yeah, it’s reacting to you. Right, so, part of that could be for this function that we just did, to be able to quickly self-adjust for you. The other one could be… not only can it follow you with very little force, it could also follow you with some force. So it could actually assist you – so you move forward, but it actually moves forward at a higher rate, so it assists you in closing your hand. And then the other way of using it would be the resistance form. You’re moving it and it’s resisting your movement like a rubber band…

Sam King: And that’s the strength training?

Peter Abolfathi: Yeah, exactly. And so in hand therapy departments, you’ll see these things called out-riggers, which are splints with rubber bands attached to fingertips. So they’re doing the same thing – keeping your hand open but creating some resistance. The problem with those systems is that with a rubber band- just like any other spring – the more you pull, the higher the force gets, because the more you stretch, the greater that force becomes, and that’s actually the opposite of what you want. You want it to release that force as you close your hand, not increase that force. You can’t really achieve that with a rubber band, but this device can apply the same exact force throughout your movement.

Sam King: Imagine the potential of a thing like this. For people who’ve lost movement in their hands; for building the hand strength of stroke patients and those who suffer from muscular degeneration or partial paralysis. This machine could give them their hands back. In fact, these robots have such an enticing promise in medicine that this is how it happens. This is how exoskeletons finally make their way from science fiction into our world.

[Science fiction/robotics sound effects]

Sam King: Not through conflict or conquering the galaxy, but through medicine. Don’t believe me? At Royal North Shore Hospital in Sydney, they have a pair of robotic legs teaching people to walk again. At the University of Technology, also in Sydney, they’re designing an exoskeleton that wraps around the entire arm, from shoulder to wrist. One day, it could help people recover the use of their arms. I was lucky enough to try it out.

Marc Carmichael: So this is the industrial controller that comes with the arm. If we tell the robot to move to this position, it controls all of that. It tells the joints how to move…

Sam King: I’m with Dr Marc Carmichael in UTS’s Robotics Lab. He’ll take it from here.

Marc Carmichael: So this is technically an exoskeleton. It’s mounted on this platform here, and operated it would sit or stand roughly aligned with the shoulder of the robot.

Sam King: See, I thought an exoskeleton was something that you’d wear.

Marc Carmichael: Not necessarily. So, you can have exoskeletons designed for individual limbs, maybe just for the wrist or elbow… an entire body exoskeleton is challenging, and we are getting to that point where I actually just recently saw someone like Honda or I can’t remember who… they had a full body exoskeleton and it’s like something out of the movie “Aliens”. It was really impressive.

Sam King: That’s a personal influence for you, right? “Aliens”?

Marc Carmichael: Yeah, it is.

[Robotic/mechanical noises]

Marc Carmichael: Growing up I loved that movie with the yellow Exoskeleton… yeah amazing.

Movie audio: Get away from her, you b—-! [robotic noises]

Marc Carmichael: …And that’s part of the reason why we have this robot as yellow – apart from it being high-viz.

Sam King: I see the resemblance.

Marc Carmichael: Yeah!

Sam King: While it’s not exactly Ripley’s “Powerloader”, this exoskeleton is remarkable in its own way. Think for a moment how complex your shoulder is – move your arm around, seriously, just to see how huge it’s range of motion is. It’s got this massive area of flexibility, but there are points of no return where the shoulder can’t bend the arm any further. A robotic exoskeleton needs to be able to mimic that insane range without dislocating your shoulder. Dr Carmichael’s robot solves this problem using a series of five rotating disc joints.

Marc Carmichael: Yeah, so that’s one of the big novelties of this design here. They’re a bit like layers of an onion in the sense that they wrap around themselves, they can fold in on themselves, and when it’s operating, it has quite a hypnotic behaviour. At a minimum, you’ll need three joints at the shoulder. We’ve looked at the idea of exploiting redundancy. So we have additional joints that are quite beneficial for a few reasons. One is if you have a few joints in your exoskeleton, there’s always going to be somewhere where when you’re wearing the device you can’t move your arm. Either the robot has to collide with itself or collide with you, or two of the joints become aligned and the phenomenon known as “gimbal lock” occurs. So all of these things limit where the robot can reach, and that means that… the human shoulder has a massive range of motion. Out of all of the articulations in the human body, it has the most. So trying to design a device that you wear which can have the same range of motion without limiting you – it’s quite challenging to try and reconcile those two.

Sam King: If this sounds a little complicated, that’s because it is. I mean, how do you sit down with a pen and paper and design an exoskeleton arm with five joints that all fall back on each other?

Marc Carmichael: Well, we got around that by not having to do it. What we did is come up with a mathematical model that describes this mechanism. Then we fed this mechanism into a computer program using genetic algorithm that told us what are actually the best models to put into this design. This is one of the designs that it spat out – we implemented it and it works pretty well.

Sam King: And I mean, even if you have a computer do your designing for you, a massive amount of effort goes into solving the issues that come with building a robot around a human. But the point is, the potential of exoskeletons to help people heal is driving scientists like Marc to solve them.

Marc Carmichael: It’s a bit of an obvious kind of place to start here. It’s very much in demand, and demand is only going to grow with an ageing population. People are predicting that the number of aged people is going to outweigh the number of people below 20 or something like that. People are foreseeing that that’s going to be a big problem. You’re going to have all of these elderly people who need this care, and there’ll be no one to give it. So the idea is that we can use a robot to be the physical embodiment, physically administering this rehabilitation, whilst people are still there overseeing the rehabilitation in a more administrative role, monitoring patient progress as they recover over time.

Sam King: And now that this groundwork is being laid, who knows where exoskeletons could take us next.

Peter Abolfathi: We haven’t seen anything like this ever, after all these years, so we think we’re really on to something. Can this be used in space to assist astronauts in opening and closing their hands in space? One of the biggest troubles that NASA has is when they create their space suits, particularly with the fingers, they require dexterity. When you’ve got that vacuum outside of the space suit, it really seizes up the gloves, so you can’t really move your hand with facility.

Marc Carmichael: The potential for where it’s going is really exciting. I can’t even think about 5-10 years ago seeing anything like this in the media. There were no videos apart from sci-fi stuff. Now it seems every few months there’s a video about a new company developing an exoskeleton that’s allowing paraplegics to walk again, and the fact that that’s happened in this small space of time is really exciting. It makes me wonder about what’s coming next. To be at the front of that is really exciting.

[Mechanical sounds]

Ellen Leabeater: Producer Sam King exploring the future of exoskeletons.

Josh Nicholas: This is Think:Health Futures. I’m Josh Nicholas.

Ellen Leabeater: I’m Ellen Leabeater. And our final story for today comes from the labs of the CSIRO in Canberra.

Nick Barnes: Yeah, so, the “VibroMat” is an array of back-worn vibration motors that can encode a visual scene based on a head-mounted camera.

Ellen Leabeater: Look, if you thought exoskeletons were out there, then wait till you hear this. You’ll have to listen closely, because this story gets a bit technical. This is Nick Barnes. He’s the project lead on VibroMat.

Nick Barnes: And I’m also an Associate Professor at the Australian National University.

[Music plays]

Josh Nicholas: Take a moment now to close your eyes. What’s in front of you? What’s behind you? How close is the nearest door? If you’re lucky enough to be able to see, you can probably remember all the major objects in the room, or the environment about you. If you’re someone who’s blind, they have none of the information that you have.

Ellen Leabeater: Now picture a back brace worn around your stomach but covered by your shirt or dress. And on your head is a relatively small camera recording the environment around you. This is the VibroMat.

Josh Nicholas: Let’s hear that description again:

Nick Barnes: Back-worn vibration motors that can encode a visual scene based on a head-mounted camera.

Josh Nicholas: So that back brace you’re wearing – it’s taking information from the camera and sending little vibrations across your back to let you know if there’s a door nearby or a table or whatever.

Nick Barnes: So we have an array of 96 tactors on the back and that’s being translated from some portion of the image, and we’re representing things spatially within that. So if they’re looking in a particular direction and they move around – an obstacle may move across their back as they look at it from different angles – and so they can visualise around and feel where the obstacle is and navigate their way around it.

Ellen Leabeater: But why vibrations?

Nick Barnes: So if someone has an impaired sense – so in this case, vision – we substitute information through another sense. Now, touch is a really good sense in that way because you often aren’t using it when you’re walking around and particularly touch on your back. If you’re navigating through an environment, you’re not using the touch receptors from your back. Vibration technology is a good way of encoding touch, and things like mobile phone motors mean that they’re cheap and very reliable.

Josh Nicholas: So it’s like that old principle: when one sense doesn’t work, the other senses compensate for it.

Ellen Leabeater: Exactly. And with some tricky machine learning, Nick and his team are able to take images from the camera and transform them into vibrations.

Nick Barnes: What we do is we take an image and look at structures within that image and the way structures are appearing. We can also use machine learning to detect different objects within it. And then emphasising things that are more important. For instance, we’re finding obstacles within the environment, we’re finding major junctions between walls and floors and emphasising those – making sure they appear in what comes through to the user. That all happens rapidly, so this is stimulating at around 10 times a second. The images are being processed more quickly from that, so they’re available for the person to see and there’s not a big lag, so the person can walk at a comfortable pace.

Josh Nicholas: Nick has been working in this space for quite a while. He also works for Bionic Vision Australia on a bionic eye. The bionic eye involves surgically implanting a microchip onto the retina. It can only be used for macular degeneration. So while VibroMat might be basic, it has the potential to benefit a wider range of people, as well as being a lot cheaper and non-invasive.

Nick Barnes: the attraction of the VibroMat is two-fold. One: it can suit any form of blindness, so you’re not restricted to diseases that have impacted on the retina. And the other one is that it doesn’t require surgery and it’s non-invasive.

Ellen Leabeater: And sensory substitution isn’t a new idea. According to Nick, we’ve been experimenting with it since the 1960s.

Nick Barnes: There were studies done by a person called Bach-y-Rita from approximately the 1960s. The difficulty then was that the technology wasn’t available in small, mobile, low-power devices, so the original prototype was embedded in a chair and it required a lot of power so it wasn’t portable obviously.

Josh Nicholas: With microchips getting smaller and smaller, the possibilities to make stuff like VibroMat have never been better.

[Music plays]

Nick Barnes: The technology in computer vision – the ability to process things quickly and find key information – is improving rapidly, and I think this is one of the things that is going to be an enabler and is going to be needed by people.

Ellen Leabeater: VibroMat is still in trial mode. They’re testing with people who aren’t blind at the moment. They’re hoping to test the tech with blind people next year.

Nick Barnes: And with Google Glass and Snapchat sunglasses all the rage, nobody would blink an eye if you were walking down the street with a wearable camera on your head.

[Music plays]

Josh Nicholas: Thanks for listening to the show. You’ve been listening to a collaboration between Think:Health and Think:Digital Futures. If you’d like to find out more about anything you’ve heard today, head to

Ellen Leabeater: Or

Josh Nicholas: If you’re listening to this on Think:Digital Futures, why not subscribe to Think:Health.

Ellen Leabeater: And if you’re listening on Think:Health, go check out Think:Digital Futures. Both shows are produced with the support of the University of Technology Sydney and 2ser. I’m Ellen Leabeater –

Josh Nicholas: I’m Josh Nicholas –

Ellen Leabeater: See you next time.


You may also like