Later this month, Rensselaer Polytechnic Institute in Troy will celebrate its bicentennial commencement.
Scheduled to speak at the event are NASA astronaut and 1997 RPI graduate Reid Wiseman, and Emily Warren Roebling – or at least, an AI-generated representation of the woman who saw through the completion of the Brooklyn Bridge in 1883.
Roebling was the wife of Brooklyn Bridge chief engineer Washington Roebling, who graduated from RPI in 1857. She assumed the role of chief project manager of the construction of one New York City’s most famous landmarks as her husband’s health declined.
To bring Emily Roebling into the 21st Century, RPI researchers used artificial intelligence to analyze her writings and create scripted dialogue based on her use of language. Roebling’s AI-generated words will be given a voice by actress Liz Wisan, who portrayed Roebling in the HBO series “The Gilded Age,” which was partially filmed in Troy.
To learn more about the project and Roebling’s AI-assisted commencement appearance, WAMC's Lucas Willard spoke with the director of RPI’s Future of Computing Institute and WAMC Roundtable panelist, Dr. Jim Hendler.
We actually spent a lot of time trying to figure out how to do this. And we found… there's a fair amount of biographical information, but we wanted to capture her style, her personality. And you know, we have some archives and RPI that have a lot of information about her. We went down to Rutgers, Jennifer went down to Rutgers…So, we have original letters she wrote, things like that. So we took a lot of that stuff and did what's called prompt engineering and fine tuning, which are the things you do to an AI…
That’s like, you give it documents, you give it a piece of text, so the computer can learn how to write in that style.
Right. So, what we would do is we would give it a lot of samples, both facts about her, things she had written, excerpts from personal letters, which we had access to, excerpts from professional statements, articles she had written, et cetera. And then the script was written between her, you know, what the conversation would be between her and the astronaut was, more or less, we came up with the questions in advance. And then we said, ‘OK, how would she answer them?’ And for some of them, we had changed the questions a little. So, it's a really exciting project where we added artificial intelligence, human intuition and knowledge, and archival information, because it really wouldn't have worked if we didn't have the information about what she sounded like, what she acted like, how she was, and we've documented as we went along some of this, so, probably in the fall, we'll give an open seminar on how we did it and, you know, sort of, what things looked like. We used the professional version of Chat GPT as our base. And, you know, we were very careful to make sure that, we've stated everywhere we can that, you know, this was not her words, these are words as if she were speaking as best we can. We want to emphasize, we're not trying to bring somebody back from the past, we're not revitalizing a dead person. What we're really doing is trying to capture the essence of her writing, and her personality, in answering these questions and in addressing our students.
So the actress that will be portraying Emily Roebling, Liz Wisan was on the Gilded Age program on HBO, and that series was filmed in RPI’s backyard, in Troy, New York.
Yes, parts of it.
So did you approach Liz and was she on board? Had she read AI script, maybe, even before, because AI is becoming more prevalent, especially in writing situations.
She was approached. Again, the leadership approached her about doing it, she checked with HBO, because obviously, once you've portrayed a character, you have to be careful. They were happy to let us do it. She wanted the script in time. So, I don't think she cared much about where the words came from. She wanted to learn the word. She has her own work she did on trying to learn a little bit about Emily and what she sounded like, but she candidly admitted that she really hadn't studied her, and that she would leave it up to us to do the getting the words in the right style, and she would work on getting into the kind of voice and feel of the times.
Now, is it also possible now that you'll have a recording of Liz reading Emily Roebling’s AI commutated dialogue and words, then that audio itself can be used by a Chat GPT application or similar application to then reproduce even more dialogue by Emily Roebling?
Yeah, you know, this is an interesting issue that a lot in AI grapple with. So what you use to train these large language models, has primarily been stuff found on the web, documents. Well, now more and more of those documents are being written either by a human with AI, some of them AI-generated, some of the pattern-generated, so more and more tools for the stuff out there to be generated by AI. And the question is, what happens when AI is just reading stuff that other AIs wrote? Where does creativity go? Where does sort of the human nature of this stuff happened? So, frankly, most of us believe…like, so for example, this this thing we did would not have worked if it was just a I just archives or just us trying to do it ourselves. So, we could read the archives and say, you know, ‘Here's an interesting thing Emily said.’ But how do I answer a different question in that style? You know, training Chat GPT not to use terminology that is more modern. So, one of the things Emily did talk a lot about was the role of women. But remember, this is pre-feminist, this is 1880s, 1890. And so it was written very carefully. And you know, when you've just first asked Chat GPT, it used very modern descriptions of feminism and STEM and things like that, after we trained it. It talked about, you know, women and engineering. How the training that she did had to be somewhat secondhand because she couldn't attend RPI. So, she was learning with her husband, that kind of thing.
How long does it take to train an AI to nail down the language of someone who's been dead for 120 years?
You know, a lot of it depends on how much information you can get, where you can get it. We've been working on this for a little more than a month. And remember, we're not trying to do major speeches are really, you know, again, capture some…you can't talk to it, you can't like have a discussion with it. We worked hard to get good answers to specific questions. There are times where we had to actually insert by hand, we find something in this letter and say, you know, ‘That phrase really should go over in this answer.’ And, you know, we're not asking Chat GPT to do that, we're just doing it. So, it was really a combination. More time would make it better. But we were able to do it pretty quickly. And if it had been all we were doing because of course, all three of us had our regular jobs at the same time, I think in a few days, we could have gotten the first draft done. We then iterated with people who knew the family, with people who knew some of the history did it again. So, you know, we did it in a very iterative nature. But it's taken us about four weeks to get to where people are very happy with. People who know the family believe it captures correctly their history. People who've heard speeches from that time on old records and things say it's capturing the thing and our archivist who's really studied Emily's letters, and we just, just today found a new one. So, that’s exciting. We’re really able to figure out what inputs we needed to do the training that would get it to…how would we do an astronaut between a dialogue between somebody who did something in the late 1800s, and an astronaut who's working on a moon mission for the future? And it was a lot of fun.
