Some of the members of the WARA Media and Language Research Arena. From the left are Konrad Tollmar, Research Director EA Games and Associate Professor KTH; Alexandra Kafka Larsson, CEO Parsd; Johanna Björkund, Associate Professor Umeå University; Sandor Albrecht from Wallenberg Launchpad and Anastasia Varava, Data Scientist SEB.
We are a tech startup, and for us, AI is a critical component to solving the needs of our users. That is also why it is critical for us to connect with the academic research community. To strengthen Sweden’s role in connecting technology development with research in ethics and bias, one of the prominent financial families (Wallenberg) has created the Wallenberg AI, Autonomous Systems and Software Program (WASP). Parsd is excited to have become a partner and joined the community. Let us tell you more about our first insights after meeting the community.
Highlights:
The research arena WARA Media and Language is an excellent opportunity for Parsd in the AI space
Parsd is a non-academic partner of WASP and WARA Media and Language since September 2022
Great networking and access to the latest research in AI for media and language
Parsd can provide exciting and relevant use cases to the research community
I believe that innovation occurs when people and ideas collide. For it to happen, we need to find networks and places to interact both online and offline. Early in the Parsd journey, I realized that it’s very important to raise awareness about the need to focus on unstructured content, which covers text, audio, video, and images. Unstructured content is at the heart of Parsd, this is where we can make a difference for both analysts and researchers. What we can provide are tools to add structure to unstructured content to make it easier to process and analyze. Artificial intelligence is one way of making that happen (the other ways are community and marketplace sharing). The development of AI is happening at break-neck speeds right now, this is why we at Parsd try to be connected to people and organizations focused on AI so that we can stay relevant and provide the tools our users need.
Sweden’s biggest AI effort so far
Research development in the AI space today is led by the big tech companies, and many popular software frameworks have been developed at these companies (such as Tensorflow and PyTorch, which then are used in AI products ready to consume over an API which l enjoy a lot). However, academia has a significant role to play, especially in linking technology development with research in ethics and bias. In a big play (6.2 Billon SEK until 2031) to strengthen Sweden´s role in this important area of research, one of the big financial families has created the Wallenberg AI, Autonomous Systems and Software Program (WASP). WASP aims to attract international expertise to Sweden and produce 600-700 (much needed!) PhDs in the area.
WASP: An AI community for media and language
One way of creating outreach to companies and organizations is through so-called ‘Research Arenas.’ Those are places where experts in AI from Academia can meet companies with use cases and business problems suitable for AI. When I heard about the research arena called Media and Language, I quickly realized that this was a unique opportunity for us to expand our network further. Finding it so close to home here in Sweden felt very nice.
Benefits of joining the WASP/WARA Community
Through the community, we at Parsd can bring the unique issues we are working on regarding challenges around AI for unstructured content to a diverse international community. We will have a chance to exchange with AI specialists from academia and other participating companies like EA Games, Spotify, Ericsson, and Nvidia. In return, we make sure we stay updated on the latest research, explore joint projects, and in the long run, invite people to do their Industrial PhD-program with us.
We have so far participated in the final industry outreach day of the Ph.D. Summer School in August, the WING Industry Days in Norrköping in September, and seminars around broadcast media in Stockholm in October. During this time, we had the opportunity to introduce Parsd at the Speaker’s Corner and also shared two of Parsd challenges (that keep me up at night). Questions we would like to solve with the help of AI to make life easier for analysts and researchers. My first question focused on how we can generate summaries of chapters, individual documents, and a set of documents. When we solve that, how can we then tailor such summaries (output) so that they are specific and relevant to the research domain of the specific users? The other research challenge was around entity extraction and how we can dynamically improve the precision of these for specific domains. Parsd believes in utilizing AI to get more time to think, and both summarizing and dynamic entity extraction are examples of existing AI technologies that are ready to use in an optimized analytical workflow. This way, we can take over at the research stage, where we need the creativity and complex analysis that only our human brains can perform. So we can focus all our energy on the essence of research for its main purpose – new insights that can make a difference!
My questions at the event led to a great discussion with several professors. We discussed the power of combining machine learning and knowledge-based/semantic approaches. I left with lots of ‘food for thought’ and a good appetite to continue engaging in the WASP community!
A GTP-3 model for the nordic languages
Over the past years, we have seen the impressive performance of large-scale language models such as GPT-3 from OpenAI. These models have opened up generative AI (i.e., text generation) where all kinds of creative texts can be created with just a few words called a prompt. It also allows for powerful question-answer interaction with large sets of texts. These models created by the leaders in AI research are often not available for small languages like Swedish, which can be seen in this presentation by Stella Biderman. That is why the new Swedish GPT-3 is in the making is so interesting. WARA Media & Language, together with Research Institutes of Sweden (RISE) and AI Sweden, have joined forces to train a model on Nordic languages using the Berzelius supercomputer at the Swedish National Supercomputer Centre. The new model is called GPTSW-3. To do that, they had to create “the Nordic pile” – a much bigger corpus of text in Swedish and Nordic languages than ever before, an effort in itself. This model means that we, in the future, can provide the same kind of AI engines to power innovation for Swedish applications as we have seen for the major languages. A first model with 3,5 B parameters will be deployed for use first, and a much bigger one with 175 B parameters has been pre-trained already and will be made available for use later on. Looking forward to seeing the next step of this work.
Prompt tuning
Zenodia Charpy from Nivida talked about another new concept in machine learning for text called P-tuning or prompt tuning. For some AI applications, it is still possible to retrain the model for special applications but for these large-scale models, that is simply not practical anymore (the Swedish GPT-3 model development depended on the Berzelius supercomputer). The training data sets are huge, and the model itself is also very large. So how can we find a way to tune applications made by this for specific use cases or domains? One answer is prompt tuning. A prompt is the first set of words (or sentences) used to trigger these models into predicting what should come next. Finding good prompts is important to get the result we desire, and it turns out that if you know the answer, we can look for the best prompt for that answer and thereby tune the input (the prompt) to the model. Look out for job ads in the future looking for “prompt tuning engineers”; that will probably be one of the new hot jobs in AI.
More to come
The event hosted by EA Games on October 13th in Stockholm was a great opportunity to discuss the challenges in the broadcasting media industry and how AI could potentially improve quality and costs. I will come back to this topic in my next blog post.
Also, stay tuned for an interview with me in the WARA Media and Language new podcast that will air in a couple of weeks.
Make sure to follow WARA Media and Language on LinkedIn
CEO Alexandra Kafka Larsson on stage at the speaker’s corner at the WASP WING Day September 20, 2022