The Alquist Technology Presentaion

We hosted an Alquist meeting for our friends and partners this Wednesday (1/30/19). Our goal was to share the Conversational AI technology of the twice in row second (Alexa Prize) Alquist. Here is a copy if you have any question about technology, or if you want to use it, or if you want to join us do not hesitate and drop us an email. Let me give you a short summary.

In the introduction, we pointed out the recent advances in AI. Deep learning improved the speech recognition accuracy. Google is one percentage point better than humans in 2017. Advances in speech accuracy enabled the new type of devices the intelligent speakers. The best examples are Alexa and Google Home. They are the target platforms for Alquist. The initial presentation slides explained the underlying architecture and how the intelligent speakers carry on the conversation. 

In the following part of the presentation, Jan explained the very basics of the Alquist Natural Language Understanding (NLU). He has shown some of the details of our keyword extraction and named entity recognition implementation. Next, he has very quickly mentioned the dialog acts recognition and the profanity detector. Profanity detection is a challenging problem essential to keep the bot pleasant to talk to. See what kind of language we need to fight.

When people meet they start chatting. There is an endless number of conversational topics. We tried to teach Alquist the most frequent ones. The current version has 30 topics, such as sport, politics, travel, etc. The Dialog Designer (DD) plays the most important role in making the topic entertaining and interesting. He invents the dialogs. Usually, the bot starts the conversation uttering an initial sentence. The user replies. We call the human bot exchange a turn. Each topic is then a set of a large number of turns. The DD prepares the turns for Alquist in the form of templates.

Petr is responsible for a dialog manager (DM) ordering the turns in a sensible dialog. In his MSc dissertation, he has developed an original hybrid DM, which we use in Alquist. The offline processing DM has two parts a graphical UI and the training set generation. The DD generates a dialog tree (composition of turns) from graphical dialog objects for a topic. The second part of the offline DM processes the topic dialog turns, and it automatically creates a rich set of sentences, the training set. We use the set to train an LSTM neural net, which predicts the next sentence on a topic. The production DM runtime is a mixture of several rules and an NN for each dialog topic. Overall such a system handles a large number of different sentences. The hybrid design makes the Alquist conversation much more robust.

We also wanted to show how to use the Alquist technology for other modality. With the Rebel & Glory, we have put together an interactive movie about Alquist. You will learn how we have chosen the name Alquist, and you can learn about Karel Capek, watch it. We also have prepared a short interactive invitation for our presentation. The team has first shot the clip and wrote the dialogs. Ondrej even acted as a guide. This movie may inspire you how to author interactive ads or simple product introductions. We are looking forward to working with creative teams on the conversational part of entertaining clips. If you have an interesting idea, let us know. In the presentation, we have pointed out some of the unexpected questions. Try them! 

The last speaker was Ondrej who is our DD. He is inventing and designing the dialogs. The dialogs have two parts the Alquist messages, and the user replies. The DD needs to work on both. He predicts what will be the user interested in and he generates the templates to react on the user's requests. Language is very complicated. The user does not want to hear repeated or tiresome replies. The conversation must be interesting. Therefor Alquist creates responses form templates as well as using generative algorithms. Ondrej has also overviewed some of the underlying problems and linguistic peculiarities he had encountered.

The Alquist presentation has attracted almost eighty people to our beautiful top floor presentation room of the CIIRC building. We were very excited by a fruitful discussion with plenty of questions at the end of our meeting. If you are interested, you may go through the presentation too. Let us know if you like it. Enjoy and if you have any questions let us know. We would also be happy to hear your ideas for new applications.

Alquist PF 2019, the Alexa Prize Finalist.

Alquist PF 2019

We have been second in a row in the Alexa Prize out of more than 100 academic teams with our social bot Alquist. Last month the students received a 100K USD prize during the Amazon re:Invent 2018 conference in Las Vegas. We are back in CTU CIIRC, in our lab again. During the long overseas flights, we started to think what to do next. We want to present our technology in a slightly different, but more entertaining form. The end of the year, the holiday season is a great chance to engage with our friends, partners, students, all people. It was a simple decision. We have prepared an interactive PF 2019, a sneak preview of how creative we can be with the Conversational AI.

Recently Amazon updated Echo Show, and Google introduced the Google Home Hub. These, screen equipped devices, are combining the Conversational AI with visual to make the user experience even more engaging. In Alquist, we believe in multimodal interfaces too. The most ubiquitous multimodal device is definitely a smartphone. The trend is clear. Google and Apple are continually reporting increasing numbers of voice interactions.

Currently, our demo needs the Chrome browser. It is a limitation, but you can run it on a smartphone and on a desktop either on macOS or Windows. One of the big problems for the Google Home or Alexa is the discoverability. Choosing the smartphone and the browser we can post just a link, no installation is required. We are not bound to existing platforms. The user only needs to allow the browser to access the mic. We still do not support the iPhone, which is not allowing the mic access. If you own an Android smartphone, you are only one click away. Try it!

We have partnered with Dazzle Pictures. They created a fantastic animation in zero time, and we designed and implemented the dialogs. At first glance, it may seem pretty simple answering ambiguously, but do not be mistaken. Give it a try, test it! For example, Alquist understands the standard English set of names only. Try how the dialog changes if it does not recognize your name correctly. Discover how the snowman replies once you utter something not understandable. We also handle a positive and negative message etc.  Have fun and let us know what you think!

Check out also our first video supported demo the Alquist Story. It is high quality movie with real actors introducing Alquist. You can also ask many questions about Karel Capek. It took us a very long time to get it to a reasonable shape. We spend time shooting the movie, a lot of time in post processing and we had to overcome various technical problems too. On the other hand the animated PF 2019 was a quick shot, all was prepared, in good shape.

We believe there are many opportunities for using our technology in many different business segments. We can imagine answering product questions, ask users for preferences, help set up devices, etc. We are looking for more ideas about how to commercialize our know-how.  Help us to discover new opportunities!

We are second again

We have made it again with the Alquist bot. Alquist competed with the Alana bot from the Heriot-Watt University, Edinburgh, Scotland and the winning Gunrock bot from the University of California Davis Davis, CA, USA.

A $500,000 prize was awarded to the winning team. We are bringing back $100,000 in prize money, and Alana receives $50,000, The challenge for additional, a $1 million research grant has not been awarded yet. It will take some time to make 20 minutes long chat. Just imagine how difficult it will be to get to a bar and talk to a stranger for 20 min.

The Alexa Prize is a $3.5 million challenge for university teams to advance human-computer interaction. Similarly, as last year the goal was to develop the best social bot conversing coherently and engagingly with humans on a range of current events and favorite topics such as entertainment, sports, politics, technology, and fashion. We continued this year with the Alquist II starting from the beginning of 2018 when the Amazon Alexa Prize was announced. We submitted our proposal, and we have made it in between eight semifinalists who were selected from more than a hundred teams from 15 countries.  Amazon has awarded us with a $250,000 research grant, Alexa-enabled devices, and free Amazon Web Services (AWS) to support our development efforts.

The research grant was significant support for our team The team leader as last year was Jan Pichl who is pursuing the third year of his Ph.D. program at the faculty of Electrical Engineering in Conversational AI. This year charged with enthusiasm the team decided to drop the first version of Alquist and started from scratch with an entirely superior, new design. We built on the latest neural network technology in combination with a small number of rules to conduct the dialogs. Alquist II knows how to react to most conversational utterances, but it excels in 26 selected topics. A great deal in the quality improvement came from a large number of users conversing with our bot. Each conversation helps to understand better the complexity and select the best matching reply.

In-depth knowledge is required to create an exciting and entertaining conversation. Where to get the content? The web is an endless source of interesting facts, but mostly in a written text. When played back it feels too long and a little tedious. To make the conversation natural, we had to solve this problem. The large part of work went to the knowledge acquisition and processing. If you are a lucky owner of Alexa device, you can test the Alquist abilities, just say let’s chat with Alquist.

The Alexa Price winners announcement was part of the AWS re:invent conference in Las Vegas. The finalists were invited. We all have enjoyed a grand celebration as well as the gathering.

Amazon is investing a lot in the development of intelligent conversational gadgets led by Alexa. The experts predict that the most natural way for communication, the speech will become in the nearest future an additional channel to control appliances, access knowledge, etc. It is fascinating and inspiring to find our team between the leading groups in the world working on the latest technology with an exciting vision. We wish our success will attract new students to join our team an pursue our adventure next year. Let us know!

We are the Alexa Prize finalists again

We have made it to the Alexa Prize 2018 finals again with our social bot Alquist. Our competitors are the Alana bot from the Heriot-Watt University, Edinburgh, Scotland and the Gunrock bot from the UC Davis, Davis, CA.

It was almost exactly one year ago I wrote the last blog. That time we were excited to get to the Alexa Prize 2017 finals, and we celebrate today again, we made it to the finals with the Alquist team again. It was a hectic time.

We have completely redesigned our bot. This year when we started the semifinals, we experienced problems with data to train the new AI. As the number of interactions was growing, we were increasing the training sets and improving the accuracy. We have augmented the dialog acts classifier processing every new user utterance. It is using the convolutional neural network and classifies the utterances to around thirty classes. The significant change in the overall architecture is the dialog manager. Last year we used a rule-based approach. It was great for cooperative users, but once the user did something unexpected we had troubles. It was also a very laborious process to write the rules. We ended with hundreds and hundreds of rules. It was also challenging to update or enhance the dialogs.  The latest Alquist uses hybrid dialog management. We have reduced the rule-based decision to a minimum and made the principal part controlled by an LSTM neural network. We have many LSTM models for different sub-dialogs. The sub-dialogs are trained and updated for excerpts of the bot user interactions. The hybrid approach significantly reduced the amount of work necessary to create a new dialogue compared to last year's rule-based approach. This fact allowed us to broaden the range of conversational topics substantially. We have also taken advantage of delexicalizing the utterances to improve the training speed. The bot includes several other neural networks helping to switch between different topics, estimating the sentiment, etc. The whole system is getting quite complicated. We have also spend a lot of efforts on improving the new information acquisition. We are crawling several social media. The discussions are an additional source of interesting facts. The social media are a great complement to knowledge databases with the factoid type of information like for example Wikipedia.

The team has changed a little compared to the last year. Roman has left, and Petr Lorenc has joined. He is helping a lot with the intent, entity recognition, which is an essential part of Alquist and has a significant impact on the overall user experience. Currently, everybody is very busy since we have another at least two months to improve the functionality. We will focus on the user experience. Since English is not our native language, we have to spend a lot of effort ironing out all conversation, adding SSML, etc.  Amazon will offer to the Alexa device owners only three first bots, which means we will get more data. More data gives us a chance to improve further the accuracy.

Amazon will announce the winners as last year at the re:Invent Amazon Conference in Las Vegas. We are looking forward to visiting Las Vegas the heart of gambling, as well as meeting our competitors and helpful Amazon Alexa Prize staff, as well as learning the latest from the Amazon technology. We were second behind the Washington team last year. Guess what are our aspirations this year. If you are a lucky Alexa device owner, try "Alexa let's chat." Keep the fingers crossed!

Alquist made it to the Alexa finals

The CVUT Alquist team managed to get with other two teams to the finals of a $2.5 million Alexa Prize, university competition. Our team has developed the Alquist social bot.

The whole team has met in the eClub during summer 2016. That time we have been working on a question answering system YodaQA. YodaQA is a somewhat complex system, and students learned the classic NLP. Of course, everybody wanted to use Neural Networks and design End to End systems. That time we have also been playing with simple conversational systems for home automation. Surprisingly Amazon announced the Alexa Prize and all clicked together.  We have quickly put together the team and submitted a proposal. One Ph.D., three MSc, and one BSc student completed a team with strong experience in NLP. In the beginning, we have been competing with more than a hundred academic teams trying to get to the top twelve and receive the 100k USD scholarship funding. We were lucky, and once we were selected in November 2016, we began working hard.  We started with many different incarnations of NNs (LSTM, GRU, attention NN, ....) but soon we have realized the bigger problem, a lack of high-quality training data. We tried to use many, movies scripts, Reddit dialogues, and many others with mixed results. The systems performed poorly. Sometimes they picked an interesting answer, but mostly the replies were very generic and boring. We have humbly returned to the classical information retrieval approach with a bunch of rules. The final design is a combination of the traditional approach and some NNs. We have finally managed to put together at least a little reasonable system keeping up with a human for at least tenths of seconds. Here started the forced labor. We have invented and implemented several paradigms for authoring the dialogues and acquiring knowledge from the Internet. As a first topic, we have chosen movies since it is also our favorite topic. Then, we have step by step added more and more other dialogues. While perfecting dialogues, we have been improving the IR algorithms. We had improved the user experience when Amazon introduced the SSML. Since then Alexa voice started to sound more natural.

While developing Alquist, we have gained a lot of experience. A significant change is a fact that we have to look at Alquist more as a product than an interesting university experiment. The consequences are dramatic. We need to keep Alquist running, which means we must very well test a new version. Conversational applications testing is by itself a research problem. We have designed software to evaluate users behavior statistically. First, a task is to find dialogues problems, misunderstanding, etc. Second, we try to estimate how happy are users with particular parts of the conversation to make further improvements. Thanks to the Amazon we have reasonably significant traffic, and while we are storing all conversations, we can accumulate a large amount of data for new experiments. Extensive data is a necessary condition for training more advanced systems. We have many new ideas in mind for enhancing the dialogues. We will report about them in future posts.

Many thanks for the scholarship go to Amazon since it was a real blessing for our team. It helped us to keep the team together with a single focus for a real task. Students worked hard for more than ten months, and it helped us to be successful.

Today we are thrilled we made it to the finals with the University of Washington in Seattle and their Sounding Board and the wild card team from Heriot-Watt University in Edinburgh, Scotland, with their What’s up Bot. Celebrate with us and keep the fingers crossed. There is a half a million at stake.

New projects for this summer

This year we are opening the eClub Summer Camp new CIIRC building. We have prepared exciting projects from the field of AI, IoT, and Internet. We will focus on conversational IA, how to program assistants to control your household, Natural Language Processing, and other topics, see the projects page.

Two years ago we started to work on the question answering engine YodaQA. Last year during the eClub Summer Camp we have designed the first bot. Our primary goal for this summer is to create a great Echo application. Echo is a voice controlled smart speaker made by Amazon. You can only ask to play music, ask factoid question, carry a simple dialog or control your household. There is an amazing technology behind the set of new Amazon devices. First of all the speech recognition, directional microphone, conversational AI, knowledge database, etc. The eClub team is among the first in the world working directly with the Amazon research group on making the Alexa even smarter. We want to make her sexy, catchy and entertaining and it requires a lot of different skills. Starting with the linguistics up to Neural Networks design. We have many well-separated problems for any level of expertise. Come to see us, we are preparing an introductory course to teach you how they do it. We will help you to create your first app with initial skills. You can meet a lot of students who work in the Conversational AI who will help you to get over the underlying problems.

We want to make the Conversational apps not only entertaining but also knowledgeable. Alexa must also be very informative. It must know for example the latest news in politics, the Stanley Cup results, what are the best movies and I am sure we can continue with many other topics. The knowledge is endless, and it is steadily growing. To handle to alway increasing data requires processing many news feeds, different sources, accessing different databases, accessing the web, etc. The news streams must be understood, and the essential information must be extracted. There are many steps before we retrieve the information. Especially today we need to be careful, and every piece of information must be verified. We try to create a canonical information using many sources of the same news. As soon as the information is clear, we need to store it in a knowledge database. The facts need to be linked to information already in the database. And how about the fake news, how to recognize them?

Building the Conversational AI does not include only the voice controlled devices. We may want to create a system automatically replying to the user email or social media requests. Imagine for example a helpdesk where users are asking many different questions from IT to HR topics. For example very frequently how to reset a password, or how to operate a printer or a projector, why not to answer them automatically? And we can be much more ambitious. Many devices are quite complex, and it is not easy to read a manual. It is much faster to ask a question such as “How do I reset my iPad,” or “How do I share my calendar.” These apps are put together from two major parts. The understanding of the question and a preparation of the answers. Both use the NLP pipeline. If you expand on this idea, you may find a million of applications with a similar scenario. An automated assistant can at least partly handle every company-customer interaction. To make a qualified decision, the executives need fast access to business intelligence. Why not ask questions such as “What was the company performance last week,” “What is the revenue of my competitors” etc.

Let me mention another aspect of our effort. The latest manufacturing lines are extensively using robots, manipulators, etc. (INDUSTRY 4.0) The whole process is controlled by a large number of computers. What if something stops working, it is a very complicated task to fix a line like this? Every robot or manipulator might be from a different manufacturer, programmable in a slightly different dialect. Is there anybody in the company who can absorb the complete knowledge to be useful in localizing the problem? Yes, it is a robot, which has all the knowledge in a structured form. The robot can apply optimization to find the best set of measurements or tests to help the maintenance technician. To make this happen, we need in addition to a productive dialog and knowledge database also an optimization to suggest the shortest path for fixing a problem. The robot can guide humans to repair the problem most efficiently.

Yes, I have almost forgotten. It is recently very popular to use the robots to control the household. Alexa, turn off all the lights. Alexa, what is the temperature in the wine seller? We want to invent and build some of these goodies to our new eClub space during the summer. Our colleagues have developed a Robot Barista application shaking drinks on demand. A voice user interface will make it even more entertaining. We have other exciting devices and small gizmos deserving voice control. You also may come with your ideas. Join us we will assist you to be successful.

These are just few use cases we will try to tackle during this season. If you want to learn the know-how behind join us, we will help you, and we also will award a scholarship.

Conversational AI for Dungeons and Dragons

we start the 2017 eClub Summer Camp. eClub has moved to a new CIIRC building. We are competing in the Alexa Prize competition. Join eClub and learn the latest machine learning, NLP algorithms.

A few years ago we have started with a question answering system YodaQA. It has been inspired by the IBM Watson beating the best player in Jeopardy. Today we continue our journey in even more challenging projects. We are creating dialogs for the latest voice-controlled appliances. We have entered the Alexa Prize competition and it helped us to develop Alquist the social bots. We have a free access to an immense power of AWS, we are in constant touch with the Amazon research staff. Every member of the Alquist team became an NLP expert. The Alquist system is day by day getting better. Currently, Alquist can conduct a sensible short dialog. The user can choose from several topics: sports, politics, celebrities, jokes, etc.

Thousands of users are using the chat and we receive valuable logs making us very busy. It takes a lot of time to get through details, to discover why the user stopped the conversation but we are learning a lot. What is the social dialog? What are the catchy questions? How to respond quickly and interestingly? There are still a lot of questions, but we are ambitious, we want to extend the Alquist knowledge to handle a long and interesting dialog. If you like tested say Alexa let's chat.

The Natural Language Understanding (NLP) underlines Alquist. NLP is also the essential part of Siri, Cortana, Alexa, Google Assitant and other latest bots. The Conversational AI is building on machine learning, optimization etc., it takes advantage of all the latest development in machine learning, starting with the classical algorithms up to the latest deep neural network, sequence to sequence and memory networks etc. This summer we want to considerably improve the Alquist capabilities. To achieve our goals we need to enlarge the Alquist team and focus on the Conversational AI. If you are a BSc, MSc or Ph.D. student join us. We have various programs including Ph.D. candidates.

In addition to AI, we also need creative individuals knowing how to handle a dialog, being innovative. We all know that carrying an interesting dialog is an art. To teach Alquist interesting dialogs is even more complicated. Creative young people with many different skills in the human to the human conversation are welcome to join us.

A dialog is also about information and experience exchange. Imagine for example a bot helping you playing an adventure game. The RPG have many rules and it is very boring to search an information in a handbook. One of the very popular RPG games is the Dungeons and Dragons. We want to design an interactive Alexa D&D handbook and improve level and XP progression. If you are interested in D&D join us helping us to design a voice-controlled interactive manual.

The NLP space is huge and we have a large number of interesting topics to work on. If you are interested in AI, machine learning, neural nets etc. join us. We have great resources, a lot of experience and funds to award you scholarships.

Alquist mission continues

The Alquist team has just returned from the Alexa Prize Summit. We have met with Alexa researchers, developers business developers, and all competing university teams. We enjoyed the three days of great fun discussing the conversational AI.

The summit opened with a visit to the Amazon Fulfillment Center. We were stunned observing the Amazon robots smoothly transporting racks with goods to pickers. The center is an excellent example of automation, optimization, and efficiency.

We have started the summit with a review of our Alquist social bot. Amazon experts commented on our accomplishment. No problems, we have passed.

The key part of the summit were the Alexa team presentations. The development of a social bot is a complex task requiring knowledge from many different disciplines. Therefore the presentations were covering many topics ranging from speech technology, NLP, Deep Learning to how to carry an engaging dialog. The Alquist team members are CVUT students with in-depth knowledge in AI. Therefore the most revealing for us were the non-technical presentations. I liked the Celeste Headlee presentation. She is a reporter on the public radio and a professional opera singer. She taught us how to Make Great Conversations. The key message:  Researchers discovered that talking about yourself activates the same pleasure centers in your brain as sex and cocaine. We learn how the Alquist needs to behave. We also got a lesson about Best Practices for Promoting Our team. The advice is simple: promote better the Alquist team and the Conversational AI.

The summit was not only about technology, on Wednesday evening we joined the screening of just released Amazon Studios Movie “The Lost City of Z” produced by Brat Pitt.

We also looked around the Seattle City on Friday afternoon. We were lucky it was one of the few days with blue sky. I gave up queuing for the lift to the Space Needle Observation Deck and instead I have climbed one of the Seattle hills. The sky was clear and the more than 4000 meters high Mount Rainier with its snow cap seemed like just behind the city.

The Alquist team was excited by the Alexa Price Summit. We had a unique opportunity to meet with the Amazon Alexa team and the competing university teams. We had the chance to experience the Amazon enthusiasm and the entrepreneurial spirit. We got back charged with fresh energy to push the conversational qualities of Alquist even further.

If you are interested in building interesting conversational applications, join eClub and the Alquist team and help us to make Alquist even better. There are many AI tasks we need to improve, some of them we have not touched yet. It does not matter if you do not have any experience, we will teach you. Thanks to Amazon we can give you a scholarship. Join us and enjoy the feeling of success!

Alexa socialbot testers wanted!

We are finishing the last details, of the Alquist social bot.  Today we are opening the first preview for testers.  If you are interested, leave your email. We will send you instructions. The total number of testers is limited. Hurry up!

If you do not own Echo or other Alexa appliance, install the Echosim.io or Reverb on your cell, and you are ready to start. Upon signing up, we will email you all required details. We are interested in your feedback.

The chatbot is not perfect but, it is the time to start learning how users interact and collect the feedback. We need real users who are not familiar with the system. Amazon is helping us with their internal testing system, and it is a unique opportunity. Help us!

What can you expect?  We have trained Alquist to carry a simple dialog, it knows how to answer factoid questions using the Wikidata DB, and how to give help. Recently, we have made advances in creating a simple dialog within limited domains with frequently changing data.

All is still under development with a lot of quirks and twists, but your feedback will help us to improve. Thanks!

New projects - join us

eClub will again organize the Summer Camp (ESC). The ESC 2016 was incredibly successful. Five eClubbers worked on a question answering system YodaQA. At the end of summer, they have entered the Amazon Alexa Prize competition and got in between top twelve selected for development of a social bot. They received $100k scholarship for a social bot development. Currently, they are busy working on the first version of Alquist.

We would like to continue in the direction of developing dialog applications in ESC 2017. The social bot task is very challenging, and many required technologies are still in development. While developing YodaQA, we have looked at the well known classical NLP algorithms as well as to new mainly neural networks based ones, such as LSTM, GRU, etc. to process text in many different ways.

It is beginning of March, but we are already prepared to incubate new, eager students interested joining us on this journey toward smarter systems. We have strong support not only from Amazon but also from the local company Seznam. Seznam is one of the few competing successfully with Google on the domestic market. They are 100% Machine Learning company with a lot of problems of mutual interest.

Here is a sneak preview of the projects for this year. Join us, and work with us tomorrow, if you are interested. We are offering scholarship equivalent to what you would earn working for a company. We are moving to the new CVUT building, it is gorgeous, as soon as it opens, it is just a matter of weeks. Join us! You can start any time.

Automatic email reply generation
In this project, we want to research methods for automatic generation of short responses to emails or social networks messages. Specifically, on a cell phone, it can be a great advantage to select from a selection of semantically diverse replies. We want first cover few words long messages. The initial steps will include a review of Recurrent Neural Networks architectures and a meaningful training set construction.

Amazon Echo conversational application
We have a set of tasks we would like to cover as a spoken dialog. We want to design interactive conversational bots for Amazon Echo. We want the application to be engaging, entertaining and informative, bringing the user latest news from specific areas, such as sports, celebrities, movies, etc. This project is OK for students who are only entering the field with no or small experience.
   
Knowledge extraction
There is vast of information on the Internet. A lot of the information is in the form of a text; the information is unstructured. In this project, we want to review the methods for retrieving and extracting the information, learning the dependencies of statements in the texts. We want to create ontologies from a selected, limited content and store the knowledge for further use. These are very challenging problems but do not hesitate to join. We have students currently working on these topics. We know what the first steps are.

Text summarization
Journalists write the Internet news in a particular language, frequently using idioms, slang or infrequent expressions. In this project, we want to extract what is important and create a summary in a clear language. Initially, we want to summarize long sentences. Next, select a suitable method, implement and test it on a chosen domain.

Events Extraction from Text
This project is only an extension of the previous one. We want to design and implement a system for extracting events from the Internet. The primary goal is selecting news messages based on identified topics (or events). Extraction of economic events like mergers & acquisitions, stock splits, dividend announcements, etc., play an essential role in decision making, in risk analysis applications and monitoring systems.

eClub Prague received the Amazon Alexa Prize!

eClub Prague students got one of their best opportunities by being selected as one of 12 teams from world leading universities to compete in the inaugural Amazon Alexa Prize with their very own Alquist chatbot. The eClub team has been selected by Amazon and it has been given the challenge to build a social bot that can converse on popular topics for 20 minutes. By being selected among such names as Princeton University, Berkeley, Carnegie Mellon and as only one of three universities in Europe this is a unique and exciting opportunity to put us on the map as one of the thought leaders on social bots. eClub Prague was awarded a $100,000 stipend, Alexa-enabled devices, free AWS services, and support from Amazon’s Alexa Skills Kit (ASK) team. All this to help us make the final three, who will be announced in August 2017. Also as part of the judging process, Alexa customers will have the opportunity to try out the Alquist chatbot themselves and their feedback will have an instrumental impact on the team making the final. The winners will be announced in November 2017 with a $1,5 million up for grabs. I am the academic advisor for a team consisting of Long Hoang Nguyen, Honza Pichl, Martin Matulík, Jakub Konrád and Petr Marek. the team is very excited about the opportunity that Amazon has given us and we hope to make the final three. Although this is a great achievement for the Prague-based team the work only starts. From now until April they will be working on developing conversational AIs such as knowledge acquisition, natural language understanding, natural language generation, context modeling, commonsense reasoning and dialog planning.

eClub login

This is a presentation for today eClub Login conference. Enjoy!

Do not look for part time job work for the university

eClub is extending the Summer Camp for the whole academic year. This is an early information.

The eClub Summer Camp is in its third month. This year we have more than twenty members. It is a little less than last year but the projects are much more focused. The mentors are more effective, they have learned how to lead. Overall we have accomplished much more. The success and the high quality of projects has ignited an idea to extend the eClub throughout the whole academic year.

We will offer students scholarships starting from the October 1st for the winter and extend for the summer semester. We are preparing joined labs with our partners Seznam, Jablotron, and many others. We are in the process of signing the contracts. Our joined labs are located in the new CVUT building in the Dejvice campus, everybody will get a seat and table. You will save the commute time. eClub and the partners too, are looking for innovative ideas and interesting projects. We also want to make the projects semestral works or final thesis to let you earn credits. Working with leading companies is a great entry to CVs. eClub will provide scholarships and we also will offer a complete infrastructure including large computer clusters. Each student will have a mentor helping to get over the usual problems no to get stucked.

We will offer similar projects as during the Summer Camp. Mainly from the field of artificial Intelligence, IoT Industry 4.0 and BigData. We do not wnat to limit our activities to this fields only, come and ask. We are inviting students from all Czech Universities. We also will support startups. Work for the leading companies in an university joined lab.

Work on great projects. Follow our FB, eClub web pages and blog, stay tuned for further information.

Bots, Home control and many more projects, eClub

The students in the eClub Summer Camp are for more than two months working on new projects in IoT and Bots. Here is the first status report describing the projects under development.

The presentation has two major parts the IoT and conversational applications projects. The IoT team has built a proof of concept application allowing intelligent control of light including the cloud infrastructure. The sensors are wirelessly connected. A cloud application offers a simple dashboard. In conversational applications, we have built several modules, services running in Docker. Combining them we can implement simple conversational bots as well as a factoid answering machine.

In the presentation, you can find a basic information and links to websites, demos or gitHub with the code. Some of the projects are in progress some of them are already finished. Some students will extend the projects to theses in the following academic year. Let us know what you think.

Bots and question answering in ESC 2016

a team of talented students formed around conversational applications in the eClub Summer Camp 2016. They continue developing the YodaQA and a simple Alquist bot. Both systems are built on top of a set of services including dialog managers up to NLP services.

We have started with the YodaQA factoid question answering system inspired by the IBM Watson. It is a fairly sophisticated engine, which builds on many NLP algorithms, Lucene search, RDF databases etc. The architecture and technology description is available on GitHub along with a test website and Android application.

The latest work is concentrated on teaching YodaQA Czech. This requires replacing some of the components with Czech versions. The most important are the Stanford syntactic parser, the Named Entity Recognition and finally the answer classifier. For syntactic parser, we use the Google Tensor Flow, the Syntaxnet and the Czech dependencies dataset. We get similar accuracy as the classical top of the line algorithms. Currently, we are developing the basic version of the entity recognition algorithm based on Conditional Random Fields (CRF). We have plans to implement NER also using Neural Nets.

The biggest problem in machine learning are the training sets. For the initial answer scoring algorithm, we have put together a set of questions-answer pairs. To make the set as rich as possible we have been enriching the set using variables for entities and synonyms, which is allowing us to algorithmically generate a large number of questions. The real system is logging questions and answers helping us creating better training sets. The sets still require some manual processing, but it is worth doing it.

The emergence of the conversational bots caught our attention too. Initially, we have tested the Wit.ai, Microsoft Luise, Meya, and Amazon Echo for English. Soon we have found many different limitations. Because the YodaQA is put together from a set of independent services, NLU processors, we have decided to use the same services to build simpler conversational bots. The bots use two essential parts intent and the entities recognizers. The bot processes the input users query and the extracted intent and entities are saved to a context object. Dialog manager (DM) uses the context to control the dialog flow.

Since we do not use the DM in YodaQA, we had to develop it. During our experiments with commercially available bots, we liked very much the Meya DM because of the simple dialog declaration in YAML. We have decided to go in a similar direction and created our own version called Alquist. It allows us writing even complicated dialogs. The implementation was fast and today we are running our first version of Alquist DM.

All this work is done by about ten students and the team grows. During the last weeks, we have made a considerable progress. We have at work several applications, stay tuned to be the first to test them.

eClub Summer Camp IoT, Machine Learning

the eClub Summer Camp is in full swing and our lab is full of students. The projects can be divided into two groups IoT and in Machine Learning.

The IoT group is busy with projecting architecture for connecting HUBs with Cloud servers. We assume the cloud will have to serve millions of HUBs collecting the information from sensors and controlling the actuators. We are discussing and making predictions how many events we will collect from sensors, how active will be the smartphone users, how much of administration traffic (heartbeats, updates, etc.) we will have to support. The users' profiles, sensors, and HUB configuration need to be maintained in databases. We also plan to save all logs to provide access to historical data. There will be probably two different systems one for handling the incoming data and another one for storing the logs. Haboop with HDFS seems to be the choice for managing the logs. SPARK for filtering and managing the events. Of course, security is one of the most important features of the system and we are busily studying communication protocols. It is a large project and many students work on preparing the specification and testing parts of the design. Our goal is to create a proof of concept showing the HUB CLOUD communication still this year.

We have also a large group focusing on conversational systems. The work is centered around the open source YodaQA factoid answering engine. It has been inspired by the Watson Jeopardy system. It already answers English questions. Our major task is to convert it to Czech and improve the functionality. We are working on the integration of WikiData knowledge DB and we have to retrain a lot of the NLU blocks to Czech. One of the students works on creating a Czech parser model for the Google SyntaxNet parser,  

We are also looking at bots, which are good for creating of simple conversational apps for example for controlling a simple home IoT. The bots technology is based on information retrieval approach. We try to search for the best answer for a particular question. In this field, we have been working on Sentence Pair Similarity algorithms, which can be trained to recognize the question intent. Students are also looking at new development packages such as wit.ai, api.ai, Microsoft LUIS, land others. We try to develop examples of small simple applications. We believe the hands-on experience will help us to understand where are the limitations of the small, IR base systems and where we need to opt for the YodaQA technology.  The laest interesting but the essential part of our effort is creating training databases. Everybody is involved in the hard work of data set collection.

If you are interested, join us visit us we may help you to select an interesting project. There is still time to join.

X.GLU startup in eClub

Last week I have visited the Pioneers festival in Vienna. This was also the first public presentation of the new eClub startup X.GLU.

The X.GLU startup has developed a revolutionary glucometer called X.GLU. It is the smallest glucose meter, it is the size of a credit card and simply slips to your wallet. X.GLU requires no batteries and no wires to read the sugar level on your smartphone. As long as your smartphone is charged, the glucose meter works. No maintenance required. The X.GLU uses a standard connector for a biomedical sensor paper. It comes in a convenient bag along with disinfection tissues, lancets, and testing strips. The read out is transmitted by the NFC technology providing a secure wireless link between X.GLU  and the smartphone. Unlike Bluetooth, Wi-Fi, and similar wireless technology, the NFC cannot be sniffed from a distance of more than several inches. The measured values are displayed and stored in the smartphone. An encrypted connection sends the X.GLU data in a cloud and makes it available to physicians providing instant feedback in treatment.

The smartphone app comes with a how-to video. It shows detailed instructions on, how to treat the skin before taking the sample and the method for properly taking the blood sample. The app also conveniently reminds the user about  the scheduled measurement time.

The mastermind of the new company is the inventor and owner Marek Novak, who came with the idea of glucometer. Marek is one of the most active students in eClub. He has worked already on several IoT-related projects, but X.GLU is the first one we want to get to production. eClub helped in complementing Marek’s knowledge and found experts in sales and marketing to create a functional company. They start their operation from our scientific incubator.

It is great news for eClub. We all will try to do our best helping to start a productive and successful path to market. We are looking for other students teams with startup ideas. Join us during the eClub Summer Camp.

Our projects part 2

This is the second part of “What we do” this time about the IoT activities. 

Our IoT effort can be roughly divided into two parts SW infrastructure and sensors. We use the standard IoT architecture combining an HUB and a cloud server. It is a typical IoT system setup allowing to collect the sensors information and control actuators over the Internet. The architecture uses an HUB. It serves as a gateway to the Internet and concentrator for the sensor data. The HUB is a simple computer with a similar power as a router  equipped with Ethernet or WiFi or both to connect to the Internet. In addition, it may have several other radios for the sensor, actuator communication. The radios are continuously listening to sensors and this typically requires power, therefore, HUBs are usually not powered from a battery. 

Our HUB is based on the Intel Edison dual core 500 Mhz Linux-based embedded computer with WiFi, BLE, and 868MHz free band radio. It also includes three USB sockets for additional peripherals. The HUB is running a simple Node JS server Zetta. This server is handling the management and communication with servers. It allows a seamless connection to similar Zetta server residing in the Cloud. The linked Zetta servers communicate using a walkable, JSON based, hypermedia Siren. The cloud-based servers allow a simple connection to the smartphone. The hypermedia Siren allows the smartphone to set the UI based on the configuration of a particular space covered by an HUB. We have designed and implemented an IoT control app for Android and it configures based on location. In practice, it means as soon as you get to a smart room or to your car the Android home page sets for the particular environment with the most frequently used control on top. 

We do not use the WiFi for communicating with sensors. WiFi usually requires a lot of battery power and it is primarily designed for TCP/IP protocol, which may not be required for the very simple sensors such as thermometers. The thermometer is sampling the environment temperature for example only every 10 min and therefore we can let the sensor sleep most of the time. The radio is waking up only for the shortest possible communication required to exchange information with the HUB. This approach is allowing us to design sensors with very low energy requirements. 

The low sensors consumption allowed us to use one of the energy harvesting approaches, the Photovoltaic Cells. We have designed and put together a set of PV powered battery-less and wireless sensors. We can measure temperature, humidity, motion (accelerometers and PIR). The PIR equipped sensor is powered just from a fluorescent tube on the ceiling and it is sensing people coming to our lab for more than one year. We are monitoring the PV accumulated energy and we have so far never run out of power. We use the accelerometer-equipped sensors to check for open windows. The outside light is also good enough to provide enough juice. The sensors communicate with the HUB using 868MHz radios. We have found this band more resistant to objects than the WiFi or Bluetooth. Currently, we use a proprietary protocol, but we are looking for LoRa and MQTT, which we use in other projects with the same HUB.

Some of the described work is part of the Bachelor’s thesis written by my students. We are looking forward to pushing our work even more ahead during eClub Summer Camp 2016. 

Our projects

Recently I was asked to review the latest development in our group and I realised how much work we have done. I have also noticed I am forgetting about my blog. Let’s fix it!

First the best news, our group has grown during the last year to five PhD and around 10 MSc students working in machine learning. 

Today I would like to start with part one and mention some of our progress in machine learning. In the second part I will describe our IoT effort. The main machine learning topics can be broken in the following categories:

• Natural Language Processing
  
• Question answering YodaQA
  
• Intelligent assistants
  
• Sentence pair similarity
  
• Multinomial classification

• Information retrieval
  
• Learning to rank 

• Information extraction
  
• Focused crawling
  
• Convolutional Neural Networks
  
• Combining text and images
  
• Image labeling 

I’ll start with the major achievement, the YodaQA answering machine. It is an open source question answering system. It implements state-of-art methods of information extraction and natural language understanding — to answer human-phrased questions! You can try the live demo. 

Along with YodaQA we have worked also on simpler Intelligent Assistants acting on a smaller number of commands. They take advantage from simpler algorithms finding the most similar answer for a given query. The sentence pair similarity is another topic of interest. The algorithms can help solving not only Answer Sentence Selection, but also other interesting problems, such as Next Utterance Ranking, Semantic Textual Similarity, Paraphrase Identification, Recognizing Textual Entailment etc. We have tested and developed a series of algorithms based on word embeddings and different architecture of Neural Networks. 

To the NLP category belongs also the multinomial classification algorithm. The use case we are testing is the products categorization to a hierarchical directory structure. Typically the e-shops are categorizing products, such as  a “14 inch screen notebook” under notebooks, computers, electronics etc. This process is handled by human beings and they do mistakes, our algorithm can find problematically categorized entries or suggest correct category. 

An exclusive position in our group has the adaptive ranking research. The web content, its information relevancy, authority and the users interests are changing constantly.  The goal of any search engine is to provide exactly what the users look for. The newly developed algorithm relies on users to find currently the best ranking. It constantly observes on what links are users clicking and it adapts based on this feedback. This is very relevant for information search and recommendation services.

Information extraction is the next topic completing our portfolio. Initially we have looked at basics of focused crawling, a strategy how to crawl internet and extract for example all mentions about AT&T and Linux. This leads to a design of a crawler with programmable search policy. Currently we work on even more sophisticated algorithm for extracting content from e-shop pages, segmentation, price, product name etc. extraction. These are known problems typically solved semi-manually constructing scripts and then running the extraction. Our goal is a high accuracy, general algorithms without any customization or training working for all e-shops.

In the part two of this blog I will review our efforts in the Internet of Things.

YodaQA

n the eClub Summer Camp, Peter Baudis has developed the YodaQA, question answering system. Today we are publishing the first presentation of the technology.